Running 4 MATLAB Scripts to read, process and plot data from many csv files

Hi,

I have a number of folders (approx. 50) each with 3 csv files in them. At the moment I have to run the 4 scripts individually for one folder at a time as the name (and hence the path) of the folder is different. I would like to be able to run all these 4 scripts from potentially a 'master' script without editing the file path to each folder.

The names of the csv files within each folder are the same. For example, every folder has csv files called file1.csv, file2.csv and file3.csv. The csv file names do not differ from folder to folder.

The folders are named in a sequence for example, the first folder is AnBn, the next AnBn+1, AnBn+2, An+1Bn, An+1Bn+1, An+1Bn+2, An+2Bn, An+2Bn+1 ........ and so on where n is a number.

The code then saves plots to a path in each folder and exports data to a master Excel sheet, where each folder should have its own sheet within the Excel document.

To clarify, I would like to be able to run the scripts so they action every folder (the files within each folder) and save the plots that each folders data generates to that folder and that folders processed data to that folders own sheet within a master Excel document without stopping.

Is it possible to do what I am asking? I would really appreciate any help or advice.

Answer 1

[Mathieu NOE]

hello

see example below

clc
clearvars

%% define path
yourpath = pwd; % or your specific path
list=dir(yourpath);  %get info of files/folders in current directory
isfile=~[list.isdir]; %determine index of files vs folders
dirnames={list([list.isdir]).name}; % directories names (including . and ..)
dirnames=dirnames(~(strcmp('.',dirnames)|strcmp('..',dirnames))); % remove . and .. directories names from list

%% demo for excel files
sheet = 1; % specify which sheet to be processed (my demo) - if needed

%% Loop on each folder
for ci = 1:length(dirnames) % 
    fileDir = char(dirnames(ci)); % current directory name
    S = dir(fullfile(fileDir,'Sheeta*.xlsx')); % get list of data files in directory according to name structure 'Sheeta*.xlsx' 
    S = natsortfiles(S); % sort file names into natural order (what matlab does not) , see FEX : 
               %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)

    %% Loop inside folder
    for k = 1:length(S) % read data in specified sheet
        data = xlsread(fullfile(fileDir, S(k).name),sheet); % or use a structure (S(k).data ) to store the full data structure
        % your own code here for data processing. this is just for my demo
        % for now :
        title_str = [fileDir ' / ' S(k).name ' / sheet : ' num2str(sheet)];
        figure,plot(data),title(title_str);
    end
        
end

Answer 2

[Andrew Feenan]

Thank you for your help.

When I run the this for loop S is empty. What should i replace Sheeta*.xlsx with? The file name I use does not work.

for ci = 1:length(dirnames) % 
    fileDir = char(dirnames(ci)); % current directory name
    S = dir(fullfile(fileDir,'Sheeta*.xlsx')); % get list of data files in directory according to name structure 'Sheeta*.xlsx' 
    S = natsortfiles(S); % sort file names into natural order (what matlab does not) , see FEX : 
               %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)

[Mathieu NOE]

hello

sure you have to adapt to your xlsx file names structure

if you want to pick all files :

S = dir(fullfile(fileDir,'*.xlsx'));

if your files start always with blabla....xlsx , and you want only those : you can filter with

S = dir(fullfile(fileDir,'blabla*.xlsx'));

[Andrew Feenan]

Yes, the file I want is called ‘data_1’ but it doesn’t see it

[Andrew Feenan]

This is what I'm getting for S. As its value is 0, the second for loop will not run.

[Mathieu NOE]

hello Andrew

I tested again the code and it works. It will look for your "data....xlsx" files inside all subfolders

make sure the matlab file is in the folder above the subfolders containing the data

for me :

yourpath = 'C:\Users\A0H36019\Documents'

and the data files are searched in the 11 subfolders :

dirnames = {'MATLAB'} {'Ma musique'} {'Mes images'} {'Mes vidéos'} {'Modèles Office …'} {'My Music'} {'My Pictures'} {'My Videos'} {'MyFolder'} {'cache'} {'users'}

NB : data files are not searched inside the main folder (yourpath)

if S is empty that means no data file (matching the file filter 'data_*.xlsx' is in the searched subfolder)

clc
clearvars

%% define path
yourpath = pwd; % or your specific path
list=dir(yourpath);  %get info of files/folders in current directory
isfile=~[list.isdir]; %determine index of files vs folders
dirnames={list([list.isdir]).name}; % directories names (including . and ..)
dirnames=dirnames(~(strcmp('.',dirnames)|strcmp('..',dirnames))); % remove . and .. directories names from list

%% demo for excel files
sheet = 1; % specify which sheet to be processed (my demo) - if needed

%% Loop on each folder
for ci = 1:length(dirnames) % 
    
    fileDir = char(dirnames(ci)); % current directory name
    S = dir(fullfile(fileDir,'data_*.xlsx')); % get list of data files in directory according to name structure 'Sheeta*.xlsx' 
    S = natsortfiles(S); % sort file names into natural order (what matlab does not) , see FEX : 
               %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
               
    %% Loop inside folder
    for k = 1:length(S) % read data in specified sheet
        data = xlsread(fullfile(fileDir, S(k).name),sheet); % or use a structure (S(k).data ) to store the full data structure
        % your own code here for data processing. this is just for my demo
        % for now :
        title_str = [fileDir ' / ' S(k).name ' / sheet : ' num2str(sheet)];
        figure,plot(data),title(title_str);
    end
        
end

[Andrew Feenan]

Hi Mathieu,

Thank you for clarifying. I have made new simplified folder directory with test data, it seems to be working now using your example.

Thank you very much for your help. I will get the code to read my own data now and I’ll be sure to let you know how it goes.

Andrew

[Andrew Feenan]

Hello again Mathieu,

How do I specify the exact sheet in each folder to read data from in each iteration of the loop.

Currently I have this but as you can see it’s the same thing. Is it possible for filename1 to point to data1 and filename 2 to point to data2 in each instance?

filename1 = fullfile(fileDir, S(k).name);
filename2 = fullfile(fileDir, S(k).name);

[Mathieu NOE]

hello Andrew

so depending on which folder you have to specify the excel sheet ? this makes the code a bit more complicated...

maybe I would suggest that you create a cell array with folder names in the first column and associated (valid) excel sheet in the second column. of course you will have to "tweak" this portion of the code each time you will have another folder/subfolder structure with another / different sheet number associated with the data files.

another comment : I don't see what you want to achieve with the code :

filename1 = fullfile(fileDir, S(k).name);
filename2 = fullfile(fileDir, S(k).name);

the two lines are exactly the same... so both filenames are identical

the sheet number (or name) is a parameter you pass in the xlsread function :

data1 = xlsread(filename1 ,sheet1);
data2 = xlsread(filename2 ,sheet2);

[Andrew Feenan]

Hi Mathieu,

filename1 = fullfile(fileDir, 'data1.csv');
filename2 = fullfile(fileDir, 'data2.csv');

V0 = readtable(filename1,'Headerlines',543);
g = readcell(filename2,'Range','L78');

This is what I used and it's working fine now.

Thank you for your help.

Andrew

[Mathieu NOE]

Hi andrew

OK - good news