loop over subfolders and saving cat parameters

Hello All,

I am trying to run my code over sub folder but facing issues. I am trying to extrac perticular variable from multiple .mat files.

I could do that for 1 folder having multiple .mat files with following code:

clear all
close all
d = uigetdir();
filePattern = fullfile(d, '*.mat');
file = dir(filePattern);
x = cell(1, numel(file));

for k = 1: numel(file)
    baseFileName = file(k).name;
    fullFileName = fullfile(d, baseFileName);
    fprintf('Now Reading file %s\n', fullFileName);
    x{k}=load(fullFileName,'Veriable1');
    j{k}=cell2mat(struct2cell(x{k}));
    
end
var1=cat(1,j{:});

Now I am trying to run this over the loop for subfolder and now facing issues and need help

Here is my current try:

clear all
close all
D = 'myPath';
S = dir(fullfile(D,'*'));
N = setdiff({S([S.isdir]).name},{'.','..'}); % list of subfolders of D.
for ii = 1:numel(N)
    T = dir(fullfile(D,N{ii},'*.mat')); % improve by specifying the file extension.
    C = {T(~[T.isdir]).name}; % files in subfolder.
    for jj = 1:numel(C)
        filePattern = fullfile(D,N{ii},C{jj})
        %filePattern = fullfile(F, '*.mat');
        file = dir(filePattern);
        x = cell(1, numel(file));
        for k = 1: numel(file)
            baseFileName = file(k).name;
            fullFileName = fullfile(D,N{ii}, baseFileName);
            fprintf('Now Reading file %s\n', fullFileName);
            x{k}=load(fullFileName,'Veriable1');
            j{k}=cell2mat(struct2cell(x{k}));
        end
    end
end
var1=cat(1,j{:});

I know it has many flaws but I am not at all able to think further and need help to make it work.

Answer 1

[Stephen23]

If you only need to loop over subfolders then why are you using three loops? One loop for the subfolders, one loop for the files... it is not clear what the third loop is supposed to achieve.

Also you do not explain if each folder only contains one .mat file or multiple .mat files, which will make a large difference to the design of your code.

Something like this will get you started:

D = 'C:\Software\Data\EDP\Rotation_3_CES_OBD\Script_for_data_analysis\19-11_matfiles\Try_subf';
S = dir(fullfile(D,'*'));
N = setdiff({S([S.isdir]).name},{'.','..'}); % list of subfolders of D.
for ii = 1:numel(N)
    T = dir(fullfile(D,N{ii},'*.mat')); % improve by specifying the file extension.
    T([T.isdir]) = [];
    for jj = 1:numel(T)
        F = fullfile(D,N{ii},T(jj).name)
        fprintf('Now Reading file %s\n',F);
        Z = load(F,'Key_swtich');
        T(jj).data = Z.Key_switch;
    end
    S(ii).data = vertcat(T.data);
end
X = vertcat(S.data)

[adi kul]

Hello,

I have updated my trial code above.

I have multiple .mat files in each sub fodler.

The updated code by me is fetching results for me, but I am not sure if it is concatenating in sequesnce.

Also Any suggestion in case of multiple variables.

Like right now it's variable1. Like that I have multiple variabl1, variabl2, variabl3, etc

[Stephen23]

"Like right now it's variable1. Like that I have multiple variabl1, variabl2, variabl3, etc"

Hopefully the variables are not really numbered like that!

Are the variables exactly the same for each .mat file?

[adi kul]

Not really. they have different names but same in all mat files.

It will be okay if you can just update your above code. Much appreciated

[Stephen23]

As long as the variables are exactly the same in every .mat file then you could do this:

for ii = 1:numel(N)
    ...
    for jj = 1:numel(T)
        ...
        T(jj).data = load(F); % load all variables (or select the ones you want)
    end
    S(ii).data = vertcat(T.data);
end
X = vertcat(S.data)

[adi kul]

nope. variable numbers are not same. some files may have 10 of them, while some have only 5 of them.

I was thinking of something like creating array of variables at the begining such as:

variables=[variable1, variable2,variable3]

and then running the above code over loop for each variable, saving it with that variable name in a mat file and moving to next variable such that. Not sure if this can be done

[Stephen23]

You can easily collect the data into one structure or one cell array, which will work regardless of how many variables are in each .mat file (or what their names are), e.g.:

for ii = 1:numel(N)
    ...
    for jj = 1:numel(T)
        ...
        T(jj).file = F;
        T(jj).data = load(F); % load all variables (or select the ones you want)
    end
    S(ii).data = T(:);
end
Z = vertcat(S.data)

[adi kul]

well that makes the process very slow if I have huge number of varaibles. Instead of having all I thought of making it variable specific.

[Stephen23]

"well that makes the process very slow if I have huge number of varaibles"

Possibly. How many variables do you have?

"...I thought of making it variable specific"

I have no idea what that means, but as you seem to know what you want, I am sure that you can get it working. Good luck!

[adi kul]

I meant somehting like this:

clear all
close all
D = 'myPath';
S = dir(fullfile(D,'*'));
N = setdiff({S([S.isdir]).name},{'.','..'}); % list of subfolders of D.
Variables={'variable1','variable2'};

for ii = 1:numel(N)
    T = dir(fullfile(D,N{ii},'*.mat')); % improve by specifying the file extension.
    T([T.isdir]) = [];
    fprintf('Now Reading Folder %s\n',D,N{ii});
    for jj = 1:numel(T)
        F = fullfile(D,N{ii},T(jj).name);
        fprintf('Now Reading file %s\n',F);
        for kk = 1:numel(Variables)
           
           Z{kk} = load(F,Variables{kk});
           T(jj).data = Z{kk}.Variables{kk};
        end
    end
    S(ii).data = T(:);
end
X = vertcat(S.data);

not able to make it work though!

[Stephen23]

"I meant somehting like this...not able to make it work though!"

And yet you don't explain enough for me to help you.

How does Variables correspond to the files in each subfolder? You defined Variables with two elements, and then within the inner loop access those two elements using indexing, so this means according to your indexing, each subfolder contains at most two .mat files, which each contains at least one variable (whose name is that of the corresponding element of Variables).

I cannot tell if that is correct, because you have not explained how Variables should correspond to each subfolder or file loop iteration, if at all.

Your code also assumes that every subfolder contains .mat files with the same variables in the same order, something like this:

Variables = {'A','B'}
sub1/f1.mat -> A
sub1/f2.mat -> B
sub2/f1.mat -> A
sub2/f2.mat -> B
etc.

Again, I cannot tell if your code matches how your files are arranged, because you have not explained how the file variables repeat, if at all.

You could easily read all of the variables into a cell array or structure, but as you earlier dismissed this idea beacuse "that makes the process very slow if I have huge number of varaibles"'.

If you want more help please give a detailied explanation of how the subfolders, files, and variable names correspond, e.g. like the list I gave above.