Have data in subfolders available

I have a folder with a MATLAB script. In that folder, I have subfolders with information that can be processed with the MATLAB script. If I want to read a specific file, the only idea I have now is to specify the complete path for each file, with it is tedious.

What I would like is to have all those files available, in the same way that you have available all the data when you are working in the current folder. Is there any way to have available all the data in the subfolders (CSV files, TXT files, etc.)?

Answer 1

[Guillaume]

Firstly, it's not a good idea to mix code and data folder. The two should be completely separate, so that your code can work regardless of the location of the data (on the local drive, on a usb drive, on a remote server, in the cloud...)

Secondly, it's not clear what you mean by "available". Matlab can access data in any folder and can easily list the content of any folder with dir. e.g. if you wanted to know all the .csv files in a specific folder you'd use:

folder = 'C:\somewhere\somefolder'
filelist = dir(fullfile(folder, '*.csv'));
%process the files one by one:
for fidx = 1:numel(filelist)
    data = readtable(fullfile(filelist(fidx).folder, filelist(fidx).name));  %read the file
    %... code to use the data    
end

[Neko Benítez]

Thank you for your reply. I checked your code, and what it does is not what I want to do. With your code, I can read all the CSV files I have in the same folder than where the Matlab script is. But what I want to do is separated the code from the data.

For instance: I have the code in 'My_folder', and in that folder, there are several subfolders 'Data 1', 'Data 2', 'Data n' that contain the data information. I want to use my script in with all the files in each subfolder 'Data i'. I do this, changing the path for each file, but that is tedious.

What I meant with 'available' is that I can do something similar to what you do when you do, for instance this:

filename = 'my_csv.csv';
data = readtable(filename);

To do this, I have to have 'my_csv.csv' in the same folder than where the script .m is.

[Guillaume]

"To do this, I have to have 'my_csv.csv' in the same folder than where the script .m is."

No, that's exactly the point I'm making, code and data should be completely separate. The data can be anywhere, it doesn't even have to be on the same computer. The code I've shown work with the data anywhere you want.

What I pass to readtable is the full path of the file to open, which is built with fullfile. Of course, not knowing the specific of your data organisation, the example code I've shown may not apply exactly as is to you. It's a starting point.

If your data is in several subfolder of a main folder, then the only change required is a small modification of the dir call (to tell it to look in subfolders:

folder = 'C:\somewhere\somefolder';  %your root data folder 
filelist = dir(fullfile(folder, '*', '*.csv'));  %look in all immediate subfolders of folder. 
% Or use '**' to look in all subfolders no matter how deep
%... rest of code unchanged
for fidx = 1:numel(filelist)
    data = readtable(fullfile(filelist(idx).folder, filelist(fidx).name));  %read the file
    %... code to use the data    
end

Again, I recommend that your m files be in a completely different location than your data folder.

[Neko Benítez]

I have to admit that working in the same folder with the Matlab script and the data is how I work with Matlab, although I did not like that procedure for reason that I am sure you know more than me. I will try to get used to working with the scripts and the data separately as soon as posible.

About the code Guillaume passed, just a comment. The loop should be this way, as the (fidx) should be for the structure.

clear all
close all
clc

folder = 'C:\Users\AbrahamA-Idener\Desktop\Matlab\Consumo stand-by\';

filelist = dir(fullfile(folder, '*.csv'));

%process the files one by one:
for fidx = 1:numel(filelist)
    data = readtable(fullfile(filelist(fidx).folder, filelist(fidx).name));  %read the file
    %... code to use the data    
end

[Guillaume]

Indeed, I made a mistake with the indexing of the structures. I've edited my posts now.

Note that if you want to look into subfolders, you'd use

dir(fullfile(folder, '*', '*.csv'))

to construct the path c:\rootfolder\*\*.csv, the * as a folder name tells matlab to look into the subfolder. And to look into subfolders any level deep:

dir(fullfile(folder, '**', '*.csv'))

which constructs c:\rootfolder\**\*.csv

Once your code is completely separated from the data it's much easier to reuse it on different data folders. You just pass the data folder as an input (via GUI, command line, whatever...). For me, the data is typically on a remote server (with is constantly backed up).

[Neko Benítez]

Thank you! I thought that with just one * you take into account all the subfolder at any level.

Today, by chance I found what I meant in my first question. For instance, when you are in the current folder to execute the Matlab script, you can click with right buttom on any folder, and then select 'Add to Path'. If you do so, all the files in that folder are now available when you use the script.

[Stephen23]

"..then select 'Add to Path'. If you do so, all the files in that folder are now available when you use the script."

The MATLAB Search Path should refer only to folders that contain MATLAB code. It is not intended for accessing data files: data files are most efficiently read using absolute/relative filenames, exactly as Guillaume showed.

[Neko Benítez]

I am going to start working that way for now on. But until now, I always kept the data and script in the same folder/subfolder.

Thank you very much again!