how to merge multiple csv files to create a timetable for synchronization?

Hello,

I have 6 oxygen and 29 temperature time series file in csv format. Both of the groups have different time steps and their row numbers also differ within themselves. My main aim to sychronize them according to certain time step.

I previously read each data as below but it is time consuming to write each file name and read them. I am new to matlab so could not figure out how to merge all the timetables and read them at once or is it really required to merge them for this purpose? Is there a smarter way to read all the directory in a loop and create a merged timetable including the original file names? I appreciate your support!

%Temperature
f015712=readtimetable("015712_20190824_0951_eng.csv")
f015713=readtimetable("015713_20190824_1013_eng.csv")
f015714=readtimetable("015714_6.8m.csv")
f015715=readtimetable("015715_20190824_1120_eng.csv")
.
% 22 more files 
.
f011563=readtimetable("011563_39.8.csv")
f011564=readtimetable("011564_40.85.csv")
f011565=readtimetable("011565_41.9m.csv")

%Oxygen
opto_2612_8=readtimetable("opto_2612_8.csv")
opto_2617_17=readtimetable("opto_2617_17.csv")
opto_2619_21=readtimetable("opto_2619_21.csv")
opto_2621_25=readtimetable("opto_2621_25.csv")
opto_2632_37=readtimetable("opto_2632_37.csv")
opto_2639_40=readtimetable("opto_2639_40.csv")

timetable1=synchronize(duo5100,f015712, f015713, f015714, f015715, f015716, f015717, f015718, f015719, ...
    f015720, f015722, f015723, f015732, f015733, f015734, f015735, f015736, f015737, f015738,f011538, ...
    f011539, f011540, f011542, f011543, f011544, f011545, f011551, f011553, f011554, f011555, ...
    f011556, f011560, f011562, f011563, f011564, f011565,duo11861,opto_2612_8, opto_2617_17, opto_2619_21, ...
    opto_2621_25, opto_2632_37, opto_2639_40,defi005,weather19, ...
    'regular','fillwithmissing',"TimeStep",minutes(60))

Answer 1

[Voss]

This will create timetables from all the .csv files in a specified directory and pass them all to the synchronize function. (If you have some .csv files that should be excluded, you'll have to modify the code.)

% folder containing the .csv files:
csv_folder = pwd(); % replace this with the full path to your folder, e.g., 
                    % csv_folder = 'C:\Users\EA\Research\csv_data';

% construct a cell array of full paths 
% to the relevant .csv files:
files = dir(fullfile(csv_folder,'*.csv'));
files = fullfile(csv_folder,{files.name});

% create a cell array of timetables containing all the 
% timetables loaded from .csv files:
n_files = numel(files);
tables = cell(1,n_files);
for ii = 1:n_files
    tables{ii} = readtimetable(files{ii});
end

% pass those timetables to synchronize, along with the other stuff:
timetable1 = synchronize(duo5100,tables{:},duo11861,defi005,weather19, ...
    'regular','fillwithmissing',"TimeStep",minutes(60));

[Cakil]

hi, thank you very much. Now I have all my timetables in "tables".

I tried to extract 33 timetables to my workspaces in a loop with different timetables names but could not figure it out.

n=1:33 %or n_files
readtimetable({tables{1,n}},[n 'Data'])
% I aimed to create 33 timetables in my workspace with Data1 Data2...Data33 

I also tried to create files with original file names to follow which table belongs to which data but it got more complicated could not find a solution.

Thank you in advance for your comments and help.

[Voss]

files contains the file names, and tables contains the timetable corresponding to each file.

I'm not sure why you want your timetables to be individual variables called Data1, Data2, etc. Doesn't that just get you back where you started, with variables f015712, f015713, etc.?

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

[Cakil]

Thank you for your reply and I can understand your point but if I do not have any timetable variable in my workspace, synchronize function does not even show up or if I open it from tasks in live editor, I can not use any variable except timetable for input.

[Voss]

Notice I use the cell array containing the timetables (the cell array is called tables) as input arguments to synchronize:

% pass those timetables to synchronize, along with the other stuff:
timetable1 = synchronize(duo5100,tables{:},duo11861,defi005,weather19, ...
    'regular','fillwithmissing',"TimeStep",minutes(60));

I'm not familiar with Live Editor, but if you can put code into it, put that line, and see if timetable1 shows up as a timetable variable in your workspace after that line is executed.

[Cakil]

hello, thank you very much it is working!

[Voss]

You're welcome!