reading multiple text file from two different folders using nested for loop

2 views (last 30 days)
Salma fathi
Salma fathi on 31 Mar 2022
Edited: Konrad on 31 Mar 2022
i am using the following lines to read multiple text files and do some comparison between them. Baiscally the first loop will read text files of certain format (we have 30 files of this type). the second loop will read text files of another format (we have 33000 of them) and we do some comparison between them.
I have my code working and it is fine but it is extremely slow because of the nested loop, and its important to mention that i dont really need these loop for anything else, only for reading the files from folders.
So is there a more effecient way of doing this.
for n = 1 :nfiles2
fprintf('.......... File - %d of %d\n',n, nfiles2)
fullFileName2 = fullnames2{n};
t2 = readtable(fullFileName2); %t2 stores the ionosonde profiles
for m = 1 :10
fprintf('.......... insideFile - %d of %d\n',m, 10)
fullFileName1 = fullnames1{m};
t1 = readtable(fullFileName1); %t1 stores the ISR profiles

Answers (1)

Konrad
Konrad on 31 Mar 2022
Hi Salma
It depends on what exactly you want to do with the data in the text files, but reading the file line by line seems to be considerably faster compared to readtable()
tic;
for k = 1:100
t = readtable('test.csv');
end
toc
Elapsed time is 1.449934 seconds.
tic;
for k = 1:100
fid = fopen('test.csv');
clear tl;
tl{1} = fgetl(fid);
while ischar(tl{end})
tl{end+1} = fgetl(fid);
end
fclose(fid);
end
toc
Elapsed time is 0.633903 seconds.
Of course readtable() converts numbers to double while fgetl() returns char and if you have to do type conversion the speed gain might disappear. But I think its worth trying.
Best, Konrad
  2 Comments
Konrad
Konrad on 31 Mar 2022
Yes, thats what I ment with type-conversion ;)
I assume your data is comma delimited.
get the header in the first row:
fid = fopen('test.csv');
tline = fgetl(fid);
header = strsplit(tline,','); % <- the variable names as cellstring
If the data is all numeric (with variable names in the first row) you can do the following:
tic;
data = csvread('test.csv',1,0);
toc
Elapsed time is 4.558796 seconds.
Else: you can read the text data, store it in a cellstring and convert the numeric part (you will have to know which columns you need). This takes much longer due to the type conversion (and the growing array, which could be avoided by preallocating), but could be made faster if you only need a small fraction of the data for your comparison. But I don't know what you want to compare, so here's how you get everything from the test.csv:
tic;
dataCellstr = cell(0);
selectedCols = 1:3;
tline = fgetl(fid);
while ischar(tline)
cols = strsplit(tline,',');
dataCellstr(end+1,:) = cols(selectedCols);
tline = fgetl(fid);
end
data2 = str2double(dataCellstr);
fclose(fid);
toc
Elapsed time is 15.281463 seconds.

Sign in to comment.

Products


Release

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by