textscan in a for loop, memory issue
1 vue (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I have a text file with 140000 lines, each line has a string at the beginning and then 10 data points. I want to read this file 500 lines at a time and do some calculations based on the string data. So I have a for loop:
for t=1:280
clearvars dataArray
startRow = 1+500*(t-1);
endRow = (500-1) + startRow;
% Open the text file.
dataArray = textscan(fileID, formatSpec, endRow-startRow+1,...
'Delimiter', delimiter, 'HeaderLines', startRow-1,...
'ReturnOnError', false);
x=dataArray{:,3:5};
y=dataArray{:,6:9};
...
...
...
end
the problem is after iteration t=25, textscan doesn't read any data and it returns empty arrays. if I change the for loop to start from 10:
for t=10:280
...
...
end
the error occurs at t=28. It seems to me that there is a memory issue. Is there any way to check what is causing this problem?
1 commentaire
Réponses (2)
Harsha Medikonda
le 8 Juil 2015
I understand that you wish to know if the behavior you are observing is related to Memory. Please execute the command 'memory' before and after executing the code. This command shows the amount of memory available.
Please refer to the link below for more details on the usage of 'memory' http://www.mathworks.com/help/matlab/ref/memory.html
Another alternative to access large files is using 'datastore'. Please refer to the following link for more details on its usage http://www.mathworks.com/help/matlab/ref/datastore.html
0 commentaires
Chris Eguires
le 21 Juin 2017
You need to do the following after you do the read; fseek(fileID,0,'bof');
0 commentaires
Voir également
Catégories
En savoir plus sur Large Files and Big Data dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!