HELP running large code.. having storage issues
Afficher commentaires plus anciens
I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...
clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? ');
year = 2009;
recs{10e4,1} = [];
ctr = 1;
while ~feof(myfile)
linestring = fgetl(myfile);
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
recs{ctr,1} = supercell;
ctr = ctr+1;
end
end
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = [];
1 commentaire
Adam
le 20 Fév 2019
Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.
Réponses (0)
Catégories
En savoir plus sur Logical dans Centre d'aide et File Exchange
Produits
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!