importing large 4GB+ tab-delimited ascii file makes MATLAB crash
1 vue (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Sean Phillips
le 17 Mai 2019
Réponse apportée : Sean Phillips
le 18 Mai 2019
i didnt have a problem until now the data files have been scaled up in complexity
i usually use
IMPORT = importdata('components.dat')
in this case components.dat is 4 GB+
it is a table of numerical values consisting of 12 columns and millions of rows
for smaller models this has worked but now MATLAB crashes when i try to import at this size.
I have 96GB of RAM and dual Xeons.
Do i need to change setting in MATLAB or is there better ways to import this data
this is an exmaple of the file
1 0.01 -50.00 0.00 -3950.00 -2.0000E+04 0.000 -4000. -1.409E-15 -2.138E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9900E+04 0.000 -4000. -1.410E-15 -2.193E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9800E+04 0.000 -4000. -1.411E-15 -2.249E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9700E+04 0.000 -4000. -1.412E-15 -2.306E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9600E+04 0.000 -4000. -1.411E-15 -2.364E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9500E+04 0.000 -4000. -1.411E-15 -2.424E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9400E+04 0.000 -4000. -1.409E-15 -2.485E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9300E+04 0.000 -4000. -1.407E-15 -2.547E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9200E+04 0.000 -4000. -1.405E-15 -2.611E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9100E+04 0.000 -4000. -1.401E-15 -2.676E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9000E+04 0.000 -4000. -1.397E-15 -2.742E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8900E+04 0.000 -4000. -1.392E-15 -2.810E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8800E+04 0.000 -4000. -1.386E-15 -2.879E-15 0.00E+00 0.00E+00
it is tab delimited
0 commentaires
Réponse acceptée
Plus de réponses (2)
Steven Lord
le 17 Mai 2019
Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.
per isakson
le 18 Mai 2019
Modifié(e) : per isakson
le 18 Mai 2019
That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?
Anyhow try
>> num = cssm( 'components.dat' );
>> whos num
Name Size Bytes Class Attributes
num 13x12 1248 double
where
function num = cssm( ffs )
fid = fopen( ffs );
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
, 'Delimiter','\t', 'CollectOutput',true );
fclose( fid );
num = cac{1};
end
If that doesn't work, I guess there is some strange stuff in the middle of your text file.
I added
, 'ReturnOnError',false
to the textscan call and the string, "an error", to the text file an run
>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
>>
textscan() found my string "an error" on the row 1005147.
0 commentaires
Voir également
Catégories
En savoir plus sur Text Data Preparation dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!