How to import big data CSV files
2 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Abhishek Singh
le 25 Mai 2019
Commenté : Abhishek Singh
le 30 Mai 2019
Hi,
Similar questions have been already asked but I wanted to know if there is an alternative to importing. I have a csv file with 3.5 million rows and 56 columns. At present while importing, I have to select the range I need. For instance I am selecting only 36 columns and barring few rows almost all. I tried textscan() but unable to achieve what I want which is without importing my code should select those rows and columns needed and also my main aim is to save time of importing and since textscan() is pretty fast I was using that. My column values are numerics.
Please let me know if I need to update with something here. I have a 8GB RAM so I think it should be easily feasible.
Thanks in advance!text
0 commentaires
Réponse acceptée
Walter Roberson
le 25 Mai 2019
Modifié(e) : per isakson
le 25 Mai 2019
These days, often the most convenient way is to use detectImportOptions, and set the SelectedVariableNames property of that to choose particular columns, and the readtable() -- or as of R2019a, readmatrix() if you are pure numeric.
textscan() is also a possibility. The easiest way to use that might be something like,
numcol = 56;
wanted_cols = [5 17:33 44:47]; %adjust to suit
fmt_cell = repmat('%*s', 1, numcol);
fmt_cell(wanted_cols) = {'%f'};
fmt = [fmt_cell{:}];
data = cell2mat( textscan(fid, fmt, 'Delimiter', ',', 'CollectOutput', 1) );
17 commentaires
Walter Roberson
le 30 Mai 2019
Shrug. You can get a wrong and unusable input quickly, or you can get a correct and usable input more slowly.
The extract you showed is not a csv file. csv files are, by definition, Comma Separated Values, never whitespace separated. RFC4180 specifically says that spaces are to be considered part of a field, not a separator.
It is not difficult to extract from a file only the lines that start with numbers. It is not all that much more difficult to divide a file into blocks of numbers, so that non-number marks the end of a block and the blocks are to be imported separately. But as soon as you start wanting to extract particular information from the non-number sections and associating it with a block of numbers, the task becomes more complicated.
Plus de réponses (0)
Voir également
Catégories
En savoir plus sur Text Data Preparation dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!