Use datastores and tall arrays.
Python-like generator for reading large CSV file?
4 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I have a large csv file, typically I just read everything into a array like this
num =csvread(filename,2,2)
if csv file is big, the num array will be big as well and sometime it blowout my memory.
In python, there is a way to read the file but without holding the data in the memory. the methods use generator object. For example,
Base case(w/o using generator object):
value=[len(x) for x in open('filename.tx')] this case will hold the data in the memory
another case(use generator object:
it=(len(x) for x in open('filename.tx'))
the it is a generator object and it only hold the memory size for the generator expression without holding memory for all the data, later, we just extract the data by using next(it).
I am wondering if Matlab has similar generator object as Python when reading csv file to save the memory? Thanks
0 commentaires
Réponse acceptée
Plus de réponses (1)
Voir également
Catégories
En savoir plus sur Standard File Formats dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!