read file variables methods
Afficher commentaires plus anciens
I got a large data matrix (3Gb) of signal data which I saved to a .mat file. Since the data size is large I figured out it will be wiser to access the file and read each time a block of data, process it and move to the next block instead of loading the complete file. I'm wondering should I use the matfile method, fopen and fread or should I use a datastore and read command. What is faster from the processing point of view and easier to work with? Or any other considerations? Thanks
3 commentaires
per isakson
le 11 Nov 2016
"data set"   is that one numerical array?
"fairly large data set"   how large compared to the physical memory?
Talmon Alexandri
le 12 Nov 2016
per isakson
le 14 Nov 2016
Given
- "large data matrix (3Gb) of signal data which I saved to a .mat file"
- "Total RAM size - 3G[B]"
- "[one] plain numerical array 10 million lines by 48 columns"
I assume the mat-file is version 7.3, because that's needed to store an array of that size.
I would say this a a perfect case for matfile, Access and change variables directly in MAT-files, without loading into memory.
It's the easiest to work with and I don't think there is any other way that is significantly faster.
Réponses (1)
Walter Roberson
le 11 Nov 2016
0 votes
Catégories
En savoir plus sur Large Files and Big Data dans Centre d'aide et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!