I got a large data matrix (3Gb) of signal data which I saved to a .mat file. Since the data size is large I figured out it will be wiser to access the file and read each time a block of data, process it and move to the next block instead of loading the complete file. I'm wondering should I use the matfile method, fopen and fread or should I use a datastore and read command. What is faster from the processing point of view and easier to work with? Or any other considerations? Thanks

3 commentaires

per isakson
per isakson le 11 Nov 2016
"data set" &nbsp is that one numerical array?
"fairly large data set" &nbsp how large compared to the physical memory?
Talmon Alexandri
Talmon Alexandri le 12 Nov 2016
data set - plain numerical array 10 million lines by 48 columns Total RAM size - 3G
per isakson
per isakson le 14 Nov 2016
Given
  1. "large data matrix (3Gb) of signal data which I saved to a .mat file"
  2. "Total RAM size - 3G[B]"
  3. "[one] plain numerical array 10 million lines by 48 columns"
I assume the mat-file is version 7.3, because that's needed to store an array of that size.
It's the easiest to work with and I don't think there is any other way that is significantly faster.

Connectez-vous pour commenter.

Réponses (1)

Walter Roberson
Walter Roberson le 11 Nov 2016

0 votes

I think maybe using memmapfile and a binary file might be easiest for access.

Catégories

En savoir plus sur Large Files and Big Data dans Centre d'aide et File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by