How to deal with a large amount of data in MatLab and create individual data sets each 30000 elements

Hi everyone,

I have a huge data set in .dat format, of about 5 GB. Of course, my computer takes a lot of time to deal with this archive if I use load to try to read it, as I usually do with shorter files. The thing is: in this file, each 30000 rows constitutes the data from a certain time step. So maybe a good way to deal with this is create through MatLab as much new files as needed, each 30000 rows, to minimize the loading time, maybe using a string in the name of each of the new created data sets to index in which time step you are. From example, the first 30000 rows, corresponding to t1, the next ones to t2... What do you think that I can do? This will reduce the loading time or it is not even to work due to the amount of data, at the end, is the same?

Thank you for your attention!

[Stephen23]

"What do you think that I can do? "

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

https://www.mathworks.com/help/matlab/datastore.html

Answer 1

[Raunak Gupta]

Hi,

From the size of dataset, I can see that the dataset involves a huge number of rows. I suggest using tall array if you want the data to be loaded without an out-of-memory error and can use gather for getting specific rows. You may look here, for how the out-of-memory data is handled in tall array.

As it is mentioned that each 30000 rows corresponds to a time instance, you may want to create separate data files for these chunks but if it is required to load all the files in a program, tall array would be a better alternative.

[Roderick]

Hey! Thank you. I have already find a way, but due to its not very fast computational time, I think that it is not suitable to write it here for future readers. I think that your answer opens the proper way, so I think your comment is the one that close this thread.

Cheers.