Loading a 484x585x6626 matrix in Matlab, computer freeze and matlab shut down, but.....

"Hello, I just tried to load a .mat file (600 megabytes, 484x585x6626) in Matlab (Linux, Ubuntu, Matlab 2023b), and it was impossible. My computer froze, and Matlab terminated the operation or just shut down. So, I downloaded GNU Octave and tried the same. I could load the file easily. Can someone please explain why? Does Matlab have issues handling large files?"

[Claudio Iturra]

I tried the same on two different computers with Matlab, and the same behavior occurred. I also attempted to load the file via Matlab/Terminal matlab -nodesktop -nojvm.

[Claudio Iturra]

I could load the file via Matlab on a MacBook computer with 32 GB of RAM. In contrast, when using Octave, my computer with 16 GB of DDR4 2600 RAM could load the file without any problems. Does this imply that Matlab consumes too much RAM on its own?

Answer 1

[Walter Roberson]

format long g
bytes = 484*585*6626*8

bytes = 
               15008685120

kilobytes = round(bytes / 1024, 1)

kilobytes = 
                14656919.1

megabytes = round(kilobytes / 1024, 1)

megabytes = 
                   14313.4

gigabytes = round(megabytes / 1024, 1)

gigabytes = 
    14

The file is not 660 megabytes.

[Claudio Iturra]

625 mb?

[Walter Roberson]

By default, MATLAB compresses data stored in .mat files. The on-disk size is not a good predictor of the size required in memory (which requires decompressing the data)

Your array has over 2 billion elements in it. Even if the elements were only 1 byte each, it would require more than 2 gibabytes in memory.

On my Mac, R2023b is using between 1 3/4 gigabytes and 2 1/2 gigabytes, depending which statistic you look at.

[Claudio Iturra]

Thanks Walter! so...I need to buy more RAM! :)

[Claudio Iturra]

Walter,. sorry...in the extream scenario that i do not have a computer with more ram, is posible to load just a part fo the 3d matrix? like load(''sst.mat",'sst(:,:,1500)'); ?

[Dyuman Joshi]

This might be helpful - https://in.mathworks.com/help/matlab/import_export/load-parts-of-variables-from-mat-files.html

[Walter Roberson]

The facilities that @Dyuman Joshi pointed out can be very useful, but they are at their best for -v7.3 files. With -v7 files (the default for most people) it is not promised that the facility will be able to do any better than to effectively load the entire variable and then trim away the unwanted parts. -v7.3 files are organized a completely different way, using a library that was designed to be able to do partial loading of variables.

This can lead to situations where you might find that you load a -v7 file on your 32 gigabyte Mac, write out a copy of the data as a -v7.3 files, getting out a file that will likely be over 1 gigabytes.. but which your other system can load in pieces.

[Claudio Iturra]

Thank you so much @Dyuman Joshi & @Walter Roberson!