Why does .mat file previewer take so long?

I often wish to preview the contents of a .mat file in the Current Folder browser. Some of my .mat files are quite large, which seems to cause a long delay for the previewer to populate. This for example, is what one of my previews should look like,

but it takes about a full minute before I see anything but this,

Why does it take so long for Matlab to inventory a .mat file on disk. Isn't all this information kept in a header?

[Steven Lord]

Just how large and/or complicated are the meta and P objects stored inside the MAT-file?

Do they overload certain methods like size? To check, load them into memory then ask which.

t = array2table(magic(4)); % sample object
which size(t) % show the location of the overloaded method 

/MATLAB/toolbox/matlab/datatypes/tabular/@tabular/size.m  % table method

[Matt J]

The P object is about 178 MB. The meta object is much smaller. The P object does overload size() but I wonder why you ask? The size() method of user-defined objects is not called by the previewer for the purpose of reporting its dimensions (much to my chagrin).

Answer 1

[Walter Roberson]

The first level elements store first a data type field and then the number of bytes in the rest of the element (excluding the 8 byte header.) Numeric items are stored with tag miMATRIX. The miMATRIX tag is followed by a mix of other tags and data, including a tag and storage for matrix dimensions.

Thus, for uncompressed data, you (typically) find the miMATRIX tag and then do a small bit of decoding after that to find the matrix dimensions.

However...

Compressed data is stored with a tag miCOMPRESSED; the element following the tag is the number of bytes of compressed data; after that is the zlib-compressed data. Compressed data does not have immediate uncompressed headers indicating the array size: the compressed data needs to be uncompressed to get the structured data that includes the miMATRIX tag and associated sub-fields.

Now, it would be theoretically possible to write code that just uncompressed enough to locate the matrix size portion, and stopped decoding after that. I tend to doubt that is happening, but I would not rule it out.

Now, in practice only very small arrays escape compression. There is a special very-small-element datatype that can represent at most 4 bytes without explicit matrix-size information. I would guess that the implied size is 1 x N where N is implied by the number of bytes stored, but it is possible also that the special element is only used for scalars.

But other than that... the headers alone for miMATRIX are almost always large enough to make the compressed version shorter, so nearly everything is compressed.

And that means that nearly everything has to be decompressed before you can peak to see what the size of the entry is.

Reference: https://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

[Matt J]

Compressed data does not have immediate uncompressed headers indicating the array size: the compressed data needs to be uncompressed to get the structured data that includes the miMATRIX tag and associated sub-fields.

Ah well, I suspected as much. Too bad.

[Walter Roberson]

I just realized that the very-small-element datatype could also cover empty 0 x 0 arrays, if the number of bytes is 0.