How to reduce the file size of MAT files?

My MAT file is around 7 GB. It has been very slow to read and write into this file. I wonder if there are ways I can make the file smaller?

Will it help to round the values to just two digits after the decimal point?

Answer 1

[John D'Errico]

NO. Rounding numbers will change nothing. MATLAB does not store double precision numbers in decimal form. So even though you think the numbers will take up less space, as long as they are doubles, then nothing happens. All you will do is lose precision for the same memory.

However, you could store your data in single precision. That will cut the memory required by half. Or you could go more extreme, if your data can be sotred in a (small) integer form.

[Leon]

Many thanks for the reply, John. Please see my comment above.

[John D'Errico]

You cannot just apply the single function to an entire struct. You would need to apply it to each field of the struct. However, you can use structfun, which will work as long as all fields are numeric.

S.A = rand(5);
S.B = rand(4);
S.C = 'The quick brown fox jumps over the sleepy cat'

S = struct with fields:
    A: [5×5 double]
    B: [4×4 double]
    C: 'The quick brown fox jumps over the sleepy cat'

Ss = structfun(@single,S,'UniformOutput',false)

Ss = struct with fields:
    A: [5×5 single]
    B: [4×4 single]
    C: [84 104 101 32 113 117 105 99 107 32 98 114 111 119 110 32 102 111 120 32 106 117 109 112 115 32 111 118 101 114 32 116 104 101 32 115 108 101 101 112 121 32 99 97 116]

As you can see, single cannot be usefully applied to a character vector. (Well, it can, but the result will no longer be character. And while you say string, many people confuse strings with chactaer arrays and vectors. So I am not sure which you intend.) Anyway, I'll make my own single variant, that will tolerate characters or strings.

function Asingle = strSafeSingle(A)
  if isnumeric(A)
    Asingle = single(A);
  else
    Asingle = A;
  end
end

I'll even add a string to the struct, in case that was your intention, just to show that it is tolerant of either case. Of course, in the event that some of your fields are stored as integer classes, well, you would need to deal with them properly too.

S.D = "The quick brown fox jumps over the lazy dog"

S = struct with fields:
    A: [5×5 double]
    B: [4×4 double]
    C: 'The quick brown fox jumps over the sleepy cat'
    D: "The quick brown fox jumps over the lazy dog"

Ss2 = structfun(@strSafeSingle,S,'UniformOutput',false)

Ss2 = struct with fields:
    A: [5×5 single]
    B: [4×4 single]
    C: 'The quick brown fox jumps over the sleepy cat'
    D: "The quick brown fox jumps over the lazy dog"

As you can see, this works, converting all numeric fields in a struct into single fields, ignoring anything that is not numeric.

Finally, a string is already stored as compactly as you can make it, unless you wanted to try doing some variety of compression, and you won't gain too much there on most short strings.

Oh, and IF you have the correct toolboxes and you are willing to suffer a great deal of precision loss in the process then half precision is even available, so you could tweek things to convert to half precision.

X = rand(1,5)

X = 1×5

0.5211 0.9595 0.5983 0.6024 0.7606

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

Xhalf = half(X)

Xhalf = 

  1×5 half row vector

    0.5210    0.9595    0.5981    0.6025    0.7607

whos X Xhalf

  Name       Size            Bytes  Class     Attributes

  X          1x5                40  double              
  Xhalf      1x5                10  half                

However, as my personal copy of MATLAB tells me, half requires one of the following:

MATLAB Coder

Fixed-Point Designer

GPU Coder

[Leon]

Thank you so much. This solution is exactly what I'm looking for.

Answer 2

[the cyclist]

It's not really as simple as "rounding", as the memory requirements are dependent on the data type. But, there are different numeric data types in MATLAB, and some take less memory than others.

By default MATLAB stores numbers in double-precision. If you don't need that much precision, then you can calculate in single-precision. (That is kinda like rounding them to a smaller number of digits.)

You can also store as integers of various sizes, or as binary, but it sounds like you need at least some floating-point numbers.

[Leon]

Many thanks for the reply. My computer is 62 bits.

I first read the input from an XSLX file and then store the output as structure arrays in a MAT file. Which step should I convert the data into single-precision? I assume it is the final step? I just realized that I can't use the "single" function on a structure array? Does that mean I would have to specify that for each of the variables within the structure array? Do I need to worry about the variables stored as strings?

Thanks.

[Edric Ellis]

It would be helpful if you could give us more details about the contents of your struct. For instance, if you have numbers stored in the struct as text (strings), then you are almost certainly better off converting these to actual numeric values using e.g. str2double . As others have said, maybe you can get away with calling single to half the storage for those fields. Another similar case might be if you have text data that consists of a few different fixed strings - in that case, you could use categorical, maybe like this:

s.myField = {'string1', 'string2', 'string1', 'string2', 'string3', 'string1', ...
    'string2', 'string3', 'string2', 'string3', 'string1', 'string2'};
whos

  Name      Size            Bytes  Class     Attributes

  s         1x1              1759  struct              

s.myField = categorical(s.myField); % convert from text to categorical
whos

  Name      Size            Bytes  Class     Attributes

  s         1x1               567  struct              

Even in this trivial case, you can see the categorical type uses much less space.

[Leon]

Many thanks for your super helpful comments! Much appreciated.