matfile operating very slowly: how to improve performance?

Hello,

I am trying to use matfile to save some variables for a behavioral experiment (including feedback and what I presented). However, when I save these pre-defined variables to a .mat file and later on access the matrix to fill in the responses for these variables, performance is very bad. It is acceptable for the first index, but saving goes slower and slower later on to the point that it takes a few seconds...

I'd still like to save after every trial... What should I do to increase performance?

Noise = cell(maxtrials, maxsessions);
Signal = cell(maxtrials, maxsessions);
Trigger = cell(maxtrials, maxsessions);
TrialNr = (zeros(maxtrials, maxsessions));
GainSignal = zeros(maxtrials, maxsessions);
GainNoise = zeros(maxtrials, maxsessions);
SNR = zeros(maxtrials, maxsessions);
Reversal = zeros(maxtrials, maxsessions);
Correct = zeros(maxtrials, maxsessions);
MouseClick = cell(maxtrials, maxsessions);
CorrectID = cell(maxtrials, maxsessions);
SubjectAnswerID = cell(maxtrials, maxsessions);
SubjectAnswer = cell(maxtrials, maxsessions);

save_name = 'test.mat' % edit
resultsfile = matfile(save_name) %edit
resultsfile.Properties.Writable = true; %edit
save(save_name, 'Exp', 'Noise', 'Signal', 'Trigger', 'TrialNr', 'GainSignal', 'GainNoise', 'SNR',...
        'Reversal', 'Correct', 'MouseClick', 'CorrectID', 'SubjectAnswerID', 'SubjectAnswer', '-v7.3');
    
%example: this becomes slow for later trials and sessions
resultsfile.TrialNr(trialnum, sesnum) = trialnum;

Answer 1

[Jan]

You explain, that the use of the matfile command is slow, but the shown code does not contain this command at all. If you access a MAT file to append data, remember that Matlab has to import the existing data at first and due to the compression triggered by the -v7.3 format the data must be decompressed at first. Then Matlab appends the new data in the RAM and inserts it into the MAT file again after compressing it. Therefore it might be required to move parts of the .mat file on the disk to get enough space to insert the grown data. Of course this takes a lot of time, when the array grows.

If you want a faster processing, use binary files, which allows to add new data by appending them at the end of the existing file.

[Walter Roberson]

7 of the variables are cell arrays. Although you are preallocating their general shape, the storage for the content is not preallocated and is potentially variable sized.

If you were to use fixed sized arrays, then potentially performance would be higher. Using fixed sized arrays would also permit you to switch to a binary file format. You might find it appropriate to use memory mapping of a struct.

Answer 2

[Tijmen Wartenberg]

Hi Jan & Walter,

Thanks for the quick replies. I added in the missing section in my example code, Jan. Hopefully, it makes my question more clear.

Although I know that no memory is allocated to the cell array, I was still wondered by the fact that even for the matrices (with zeros), altering only one index (row, col) of the matrix variable, takes in the order of tens of ms and increases significantly for further indices, but I presume this is due to the uncompressing/compressing behavior that Jan explained? Is there is a specific reason why matfile was only enabled for compressed data of type 7.3?

For now, I am using a struct and saving at the end of my experiment, since I found matfile too slow for intermediate saving, so in that sense my problem is solved. However, I am still open for a fast solution for intermediate saving.

[Walter Roberson]

The storage arrangement for -v7.3 is very different, using a modified HDF5, which has facilities for accessing and modifying and adding (chunks of?) substructures without having to rewrite the entire variable.

Compression is on by default in -v7.3, but in R2017a a -nocompression flag was added for save()