Adding/editing variables in existing .mat file increases memory size on harddisk if datatype table is used, even if it should just overwrite.

Adding/editing variables in existing .mat file increases memory size on harddisk if datatype table is used, even if it should just overwrite.

How can this be explained? I guess this happens if you save objects and not only data arrays to a .mat file.

I have a dataset with nearly one gigabyte, and every time i want to add a small Table to the existing .mat file, it doubles the size. My workaround is to load all the data to the workspace and overwrite the file. But there must be a more elegant solution i guess. Especially if you work with datasets bigger than the memory.

%Code Example
%% Create TestData
fileName = 'testData.mat';
Data = zeros(4000);
Data2 = array2table(Data);
%% Create .mat file with data and get handle
save(fileName,'Data','Data2','-v7.3'); %Save in -v7.3
m = matfile(fileName,'Writable',true);
s = dir(fileName);
fprintf('Original Datasize = %s bytes\n',s.bytes);

%% Save/overwrite double datatype in file (Does not increase Size)
x = ones(50000,1);
for k = 1:10
    m.data = x;
    s = dir(fileName);
    fprintf('Datasize Array = %s bytes\n',s.bytes);
end

%% Save table datatype in file (Increses size, without adding data) Does not overwrite?
tab = array2table(x);
for k =  1:10
    m.data = tab;
    s = dir(fileName);
    fprintf('Datasize Table = %s bytes\n',s.bytes);
end

[Florian Zacherl]

Answer from the Matlab Support facing this issue.

This is a known issue with MAT-file v7.3, which is a current limitation of the HDF5 format. MAT-file v7.3 is based on HDF5, which does not manage freespace as effectively as it should. When a dataset is frequently rewritten or frequently added and deleted as what you were doing in the reproduction code, the file can develop large numbers of holes and grow unnecessarily large. Please refer to the following documentation from the HDF5 group for more information.

https://web.archive.org/web/20150906142926/http://www.hdfgroup.org/HDF5/doc/H5.user/Performance.html

A possible workaround to remove the holes in the file is to load the contents of the MAT-file into workspace and resave them to a MAT-file after you are done rewriting or modifying data in the file.

Another possible workaround is to use the -v7 flag instead, although this removes the functionality of HDF5 to partially load/save variables.

Our development team is also actively working on this issue and I linked your case to an open enhancement request and you will be notified when there is any enhancement existing in a future release regarding this.