How to iteratively save a 3d array to a .mat file without storing the array in the workspace to conserve memory?

I've tried this two ways. The first way is to create a structure Psi, then iteratively save the fields p1, p2,... pn into the same structure in the .mat file; however, this forces me to create a new Psi variable while the original psi is still in memory. I'd like to avoid that step if I can.

I've changed some things about the original code to make it more readable.

blksize = 100;
for iblks = 1:blksize:nprofiles
    blockids = iblks : min(iblks+blksize-1,nprofiles);
    psi = nan(1000,3000,blksize);
    zout = nan(1000,1,blksize);
    rout = nan(1,3000,blksize);
    
    parfor issp = 1:blksize
        [psi(:,:,issp), zout(:,:,issp), rout(:,:,issp)]=ram(inputs);
    end

    Psi = struct(['p' num2str(iblks)], psi);
    clearvars psi
    save('F:\Thesis_data\winter_psi', '-struct', 'Psi', ['p' num2str(iblks)])
end

The second way I used matfile.m like this:

m = matfile('directory\winter_psi_', 'Writable', true);
m.psi = [];

for iblks = 1:blksize:nprofiles
    
    blockids = iblks : min(iblks+blksize-1,nprofiles);
    psi = nan(1000,3000,blksize);
    zout = nan(1000,1,blksize);
    rout = nan(1,3000,blksize);
    
    if iblks == 1
        
        parfor issp = 1:blksize
          [psi(:,:,issp), zout(:,:,issp), rout(:,:,issp)]= ram(inputs)
        end
        
        m.psi = psi;
        clearvars psi
        
    else
        
        parfor issp = 1:blksize
          [psi(:,:,issp), ~, ~]= ram(inputs);
        end
        
        m.psi(:,:,iblks) = psi;
        clearvars psi
        
    end
end % iblks

Unfortunately, m.psi is saved in the workspace in the Matfile object m.

Is there a way to save psi iteratively to a .mat file without storing a new variable in the workspace?

Answer 1

[Abhijeet]

Hi,

Yes, you can use the matfile function in MATLAB to iteratively save a 3D array to a .mat file without storing the array in the workspace to conserve memory.

Here's a sample code :

m = matfile(filename, 'Writable', true);
for i = 1:N
    % calculate the ith slice of psi
    psi_i = calculate_psi_i(i);
    % write psi_i to the matfile
    m.psi(:,:,i) = psi_i;
end

Inside the loop, the calculate_psi_i function is used to calculate the ith slice of the 3D array psi. This slice is then saved to the matfile using the syntax m.psi(:,:,i) = psi_i.

Note : This syntax does not create a new variable in the workspace, but instead writes directly to the matfile on disk. The matfile is automatically closed when the script finishes running.

Thanks