Parallel pool never releases used memory leading to instability

Running code like:

rxs = rxsite("AntennaHeight",1.5, ...
            "Latitude", 38.9899587, "Longitude", -76.9353889);


for scenario = 1:2
    switch scenario
        % these are not meaningful paths
        case 1
            txPositions = [linspace(38.9857294, 38.9857294, 10000).', linspace(-76.9442905, -76.9407240, 10000).'].';
        case 2
            txPositions = [linspace(38.9895908, 38.9903186, 10000).', linspace(-76.9432997, -76.9446669, 10000).'].';
    end

pm = propagationModel("raytracing");
data = [];
parfor point = 1:size(txPositions,1)
txs = txsite("AntennaHeight", 1.5, ...
            "Latitude", txPositions(1,point), "Longitude", txPositions(2,point), ...
            "TransmitterPower", 10);
rays = raytrace(txs, rxs, pm, "type", "power"); % rays is Na x Ns cell array
if ~isempty(rays{1}) % only ever 1 tx/1 rx
    rayPower = 0;
    % phasor sum of rec'd power
    for rayPath = 1:numel(rays{1})
        pWatts = 10^(-rays{1}(rayPath).PathLoss/10)*10;
        rayPower = rayPower + pWatts*(cos(rays{1}(rayPath).PhaseShift) + 1i*sin(rays{1}(rayPath).PhaseShift));
    end
    % convert to RSS in dBm and resultant signal phase offset
    RSS = 10*log10(abs(rayPower)/.001); % dBm
    phaseOffset = angle(rayPower) ; % rads
    % collate data
    data = [samplePoints(step, 1), samplePoints(step, 2), RSS, phaseOffset];
end
end

close all force % this is an attempt to fix this issue
save("data.mat","data")
clear data
delete(gcp)

end

This code will spawn a transmitter, move it along a path, sample the RSS at a receiver then append that data into a list which gets saved. After that, the parallel pool is reinitialized and the data variable is cleared before starting the next scenario. The delete(gcp) line was from another about a similar issue which I cant find anymore - this helps but doens't solve the problem.

Running this code, I would expect the memory usage to increase during a single scenario (which is what I see), but I would also expect to see memory usage drop considerably between scenarios. However, this code causes the memory usage to steadily increase across scenarios, regardless of if I delete the parallel pool and clear variables. This almost always results in MATLAB either locking up the computer or just crashing outright.

The issue never occurs on the first (or even the first few) scenario, but will always happen after some number have been run, so I know that the individual scenarios are totally fine.

Am I going about this wrong or is there something I should do to fix this? This issue is present on R2024b on Ubuntu22 and Win10.

[Alex B]

Also I have tried a variety of parallel pool settings and while some seem to crash faster than other, they all still crash.

[Walter Roberson]

    data = [samplePoints(step, 1), samplePoints(step, 2), RSS, phaseOffset];

variable step is not defined at that point, so it is invoking the function step() with no parameters.

Answer 1

[Alex B]

For anyone else facing this same issue, the only way I found to reliably make parallel pools give up memory when doing this sort of of large data sampling problem is to:

• Make a 'storage' array which will never be accessed from inside the parfor loop
• Use the parfor loop to iterate over small chunks of the array, concatenating results at the end
• Delete the parallel pool between parfor calls! If you don't the memory will never release

Doing only the second or only the third step does not seem to fix the problem - you'll end up with parfor using progressively more memory over time.

The resulting code looks something like:

data = []; maxDataSamples = 5000; currentDataSample = 1;
maxSamplesPerStep = 1000; simNotComplete = 1;
while simNotComplete
    % get num steps for this batch of data
    numStepsInBlock = min(maxDataSamples - currentDataSamples, maxSamplesPerStep)
    
    % make empty storage
    dataTemp = [];
    
    % generate data in the parfor loop
    parfor step = currentStep:(currentStep+numStepsInBlock)
        % logic goes here
        dataTemp(step,:) = f(step);
    end
    
    % concatenate data and update the step counter
    currentStep = currentStep = numStepsInBlock;
    data = [data; dataTemp];
    
    % delete parallel pool to free memory
    delete(gcp);
    
    % check if sim is done
    if currentStep == maxDataSamples
        simNotComplete = 0;
    end
    end
end

Answer 2

[Walter Roberson]

            txPositions = [linspace(38.9857294, 38.9857294, 10000).', linspace(-76.9442905, -76.9407240, 10000).'];

That is 10000 x 2

            txs = txsite("AntennaHeight", 1.5, ...
            "Latitude", txPositions(1,point), "Longitude", txPositions(2,point), ...
            "TransmitterPower", 10);

That tries to access txPositions(1,point) where point is up to 10000. Because txPositions is 10000 x 2, you cannot access txPositions(1,3)

[Alex B]

I added the proper transposition to the code, apologies for any confusion. However, thats not the problem I'm facing. This code is a snippet of a larger file which contains almost this exact loop plus some pre and post processing that happens outside of parfor and gets cleared when its done.

My problem is that each time a new parfor loop starts it never releases the memory used by the last loop, leading to memory usage continually increasing until MATLAB eventually crashes.