Parallelize calculations on a big cell array without making N input copies

Hello,

The existing answers on matlab answers, also due to their age, have not made it clear to me if it is possible to parallelize CPU-intensive calculations done with either cellfun or a for loop over a large (15.000.000x1) cell array with each cell containing a 24x24 matrix, and then writing the calculation results for each into a 15.000.000x1 vector.

Using just parfor is no use since my PC runs out of memory (I have enough memory for 1 copy of the cell array but not 6 copies for the 6 workers). Is there a way to perhaps copy it only once, with each of the 6 workers receiving only a 6th of the total array as copy?

An older comment said there was a userwritten solution with shared memory, but that this would not work anymore with newer Matlab versions (I use 2016a).

Thank you for your help!

Answer 1

[OCDER]

If each calculation is independent (as in, you don't need the entire 15000000x1 cell to make 1 calculation), then you could rewrite the parfor loop to work on a sliced variable. Workers will only receive a "slice" of the large 15000000x1 cell. This prevents passing the large cell as a broadcast variable, which will use too much memory.

The parfor loop should look something like this:

LargeCell = repmat({zeros(24)}, 100, 1); %Represent your 15000000x1 cell
Results = zeros(size(LargeCell)); %Store results here
parfor k = 1:length(LargeCell)
   Results(k) = complex_function( LargeCell{k} ); %Do your CPU-intensive calculation here
end

[Flora Feldner]

Yes, each calculation is independent. The sliced variable concept might exactly be what I need. Now I just need to monitor the memory usage and hope for the best. Thank you!