Passing big matrix to workers

I'm trying to pass very large matrix (65000x460x150) to workers with Worker Object Wrapper, with no luck.

The code:

Imat_worker = WorkerObjWrapper(Imat);
parfor kk = 1:size(num_iterations,2)
   Imat_par = Imat_worker(:,:,1:num_iterations(kk));
   some code...
end

I get the following message while running the code:

Error using WorkerObjWrapper/workerInit (line 156)
The parallel pool that SPMD was using has been shut down.
Error in WorkerObjWrapper (line 97)
            WorkerObjWrapper.workerInit( tmpId, ctor, args, dtor ); 
A write error occurred while sending to worker 2.

Any ideas what am I doing wrong?

It is worth noting that I'm running the code on a server with enough RAM for the matrix.

Thanks in advance!

[Matt J]

What version of MATLAB is this?

[or ohev shalom]

2017b

[Matt J]

If you have 2017b, you probably want to be using parallel.pool.Constant instead.

[or ohev shalom]

Tried it also but it still gives an error.

Correct me if I'm wrong, but when I use the parallel.pool.Constant:

      Imat_worker = parallel.pool.Constant(Imat);

the amount of used RAM should be the same as before the command.?

[Matt J]

I don't know at what point the increase in RAM consumption is supposed to hit, but Imat will eventually be cloned on all the workers, so you will end up consuming as much RAM as numlabs copies of Imat. Is Imat sparse?

[or ohev shalom]

Imat is double.

Then if I understand it correctly, there is no way to avoid the cloning (and if there isn't enough memory for all the cloning I can't go on that way)...

In that case, since I need only a slice of the matrix for each worker, shouldn't I clone only the relevant data somehow before the parfor ?

[Matt J]

Imat is double.

Yes, but are most of those double values zeros?

[or ohev shalom]

Unfortunately not. There isn't even 1 single zero value inside.

Answer 1

[Matt J]

In that case, since I need only a slice of the matrix for each worker, shouldn't I clone only the relevant data somehow before the parfor ?

Yes you should, and that's something parfor will do for you automatically as long as the operations you do on Imat inside the loop satisfy certain restrictions.

[or ohev shalom]

Ok, Thank you very much.

Now it is much clearer, and I'm pretty sure that I can't run the code the way I wanted (not enough RAM...).

[Matt J]

Yes, making Imat a sliced variable is much more sensible if it is possible to do so.

If you can afford the loss in precision, you could also consider storing the original Imat as single floats. You could still do the calculations in double precision on the workers by converting the current slice being processed to double in the course of the loop. The total amount of double data being held anywhere simultaneously will then be much smaller.