Error caught during construction of remote parfor code

I have four vectors each about 2Gb outside of a parfor loop. Then Matlab gives me the warning message:

Warning: Error caught during construction of remote parfor code. The parfor construct will now be run locally rather than on the remote matlabpool. The most likely cause of this is an inability to send over input arguments because of a serialization error.

I have a system with 16core CPU and 128Gb memory, and it's planned to do parallel computations. I can move the vectors into the parfor loop, but then this will increase the computation time greatly.

Anyone has some suggestions please?

Answer 1

[Edric Ellis]

The error message indicates that perhaps you're trying to send too much data into the body of the PARFOR loop. For example, the following fails with that message for me:

    largeArray = zeros(3e9,1,'uint8');
    parfor ii=1:2
        disp(size(largeArray))
    end

because largeArray is too large. The limit for sending a whole array into a PARFOR loop is around 2Gb, so it's highly likely that you're hitting the same problem.

How to work around this limit depends very much on the structure of your code...

[James]

Thanks Edric. The problem I have is exactly about the size of the vectors and matrices. The reason that I bought this 128Gb memeory system is to model large domain wave propagation problem using the Finite Element Method (FEM). Big size matrices are inherent in a FEM problem and there is no way to get around this.

I think Matlab should improve on this PARFOR function, since 2Gb is really not so big for FEM problems. The total data size outside the PARFOR loop is about 8Gb. I try to open just 2 workers, and it's not working.

[Daniel Lyddy]

I agree with James. The whole point of using the parallel processing toolbox is to solve large problems more quickly. Having a puny limit like 2GBytes completely defeats that.

Why was this 2GByte limit imposed? Is there some hardware restriction, did the MATLAB programmers believe that serializing arrays larger than that would bog down or crash my system (96 CPUs, half a Terabyte of main memory), or were those programmers just being lazy?