Can I share a copy of a CONSTANT large array between workers of a parpool?

I have an N-by-N array, data, of constant data that needs to be accessed by each worker of a parallel pool in a parfor loop. The loop will be called several times. Unfortunately each worker does need access to the entire array.

I just learned about the parallel.pool.Constant function which helps avoid multiple transfers of data and this is already helpful. However it seems that each worker keeps its own copy of data so I am limited by available memory to a small parallel pool.

Is there a way to use only a single copy of data in a parfor loop? To make things easier let's assume we will use a local cluster only, i.e. a multi-core desktop.

thanks!

Answer 1

[Naor Movshovitz]

Answering my own question after some more googling and doc search. I am not 100% sure but I think the right answer is: no, this is not possible with a parfor construct. The ease of use of parfor comes with some limitations and this is one of them. For sharing a large array between workers the code would need to be rewritten as a spmd region and use the distributed function. I expect this might not be ideal for small-to-medium arrays though, that we would not want fragmented across memory, but possibly ML is smart enough to optimize this behavior behind the scenes.

As a last resort the sharedmatrix fx submission suggested above might work but it sounds pretty risky. A lot of talk of segmentation faults in the comments ;)

Answer 2

[Edric Ellis]

You might be able to combine use of parallel.pool.Constant with the sharedmatrix package (disclaimer: I haven't tried to do this!).