memmapfile in parfor loops

Hello, I'm trying to use mem mapped files for input and output of data from the body of a parloop.

If I understand correctly, this seems impossible, as the way memmapfile data is accessed (struct) violates the 'first indexing' rule for sliced variables in parallel loops.

As the memmapfile can be accessed by concurrent processes (as in the demo), is there a workaround to use it in parfor loops?

Thanks. L

Answer 1

[Sarah Wait Zaranek]

You should be able to use memmapfile in a parfor loop. The trick is to pass m into a function within the parfor loop. You can get around the restriction that way. See simple example below:

function parMemMap
% randData = gallery('uniformdata', [100, 1], 0, 'double');
% fid = fopen('myfile.dat');
% fwrite(fid, randData, 'double');
% fclose(fid);

m = memmapfile('myfile.dat','Format', 'double','Writable',true);

parfor ii=1:length(m.Data)
    output(ii,1) = myTestFunc(m,ii);
end

data = m.Data(:);

disp(output)
isequal(output,data)

end

function test=myTestFunc(m,ii)

m.Data(ii) = m.Data(ii)*rand;
test = m.Data(ii);

end

[Adam Wyatt]

Is this safe to do?

I've had issues with corrupting files when accessing inside par-for loops. I can imagine however that because memmapfiles utilize native OS virtual memory routines, it should be multithread safe.

[Keith]

11/11/11! Suppose there's a need to share data not only inside a parfor but between ML instances? Will a memmapfile, specifiying the desired disk file, in one thread facilitate reading that disk file as written by another(cooperating) thread?

[Walter Roberson]

Will a memmapfile, specifiying the desired disk file, in one thread facilitate reading that disk file as written by another(cooperating) thread?

No.

However, a memmapfile in one thread could obtain data that you exchange with the client using parallel.pool.DataQueue and your client could then make that data available to another thread using a different set of parallel data queues. It is a somewhat ackward process.