Can't find file on worker - Parallel Programming

Hi, I'm attempting to parallelize part of my code to speed up the process. But have become stuck when one of the attached files cannot be found by a worker. The file is being opened and modified on each driver for calculation before being reverted back to it's original state. The missing file is the "hydroNet.NetworkFile". Do I have to create a duplicate file for each individual worker?

parpool('local','AttachedFiles',{'epanet2.dll',hydroNet.NetworkFile});
    spmd
        folder = getAttachedFilesFolder(hydroNet.NetworkFile);
        if not(libisloaded('epanet2'))
            loadlibrary('epanet2');
        end
        oldFolder = cd(folder);  % Change to that folder
        [OK,output] = system(x(hydroNet,pumpCombo,nCluster,pumpCluster,idxSysLink,logger));
        cd(oldFolder);            % Change back to the original folder
    end

Answer 1

[Raymond Norris]

You mean to say that folder is empty?

At the top of the spmd, add this

spmd
    getAttachedFilesFolder
    hydroNet.NetworkFile
    getAttachedFilesFolder(hydroNet.NetworkFile)
    ...
end

[Thomas Clifford]

I think the issue is more to do with having the file open on mutliple workers at one time. Because it runs succesfully if I set it to only one worker and occasionally it runs through a couple of iterations before producing the error. So do I need to create a copy of the file for each worker to run from? If so how would you recomend I go about that?

[Raymond Norris]

I see. It's not that the file can't be found. It's that you're running into issues with multiple processes (i.e., workers) all writing to the same file at potentially the same time. Correct?

[Thomas Clifford]

I believe so. I'm using a toolkit that simulates a particular piece of software. The error i'm getting is that it can't find any data in the file. So I think it succesfully finds the file, but if it is in use by another worker then the file reads blank.

[Raymond Norris]

You can have many processes reading the same file, but need to have semiphore or another way to have simultaneous writes (e.g. parallel database). I'm not sure why with one worker it will on occasion throw an error.

In regards to your situtation, each worker ought to be able to read from the same file, but then they might write to their own file, followed by a way to "true" up the output. Or maybe send everyone their local part and then write to one file. For example (I'm leaving off error handling)

spmd
    % Import data
    fid1 = fopen('input.txt','rt');
    A = fscanf(fid1,'%s');

    % Modify data
    len = length(A);
    variantB = A(randperm(len));

    % Send each local part to everyone
    replicatedB = gcat(variantB);

    % Write to disk one output file
    fid2 = fopen('output.txt','wt');
    fprintf(fid2,[repmat('%c',1,len) '\n'],replicatedB);
    fclose('all');
end