Remote workers still access data files from the client despite the files have been copied to the remote host beforehand

Hi,

I have two PCs, and I use Parallel Server and Parallel Computing to utilize the cores in both PCs simultaneously. Since the script I'm running load relatively large amount of data files (.mat), I copied the files to the remote host beforehand, and then use the following command:

MJSName = 'AdibMJS';
ScriptName = 'SatelliteDataProcessing';
PathInRemoteHost = 'D:\SatelliteData';
NumWorkers = 8; % 4 from each PC

C = parcluster(MJSName);
Job = batch(C, ScriptName, 'Pool', NumWorkers - 1, 'AdditionalPaths', PathInTheRemoteHost);

parfor i = 1:N
    load('SomeData.mat')
    % Some codes
end

By doing this, I thought I could avoid the files to be transferred during runtime through WLAN, which is slow in my setup. When I started the job, I could see that, through Task Manager of both PCs, there are Network usage by MATLAB, indicating that there are still some data being transferred through WLAN. This cause the parfor-loop to run slower compared to using less workers (e.g., 4) on just the local cluster. What I want to happen instead is for both PCs to load the data files locally and run the parfor-loop simultaneously. Note that in both PCs, the path that stores the data files is identical.

Please advise where did I do wrong? Thank you!

Answer 1

[Jason Ross]

It sounds like there might be some dependency analysis still going on that's sending the files over. You can turn that off by setting 'AutoAddClientPath' and 'AutoAttachFiles' to 'false' in your batch command.

See the batch documentation for more info.