Dramatic but intermittent slow down of parallel system calls

I have a program that I run in parallel using the matlab parallel toolbox in a linux environment. We've put together a large run set, and we execute the program by calling something like

parfor iRun=1:nRuns
    system(commandStringsForRun{iRun})
end

In most situations this runs with no problem, and each system call takes something like 10 seconds to complete. We've seen an issue recently where individual calls in this loop become dramatically slower than their counterparts, and are trying to understand why.

The only change in our process seems to be that we're now running 100k runs in sequence, vs. a smaller number in the past (possible the problem existed before but we were unaware). What we know is that the problem:

• Is not deterministic. The same command string might slow down one time, but not another.
• Is not isolated to the parpool - we've had instances where the slow down occurs with a single run in progress in matlab.
• Does not happen at the same point in the run each time
• Does not happen during parallel execution in a bash loop outside of matlab or on the command line outside of matlab
• Does not crash the run - our 10s runs might take 3-5 hours, but they will finish

Any thoughts or suggestions on what might be causing this? Best guess is something to do with how matlab is managing the system call under the hood, but that's not an area I'm very familiar with.

Thanks!

Answer 1

[Edric Ellis]

This sort of sporadic slowness can be extremely hard to track down. I'm not an expert on how MATLAB manages system calls, but I'm not aware of anything that could be causing this. I suppose one area that might be different is how stdin and stdout are handled. Maybe you could try wrapping your executable in a shell script that does no I/O?

If you are able to, it might be worth attaching strace to one of the "slow" process executions to see if there's anything unusual going on.

[Daniel Schwartz]

We were unable to really resolve this problem, but I have come to believe it was not a Matlab issue as previously thought.

Thanks for your reply

Answer 2

[Benjamin Thompson]

Windows can have any number of tasks the run apart from MATLAB. Anti virus, caching the file system for searches. Try researching Windows performance improvements or Windows slowdowns and start ruling those out.

[Edric Ellis]

The OP mentioned that they are running on a Linux system...