How does an active parpool influence serial / threaded mex computation? (runtime/memory)
6 views (last 30 days)
I am running a program right now that uses part non-paralllized serial code, part a threaded mex function, and part matlab parallel pool. The exact code is not really of interest and I already checked: The non-parallized part cannot run parallel, the threaded mex part can not run parallel in Matlab (it could, but way slower because of additional linearization that would need to be done), and the part running parallel in Matlab is just fine and fast there.
My question now is, if I have a parallel pool active, how does this influence the first two parts of my program? Will they be slower? Do they have less memory / less cpu kernels available? Or in general, what does an active parpool do/need/block when its active but no parallel code is running at the moment? From the test I have done it seems that there is a slight speed loss (about 1-5%) in both the serial code and the mex function, but nothing too serious. The available memory also seems to be fine. However, I was looking for some official information as the code I am writing will not only run on my Computer, but will be shared with some coworkers.
As example, lets say my program looks like this:
function out = myProgram(in)
out = serial_function(in);
out = mex_threaded_function(out);
out(k) = some_function(out(k));
I could start the parpool directly before the parfor-loop so the first two parts are not influenced by it. However, the program runs a simulation that will be called many times, so it would mean shutting down and restarting the parpool over and over, what is time consuming in itself. Including the parfor in the mex-file is also not an option as some_function is way too complicated for me to code it in C. So the question remains, what does an active parpool do when its not needed at the moment? And how does this influence speed/memory?
(I was already searching a lot but didn't find anything. Maybe I was looking for the wrong keywords.)
Edric Ellis on 25 May 2022
Edited: Edric Ellis on 25 May 2022
Simply: an idle parpool should be doing nothing very much other than sitting there consuming some memory. It should consume only minimal CPU cycles, and should not adversely affect code running in your desktop MATLAB.
If you are using the 'local' cluster (the default with Parallel Computing Toolbox), then there is almost certainly no disadvantage to keeping the pool running even when you're not using it - providing the memory consumption is acceptable.
If you are using a pool running on a remote cluster (such as MJS or Slurm) through MATLAB Parallel Server, then an idle parallel pool is a bad idea since it will stop other work from running on the cluster.