Parfor number of workers, CPUs, how are they related? What are the limits?
51 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
In-chan Kim
le 30 Juin 2021
Commenté : In-chan Kim
le 2 Juil 2021
Hi,
Using parfor specifying the number of parallel workers, how does it relate to the number of CPUs?
Where M specifies the Maximum number of workers running in parallel, as both a cap, and a limit.
It may seem like a very obvious question and answer: is the number of workers equal to the CPUs available? That is, on a quadcore computer, the maximum number of workers would be 4?
A separate question, I've been advised I can set some settings using this written in somewhere before the parfor coding:
tempdir = getenv('TMPDIR')
c = parcluster
c.JobStorageLocation = tempdir
c.NumWorkers=8
parpool(c.NumWorkers);
Would this specify the pool to 8 workers? In which case the code below for parfor would cap at 8 even if M is specifed as 10? What would happen if M was 4? Would it be capped at 4 or 8?
parfor (i=1:10,10)
...
end
parfor (i=1:10,4)
...
end
And where should theTMPDIR in tempdir = getenv('TMPDIR') be located?
Thanks a lot!
0 commentaires
Réponse acceptée
Raymond Norris
le 30 Juin 2021
A quadcore computer should max at 4 workers. We don't suggest throttling it to include hyperthreads. With that said, let's say you have 4 additional HT and you start a pool of 8 workers, perhaps you get even a 10% increase (for example), then maybe it's worth it. You just shouldn't expect an 8x improvement.
Setting the JobStorageLocation is only a suggestion if you are running MATLAB jobs on an HPC cluster and in some sort of rapid batch mode. For example, you write a PBS job script to run MATLAB code that uses a local pool. And then you submit a slew of those at once. In that case, setting the JSL can be helpful (due to a race condition).
The number of workers used is the min(M,size-of-pool). In your example, that would be 8 and 4 respectfully.
3 commentaires
Raymond Norris
le 30 Juin 2021
Correct, you could set M to a high number. However, it's really designed more for setting it to be lower than the size of the pool. Otherwise, there's no need to set it. That is
parpool('local',8);
parfor (i = 1:10,100000)
Is the same as just
parpool('local',8);
parfor i = 1:10
I'd just assign JSL to tempname. For example
c = parcluster;
jsl = tempname;
[FAILED,emsg,eid] = mkdir(jsl);
if PASSED==false
error(eid,emsg)
end
c.JobStorageLocation = jsl;
c.NumWorkers = 8;
c.parpool(c.NumWorkers);
tempname will generate a temporary name of the directory to use.
Plus de réponses (0)
Voir également
Catégories
En savoir plus sur Parallel Computing Fundamentals dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!