Effacer les filtres
Effacer les filtres

How is the work distributed in parfor loop?

5 vues (au cours des 30 derniers jours)
Luqman Saleem
Luqman Saleem le 15 Fév 2024
Commenté : Walter Roberson le 17 Fév 2024
In a scenario with a parfor loop consisting of 10000 iterations distributed over 10 workers, is each worker initially assigned 10000/10=1000 tasks at the beginning of the parfor loop, or does each worker receive one task initially and subsequently get assigned new tasks upon completion?
Specifically, I am encountering a situation where approximately 500 iterations of a parfor loop (of total 10000 iterations) are computationally heavy while the remaining ~9500 iterations are computationally lighter. When running this parfor loop, I notice that the execution is initially extremely fast, with around 90% completion within 5 minute. However, the remaining 10% takes significantly longer, up to 20 hours. My suspicion is that the computationally heavy iterations are being assigned to a single worker, which handles them sequentially.

Réponse acceptée

Edric Ellis
Edric Ellis le 16 Fév 2024
If you know in advance which iterations are the time-consuming ones, then you could consider running those together as a separate parfor loop, and the default iteration partitioning described by @Walter Roberson will probably do a decent job of keeping all workers busy.
If you do not know in advance, then there are two strategies that you could use, and maybe you could use both together:
  1. Shuffle the order of the iterations using randperm or similar to try and end up with the long-running iterations in different sub-ranges.
  2. Use parforOptions to force small sub-ranges.
  2 commentaires
Luqman Saleem
Luqman Saleem le 17 Fév 2024
Thank you. I have a general idea of which iterations are computationally intensive. My current parfor loop structure is as follows:
parfor ix = 1:10000
% code
end
Specifically, I have identified the iterations from ix = 2000 to 2500 and from ix = 7000 to 7500 as computationally heavy. Could you please provide a sample code illustrating how I can instruct the parfor loop to distribute these iterations among multiple workers?
Walter Roberson
Walter Roberson le 17 Fév 2024
For example, set the RangePartitionMethod to [50*ones(1,(1950-0)/50), 20*ones(1,(2500-2000)/50), 50*ones(1,(7000-2500)/50)), 20*ones(1,(7500-7000)/20), 50*ones(1,(10000-7500)/50)]
or something like that.

Connectez-vous pour commenter.

Plus de réponses (1)

Walter Roberson
Walter Roberson le 15 Fév 2024
You can control how parfor divides up the iterations by using parforOptions() RangePartitionMethod https://www.mathworks.com/help/parallel-computing/parforoptions.html#mw_5eb7f106-9fa3-45da-a7ae-6f5b0db4bef0
The default is "auto", which divides most iterations up into chunks, then divides most of the remaining iterations into smaller chunks, then hands out the remaining iterations one at a time.

Catégories

En savoir plus sur Loops and Conditional Statements dans Help Center et File Exchange

Tags

Produits


Version

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by