Effacer les filtres
Effacer les filtres

Parfor Loop Memory Issues

26 vues (au cours des 30 derniers jours)
Carlos Acasuso
Carlos Acasuso le 31 Mar 2021
Commenté : Tim Johns le 27 Avr 2021
My code runs on a parfor loop which runs with no issues for most cases.
The issue is, I am intending to execute this loop in some cases for vast amounts of data, the loop has to run for 7,805,662 iterations, to give an idea. The parfor loop then executes a significant number of calculations, which are all stored in arrays that are then post-processed after the parfor loop.
When executing this super large iterations, I am finding that my Matlab "freezes", and if I try to pause the run Matlab becomes unresponsive and has to be closed. No error is dropped whatsoever, which does not help me to try identify the problem. I have then done a few tests. The parfor loop runs perfectly fine when the number of iterations is not so large, and also I have found that if I run on a standard, serial "for" loop instead of parfor, it runs with no problems, but it takes way too much time and is unfeasible.
This is then narrowing the problem down to the parfor loop, and makes me think that the workers might be becoming "overwhelmed" and causes the run to stall. I am trying to find a way of partitioning the parfor loop so that this doesn't happen using the parforOptions, but I am struggling to use this effectively. Just to start, I would like to understand how Matlab partitions the loop by default, when no options are set. Is anyone able to answer that? What might be a reasonable number to partition the loop?
My machine has capacity for 8 Workers, 128GB Ram, 64-bit, and an Intel(R) Xeon W-2145 CPU @ 3.7GHz.
Many thanks for your help, really appreciate it.
  2 commentaires
Mohammad Sami
Mohammad Sami le 31 Mar 2021
Without specific details of what you are doing, it's a bit challenging to see how you might solve the problem.
One idea is perhaps use tall arrays if your data does not fit into memory. This will work if your calculations can be vectorised.
Tim Johns
Tim Johns le 27 Avr 2021
Hi Carlos,
My guess is that it runs in serial ok because you have 1 worker with 128 - 6 (ish, for OS) = 122 GB RAM available. When you divide that up by 8 workers, you only have 15 GB. If your calculations are loading/generating a lot of data, you could be running out of RAM and so the OS has to start swapping data on and off the hard disk - this is very slow.
Try running in serial and record what the peak RAM usage is, round up, and then divide the total available RAM by that to get the number of workers.
If this doesn't work, please file a help ticket and someone will be able to work with you more closely.

Connectez-vous pour commenter.

Réponses (0)


En savoir plus sur Parallel for-Loops (parfor) dans Help Center et File Exchange





Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by