Parfor Loop Memory Issues
6 views (last 30 days)
My code runs on a parfor loop which runs with no issues for most cases.
The issue is, I am intending to execute this loop in some cases for vast amounts of data, the loop has to run for 7,805,662 iterations, to give an idea. The parfor loop then executes a significant number of calculations, which are all stored in arrays that are then post-processed after the parfor loop.
When executing this super large iterations, I am finding that my Matlab "freezes", and if I try to pause the run Matlab becomes unresponsive and has to be closed. No error is dropped whatsoever, which does not help me to try identify the problem. I have then done a few tests. The parfor loop runs perfectly fine when the number of iterations is not so large, and also I have found that if I run on a standard, serial "for" loop instead of parfor, it runs with no problems, but it takes way too much time and is unfeasible.
This is then narrowing the problem down to the parfor loop, and makes me think that the workers might be becoming "overwhelmed" and causes the run to stall. I am trying to find a way of partitioning the parfor loop so that this doesn't happen using the parforOptions, but I am struggling to use this effectively. Just to start, I would like to understand how Matlab partitions the loop by default, when no options are set. Is anyone able to answer that? What might be a reasonable number to partition the loop?
My machine has capacity for 8 Workers, 128GB Ram, 64-bit, and an Intel(R) Xeon W-2145 CPU @ 3.7GHz.
Many thanks for your help, really appreciate it.