As it has been already advised, parfor within for isn't really recommended, but I am somewhat bounded by the intention of my code. I am running an optimization algorithm, and within each iteration of the algorithm I need to update around ~1k variables.
So, my code (please bear with me for the rough pseudo-code) -
parfor (i=1:1000, 10)
If I replace the parfor loop with a for loop, I see that one inner iteration takes roughly around ~30mins. I managed to get it under control using a parfor loop, and I happily see a reduction in simulation time to ~6-7mins. However, I am observing that this simulation time keeps on increasing as it goes for another round of optimization (next outer iteration), and it keeps on happening.
On some further checks, I checked that for the first outer loop, my cores/workers are nearly running at ~100%, but for the subsequent ones, they run at around ~20-60%, which I think is contributing to the increment. However, I am not able to figure out what is the problem, as the inner process is nearly the same for every outer iteration.
I have even tried parfevalOnAll('clear', 0, 'all') to clear the workers' data and delete(gcp('nocreate')) methods to even shut down the parpool after the inner loop completes, but nothing seems to be stopping the increment. Can somebody please help me out??
Thanks so much!