Graceful exit from a parfor loop

If I detect an error while processing data within a parfor loop, I would like to gracefully exit the program. Can I delete the parallel pool object using "delete" to shut down all of the workers before exiting?

Answer 1

[Matt J]

It sounds like a job for onCleanup.

[Clay Fulcher]

Thanks Matt. I'll give this a try. I'm not sure it will destroy all the independent workers in a parallel loop, but I'll give it a try. Do you know if simply deleting the parallel pool object will accomplish the destruction of the tasks being performed by all the independent workers in the parfor loop?

[Matt J]

Do you know if simply deleting the parallel pool object will accomplish the destruction of the tasks being performed by all the independent workers in the parfor loop?

A simple error will do that. If even one of the workers hits an error, the whole loop aborts and all of the workers stop what they are doing. The only question at that point is whether you want the pool left open. So your code could look like this:

function output=runSomething(poolSize)

  objPool = parpool(poolSize);

  objCleanup=onCleanup( @() delete(objPool));

  try
      parfor i=1:N
          
          
      end
      output=...
  catch
       disp 'The loop failed. Aborting'
       output=[];
       return
  end
  
 

end

This will completely close the pool whether the function runs to successful completion or not.

[Matt J]

If even one of the workers hits an error, the whole loop aborts and all of the workers stop what they are doing.

Note that not all Parallel Computing Toolbox tools behave this way. With parfeval, for example, it is possible for some workers to keep going when others fail.

[Catalytic]

A simple error will do that

Here's a demo of that. When worker #3 aborts with an error, you can see that all the other workers whose task is too long never complete it --

parpool(6)

runThis(0)  %Short task. All workers except #3 complete

Worker 4 Completed

Worker 6 Completed

Worker 1 Completed

Worker 2 Completed

Worker 5 Completed

Error using test>runThis (line 11)

Worker 3 failed

runThis(30) %Only workers 4,5,6 complete

Worker 4 Completed

Worker 6 Completed

Worker 5 Completed

Error using test>runThis (line 11)

Worker 3 failed

function runThis(N)

    t=[ N,N,0,0,0,0];

    parfor i=1:6
    
        if i~=3
            pause(t(i))
            disp("Worker " +i+" Completed")
           
        else 
            error 'Worker 3 failed'
        end
    end

end

--

[Clay Fulcher]

Thank you both, Matt and Catalytic. I appreciate the expertise and your time. I will implement the try/catch with onCleanup to clean everything up upon hitting an error in the parfor loop.