Making multiple batch calls quickly

Hi all,

I have built a GUI which performs 120 function calls at around 5 seconds execution time each. What I've tried to do is send them all out to batch jobs so that I can keep using the GUI while the crunching happens. The problem that I'm running into is that it takes almost as long as the execution time itself to send them all out to batch jobs. Here is the code snippet:

for i = 1:6
  for j = 1:20
    workerTable{i,j} = batch('lengthyFxn',7,{inputs related to i,j values});
    disp(num2str(j));
  end
end

My thought process is that I should be able to farm out all of the tasks quickly, which are just calling the same function with different input arguments based on the values of i,j in the loops, and then retrieve them later after the user has taken some time working in the GUI. I added the disp call just to see if anything was actually happening, because this assignment of tasks takes upwards of 5 minutes and the gui and workspace are blocked/unusable in the process. Is there a better or smarter way to do this?

[Edric Ellis]

Which cluster type are you using? It's possible that if you're using the 'local' cluster (the default), then you might be able to add the arguments

batch(..., 'AutoAttachFiles', false)

to stop some of the dependency analysis. You might also be better off creating 120 tasks of a single job object, something like this:

j = createJob();
for ...
  for ...
     createTask(j, @lengthyFxn, 7, {args});
  end
end
submit(j)

[Stephen]

Yes, sorry for not remembering to clarify that - I am using the local cluster. Changing the granularity of the problem by breaking it down into tasks like that did indeed solve the problem I was seeing! What I did was break it down into six jobs assigned 20 tasks each. Thank you so much!!

What is the reason that I'm seeing only one job running at a time? Basically, matlab runs job 1 while 2-6 remain queued, and will only start running the next job when the previous job has finished. Why would this be the case if my cluster has 8 workers? I would ideally like to have all of the jobs running simultaneously.