running distributed jobs

I'm trying to prepare and run a distributed job (not parallel). Using

%sched = findResource('scheduler','type','jobmanager')

%job1 = createJob(sched)

At the last step when I try to create task, using my scripts:

%createTask(job1, scriptFileName);

I get this error:

Attempt to execute SCRIPT high_dimensional_warping_job_1st as a function

I was wondering how I could submit my scripts as distributed jobs in MATLAB's parallel computing toolbox. It seems that running my scripts as batch jobs does not run as distributed job, but parallel job. I tried to run my script as a batchscript in matlabpool but it even took longer than running the script on a single machine (when comparing it to cluster of machines), that's why I have tried to construct different tasks in a job using my script files and then submit the job in a distributed way.

Any input on this is greatly appreciated.

Answer 1

[Edric Ellis]

You need to give the name of the script to the createTask function, like so:

    sched = findResource(...);
    job = createJob(sched); 
    task = createTask(job, 'scriptFileName');
    submit(job);

This is basically equivalent to running

    job = batch('scriptFileName', sched);

A single isolated task might run more slowly on a cluster because MATLAB workers run in "single computational thread" mode. You might be better trying to run lots of scripts simultaneously.

[Arman]

Thanks for your reply. What would you recommend for running lots of scripts simultaneously? because if I use all the available workers for my scripts, second script will be "queued" until the first script finishes running. Shall I somehow limit my workers for every script so I could run lots of them, what would be the best command to do it? suppose I have 50 workers.

[Edric Ellis]

Your second and subsequent scripts should be able to start executing straight away if you have sufficient workers available. You should only execute "wait(job)" once you've submitted everything.