How to run MATLAB on a Sun Grid Engine cluster to use parallel optimization

I am trying to use the Global Optimization Toolbox in MATLAB to minimize a non-smooth function. The function computation is quite expensive and hence it makes sense to find local minimas by starting from different points in parallel.

I am trying to run my matlab script on a cluster which uses the Sun Grid Engine for scheduling jobs. However, it seems that I am capped by the number of cores in each node to run parallel optimizations. Is there any way to share cores across nodes to run my job in parallel with several cores?

Answer 1

[Kojiro Saito]

You need 2 more products to run parallel job in a cluster.

1. Parallel Computing Toolbox ( https://www.mathworks.com/products/parallel-computing.html)
2. MATLAB Distributed Computing Server ( https://www.mathworks.com/products/distriben.html)

Parallel Computing Toolbox enables you to run parallel jobs in a local machine. In adddition to it with MATLAB Distributed Computing Server, you can run parallel jobs on cluster nodes.

About, Grid Engine can be integrated with these 2 products with the following support package. https://www.mathworks.com/matlabcentral/fileexchange/52816-parallel-computing-toolbox-integration-for-matlab-distributed-computing-server-with-grid-engine

[Akhil Lohia]

Thanks Kojiro. I have the parallel computing toolbox but can you tell me how to check if I have the MATLAB Distributed Computing server?

Also, if I don't have that, then is there any way to use cores across different nodes in the cluster instead of depending on the number of cores available on one node? So by using something similar to MPI maybe?

[Kojiro Saito]

You can find whether you have a MATLAB Distributed Computing Server license in the license file (license.lic or license.dat) if "INCREMENT MATLAB_Distrib_Comp_Engine" line is written in the file.

Also, You can find whether MATLAB Distributed Computing Server is installed on the server if $MATLAB_INSTALL/toolbox/distcomp/bin/mdce (if UNIX) or $MATLAB_INSTALL\toolbox\distcomp\bin\mdce.bat (if Windows) is existing.

Without MATLAB Distribued Computing Server, I have no good idea. Maybe you can implement by using MPI across nodes...

[Walter Roberson]

It is possible to add MPI calls to transfer data, either through loadlibrary() or through a mex call. Or you could use tcp() or udp() to communicate. Or if you have a common file system, you could use memmapfile to communicate.

If you are transferring MATLAB data structures, you will have to find some way to serialize them. The official serialization routines are not exposed, but you can find a decent "fast serialize" contribution in the File Exchange.