Eigs in multinode cluster

I am new to high performance computing and I am now trying to reconfigure one of my matlab code to a HPC.(I know it is probably not preferred, but I kind of need it in urgent.)

The task is to solve the lowest 6 eigen values for a 42500*42500 complex sparse Hermician matrix.

I have access to minimum 4 nodes and each node have more than 16GB memory.

I tried to submit this job but get an error

Out of memory. Type HELP MEMORY for your options.

Error in eigs/checkInputs/LUfactorAminusSigmaB (line 987)
                [L,U,pp] = lu(AsB,'vector');

Error in eigs/checkInputs (line 809)
            [L,U,pp,qq,dgAsB] = LUfactorAminusSigmaB;

Error in eigs (line 96)
    checkInputs(varargin{:});

Error in TwoDsolve (line 65)
    [psi,E]=eigs(H,NOS,'sm');

Error in Loop_Main (line 9)
    [psi,En] = TwoDsolve( V,m2Dx,m2Dy,Grd,'eigs',NOS);

Apparently it only use My question is: is there a way to configure eigs so that it can use all the memory I required from different nodes? I attach my submission file here as well.

#SBATCH --mail-user=[e-mail]
#SBATCH --mail-type=ALL
#SBATCH --time=00:60:00
#SBATCH --nodes=4
#SBATCH --mem=16384
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=4

echo "myjob.sub"
hostname

module load matlab/2016a
cd $PBS_O_WORKDIR
unset DISPLAY

matlab -nodisplay -r Control

Thanks, Wister

Answer 1

[Andrew Knyazev]

EIGS has limited support for distributed memory, so you can run it only on a single node, but see the answer from Christine Tobler below.

Google, e.g., for LOBPCG implementations in BLOPEX, SLEPc, Trilinos, hypre, MAGMA for parallel MPI-based eigensolvers that scale to thousands of nodes and run on GPUs.

https://www.mathworks.com/matlabcentral/fileexchange/48-locally-optimal-block-preconditioned-conjugate-gradient

Revision 4.16 adds support for distributed or codistributed arrays available in MATLAB BigData toolbox, e.g.,:

A = codistributed(diag(1:100)); B = codistributed(diag(101:200));

[blockVectorX,lambda]=lobpcg(randn(100,2),A,1e-5,5,2)

[Christine Tobler]

That's great!

Answer 2

[Christine Tobler]

The eigs function is supported for codistributed matrices in the Parallel Toolbox (see help codistributed/eigs. However, this version of eigs does not support the 'sm' / 'smallestabs' option (except for the case of a diagonal or triangular input matrix).

You could try to use the 'smallestreal' option instead, but this often has convergence issues, as it does not compute a decomposition of A, and also doesn't use a preconditioner, as LOBPCG typically does. However, this should be quick enough to try out, as it is just a one-liner.

[Christine Tobler]

Sorry, I realize I started out with the details when I should have started more generally: To distribute a MATLAB array over several compute nodes, use the Parallel Toolbox, and its distributed or codistributed classes (<https://www.mathworks.com/products/parallel-computing.html Parallel Toolbox Documentation>).