I wanted to rum my code on parallel HPC cluster however I face with an erorr.

Starting parallel pool (parpool) using the 'carya-multinode' profile ...

Connected to parallel pool with 48 workers (PreferredPoolNumWorkers).

[Damian Pietrus]

What is the error you are receiving? Once the pool is open as stated in your post, you can interactively run your parallel code. You can verify the pool is open and get more information about each worker by running the getCurrentWorker command:

spmd, getCurrentWorker, end

[Abdolrazzagh]

The code that I'm running is added below. It is told that I should add the following commend to my code:

c = parcluster parpool(c)

Bt I don't know how to add this function to my code.

clear, close all
clc
LogData = readtable("all_logs.xlsx","Sheet","Sheet1");
% Depth	RHOP	DTP	DTS	LLD	PHIE	SWE	So	Sg	Vol_Cal	Vol_Shale
warning off
SGg = 0.6;
rh0 = 1.0;
depth = LogData.DEPTH;
rhob = LogData.RHOB;
dtp = LogData.DT;
res = LogData.RT;
phie = LogData.PHIE;
swe = LogData.Swe;
so = LogData.So;
sg = LogData.Sg;
vol_cal = LogData.VolCal;
vol_shale = LogData.VolShale;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
C_minerals(:,:,1) = [46.4117225794362,12.6621444491628,12.6621444491628,0,0,0;...
    12.6621444491628,46.4117225794362,12.6621444491628,0,0,0;...
    12.6621444491628,12.6621444491628,46.4117225794362,0,0,0;...
    0,0,0,12.7844541038098,0,0;...
    0,0,0,0,12.7844541038098,0;...
    0,0,0,0,0,12.7844541038098];
C_minerals(:,:,2) = convert_ku_c(62,26);

fr=0;
am1=[1, 1];
bm2=[1, 1];
cm3=[1, 1];
orientation_type_mineral={'chaotic','chaotic'};
theta_mineral=[];
phi_mineral=[];
psi_mineral=[];
mu_theta_mineral=[];
sigma_theta_mineral=[];
mu_phi_mineral=[];
sigma_phi_mineral=[];
alpha_theta_mineral=[];
beta_theta_mineral=[];
alpha_phi_mineral=[];
beta_phi_mineral=[];
aM_theta_mineral=[];
aM_phi_mineral=[];
%$
theta_pore=[0 0];
phi_pore=[0 0];
psi_pore=[0 0];
mu_theta_pore=[];
sigma_theta_pore=[];
mu_phi_pore=[];
sigma_phi_pore=[];
alpha_theta_pore=[];
beta_theta_pore=[];
alpha_phi_pore=[];
beta_phi_pore=[];
aM_theta_pore=[];
aM_phi_pore=[];
orientation_type_pore={'chaotic','chaotic'};
%$
phi=0;
theta=0;

R0=10^5*eye(3,3);
pH=7;
Z=[1 -1 1 -1]';
Cf=160000;
T=25;
beta= [5.2 7.9 36.3 20.5]'*10^(-8);
eta=10^(-3);

iso_aniso=1;

Km=zeros(3,3,2);
K0 = 9.8692326671601e-16 * eye(3,3);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
reza = numel(depth);
parfor i = 1:reza
    C_matrix=c2dto4d(vol_cal(i).*C_minerals(:,:,1) + vol_shale(i).*C_minerals(:,:,2));
    Vfrm=[vol_cal(i), vol_shale(i)];
    pt=phie(i);
    rho= rhob(i);
    VpLog = 304.8/dtp(i);
    ResLog= res(i);
    p = 2.71*9.81*depth(i)*0.001;
    T = 25/1000*depth(i);
    [~, ~, Ko] = PVT_oil2(rh0,p,T);
    [~, ~, ~, ~, ~, Kg, ~, ~] = PVT_gas(SGg,p,T);
    K_fluid = [2.2, Ko, Kg];
    SAT = [swe(i), so(i), sg(i)];
    lb = [0, 0, 1e-12, 1e-12, 1e-12, 1e-6, 1e-6, 1e-6, 3, 3];
    ub = [pt/2.5, pt/2.5, 1e-6, 1e-6, 1e-6, 1e-3, 1e-3, 1e-3, 100, 100];
    x0 = (lb+ub)/2;
    x0 (9) = 5;
    x0(10) = 5;
    Yexp = [VpLog, ResLog];
    options = optimoptions("fmincon","Display","none","UseParallel",true);
    [xSol, err(i)] = fmincon(@(x) PosteriorDensityRockPhysics_DeltTResPerm(x,...
        K_fluid,...
        SAT,C_matrix,fr,am1,bm2,cm3,...
        C_minerals,orientation_type_mineral,theta_mineral,phi_mineral,...
        psi_mineral,mu_theta_mineral,sigma_theta_mineral,mu_phi_mineral,...
        sigma_phi_mineral,alpha_theta_mineral,beta_theta_mineral,alpha_phi_mineral,...
        beta_phi_mineral,aM_theta_mineral,aM_phi_mineral,Vfrm,pt,theta_pore,phi_pore,...
        psi_pore,mu_theta_pore,sigma_theta_pore,mu_phi_pore,sigma_phi_pore,...
        alpha_theta_pore,beta_theta_pore,alpha_phi_pore,beta_phi_pore,...
        aM_theta_pore,aM_phi_pore,orientation_type_pore,rho,phi,theta,VpLog, ...
        ResLog,R0,pH,Cf,T,beta,eta,Z,Km,K0),x0,[],[],[],[],lb,ub,[],options);
    RFP(i,:) = xSol;
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    [Vpe, ~, Reff, ~] = RockPhysicsModel_DeltTResPerm(xSol(1:8),K_fluid,...
        SAT,C_matrix,fr,am1,bm2,cm3,...
        C_minerals,orientation_type_mineral,theta_mineral,phi_mineral,...
        psi_mineral,mu_theta_mineral,sigma_theta_mineral,mu_phi_mineral,...
        sigma_phi_mineral,alpha_theta_mineral,beta_theta_mineral,alpha_phi_mineral,...
        beta_phi_mineral,aM_theta_mineral,aM_phi_mineral,Vfrm,pt,theta_pore,phi_pore,...
        psi_pore,mu_theta_pore,sigma_theta_pore,mu_phi_pore,sigma_phi_pore,...
        alpha_theta_pore,beta_theta_pore,alpha_phi_pore,beta_phi_pore,...
        aM_theta_pore,aM_phi_pore,orientation_type_pore,rho,phi,theta,VpLog, ...
        ResLog,R0,pH,Cf,T,beta,eta,Z,Km,K0);
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    lambda_prior = [4;1/5;4;1/5;0.5;0.5*(1-pt/2)/(pt/2);0.5;...
        0.5*(1-pt/2)/(pt/2);0.5;0.5*(1-1e-9)/(1e-9);0.5;...
        0.5*(1-1e-9)/(1e-9);0.5;0.5*(1-1e-9)/(1e-9);0.5;...
        0.5*(1-1e-4)/(1e-4);0.5;0.5*(1-1e-4)/(1e-4);0.5;...
        0.5*(1-1e-4)/(1e-4)];
    lambdaLB = 0.5;
    lambdaUB = 1.5;
    ShapeFactor = 1 - 2*xSol(1:8);
    ScaleFactor = (2*xSol(1:8).^2 - 3*xSol(1:8) + 1)./xSol(1:8);
    ShapeFactorLB = 1 - 2*lambdaUB*xSol(1:8);
    ScaleFactorLB = (2*lambdaLB*xSol(1:8).^2 - 3*lambdaLB*xSol(1:8) + 1)./(lambdaUB*xSol(1:8));
    ShapeFactorUB = 1 - 2*lambdaLB*xSol(1:8);
    ScaleFactorUB = (2*lambdaUB*xSol(1:8).^2 - 3*lambdaUB*xSol(1:8) + 1)./(lambdaLB*xSol(1:8));
    %
    lambda = [3;2*1/xSol(9);3;2*1/xSol(10);ShapeFactor(1);ScaleFactor(1);...
        ShapeFactor(2);ScaleFactor(2);...
        ShapeFactor(3);ScaleFactor(3);...
        ShapeFactor(4);ScaleFactor(4);...
        ShapeFactor(5);ScaleFactor(5);...
        ShapeFactor(6);ScaleFactor(6);...
        ShapeFactor(7);ScaleFactor(7);...
        ShapeFactor(8);ScaleFactor(8)];
    lambdaLB = lambda/2;
    lambdaUB = 2*lambda;
    lambda = lambda + randn(20,1).*lambda/5;
    %
    [lambda_final(i,:),~,~, ~] = runfmincon_DeltTResPerm(lambda,lambda_prior,...
        lambdaLB,lambdaUB,K_fluid,SAT,C_matrix,fr,am1,bm2,cm3,...
        C_minerals,orientation_type_mineral,theta_mineral,phi_mineral,...
        psi_mineral,mu_theta_mineral,sigma_theta_mineral,mu_phi_mineral,...
        sigma_phi_mineral,alpha_theta_mineral,beta_theta_mineral,alpha_phi_mineral,...
        beta_phi_mineral,aM_theta_mineral,aM_phi_mineral,Vfrm,pt,theta_pore,phi_pore,...
        psi_pore,mu_theta_pore,sigma_theta_pore,mu_phi_pore,sigma_phi_pore,...
        alpha_theta_pore,beta_theta_pore,alpha_phi_pore,beta_phi_pore,...
        aM_theta_pore,aM_phi_pore,orientation_type_pore,rho,phi,theta,VpLog, ...
        ResLog,R0,pH,Cf,T,beta,eta,Z,Km,K0);
end

save all

[Damian Pietrus]

Hopefully I can provide some more insight into how a pool operates. Let's take a look at the parcluster command first:

c = parcluster;

What this command does is takes your default cluster profile and brings it into the workspace into the variable c. We can then operate on that in a few different ways.

parpool(c)

What this does is open an interactive parpool based on that cluster profile (c) that we just initialized. Once the pool is open, we can run our parallel code. One thing to note is that you can only have one parpool open at a time. If you already have one open and attempt to open a second, then you will run into an error. Therefore what I recommend doing is to use gcp at the top of your code, which gets a handle to the current pool if one exists. If not, we can create one:

clear, close all
clc

c=parcluster;
pool = gcp('nocreate');
if isempty(pool)
    pool = parpool(c);
end

LogData = readtable("all_logs.xlsx","Sheet","Sheet1");
...
(rest of your code here)
...
    

By adding this to the top of your code, you should be able to run it multiple times. Please give that a shot and let me know how it goes!

[Abdolrazzagh]

Thank you for answering my question. I dont want to rum my code multiple times, I want to rum my code on multiple nodes, which have myltiple cores???

[Damian Pietrus]

This is where things can get a little bit more complicated, so I'm going to generalize and make some assumptions about your setup.

Based on your first post, it looks like your default cluster profile is called 'carya-multinode'. I'm going to assume that this is an HPC cluster of some sort, and that it has a scheduler such as Slurm or PBS. These schedulers can accept lots of arguments on how to shape your job (which queue it goes to, how much memory your job needs, etc). At the very least, what we want to do is tell the scheduler how many MATLAB workers, and by extension, how many cores we want. We can do this with the parpool command:

c=parcluster;
pool = gcp('nocreate');
numWorkers = 50;
if isempty(pool)
    pool = c.parpool(numWorkers);
end

As you can see, we can change the numWorkers value to indicate how many cores we'd like the job to run on. It's then up to the scheduler to determine where those cores are going to land on the HPC cluster. Once the pool is open on one or more nodes, the parfor in your code will run across those workers.

There's more to consider when working with HPC, but that's the quick version of choosing a worker count.

If you'd like to reach out to me directly, please feel free to send an email to [e-mail]. Please include a link to this post and ask for me (Damian Pietrus), and the front-line technicians should pass it along.

[Walter Roberson]

See https://www.mathworks.com/help/parallel-computing/discover-clusters-and-use-cluster-profiles.html for information on discovering and managing cluster profiles.

Each cluster profile is named. To choose to execute on a specific cluster configuration, you would use

c = parcluster('ClusterNameGoesHere');

To run your code on multiple nodes with multiple cores, you need to first find out the name that has been associated with that configuration.