Keep parpool open instead of open/close for each iteration. Possible?

Good day and thank you in advance for your time and help. I am running a surrogate optimization using the surrogateopt script, that runs a ray tracer which uses a parpool to trace the rays.

An exemplary code of the ray tracer would be the following:

parpool('local',4);

Number_of_rays = 100000;
spmd
    for i=drange(1:Number_of_rays)
      % ray's are traced
      
      %if hit, a counter is raised
      if hit==true
        absorption_counter=absorption_counter+1;
      end
      
    end
    
labBarrier % we wait for all the workers to stop tracing

absorption_counter = gplus(absorption_counter); %we add the counters of each worker

end %spmd

delete(gcp('nocreate'))

So surrogateopt call the ray tracer and run it many times. However, I've noticed that the time that takes to start the parallel pool is quite high.

Starting parallel pool (parpool) using the 'local' profile ...

I was just wondering if you can somehow keep the pool open and run the optimization iteratively but opening the parpool once, and then not closing it.

I guess that the absorption_counter would need to be resetted each time (set to 0 before running again), but I don't know if this would work.

Thank you for your input!

Best,

Sebastian

[Mario Malic]

This line stops parpool. It will be automatically turned of after 30 mins of inactivity I think.

delete(gcp('nocreate'))

Answer 1

[Hassaan]

% Initialize the parallel pool with the desired number of workers
% Ensure this number does not exceed the maximum number of workers allowed on your system
poolObj = gcp('nocreate'); % Get the current parallel pool object
if isempty(poolObj)
    poolObj = parpool('local', 2); % Adjust the number of workers according to your system
end

% Define the number of rays and iterations
Number_of_rays = 100000;
number_iterations = 50; % Set the number of iterations for the secant method

% Preallocate the matrix to store iteration results
results_matrix = zeros(number_iterations, 3);

% Main loop for the secant method
for iter = 1:number_iterations
    % Initialize the absorption counter for this iteration
    absorption_counter = 0;
    
    % Run the computation in parallel
    spmd
        local_absorption_counter = 0; % Initialize local counter for each worker
        
        % Parallel loop for rays tracing
        for i = drange(1:Number_of_rays)
            % Simulate ray tracing here...
            hit = rand < 0.5; % Dummy condition for hit
            
            % If hit, increment the local counter
            if hit
                local_absorption_counter = local_absorption_counter + 1;
            end
        end
        
        % Synchronize the workers and sum the local counters
        spmdBarrier;
        absorption_counter = spmdPlus(local_absorption_counter);
    end
    
    % Collect the result from the parallel block (spmd)
    total_absorption = absorption_counter{1}; % Get the result from the first lab
    
    % Calculate the error (placeholder values for exact and approx)
    % You would replace 'exact_solution' with the value of your exact solution
    exact_solution = 12345; % Placeholder for the exact solution
    approx_solution = total_absorption; % Placeholder for the approximated solution from the current iteration
    error = abs((exact_solution - approx_solution) / exact_solution) * 100;
    
    % Store the iteration results
    results_matrix(iter, :) = [iter, approx_solution, error];
end

% Display the results matrix
disp('Iteration results:');
disp(results_matrix);

% Close the parallel pool if you're done with all parallel computations
% delete(poolObj); % Uncomment this line if you want to close the pool

Please replace the placeholders like exact_solution, approx_solution, and the dummy condition for hit with your actual logic and calculations.

Note:

I used MATLAB R2023b

Output:

Iteration results:

1.0e+04 *

0.0001 4.9976 0.0305

0.0002 4.9877 0.0304

0.0003 5.0095 0.0306

0.0004 4.9981 0.0305

0.0005 4.9823 0.0304

0.0006 4.9939 0.0305

0.0007 4.9803 0.0303

0.0008 5.0107 0.0306

0.0009 4.9869 0.0304

0.0010 4.9901 0.0304

0.0011 5.0248 0.0307

0.0012 5.0106 0.0306

0.0013 5.0295 0.0307

0.0014 4.9952 0.0305

0.0015 5.0119 0.0306

0.0016 4.9926 0.0304

0.0017 4.9979 0.0305

0.0018 4.9810 0.0303

0.0019 5.0103 0.0306

0.0020 4.9841 0.0304

0.0021 5.0484 0.0309

0.0022 5.0156 0.0306

0.0023 4.9970 0.0305

0.0024 4.9889 0.0304

0.0025 4.9937 0.0305

0.0026 5.0417 0.0308

0.0027 4.9775 0.0303

0.0028 4.9955 0.0305

0.0029 5.0080 0.0306

0.0030 4.9746 0.0303

0.0031 5.0127 0.0306

0.0032 4.9700 0.0303

0.0033 5.0011 0.0305

0.0034 4.9860 0.0304

0.0035 5.0043 0.0305

0.0036 5.0167 0.0306

0.0037 4.9996 0.0305

0.0038 4.9999 0.0305

0.0039 4.9877 0.0304

0.0040 4.9733 0.0303

0.0041 4.9969 0.0305

0.0042 4.9934 0.0304

0.0043 4.9995 0.0305

0.0044 5.0303 0.0307

0.0045 5.0180 0.0306

0.0046 4.9946 0.0305

0.0047 5.0026 0.0305

0.0048 4.9867 0.0304

0.0049 5.0093 0.0306

0.0050 4.9710 0.0303

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems | Firmware Developement | Simulations
• Electrical and Electronics Engineering

[Sebastian Sas Brunser]

Hi Muhammad,

This wouldn't work, because the iterations happend outside this script.

Maybe I should have added that the iterations happen within a surrogate optimization (surrogateopt) script. I'll add this info to my original post. Thanks a lot for such a complete answer.

[Mario Malic]

There is an option for surrogateopt to run the optimization in parallel which is UseParallel.

[Sebastian Sas Brunser]

Hi Mario,

Yes, true. But you can't run parallel job on parlallel optimization. It doesn't branch, so I would need to change the code. Anyways, I don't know if changing which part is parallelize will solve the problem. But yes, thanks!

[Mario Malic]

When you say, it doesn't branch - do you expect parallel in that case means that each worker should have an initial guess and start optimization process separately? If that's what you want, this is an option https://uk.mathworks.com/help/gads/multistart.html

Answer 2

[Hassaan]

@Mario Malic A basic approach. Let me know if it helps. Thanks.

% Initialize the parallel pool
poolObj = gcp('nocreate'); % If no pool, do not create new one.
if isempty(poolObj)
    poolObj = parpool('local', 2); % Modify based on your system's capability
end

% Set the number of simulations and other parameters
num_simulations = 10000;
% ... (Other parameters initialization)

% Define the surrogate optimization function
opts = optimoptions('surrogateopt', 'UseParallel', true);

% Define the objective function for surrogate optimization
objectiveFunction = @(x) myObjectiveFunction(x); % Define your objective function properly

% Call the surrogate optimization
[sol, fval, exitflag, output] = surrogateopt(objectiveFunction, lb, ub, opts);

% After the optimization
% ... (Do something with the results)

% Optionally close the parallel pool if you're done with all parallel computations
% delete(gcp('nocreate'));

• The parallel pool is checked and created if not already initialized.
• The optimization options are set to use the parallel pool.
• The objective function is defined, which should include the logic for your specific optimization problem.
• surrogateopt is called with the objective function, lower and upper bounds (lb and ub), and the options structure.
• After the optimization, you can process the results as needed.

You need to replace myObjectiveFunction with your actual objective function and define the lower and upper bounds (lb and ub) according to your problem.

The parallel pool will remain open throughout the surrogate optimization process. Once all computations are complete, you can choose to close the pool or leave it open if further parallel computations will follow. Remember that keeping the parallel pool open consumes system resources, so close it when it's no longer needed.

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems | Firmware Developement | Simulations
• Electrical and Electronics Engineering