Running on AWS server does not benefit from more cores in parallel pool?

I am using the built in Genetic Algorithm functionality in the Optimization Toolbox and time is a serious constraint when running the code.

The problem in question is an integer (binary) problem with linear constraints.

The solutions are on the order of a 400x1 vector of ones and zeroes, and the population size is 500. All other (relevant) Genetic Algorithm options are left as default.

The main bottleneck on run time is within the fitness function (it does not receive the population in vectorized form because this conflicts with using a parallel pool). It has to perform numerous operations on a 3D matrix that is roughly 300000x4x10. These operations are vectorized as much as possible to take advantage of Matlab's speed.

I start a parallel processing pool before calling the algorithm as follows:

    delete(gcp('nocreate'));
    c = parcluster('local'); % build the 'local' cluster object
    numberOfCores = 6;
    parpool(numberOfCores);

With 6 cores it takes roughly five minutes for each generation to complete (when running as a compiled DLL called by .NET). I have tried to set up an AWS server with 20 cores in hopes of speeding up the run time of each generation, but instead the run time per generation was roughly the same or slightly worse. (Matlab verified that the 20 core pool was properly created).

Does anyone know why that would be the case? Is there some way to estimate if there is a "sweet spot" for number of cores in a parallel processing pool compared to any potential overhead associated with parallelizing operations?

[Run Zhu]

Maybe it is off the topic. But the performance one would obtain is typically: optimization model > optimization algorithm > parallel computing. As in your case, any particular reason to choose GA? Since it is usually slower than intlinprog and patternsearch, consider other solver would be more beneficial.

Answer 1

[Doug Rank]

I believe I have investigated using other options, but none of them had the necessary components.

The objective function is way too complex to be formulated in a way that would work for intlinprog. (In reality, the solution x for the problem I'm working on only defines a set of rules for how an algorithm within the fitness function will behave to calculate the ultimate value of a given solution).

Patternsearch, if I understand correctly, does not allow for the solution vector to be constrained to integer values, so it would not work in this situation either. If it's particularly susceptible to local minima - that's also a problem.

Answer 2

[Walter Roberson]

patternsearch is not "particularly susceptible to local minima": it is designed to have a reasonable chance of escaping local minima (but will certainly get stuck if they are deep enough.)

Since all of your inputs are integer (not a mix of integer and non-integer) you can use patternsearch using the techniques described at https://www.mathworks.com/matlabcentral/answers/285753-pattern-search-with-integer-decision-variable#answer_223391