Fmincon error "Provided objective gradient does not have the correct number of elements."

Hello,

I am optimizing my propeller design by fmincon function with sqp algorithm. In order to decrease the computational cost, I loose the constraint to an acceptable level. I use XFOIL + XROTOR to design my propeller. Those Fortran programs only have 4 demical precision. Below is my setup:

function [x, f, eflag, output, grad] = runoptimiser(x0, lb, ub, rOverR, Oper, const)

xLast = [];
myf   = [];
ineq  = [];
eq    = [];
                              
options = optimoptions(@fmincon,'Display','iter-detailed', ...
                                'Algorithm','sqp', ...                                
                                'FunValCheck', 'off',...
                                'FiniteDifferenceType', 'forward',...
                                'UseParallel', true, ...
                                'DiffMinChange', 0.05,...
                                'OptimalityTolerance', 1e-3, ...                                
                                'StepTolerance', 1e-3, ...
                                'ConstraintTolerance', 1e-3, ...
                                'PlotFcn',  {@optimplotfval, @optimplotx, ...
                                @optimplotfirstorderopt, @optimplotfunccount, ...
                                @optimplotstepsize, @optimplotconstrviolation, ...
                                @optimplotfunccount}, ...
                                'OutputFcn', @outputFcn_global, ...
                                'MaxIterations', 15);
[x,f,eflag,output,~,grad,~] = ...
fmincon(@(x) objfun(x),x0,[],[],[],[],lb,ub,@(x) constr(x),options);

    function f = objfun(x)
        if ~isequal(x,xLast)
            [myf, ineq, eq] = propeller_analysis(x, rOverR, Oper, const);
            xLast = x;
        end
        f = myf;
    end

    function [c,ceq] = constr(x)
        if ~isequal(x,xLast)
            [myf, ineq, eq] = propeller_analysis(x, rOverR, Oper, const);
            xLast = x;
        end
        c = ineq; 
        ceq = eq;
    end

If there is a infeasible design, I will return NaN to objective function and constraints.

However, when I use the code, this error appears sometime:

Provided objective gradient does not have the correct number of elements.

I have checked the code and try to play with DiffMinChange, but it doesn't really solve the problem. Does anyone have experience on this error?

I really appreciate any help. Thanks!

[Torsten]

We don't see from your code where you return the gradient. Do you return a vector of NaN which has the same size as x in case of an infeasible design ?

[Yi-Kai Peng]

Hello Torsten,

I didn't have the code to return the gradient since I didn't use a user-defined gradient funtion. The error message happens after 3 or 4 iterations usually.

[Torsten]

Does the problem also arise if you set "UseParallel" to "false" ?

[Yi-Kai Peng]

Hello Torsten,

I didn't try to turn "UseParallel" to "false" and see if it happens. I will try it. But how does it relate to the problem?

[Torsten]

In the course of time, I read about some "strange things" happening in parallel computations.

If this is not the reason, why else should the error message appear if you don't provide the objective gradient ?

[Yi-Kai Peng]

I think it is hard to provide the objective gradient in my case because there are two external programs porviding values and influencing the search direction in fmincon.

[Yi-Kai Peng]

Hello Torsten,

Thank you. You are right! I run the failed optimization without parallel pool and it works perfectly.

However, I also found out the final objective function value is different between parallel and non-parallel optimization.

[Torsten]

However, I also found out the final objective function value is different between parallel and non-parallel optimization.

I'm not surprised. And which value is better ?

[Yi-Kai Peng]

It seems like the value from non-parallel pool is better. However, I have to rerun some data points to compare to my reference again.

Do you have any link or literature that I can refer to debug the code and make parallel pool works? It seems my setting is fine.

[Torsten]

https://uk.mathworks.com/help/optim/ug/using-parallel-computing-with-fmincon-fgoalattain-and-fminimax.html

The fact that you coupled MATLAB with two external programs will of course complicate things further.

[Yi-Kai Peng]

Thank you! I will take a look at it. Also, I think it could be because I am returning NaN to objective function and constraints. This might make the parallel pool unstable.

[Walter Roberson]

Is it possible that when you return NaN that you are returning scalar NaN in a situation that expects a vector of NaN ?

[Yi-Kai Peng]

Hi Walter,

This is my way to return NaN

myf = NaN;
ineq = [NaN NaN];