Fmincon + Interior point algorithm + Memory problems---How to fix this problem please>

Dear All,

I keep receiving this error message when implementing an optimization with fmincon (interior point algorithm). I seems that it is a memory problem, any clue of how to fix this one please?

??? Error using ==> ldl Out of memory. Type HELP MEMORY for your options.

Here is my code:

beq = [ 1 const1 const2]' ;

months={'Jan', 'Feb', ...,'Dec'};

     for i=1997

        for j=1:length(months)

            w0=xlsread(P:\excelfile.xls ,'Begin','S5:S5000');
            a1=xlsread(['P:\excelfile.xls,'Begin','H5:H5000');
            a2=xlsread(['P:\excel file.xls'],'Begin','Z5:Z5000');

            a = [a1, a2];
            n = size(a,1) ; 

            Aeq = vertcat(ones(1,n),a') ; 
            lb = zeros(n,1) ; 
            H = eye(n) ;             
            [w,fval, exitflag] = fmincon(@(x)norm(H*x-w0)^2,w0,[],[],Aeq,beq,lb,[],[],options);

        end

     end

Any guidance or advice is much appreciated. Regards

Saad

[Walter Roberson]

What is changing in the loop? I do not see anything that is varying wit h"i" or "j" ? And I do not see you storing the results anywhere ?

[Saad]

nothing changes in the optimization. The only thing changing in the loop is the excel file but the optimization (i.e. constraints etc) stays the same.

Regards

[Saad]

Basically the loop changes the data but the optimization procedure is the same.

[Walter Roberson]

Where in the code is the data file changing?

The code you posted is broken for the first two xlsread() calls, as the filename lacks all quotation marks in the first one and it lacks the closing quotation mark and closing ] in the second call. Please post accurate code as the solution might happen to be in something subtle.

[Saad]

Hi Walter,

Sorry I should have been more specific from the beginning: Here is the code:

beq = [ 1 const1 const2]' ;

options=optimset('TolFun',0.00001,'TolX',0.00001,'Algorithm','interior-point','MaxIter', 1000000,'MaxFunEvals', 1000000,'Hessian','lbfgs');

months={ 'Jan', 'Feb','Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'};

     for i=1997:2012

        for j=1:length(months)

            k=(i-1997)*12+j;
             baseFileName = sprintf('%s%d.xls',  months{j}, i);
            fullFileName = fullfile('P:\indexbondanalysis\New Bloomberg Data CL Analysis\Data\', num2str(i), baseFileName);

             if exist(fullFileName, 'file')
  a1 = xlsread(fullFileName, 'Begin', 'X5:X5000');
            a2 = xlsread(fullFileName, 'Begin', 'Z5:Z5000');
            w0=xlsread(fullFileName ,'Begin','S5:S5000');

            end  

            a = [a1, a2];
            n = size(a,1) ; 

            Aeq = vertcat(ones(1,n),a') ; 
            lb = zeros(n,1) ; 
            H = speye(n) ; 

            [w,fval, exitflag] = fmincon(@(x)norm(H*x-w0)^2,w0,[],[],Aeq,beq,lb,[],[],options);

        end
     end

This problem should be a better fit for lsqlin function in matlab but the code crashes and it says "Warning: Matrix is singular, close to singular or badly scaled. Results may be inaccurate. RCOND = NaN. " I am trying to use quadprog for this optimization but I don't know how I could change it so that it fits the function quadprog. Any advice would be much appreciated

Regards

S

Answer 1

[Chetan Rawal]

Check out:

http://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html

http://www.mathworks.com/help/matlab/matlab_prog/strategies-for-efficient-use-of-memory.html

[Saad]

Thank you I will have a look

Regards

S

Answer 2

[Walter Roberson]

People have encountered memory problems when using xlsread() for many files in the same session. I do not know if the problem has been tracked down. If you are using MS Windows then more efficient and less of a memory problem is to open Excel through ActiveX and then to control it to do the reading.

Your code could also run out of memory if the files were sufficiently big.

In the case where an input file does not exist, your code repeats the previous optimization.

[Saad]

Hi Walter, Thanks for the advice (Indeed each excel file i have contains lot of information), this might explain why the code stops. But leaving the memory issue aside, do you think it is possible to reformulate the fmincon problem into quadprog? I have trouble doing that because I cannot change the "x" in quadprog :s...Thanks a lot for any reply or hint you could provide.

S

[Walter Roberson]

Sorry I am not familiar with quadprog()