Objective function for Lasso

Hi,

We would like to compute the Bayesian Information Criterion (BIC) for some models estimated using the lasso function in the statistics toolbox. We have been trying to work out exactly what the objective function is that the lasso function is optimising. We have used the following formulation, based on what we can glean from doc lasso:

% generate random dependent variable and design matrix
Y = rand(50,1);
P = rand(50,100);
L = .5; % arbitrary lambda value
A = 1; % set alpha to 1

% Fit lasso model for arbitrary lambda
[Beta Info] = lasso(P, Y, 'Lambda', L, 'Alpha', A); 

Yh = P*Beta; % compute predicted scores
R = Y-Yh-Info.Intercept; % compute residual term based on formulation in help lasso

    SR = sum(R)./(2*length(Y));
    K = size(Be,1);
    n = length(Y);
    F = SR+L*(((1-A)/2)*sum(Beta.^2)+(A.*sum(abs(Beta)))); % objective function for elastic net as per help lasso

   BIC = -2.*log(F)+(K*(log(n))); % compute BIC

One thing that seems a little odd to us is that SR is computed using actual rather than squared residuals. This means that positive and negative values can cancel each other out. The fomrmula in help lasso does not indicate that squared residuals are used, so we were wondering whether the above interpretation is correct.

Thanks and regards,

Alex

Answer 1

[Ilya]

You are asking for a definition of BIC for lasso. This is not really a MATLAB question. You might have better luck posting it to sci.stat.math or a similar group.

I am not an expert. For what it's worth, using residuals instead of residuals squared looks odd to me too. (But I don't see a problem with lasso documentation; it's a problem with your definition of BIC.) Two other things look odd to me as well: Including the lasso penalty in the variance term and using the number of all predictors in the model as the number of degrees of freedom. A quick Google search finds this paper:

http://www.stanford.edu/~hastie/Papers/dflasso.pdf

Their definition of BIC for lasso looks nothing like yours.