The variable s in a parfor cannot be classified.

I would like to run the following code snippet in parallel as the size of the data is too huge.

    data_no = 10099;

    alpha = 0.5;

    m = 1;

    parfor i=1:data_no

        parfor j=i+1:data_no
            s1 = norm(data_corr(i,:)-data_corr(j,:));
            s2 = norm(data(i,:)-data(j,:));
            d = s1 + alpha*s2;
            s(m,1) = i;
            s(m,2) = j;
            s(m,3) = -d;
            m = m+1;
        end
    end

But I get the error "The variable s in a parfor cannot be classified". I have not had much luck with understanding the matlab help unfortunately.

Can somebody please help.

Thanks,

Janki

Answer 1

[Matt J]

Here is a method that avoids looping altogether. I expect it would be much faster than a parfor approach.

    data_no=size(data_corr,1);

    [I,J]=ndgrid(1:data_no);
    idx=(J>I);

    D=normMatrix(data_corr)+alpha*normMatrix(data);

    s=[I(idx), J(idx), -D(idx)].';

         function S=normMatrix(data)

              normterms=sum(data.^2,2);
              crossterms=data*data.';

              S=bsxfun(@minus,normterms, 2*crossterms);
              S=bsxfun(@plus,S,normterms.');
              S=sqrt(S);

          end

[Janki Mehta]

Thanks a lot Matt. The code took about 100 times less time to run.

But I still have one more issue. My data (fMRI time series) is about the size 60000x100, which gives me errors of Out of Memory. I guess this can possibly be solved if use some high-performance workstation.

But the bigger problem is the following error for the line

[I,J]=ndgrid(1:60000);.

Maximum variable size allowed by the program is exceeded.
Error in ndgrid (line 63) 
varargout{1} = x(:,ones(size(y)));

Can you please help me this?

Thanks, Janki

[Matt J]

With data_no=60000, the 's' you are trying to compute would be 40GB in double floats or half that in single floats. You could get that much RAM if you were really determined, I suppose, but you should probably be rethinking your goals instead...

Answer 2

[Matt J]

    [I,J]=ndgrid(1:data_no);
    idx=(J>I);
    I=I(idx); J=J(idx);
    N=length(J);

    s=nan(3,N);

    parfor m=1:N

           i=I(m);
           j=J(m);

            s1 = norm(data_corr(i,:)-data_corr(j,:));
            s2 = norm(data(i,:)-data(j,:));
            d = s1 + alpha*s2;
            s(:,m)=[i;j;-d];

    end

     s=s.';