Lowess smoothing has 49,226 calls , taking forever, anyway to speed up?

I have an array that is 24613 cells of data and I am trying to run:

if true
  Smooth=smooth(GraphTime, Data(:,1),800,'lowess');
end

The problem is lowess looks at every cell twice for the comparison and smoothing, so thats 49226 calls which according to the profiler takes 117 seconds to complete this and I am doing it twice for separate data sets in each run! How can I shrink this time?

[matlabuser12]

ds=sort(d(in))

this line seems to be taking up 75% of the time lowess run time. sorting a lot of cells should take a while i get that, can i change the sorting algorithm in any way to speed things up?

[matlabuser12]

I should mention that I have the parallel computing toolbox too, can I use that somehow?

Answer 1

[matlabuser12]

I found the problem. In my smooth call: I am doing a double smoothign for both the (x,y) variables. so it is comparing all of them bewteen graphtime and data. if i delete graphtime it looks about the same and is 0.6 seconds fast. basically do just this:

    Smooth=smooth(Data(:,1),800,'lowess');

Answer 2

[Sean de Wolski]

Yes, you should be able to use a parfor-loop (rather than for or cellfun) to loop over your cells and do the analysis:

C = your_cell_array
C2 = cell(size(C));

parfor ii = 1:numel(C)

    C2{ii} = your_smoothing_algorithm(C(ii))

end

[matlabuser12]

I dont have a for loop though. should I just add one? my line is literally the one i posted:

     Smooth=smooth(GraphTime, Data(:,1),800,'lowess');

[Sean de Wolski]

Well what is each cell then? Terminology can be very important. You said there are 24613 cells of data which implies to me you need to loop over each cell of data and call smooth on it.

[matlabuser12]

Each cell has a number in it. It is a 24613x1 array of double.

[Sean de Wolski]

So it's not cells! It's doubles. You might be able to see a speedup with blockproc and 'UseParallel'. That would be the next thing I would try, make sure you provide enough block overlap for lowess to work.

[matlabuser12]

that seems to be part of the image processing toolbox. i have it, but I am confused where/how i implement it into my code

[Sean de Wolski]

I don't even think you need PCT:

tic
smooth(rand(30000,1),800,'lowess');
toc

On my laptop:

Elapsed time is 0.117642 seconds.

Why do you need it to be faster than that?

On my machine, the data communication for Parallel is more than the gain.

tic
x = blockproc(rand(30000,1),[5000,1],@(ds)smooth(ds.data,800,'lowess'),'BorderSize',[800,0],'UseParallel',true);
toc

Elapsed time is 0.527862 seconds.

[Dan Gianotti]

Recognizing that this post is more than three years old, but just in case anyone out there reads this, try the more general usage instead:

tic
smooth(rand(30000,1),rand(30000,1),800,'lowess');
toc

This takes a couple of orders of magnitude longer. If anyone has solutions still in 2018, I'd be interested.