Speed up my "for" loop

Hi Everyone,

Could you please help to speed up this piece of code? What it does, it basically converts 50 Hz data to a 5 Hz data, averaging each 10 values of each field. Second and last field names are not under this conversion. And the last # of values of each field is less than ten, therefore , I use last two lines inside the loop to take care of it.

so, in total there are 11 fieldnames, each containing 12666529 values

tic

for n = [1, 3:length(names)-1]

    for k = 0:count
        data.(names{n})(k+1) = mean(data.(names{n})((1+10*k):(10*(k+1))));
    end
    data.(names{n})(count + 2) = mean(data.(names{n})(total - (total - (count+1)*10 + 1):total));   
    data.(names{n})(count+3:total) = [];

    toc
end

The time I spent right now is 304 secs:

each loop takes around 25 secs

Elapsed time is 25.251421 seconds.

Elapsed time is 50.271446 seconds.

Elapsed time is 75.096751 seconds.

Elapsed time is 100.987307 seconds.

Elapsed time is 126.373710 seconds.

Elapsed time is 151.917404 seconds.

Elapsed time is 177.191153 seconds.

Elapsed time is 202.475424 seconds.

Elapsed time is 227.890811 seconds.

Elapsed time is 253.500255 seconds.

Elapsed time is 278.895494 seconds.

Elapsed time is 304.218428 seconds.

Elapsed time is 304.268410 seconds.

Thanks a lot! Nurlan

[Mazin Mustafa]

Hi,

I think that if you can use vectors instead of the for loop, this might speed up the calculations. Matrix operations are the fastest in MATLAB.

Answer 1

[Jan]

You forgot to pre-allocate the results. Letting the arrays grow iteratively requires a huge amount of memory allocations and copies. For 12666529 values the resulting vector grows 1266652 times and this requires sum(1:1266652) * 8 bytes to be allocated and copied: 802 GigaByte!

So in a first step a pre-allocation is recommended:

result = zeros(1, count + 1);  % Pre-allocate !!!
value  = data.(names{n});      % Faster shortcut
for k = 0:count
   result(k+1) = mean(value((1+10*k):(10*(k+1))));
end

For testing measure the timings with this method at first. Then in a next step the blockwise sum can be calculated in a vectorized form:

result = sum(reshape(data.(names{n})(1:10*(count+1), 10, []), 1) / 10;

[Nurlan Mukanov]

Hi,

Sorry, I didn't add that, but I preallocate data with size 12666529 - basically I copy the original struct with all the fields to data. This is what I put before the loop:

data = originalStruct; names = fieldnames(data);

tic count = floor(length(originalStruct.oneField)/10)-1; % count+2 will become a new size for data

total = length(originalStruct.oneField); % size of each field

Your code actually works pretty nicely and it does it twice as fast, but problem is that I get is: result - is a matrix, but I need a struct with the same fields as the original.

Thanks,Nurlan

[Jan]

Sorry, Nurlan, I thought this is clear. After my code you have to copy result back to your struct, of course:

tic
c1 = 10 * (count + 1);
for n = [1, 3:length(names)-1]
  value             = data.(names{n});
  result            = sum(reshape(value(1:c1, 10, []), 1) / 10;
  result(count + 2) = mean(value(c1 - 1:total));   
  data.(names{n})   = result(1:count + 2);
  toc
end

I've simplified the indexing a little bit.

[Nurlan Mukanov]

hey Jan,

In your first comment the last line didn't quite work...(or maybe I was doing something wrong) So, using your code from the first comment I got this(below) But it doesn't save me much...instead of 300sec I get 250secs, which is good, but not so much...

    tic

    names = fieldnames(originalStruct);
    % initialize a empty struct 
    size = [1:count+2]';
    data = struct(names{1},size, names{2},originalStruct.twoField,...
                  names{3},size, names{4},size,...
                  names{5},size, names{6},size,...
                  names{7},size, names{8},size,...
                  names{9},size, names{10},size,...
                  names{11},size, names{12},size,...
                  names{13},size, names{14}, size(end));

count = floor(length(originalStruct.oneField)/10)-1;  % size of each field divided by 10 and minus 1
total = length(originalStruct.oneField); % size of each field

    col = zeros(count+2,1);  % Pre-allocate !!!

    for n = ([1, 3:length(names)-1])
        for k = 0:count
           col(k+1) = mean(originalStruct.(names{n})((1+10*k):(10*(k+1))));
        end
        col(count+2)= mean(originalStruct.(names{n})(total - (total - (count+1)*10+1):total));

        data.(names{n}) = col;
        toc
    end

    toc

I didn't have time yet to work with your last comment... But i think it's close to what I did...

Thanks, Nurlan

[Jan]

I do not get it. Instead of calling mean() inside a loop, it is much faster to use reshape to convert the vector to a matrix with 10 rows, calculate the sum() and finally divide by 10.

Answer 2

[Ken Atwell]

Do you have access to Signal Processing Toolbox? Does resample do any better?

[Nurlan Mukanov]

Hi...Yes I do. never tried working with it though...How can i use resample?

[Ken Atwell]

I have not used is much, but from the doc:

• First argument is the signal
• Second argument is the desired sample rate
• Third argument is the current sample rate

It could actually be slower because it may be doing something more mathematically sophisticated than taking the mean, but it is worth trying. More on resampling here.

[Nurlan Mukanov]

Thanks Ken,

I think it's quite complicated to use resample for my case... I tried using it, but there is something complicated going on inside the function...

Thx, Nurlan