Correlating moving window averages. Spurious correlations.

I am taking moving window averages of two variables and then correlating the new data. I was noticing extreme pseudo-cyclical changes in the correlation values with changing window size. I injected random data instead into the algorithm and a very strong correlation appeared when the window size became large enough. This was reproducible with other random datasets. I am guessing this is simply because the windowed values tend toward the mean with increased size, but does anyone have experience with this or have a better idea of the dynamics involved.

Here is the main code. Inputs are one circular variable (phase angle 0-2π) and one linear variable.

for a = 1:2:(length(rt))-1
k=waitforbuttonpress;
window=a

j=1

for i = window:length(rt); % This loop 

    rt_std(j) = std(rt(j:i));%/mean(rt(j:i));
    angs=resp_angles(j:i);
    %av_ang(j)=mean(angs);
    angs=mod(angs,2*pi); % convert -pi/+pi to 0-2pi
    ang_sin = sin(angs);
    ang_cos = cos(angs);
    ang_sin_sum = sum(ang_sin);
    ang_cos_sum = sum(ang_cos);

    av_ang(j)=mod(atan2(ang_sin_sum,ang_cos_sum),2*pi); % average phase angles and convert to 0-2pi

    %k=waitforbuttonpress;

    j=j+1;

end

[r,p]=corrcoef(av_ang,rt_std)

This loops through for all iterations of window size from 1 to n step 2.

Thanks (:

[the cyclist]

My first instinct would be to plot the data to see what is going on.

[Chad Greene]

I recommend using Aslak Grinsted's moving function. It's computationally much faster than computing in a loop, and it's simple to use.

[Miguel Melnychuk]

Yes have plotted excessively with iterating incremental window sizes and all possible combinations of variables. It is very possible to create a non-existent correlation using this method. I would think, having looked at the emerging patterns from incrementing window size across all possible window sizes, that some further statistic applied to the resulting correlation graph would indicate something meaningful, but by itself a moving-window average, when applied in a correlation analysis, seems somewhat suspect to me. I would love to hear from someone with more experience than myself.

[Michael Melnychuk]

If anyone is curious I came back to this topic a few months ago, and did extensive analysis on it. Basically, I found that if the moving window average is overlapped then spurious correlations tend to arise. If the window is not overlapped, however, the correlation does not tend to increase. I'm only jotting this down in case someone is looking for an answer to this question. I'm surprised it isn't more commonly asked.

Answer 1

[Image Analyst]

Well we don't have your data so I can't say much. But what I can say is that a high correlation does not necessarily mean the signal and the template look like each other at that location. Unfortunately that is a very common misperception. All it means is that the sum of the products is a big number, which can happen for example if you have a huge noise spike in either the signal or template. And you said "I injected random data", so there you go.