Removing noise from a signal

I have a following signal, and I want to correct the part which has a sharp jump, based on the value of the signal sometimes before the jump and its future values.

Instead of that jump I want something like a red line or green line, or something in between (sth that is most likely to be the signal instead of that noise based on the behavior of the signal in the pas and in the future.) Any suggestions?

Answer 1

[Image Analyst]

You can use a modified median filter. Basically compute the median. Then find any points where the actual signal is some factor higher than the median signal. Replace those points with the median. Something like

medianSignal = medfilt1(signal, 5); % Smooth signal
badIndexes = abs(signal-medianSignal) > someValue; % Find bad ones
signal(badIndexes) = medianSignal(badIndexes); % Repair signal

[MINA]

I want something more complicated algorithm. I can already find the maximum point which diverges from the signal (in this case the point at lfp=8000) but I don't know how much I should go left and right and call the other points as noise. And I don't want to just replace those noise with a fix value.

[Image Analyst]

You want something more complicated? Usually we hear the opposite - the solutions we offer are too complicated and the poster wants something simpler. There are tons of denoising algorithms out there. I'm not as familiar with 1-D signal denoising algorithms as with 2-D image denoising algorithms. You could do Fourier filtering in the spectral domain like Star suggested. This will filter the entire signal. So you have to ask yourself if it's okay to filter the entire signal or if you want just the spike fixed and all the other values to remain untouched. Keep in mind though that Fourier filtering will get rid of the spike more than it will affect the small little humps. You may not observe too much change in those small humps. I'm not sure if 2-D methods will work on a 1-D vector but they might. And you'll be happy to hear they're much more complicated. And they work better. State of the art image denoising algorithms include BM3D, non-local means, wavelets, K-SVD, K-LLD, UINTA, etc.

For a whole bunch of info on denoising, click here or here

A nice overview of a bunch of image filters can be found here: https://users.soe.ucsc.edu/~milanfar/publications/journal/ModernTour.pdf

Usually BM3D is found to be the best and state of the art, except of course when someone is publishing a paper trying to top it. However non-local means might be good in your case. There is a version of non-local means in the File Exchange that has been applied to 1-D ECG signals http://www.mathworks.com/matlabcentral/fileexchange/41762-non-local-means--nlm--denoising-for-time-series--applied-to-ecg

[Greg Dionne]

If you have a recent copy of the Signal Processing Toolbox you can try using hampel and fillgaps. First, try hampel directly:

hampel(x)

If that fails, and you want to do a little more processing... at first glance you might benefit from AR modeling.

Something like:

% find outliers
[~,isout] = hampel(x);

% clobber with nan
x(isout) = NaN;

% use two-sided AR modeling to extrapolate into each gap
fillgaps(x,100)

[Image Analyst]

Cool. I didn't know they had added those functions. Thanks Greg.

[Chappi]

aahhh. I want to try out your suggestion but I dont have signal processing toolbox. Would you please recommend sth else silimar with medfilt ? THank you

[Christine Tobler]

The function smoothdata in basic MATLAB (introduced in R2017a) has a median filter.

Answer 2

[Star Strider]

The ‘spike’ creates broadband, high-frequency noise. I would first do a fft of your signal to understand the frequency composition, then design a lowpass or bandpass filter to keep your frequencies of interest, and use it to filter your signal.

There are many ways to design filters in MATLAB, probably the easiest being designfilt. My IIR filter design procedure is outlined in: How to design a lowpass filter for ocean wave data in Matlab? Remember to do the actual filtering with the filtfilt function.

[632541]

Can i use built in function for LPF in matlab instead of designfilt or filtfilt

[Star Strider]

I would use the lowpass function (introduced in R2018a) if you have it. Otherwise, designing the filter from command-line functions is the only option.