How can I filter/remove peaks in my curve? (not a signal)

Hi all.

I am trying to remove the peaks from my data using the Signal Processing Toolbox, without success.

In my data, which values are attached, there are peaks which values are significantly lower than the "correct data" at the beginning, and afterwards I have peaks with both higher and lower values. Also, there are infinite values which are not plotted and I want to ignore.

I have tried different commands but as I am quite new to MatLab and my curve is not a signal, I am not sure if I am using the ToolBox correctly.

Every help will be very much appreciated. Thank you very much.

P.S. I know the image is from Excel but of course I am managing the data in MatLab.

Answer 1

[Star Strider]

It is not obvious what result you want. If you just want a regression line through your data, first use the fillmissing (R2016b and later) function to eliminate the non-finite gaps (first replace the Inf values with NaN), then use polyfit and polyval to do a simple regression and evaluation.

I have not looked at your data because I do not know what you want to do. However the best approach to filter design is first to do a Fourier transform (fft) on your data to understand its spectral characteristics, then design your filter to elimiinate the unwanted frequencies. Your data have to be regularly-sampled (constant sampling interval) of none of these will work correctly. If they do not have a constant sampling interval, use the resample function (after fillmissing) first to correct that.

[Manuela Garcia]

Thank you very much for the answer and your time!

I am sorry that I couldn't be clear enough. What I would like to get, is a regression curve like the one I drew in the image and you can see here (sorry for the unprofessional explanation!).

The peaks mainly at the beginning are so pronounced that I don't want to take them into account, and therefore I want to delete them for the following analysis.

[Star Strider]

My pleasure.

Unless you have a mathematical model of the process that created your data that it would then be possible to fit, use polyfit and polyval to create a simple polynomial regression through your data. (Use fillmissing first.)

I do not have a good idea of what you want to delete from your data and how you want to do it.

Another option is to use the sgolayfilt function.

[Manuela Garcia]

The data was recorded from an experiment, so I didn't create it.

I guess I have to delete the points surrounded by a red circle, before applying the regression? (i.e. the very pronounced peaks)

Otherwise my results so far consist in a curve which is lowered in y-values compared to the black one which I would like to get.

Thank you again!

[Star Strider]

It took me a while to figure out how to approach this. Note that the data in the ‘data.mat’ file does not appear to bear a strong resemblance to the data in the plot you posted. (It is always a good idea to post the code you use to plot the posted plot images.) My code initially detrrends the data with a polynomial fit, then removes the outliers, adds the polynomial trend back, and then fits an other arbitrary polynomial to the ‘cleaned’ data.

The Code —

D = load('data.mat');
x = D.data(:,1);
y = D.data(:,2);

y(isinf(y)) = NaN;                                          % Replace ‘Inf’ with ‘NaN’
y2 = fillmissing(y, 'linear');                              % Interpolate ‘NaN’ Values

P1 = polyfit(x, y2, 5);                                     % Fit For Detrending
Yp1 = polyval(P1, x);                                       % Evaluated Fit For Detrending

y3 = y2 - Yp1;                                              % Detrend
[y4,TF] = rmoutliers(y3, 'median');                         % Remove Outliers From Detrended Data
yclean = y4 + Yp1(~TF);                                     % Reconstitute Original Data

P2 = polyfit(x(~TF), yclean, 7);                            % Fit ‘Cleaned’ Data To Arbitrary Polynomial
Yp2 = polyval(P2, x);                                       % Evaluate Fit Of ‘Cleaned’ Data To Arbitrary Polynomial With Original Data

figure
plot(x, y2, '.')
hold on
plot(x, Yp2, '-r')
hold off
grid

The rmoutliers function was introduced in R2018b, and fillmissing in R2016b.

I am not posting the figure that this code produces because it does not appear to be the data in the plot figures you posted. My code should work regardless.

[Manuela Garcia]

This is exactly what I needed!

Thank you very much for your time and patience understanding my issue!

P.S. I just realised I uploaded a different graph than the data I attached, as I have around 40 similar curves to analyse, but they all have the same trend and code can be applied to all, so again many thanks!

[Star Strider]

As always, my pleasure!

[Manuela Garcia]

Hey again,

This might be a stupid question, but as you helped me so much I thought it was worth asking!

As you can see in the plot, I get the "blue points" which are the initial data and the "red line" which is the solution after applying polyval and polyfit, if I correctly understood your code.

My problem is, when plotting Yp2 I get these two curves, connecting points with different spacings, and I would like to have only the one which connects all points. I tried to look in MatLab answers but can't find this specific case.

Thank you very much again! Your help saves me a lot of time!

[Star Strider]

One problem I noticed with the data set you posted was that the data doubles back on itself. If you plot it as a line instead of points, that is easily apparent.

I am not certain how you want to deal with that problem.

One way might be to sort the data by the x-values, although with some of your data that might not be the most optimal solution.

Another way would be to only use the initial data to the maximum value of the x-vector (not the last value).

Using your posted ‘data.mat’, that would be:

D = load('data.mat');
x = D.data(:,1);
y = D.data(:,2);

y(isinf(y)) = NaN;                                          % Replace ‘Inf’ with ‘NaN’
y2 = fillmissing(y, 'linear');                              % Interpolate ‘NaN’ Values

[max_x,idx] = max(x);                                       % Value & Index Of Maximum X-Value
xe = x(1:idx);                                              % Trim Vector
y2e = y2(1:idx);                                            % Trim Vector

P1 = polyfit(xe, y2e, 5);                                   % Fit For Detrending
Yp1 = polyval(P1, xe);                                      % Evaluated Fit For Detrending

y3e = y2e - Yp1;                                            % Detrend
[y4e,TF] = rmoutliers(y3e, 'median');                       % Remove Outliers From Detrended Data
yclean = y4e + Yp1(~TF);                                    % Reconstitute Original Data

P2 = polyfit(xe(~TF), yclean, 7);                           % Fit ‘Cleaned’ Data To Arbitrary Polynomial
Yp2 = polyval(P2, xe);                                      % Evaluate Fit Of ‘Cleaned’ Data To Arbitrary Polynomial With Original Data

figure
plot(xe, y2e, '.')
hold on
plot(xe, Yp2, '-r')
% plot(x(~TF), yclean)
hold off
grid
ylim([-1E+3 1E+4])

That seems to work well with this data set.