Outlier Detection of a matrix depending on spatio coordinates

Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.

I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)

with rather average level of satisfaction.

Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?

Thanks,

Chris

[Dyuman Joshi]

Have you tried the built-in function rmoutliers?

[CSCh]

Thank you for your quick answer. Yes, the detection method you can choose in rmoutliers is the same like in filloutliers, if I have not missed anything.

[Dyuman Joshi]

Yes, the methods for finding outliers are the same for both.

And none of those worked for you?

Could you attach a sample of your data?

[Constantino Carlos Reyes-Aldasoro]

Have you tried the traditional mean +- 3 * standard deviation criterion to determine what is an outlier?

[CSCh]

Dear Joshi & Constantino,, thank you for your reply.I have tried out all the methods of "rmoutliers". It varies from dataset to dataset, sometimes one works well (eg nearest neighbour) sometimes the other e.g. quantiles). It's difficult to upload all the data sets. Yes, I tried 3Sigma, but it's the same here: for some it works very well, then again it doesn't find all of them. Isn't there something available based on machine learning , I have a lot of historical data available?

Chris

[CSCh]

Maybe the SVM classyfier could work...

[Alan]

Hi CSCh,

I understand that you want to detect outliers within each matrix rather than considering each matrix as datapoints.

In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://in.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2

Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:

x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));

% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");

% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);

% Continue to calculate z-score and exclude outliers

In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:

1. isoutlier() - https://in.mathworks.com/help/matlab/ref/isoutlier.html
2. rmoutliers() - https://in.mathworks.com/help/matlab/ref/rmoutliers.html
3. filoutliers() - https://in.mathworks.com/help/matlab/ref/filloutliers.html

Some parameters to experiment with could be:

1. dim : Dimension along which the outliers should be detected.
2. movemethod : To detect outliers using a moving window.
3. percentiles : To identify the extreme percentile of data.

As for machine learning methods, you could check out this resource: https://in.mathworks.com/help/stats/unsupervised-anomaly-detection.html. It contains various supervised methods for multivariate data.

[CSCh]

Dear Alan, many thanks for your help.

The fit doesn't work in your example:

Error: "Incorrect number or types of inputs or outputs for function 'fit'.

[Alan]

Could you try running the exact code? It is supposed to give the following output:

If you are providing your own input data to the code, ensure that you are maintaining the right format.

[CSCh]

Hi Alan, I just copied you code under the script name "tests".

[Bruno Luong]

fit belongs to curve fitting toolbox do you have access to it?

[CSCh]

No, OK, thats the problem. i'am wondering why I got such an missleading error message in that case. For example is I use:

wgs = sgolayfilt(Z',3,7);

I got a clear error message:

'sgolayfilt' requires Signal Processing Toolbox.

[Bruno Luong]

Because you have another fit somewhere else I guess.

[Alan]

I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.

Answer 1

[Alan]

(This is a repost from the comments section)

If you want to detect outliers within each matrix rather than considering each matrix as datapoints.

In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2

Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:

x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));

% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");

% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);

% Continue to calculate z-score and exclude outliers

In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:

1. isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
2. rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
3. filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html

Some parameters to experiment with could be:

1. dim : Dimension along which the outliers should be detected.
2. movemethod : To detect outliers using a moving window.
3. percentiles : To identify the extreme percentile of data.

As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.