Asked by Mohamed Ibrahim
on 11 Sep 2019

Hi All,

I would like to fit the data shown in the attached photo. The two plots are from two temperature sensors (resistance sensor) against time. From the figure, at one point, you can see the lines start to broaden. This happens due to high variation of the temperature (resistance of the sensor) at low temperatures. This broadening is actually a sin function variation with very small frequency 1.15Hz (0.0145 min). I would like to find the best fit or someway to find the average of these points to elimenate the broadening.

Thanks

Answer by John D'Errico
on 11 Sep 2019

Edited by John D'Errico
on 11 Sep 2019

Accepted Answer

You don't have any simple model that you can use, at least, you have not suggested any. And, even if somebody suggests using polyfit, DON'T BELIEVE THEM! A polynomial model will be poor here.

Therefore, you need to ue a spline model. A problem is the heteroscedasticity. (]Yes, it hurts even to try to type that word.) But I would suggest a simple set of weights that decrease linearly with time. So small weight means that you don't trust that point much. High weight means you think the point is likely to be accurate.

You have not included your actual data, only a picture. Else, I would show how to fit it using my SLM toolbox. But a wild guess as to the call would be...

W = 1.25*max(Time) - Time;

slm = slmengine(Time,SR,'knots',12,'increasing','on','weights',W,'plot','on');

If you don't want to see the curve fit plot, then don't turn it on. Note that you should ALWAYS plot your data, and the curve fit through it. Decide for yourself if the fit is adequate.

More knots will give you a better fit, but less smooth, chasing the bumps in the curve.

If you want it to follow those bumps in the top end, where the curve seems to jump down a little, then you can leave increasing to be 'off' (the default), or you can set a specific interval where it should only be on.

Find SLM for download (free of course) on the file exchange here:

Note that SLM does use the optimization toolbox.

A simpler alternative to SLM would be to use a moving window average or moving median filter. Without the data, it is difficult to know how well it would work. And of course, you would need to choose the window size. A Savitsky-Golay filter might also be reasonable. The problem with any of the latter tools is the window length might need to vary with Time, so where it is more noisy, you need a wider window to smooth things out.

Mohamed Ibrahim
on 1 Oct 2019 at 10:55

John D'Errico
on 2 Oct 2019 at 13:24

I TOLD you that polyfit won't do anything of value. Sometimes people actually believe me.

If you usually get a poor fit using SLM, then you did not use it right.

As well, I would claim that your data has problems in it, that need to be corrected before you use ANY fitting tool.

For example, I ran your script, then I plotted the data.

- ALWAYS PLOT THE DATA.
- THINK ABOUT WHAT YOU SEE.
- Only then should you bother to use any fitting tool.

So here is your data.

whos

Name Size Bytes Class Attributes

SROhm1BCal 178898x1 1431184 double

SROhm2AAfter 178898x1 1431184 double

SROhm2ABefore 178898x1 1431184 double

SROhm2AMean 178898x1 1431184 double

Timems 178898x1 1431184 double

assigned 178898x3 4293552 double

Now plot it. I've just picked one curve.

plot(Timems,SROhm2AMean,'.')

What I see is a moderately nice curve, but one that then drops off due to somethign at the end. That drop-off will NOT be meaningfully fit. So if you do try to fit that with ANY tool, expect COMPLETE CRAPOLA. That is stated without any need to try to fit it.

Lets zoom in on the plot.

axis([156 158 800 1100])

grid on

Ok, so something completely different is happening here. If you hope to fit that fall-off, then think again. As well, if your hope is that you will be able to fit the bumps and wiggles in that curve, then you will neem to hope again. For example, if we look at this interior section:

Then are you looking to git the extremelely rapidly varying stuff there? Do you want to fit the jogs in the burve? OR are you looking to fit the overall shape?

Only you know what you meant by the statement that somethign did not fit well.

So first, get rid of the crap.

k = Timems <= 157.1;

plot(Timems(k),SROhm2AMean(k),'.')

SLM = slmengine(Timems(k),SROhm2AMean(k),'increasing','on','plot','on','knots',[0:10:50, 54:2:90, 95, 100:10:160]);

That seeems to be an eminently reasonable fit. If you want to get the same fit, with slightly more CPU time (only a fraction of a second) needed, then this would have worked as well, with no real thought necessary:

SLM = slmengine(Timems(k),SROhm2AMean(k),'increasing','on','plot','on','knots',75);

So I fail to see that you would have had a problem, IF you used the tool correctly, and thought about your data.

Sign in to comment.

Answer by Mohamed Ibrahim
on 2 Oct 2019 at 16:23

Edited by Mohamed Ibrahim
on 2 Oct 2019 at 16:23

Thanks for your help. I really appreciate the time and effort you put into your answer but still there are some problems.

For this zoom in figure:

I actually don't need to fit this drop-off.

For this interior section:

I am actually interested in following the bumps in this curve (i.e. I would like the fitted line to have the same trend as this broadened curve). I already used your settings but I guess the number of knots was little to really follow the kinks in the curve. I increased it to 300 and it looks nicer now but still some sharp bumps not followed. I am afraid to increase it more and get it overfitted somehow. Another naive question, how do you extract the fitted line vectors? I saw variables x, y in the SLM struct but, when I plot them, I got the row data plotted again with the fitted line (in other words, how I show the fitted line alone and extract its data to use).

John D'Errico
on 2 Oct 2019 at 22:47

Please, learn to use COMMENTS! This is not an answer, just a response to my answer.

There is a tool in the toolbox to evaluate a spline. Use slmeval. If you had read the copius help and examples, you would have seen that.

As far as fitting the bumps there, you just need to use more knots. Or, you need to put sufficient knots in places where it is important.

Note that if you have 'increasing' set to be 'on', as I did in my example, then it cannot follow a decrease as you see there.

Sign in to comment.

Opportunities for recent engineering grads.

Apply Today
## 2 Comments

## darova (view profile)

## Direct link to this comment

https://fr.mathworks.com/matlabcentral/answers/479911-fitting-of-raw-data#comment_744923

## Mohamed Ibrahim (view profile)

## Direct link to this comment

https://fr.mathworks.com/matlabcentral/answers/479911-fitting-of-raw-data#comment_751965

Sign in to comment.