Fitting of raw data

Hi All,

I would like to fit the data shown in the attached photo. The two plots are from two temperature sensors (resistance sensor) against time. From the figure, at one point, you can see the lines start to broaden. This happens due to high variation of the temperature (resistance of the sensor) at low temperatures. This broadening is actually a sin function variation with very small frequency 1.15Hz (0.0145 min). I would like to find the best fit or someway to find the average of these points to elimenate the broadening.

Thanks

[darova]

Did you fit() function?

[Mohamed Ibrahim]

yes but it does not work nicely.

Answer 1

[John D'Errico]

You don't have any simple model that you can use, at least, you have not suggested any. And, even if somebody suggests using polyfit, DON'T BELIEVE THEM! A polynomial model will be poor here.

Therefore, you need to ue a spline model. A problem is the heteroscedasticity. (]Yes, it hurts even to try to type that word.) But I would suggest a simple set of weights that decrease linearly with time. So small weight means that you don't trust that point much. High weight means you think the point is likely to be accurate.

You have not included your actual data, only a picture. Else, I would show how to fit it using my SLM toolbox. But a wild guess as to the call would be...

W = 1.25*max(Time) - Time;
slm = slmengine(Time,SR,'knots',12,'increasing','on','weights',W,'plot','on');

If you don't want to see the curve fit plot, then don't turn it on. Note that you should ALWAYS plot your data, and the curve fit through it. Decide for yourself if the fit is adequate.

More knots will give you a better fit, but less smooth, chasing the bumps in the curve.

If you want it to follow those bumps in the top end, where the curve seems to jump down a little, then you can leave increasing to be 'off' (the default), or you can set a specific interval where it should only be on.

Find SLM for download (free of course) on the file exchange here:

https://www.mathworks.com/matlabcentral/fileexchange/24443-slm-shape-language-modeling

Note that SLM does use the optimization toolbox.

A simpler alternative to SLM would be to use a moving window average or moving median filter. Without the data, it is difficult to know how well it would work. And of course, you would need to choose the window size. A Savitsky-Golay filter might also be reasonable. The problem with any of the latter tools is the window length might need to vary with Time, so where it is more noisy, you need a wider window to smooth things out.

[Mohamed Ibrahim]

I actually tried sigmoid, polyfit and a smooth bent step functions but nothing works. I tried also the SLM toolbox suggested by you but I usually get a poor fitting. I have attached the import data code and the data file. In the code, you need to change the directory to the data file directory. From the data file, you will find 5 columns. I am interested in the data of time (ms)(the unit of time is written wrongly in ms but it is min) and the last two columns SR(ohm)2AMean and SR(ohm)1BCal.

[John D'Errico]

I TOLD you that polyfit won't do anything of value. Sometimes people actually believe me.

If you usually get a poor fit using SLM, then you did not use it right.

As well, I would claim that your data has problems in it, that need to be corrected before you use ANY fitting tool.

For example, I ran your script, then I plotted the data.

1. ALWAYS PLOT THE DATA.
2. THINK ABOUT WHAT YOU SEE.
3. Only then should you bother to use any fitting tool.

So here is your data.

whos
  Name                    Size              Bytes  Class     Attributes

  SROhm1BCal         178898x1             1431184  double              
  SROhm2AAfter       178898x1             1431184  double              
  SROhm2ABefore      178898x1             1431184  double              
  SROhm2AMean        178898x1             1431184  double              
  Timems             178898x1             1431184  double              
  assigned           178898x3             4293552  double          
  

Now plot it. I've just picked one curve.

plot(Timems,SROhm2AMean,'.')

What I see is a moderately nice curve, but one that then drops off due to somethign at the end. That drop-off will NOT be meaningfully fit. So if you do try to fit that with ANY tool, expect COMPLETE CRAPOLA. That is stated without any need to try to fit it.

Lets zoom in on the plot.

axis([156 158 800 1100])
grid on

Ok, so something completely different is happening here. If you hope to fit that fall-off, then think again. As well, if your hope is that you will be able to fit the bumps and wiggles in that curve, then you will neem to hope again. For example, if we look at this interior section:

Then are you looking to git the extremelely rapidly varying stuff there? Do you want to fit the jogs in the burve? OR are you looking to fit the overall shape?

Only you know what you meant by the statement that somethign did not fit well.

So first, get rid of the crap.

k = Timems <= 157.1;
plot(Timems(k),SROhm2AMean(k),'.')

SLM = slmengine(Timems(k),SROhm2AMean(k),'increasing','on','plot','on','knots',[0:10:50, 54:2:90, 95, 100:10:160]);

That seeems to be an eminently reasonable fit. If you want to get the same fit, with slightly more CPU time (only a fraction of a second) needed, then this would have worked as well, with no real thought necessary:

SLM = slmengine(Timems(k),SROhm2AMean(k),'increasing','on','plot','on','knots',75);

So I fail to see that you would have had a problem, IF you used the tool correctly, and thought about your data.

Answer 2

[Mohamed Ibrahim]

Thanks for your help. I really appreciate the time and effort you put into your answer but still there are some problems.

For this zoom in figure:

I actually don't need to fit this drop-off.

For this interior section:

I am actually interested in following the bumps in this curve (i.e. I would like the fitted line to have the same trend as this broadened curve). I already used your settings but I guess the number of knots was little to really follow the kinks in the curve. I increased it to 300 and it looks nicer now but still some sharp bumps not followed. I am afraid to increase it more and get it overfitted somehow. Another naive question, how do you extract the fitted line vectors? I saw variables x, y in the SLM struct but, when I plot them, I got the row data plotted again with the fitted line (in other words, how I show the fitted line alone and extract its data to use).

[John D'Errico]

Please, learn to use COMMENTS! This is not an answer, just a response to my answer.

There is a tool in the toolbox to evaluate a spline. Use slmeval. If you had read the copius help and examples, you would have seen that.

As far as fitting the bumps there, you just need to use more knots. Or, you need to put sufficient knots in places where it is important.

Note that if you have 'increasing' set to be 'on', as I did in my example, then it cannot follow a decrease as you see there.