Help correcting a messy time series data

Hi,

I have multiple files of "daily" minimum temperature. The time series is non-continuos. Files start and end in different dates, some days in the middle are missing (the rows are missing), and some days have more than one measurement.

This is an example of what I have:

1999 01 01   5.2
1999 01 02   4.3
1999 01 02   5.0
1999 01 02   4.1
1999 01 03   3.8
1999 01 05   3.2
...

So day 02-jan has 3 different meassurements and day 04-jan is missing. Say that I need all the files to begin at 31-dec-1998 and end at 07-jan-1999, my files should end up looking like this:

1999 12 31   6.6
1999 01 01   5.2
1999 01 02   4.1
1999 01 03   3.8
1999 01 04   NaN
1999 01 05   3.2
1999 01 06   NaN
1999 01 07   NaN

So far I managed, using this thread , to fill the missing dates with NaNs.

I still need to:

1) take the minimum value of the days with more than one measurement. I have absolutely no idea how to do this...

2) complete the period with NaN or crop the data between the starting and the ending date that I need. So far I managed with a long and messy script to crop and/or add the missing data at the end. But I couldn't make that work for the beggining, and I'm sure there must be an easier, more effective way of doing so.

I would appreciate any help you can give me!

Answer 1

[Walter Roberson]

YMD_data = YourData(:,1:3);
temperature_data = YourData(:,4);  %must be column vector

first_wanted = datenum('31-dec-1998');
last_wanted = datenum('07-jan-1999');
dayspan = last_wanted - first_wanted + 1;

dnum = datenum(YMD_data);
in_range = first_wanted <= dnum & dnum <= last_wanted;
useful_dnum = dnum(in_range);
useful_temperature = temperature_data(in_range);

relday = useful_dnum - min(first_wanted) + 1;  %so first day is 1, next is 2, etc
mintemp = accumarray(relday, useful_temperature, [dayspan, 1], @min, NaN);

wanted_dates_vec = datevec(first_wanted : last_wanted);
results = wanted_dates_vec(:,1:3, mintemp);

That's it. All of the real work is being done by the accumarray() call, which is going to construct an entry for each consecutive day, and the entry is going to be the min() of all of the data entries with the same date relative date number. The entries for which there is no information will be filled with NaN.

The bit after that constructs the output table. datevec() produces an N x 6 array in which the first three entries are Y M D (then H Min S). Extract those three, paste on the column output from accumarray and the task is done. The entries will appear in sequence, the minimums taken, the missing data NaN filled. No subselection of the output is needed because we selected what we wanted before passing it into accumarray.

[lightworks]

Thank you so much for all the explenation and everything! Sadly it's not working for me. I get the error Subscript indices must either be real positive integers or logicals

It's coming from in_range , which for some reason it's all full of zeros. Not sure if that would cause it, but I have lots of days in the asked range in my data.

Thanks a lot again!

[Thorsten]

Try

whos in_range

Should be of class logical; It has zeros at all positions where dnum is not within the specified range.

And in the next line

 useful_dnum = dnum(in_range);

logical indexing is used, i.e., all values from dnum are picked for which in_range is 1.

[lightworks]

It works perfectly!!

:D

Thank you so much!

Answer 2

[Andrei Bobrov]

d = [
1998 12 14   6
1999 01 01   5.2
1999 01 02   4.3
1999 01 02   5.0
1999 01 02   4.1
1999 01 03   3.8
1999 01 05   3.2
1999 02 12   8]

[y,m,dy] = datevec((datenum([1998 12 31]):datenum([1999 1 7]))');
daout = [y,m,dy];
[a,~,c] = unique(d(:,1:3),'rows');
d2 = accumarray(c,d(:,end),[],@min);
[lo,ii] = ismember(a,daout,'rows')
daout(:,end+1) = nan;
daout(ii(lo),end) = d2(lo);

[lightworks]

Thanks a lot! Although the example is working just fine, when I apply it to my data I get a Subscripted assignment dimension mismatch.

It comes from the last line. The problem is daout has as many rows as needed and d2 has not the same amout of rows. When data begin before the first day asked (31-dec-1998) or ends after, d2 ends up having more rows than daout.

I've been trying to make it work but so far I couldn't. I'll keep trying though.

[Andrei Bobrov]

corrected