cannot concatenate variable in a table

My code is attached. There is an issue concatenating the two tables in the first loop. I think the problem is because in some of the data files that are being read, some variables appear as string NaNs and then as numbers in other files. How would I make it so that any NaNs are numerical instead of strings before combing the tables in line 20? As you can see, I change the NaNs to numerical types later in the code but had an issue when I tried doing it in the same way in the loop. I think the indexing was really confusing me.

[dpb]

Would be much simpler to see the data files that cause the problem than try to divine a coding error in so much code... :)

You should be able to import/read the data such that NaN inputs are automagically convered when read rather than having to go back and fixup an import error.

Why that isn't happening is buried in how the data are encoded--maybe they aren't actually NaN but some other ill-formed text that fails? but we can't tell that from here.

[Susan Santiago]

This .zip contains the files that are giving me error. Thanks for the help

Answer 1

[dpb]

Try the following:

VarNames = {'TStart','TEnd','CO2','CO2_SIGMA','H2O',...
    'H2O_SIGMA','FC','FC_SSITC_TEST','LE','LE_SSITC_TEST','ET','ET_SSITC_TEST',...
    'H','H_SSITC_TEST','G','SG','FETCH_MAX','FETCH_90','FETCH_55','FETCH_40',...
    'WD','WS','WS_MAX','USTAR','ZL','TAU','TAU_SSITC_TEST','MO_LENGTH','U',...
    'U_SIGMA','V','V_SIGMA','W','W_SIGMA','PA','TA_1_1_1','RH_1_1_1',...
    'T_DP_1_1_1','TA_2_1_1','RH_2_1_1','T_DP_2_1_1','VPD','T_SONIC','T_SONIC_SIGMA',...
    'PBLH','SWC_1_1_1','SWC_2_1_1','TS_1_1_1','TS_2_1_1','ALB','NETRAD','SW_IN',...
    'SW_OUT','LW_IN','LW_OUT'};

d=dir('*Flux*.dat');
opts=detectImportOptions(d(1).name);                     % base best guess import options
opts.VariableNames=VarNames;                             % set names
opts=setvartype(opts,1:2,'datetime');                    % first two colums are datetime
opts=setvaropts(opts,1:2,'InputFormat','yyyyMMddHHmm');  % input format for 'em
opts=setvaropts(opts,1:2,'DatetimeFormat','dd-MMM-uuuu HH:mm'); % and output format
opts=setvartype(opts,3:numel(opts.VariableTypes),'double');     % all the rest are double

T=table;
for j=1:length(d)
  T = [T;readtable(d(j).name,opts);
end

The key is to create the import options stucture for the files first, then apply it to every file of that type.

The detectImportOptions function is pretty good, but here it needs some help because there's a double variable with the string "NAN" in the first row for some columns so it tries to import them as character. That's ok for the first record of the first file, but there's nothing that says those same columns will be the only ones for other files or even elsewhere in the same file. So, if everything except the first two datetime columns is numeric, say so...

Then, all that's left after that is to parse the datetime field as it's given on input and choose a useful output format. I think it's a bug that the use of 'Default' in the options object doesn't apply system default but I guess that may be consistent with the other behavior that it echoes the input unless told different.

NB: I did change the time var names while testing to something shorter..