Import multiple .csv files with different of variables and rows

I have a large amount of .csv files that have basically the same format as the one I attached, except the number/type of variables ( and number of data points is different depending on the file. The function I attached is able to read in the current file I have and assign column headers, which Matlab gathered by reading the 3rd line of the .csv when I used the 'Import Data' then 'Generate Function' buttons. My question is how can I make this function general so that it reads the file, then creates a table with the same column headers as the current file. I do not need any of the information in the top line of the .csv's, I only need the 3rd line (variables) and below (data).

There is a space between the 3rd and 5th lines of the .csv's, which I would like not to change manually because I have so many files, which may affect reading variable names using alternative methods.

I tried removing the variable naming from line 95 of the importfile function I created, and then using textscan() to read the 3rd line of the .csv to use as variable names, but for some reason textscan stops after the first column (Date-time); most likely because it is recognizing the 'time' cell as NaN.

I have also tried importdata(), but all it did was import the very first cell of the .csv.

Any help is appreciated, thank you

Answer 1

[Mathieu NOE]

hello

following your first idea, I modified the import function so that it can extract the varaible names from the 3rd line of the header

so far it works , but if you test it with a csv file with another number of columns or differently organized we will have an issue

with formatSpec = '%s%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%C%f%C%f%f%f%f%C%f%C%f%C%f%f%f%f%f%f%f%C%[^\n\r]'; that also need to be addressed in case of change of dimensions of the csv file

maybe in a 2nd attempt ?

all the best

[Benjamin Azrieli]

Thank you for trying - I ended up just cutting out the problem column (first column) in all the csv files and then used readtable(), which worked perfectly. Hoping someone else comes along who knows how to skip columns when reading in csv's.

[Mathieu NOE]

ok

glad you have a solution that works

Answer 2

[Jeremy Hughes]

More recent versions of MATLAB generate code with the Import Tool which used the Import Options and readtable. The code is simpler, and has more configuration options available.

I added a line to sub-select variables to the base generated code.

opts.SelectedVariableNames = [...]

The code should still work in some older releases (those where import options are available).

[Benjamin Azrieli]

>> x = importfile_new(filelist(4,1))
%Text below is red in my command prompt
Operands to the || and && operators must be convertible to logical scalar values.

Error in matlab.io.ImportOptions/setvaropts (line 474)
            elseif isempty(selection) || strcmp(selection,':') % ':' allows pass through from
            indexing.

Error in importfile_new (line 38)
opts = setvaropts(opts, ["LambdaSensorStatus", "CoolantSwitchStateCon",
"RS422RCVMessage01TImeout", "SystemStatus", "RS422IgnitionCheckLastMessage",
"BlockHeaterState"], "EmptyFieldRule", "auto");

This is the error I am getting when using your function on one of my csv's.

[Jeremy Hughes]

That seems like a bug. What release are you using?

[Benjamin Azrieli]

R2018b

[Jeremy Hughes]

I can see it was fixed in R2019a. I'll make a report, and I've made some changes to work around the issue.