Textscan and csv fitness data problem

Challenge: Convert attached csv-file into vectors for timestamps, power, and heart_rate data. The file originates from a FIT-file (from a fitness device) converted via FitSDKRelease into a CSV-file.

Problem: I am not able to generate an output via textscan that corresponds to the number of lines (rows) in the csv-file. E.g. the variable DataMat generated via examples below holds values from various rows in the first line (DataMat(1,:)). It is like some lines wrap around.

Example code

    fileID = fopen(PathTargetFile,'rt');
    Header = fgets(fileID); %line by line read
    %....
    LengthHeadder = 126;
    StrReadFormat = repmat('%s',1,LengthHeadder);
    Data = textscan(fileID, StrReadFormat, 'Delimiter', ',',  'ReturnOnError' , false);
    DataMat = cell2mat(Data);
    DataMat(1,:)

Another iteration:

    fileID = fopen(PathTargetFile,'rt');
    Header = fgets(fileID); %line by line read
    %....
    LengthHeadder = 126;
    StrReadFormat = [repmat('%s',1,min(LengthHeadder,40)) '%*[^\n]'];
    Data = textscan(fileID, StrReadFormat, 'Delimiter', ',',  'ReturnOnError' , false);
    DataMat = cell2mat(Data);
    DataMat(1,:)

Answer 1

[Walter Roberson]

Your input has 127 columns, not 126. Column 127 is empty where it exists at all -- but the fact that it exists is throwing off textscan, because textscan always picks up "mid-line" if the format did not use up the entire line.

[Thomas Esbensen]

Even if I extract only 40 columns via textscan, and skip until end og line, the output is incorrect and somehow wraps around. See my second example

Example:

   StrReadFormat = [repmat('%s',1,min(LengthHeadder,40)) '%*[^\n]'];

How shall I call textscan, or other read function?

[Walter Roberson]

I used

fid = fopen('2017-08-11-143641-ELEMNT BOLT 2779-2-0.csv','rt','n','UTF-8')
nc = 127
fmt = repmat('%s',1,nc)
data = textscan(fid,fmt,'delimiter',',');
fclose(fid)

and it looks fine when I compare it to the file contents.

[Thomas Esbensen]

Hmm. I am jumping a bit between MATLAB and Octave. I wasn't aware of this part being very different.

In Octave

    fileID = fopen(PathTargetFile,'rt');
    LengthHeader = 15;
    StrReadFormat = [repmat('%s',1,LengthHeader) '%*[^\n]'];
    Data = textscan(fileID, StrReadFormat, 'Delimiter', ',',  'ReturnOnError' , false, "TreatAsEmpty", false);
    fclose(fileID);
    DataMat = cell2mat(Data);  % you can't do this in MATLAB 
    size(DataMat) 
    %OUTPUTS      2784     15

    fileID = fopen(PathTargetFile,'rt');
    LengthHeader = 40;
    StrReadFormat = [repmat('%s',1,LengthHeader) '%*[^\n]'];
    Data = textscan(fileID, StrReadFormat, 'Delimiter', ',',  'ReturnOnError' , false, "TreatAsEmpty", false);
    fclose(fileID);
    DataMat = cell2mat(Data);  % you can't do this in MATLAB
    size(DataMat) 
    %OUTPUTS      2762     40

    fileID = fopen(PathTargetFile,'rt');
    LengthHeader = 107;
    StrReadFormat = [repmat('%s',1,LengthHeader) '%*[^\n]'];
    Data = textscan(fileID, StrReadFormat, 'Delimiter', ',',  'ReturnOnError' , false, "TreatAsEmpty", false);
    fclose(fileID);
    DataMat = cell2mat(Data); % you can't do this in MATLAB
    size(DataMat)  
    %OUTPUTS       2182    107

In MATLAB, it is consistent to the file length of 2784 rows.

[Walter Roberson]

I have lost track of what the question is? The code I posted works on MATLAB; the behavior of Octave is mostly off-topic for this forum.

Answer 2

[Jeremy Hughes]

Hi Using textscan is great if you have files with varying formats or if you need certain special behaviors.

I found this one pretty straight forward with readtable and import options. Although it picks '/' for the delimiter.

opts = detectImportOptions(filename,'Delimiter',',','TextType','string');
opts.ExtraColumnsRule = 'ignore';
T = readtable(filename,opts);
head(T)