textscan: skipping rows with missing data

I'm using textscan() to open large txt files. Every so often I get a line where the value in the third column is missing, so everything gets shifted and text scan stops reading the data and just returns everything up until this point. The data kind of looks like this, although really it's 5 string columns followed by 30 double columns. It doesn't seem to want to preserve my formatting which would make it easier to see but I think this is still helpful

    string   string   double   double   double
    string            double   double   double 
    string   string   double   double   double 
    string   string   double   double   double 
    string   string   double   double   double

Does anyone know a way to deal with that other than going in and manually deleting those incomplete lines from the individual text files?

[per isakson]

The columns are they separated by tab or space?

[Kendra Wright]

columns are separated by tabs

[Kendra Wright]

Then when there is the bad line it has tab+ a few spaces in place of the missing strings

[Thorsten]

It would be helpful if you post your file, or at least some of the lines with the different number of columns.

Answer 1

[per isakson]

"columns are separated by tabs"

textscan should take care of missing data automagically! &nbspReplace 'Delimiter', ',' by 'Delimiter', ','\t' in these examples

    >> cac = textscan( 'abc,,1,2,3', '%s%s%f%f%f', 'Delimiter', ',' )
    cac = 
        {1x1 cell}    {1x1 cell}    [1]    [2]    [3]
    >> cac{2}
    ans = 
        {''}
    >>

and

    >> cac = textscan( 'abc,def,1,,3', '%s%s%f%f%f', 'Delimiter', ',' )
    cac = 
        {1x1 cell}    {1x1 cell}    [1]    [NaN]    [3]

&nbsp

"Then when there is the bad line it has tab+ a few spaces in place of the missing strings"

    >> cac = textscan( 'abc,    ,1,2,3', '%s%s%f%f%f', 'Delimiter', ',' )
    cac = 
        {1x1 cell}    {1x1 cell}    [1]    [2]    [3]
    >> cac{2}
    ans = 
        {''}

and

    >> cac = textscan( 'abc,def,1,    ,3', '%s%s%f%f%f', 'Delimiter', ',' )
    cac = 
        {1x1 cell}    {1x1 cell}    [1]    [NaN]    [3]

&nbsp

@Star's proposal: MultipleDelimsAsOne

    >> textscan( 'a b c,d e f,1,2,3', '%s%s%s%s%s%s%f%f%f' ...
               , 'Delimiter', ', ', 'MultipleDelimsAsOne', true )
    ans = 
      Columns 1 through 6
        {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}
      Columns 7 through 9
        [1]    [2]    [3]

So far so good. However, with "b" missing

    textscan( 'a    c,d e f,1,2,3', '%s%s%s%s%s%s%f%f%f' ...
            , 'Delimiter', ', ', 'MultipleDelimsAsOne', true )

    ans = 
      Columns 1 through 6
        {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}
      Columns 7 through 9
        [2]    [3]    [0x1 double]

it returns an erroneous result

[Kendra Wright]

Ok I'm getting close now. But I just realized it's actually delimited like this space space tab space space tab tab tab tab tab (30 times)

[Kendra Wright]

Can textscan do that?

[per isakson]

"space space tab space space tab tab tab tab tab (30 times)" &nbsp That's weird! Is there a tab in front of the first numeric?

"Can textscan do that?"

I'm not sure. Some spaces are delimiters and other spaces happens to be there because of missing data. The latter should be ignored. It's worth trying to read the file with tab as delimiter and in a second step parse the string columns.

[Star Strider]

In the online documentation for textscan there’s an option (under ‘Examples’ > ‘Treat Repeated Delimiters as One’) to use the 'MultipleDelimsAsOne',1 name-value pair.

Seems worth a try, anyway.

[Kendra Wright]

I have a friend helping me out who is much more savvy with matlab than I, and it turns out this is much more complicated that I thought...

[per isakson]

Consider the strings containing space separated letters

    'a b c'
    '  b c'
    'a   c'

one need additional information to be able to decide whether 'a' or 'b' is missing. That has little with limitations of Matlab to do.

Why don't you attach a small sample file.

Answer 2

[Thorsten]

I think that you have to read individual lines with fgets and then deal with the cases of missing values.

[Kendra Wright]

So use fgets instead of textscan?

[Thorsten]

Yes.