Textscan with blocks of data with headers to individual block

Hi All,

I am trying to extract to following data with textscan. But I am not able to implement it correctly. Each block consists of 11 data lines and 8 unnecessary lines to exclude.

Data file (testdata.dat) is :

Title

Data Header:                                                Time:     100.16064 Sec       10/8/2012 3:32:17 PM
Data Acquisition: T
Station Name: default
Test File Name: 
Axial  Axial  Axial 
segments  mm        kN
0         -0.19470811 -0.0067140702
0         -0.19806515 -0.0033570339
0         -0.19806515 -0.0033570339
0         -0.19470811 -0.0067140702
0         -0.16785182 -0.0033570339
0         -0.16785182 -0.0067140702
0         -0.13763849 -0.0067140702
0         -0.12421035 -0.0067140702
0         -0.10742517 -0.0033570339
0         -0.11413924 -0.0033570339
0         -0.14099553 -0.0033570339

Data Header:                                                Time:     100.16064 Sec       10/8/2012 3:32:17 PM
Data Acquisition: T
Station Name: default
Test File Name: 
Axial  Axial  Axial 
segments  mm        kN
0         -0.16785182 -0.0033570339
0         -0.20813625 -0.0067140702
0         -0.2316355 -0.0067140702
0         -0.25513476 -0.0067140702
0         -0.29206216 -0.0033570339
0         -0.35584584 -0.0067140702
0         -0.40620136 -0.0067140702
0         -0.47334209 -0.0067140702
0         -0.55055392 -0.0067140702
0         -0.61098057 -0.0067140702
0         -0.70162052 -0.010071106

My implementation is

fileID = fopen('testdata.dat');
formatSpec = '%d %f %f';
clear C
k = 0;
    N = 11;%Block size
while ~feof(fileID)
  k = k+1;
  C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 8);%'CommentStyle','##',
end
fclose(fileID)

Please correct my code. Thanks in advance.

[Raj Raj]

p.s.

I tried the code given in Here However, I am getting error with my data file as

Index exceeds matrix dimensions.

at

ncol = numel( regexp( buf{1}, ' +', 'split' ) );

Appreciate your comments.

[per isakson]

That function read one trailing block of data after a file header, the last word of which was "END".

Answer 1

[per isakson]

Modified version of your code

    fid = fopen('RajRaj.txt');
    formatSpec = '%d%f%f';
    clear C
    k = 0;
    N = 11; %Block size
    while ~feof( fid )
      k = k+1;
      C( k, 1:3 ) = textscan( fid, formatSpec, N, 'HeaderLines', 8 ); 
    end
    fclose( fid );

check the output

    >> whos C
      Name      Size            Bytes  Class    Attributes
      C         2x3              1112  cell

"Please correct my code" &nbsp I didn't try your code and isn't sure why it doesn't work. It's tricky to count header-lines. An ending new-line matters.

&nbsp

Here is another function that uses &nbsp fileread &nbsp and &nbsp regexp

    >> g = read_blocks_of_numerical_data( 'RajRaj.txt', 150 );
    >> g{:}
    ans =
             0   -0.1947   -0.0067
             0   -0.1981   -0.0034
    ...
    ans =
             0   -0.1679   -0.0034
             0   -0.2081   -0.0067
    ...

Where both the function and the sample text file are attached. The magical number &nbsp 150 &nbsp is the minimum number of bytes in a block of numerical data. The entire file must fit in memory together with the result.

&nbsp

And one more function

    >> M = RajRaj()
    M(:,:,1) =
             0   -0.1947   -0.0067
             0   -0.1981   -0.0034
             ...
    M(:,:,2) =
             0   -0.1679   -0.0034
             0   -0.2081   -0.0067
             ...

where

    function  M = RajRaj()
        xpr = '(?<=kN)[ 0-9\.\+\-e\r\n]++(?=(Data Header:)|(\s*$))';
        str = fileread( 'RajRaj.txt' );
        cac = regexp( str, xpr, 'match' );
        M = nan( 11, 3, length( cac ) ); 
        for jj = 1 : length( cac )
           M( :, :, jj ) = str2num( cac{jj} ); 
        end
    end

&nbsp

I believe there is a reason why Matlab doesn't include a good functions to read a file with many blocks of numerical data.

[Raj Raj]

Hi Thanks for attention and time to respond. Here comes an issue after dealing with large data file.

My data is in chunks of 11 lines. However, by some reason machine wrote down a data block whose size is 8. However, the "textscan" with chunks data option also tries to match format of data. Hence, I am saved and read the second chunk of data whose size is 8. Here comes the trouble in reading the rest of the data.

The problem is in dealing with the 'Headerlines' option in textscan. Is there any way to tell textscan to go one line backwards ?

Appreciate your inputs. Thanks in advance.

Answer 2

[Geoff Hayes]

Raj - presumably, each block of 8 lines that you wish to ignore is of the format

 <empty line>
 Data Header:                                   
 Time:     100.16064 Sec       10/8/2012 3:32:17 PM
 Data Acquisition: T
 Station Name: default
 Test File Name: 
 Axial  Axial  Axial 
 segments  mm        kN

So there is a blank/empty line followed by seven lines with text. Each of these blocks is then followed by 11 lines that you wish to keep.

I stepped through code (using the debugger) and noticed that the first iteration of your loop worked fine - three columns of 11x1 was read successfully. However, subsequent reads failed with either nothing read, or only 4x1 columns, etc. I repeated the above and once the first block was successfully read in, I typed the following in the Command Window to see the contents of the current line

 K>> fgetl(fileID)

 ans =
     ''

which makes sense if the next line is blank. I then repeated it twice more

 K>> fgetl(fileID)

 ans =
     ''

 K>> fgetl(fileID)

 ans =
     Data Header:

So there seemed (for some reason) to be two blank lines between each block, rather than the expected one. That suggests a problem with the format specification string which you have initialized to

 formatSpec = '%d %f %f';

If I were doing this, reading in data line by line, I would add the newline "character" to the end of the format as

 formatSpec = '%d %f %f\n';

And that seems to be what was missing. Using your above data as an example, both blocks were read in successfully.

Just change your formatSpec to the above, and try again.

[Raj Raj]

Hi Thanks alot for your time. I tried but not able to reproduce the result as your mentioned. Instead I used following code to get the result:

fileID = fopen('testdata.dat');
formatSpec = '%f %f %f';
clear C

k = 0;N = 11;%Block size
while ~feof(fileID)
  k = k+1;
    if k==1,   
  C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 8);%'CommentStyle','##',
    else, 
    C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 9);%'CommentStyle','##',
    end
end
fclose(fileID)

Answer 3

[Gabriel Felix]

I had to use \n at the end of each line. Without it I couldn't make textscan() work properly, even thoug the "HeaderLines" was configured according to the text file lines. This was the only solution I found after struggling with the code for an intire day.

This was the text:

!
!
! alfa (graus) =  5.0
!
! Id.      x/s       z/s     alfai       cl     c*cl/cmed    cdi      cmc/4
!                           (graus)
   1      .246      .050    -1.209      .255      .332     .00538     .0170
   2      .292      .150    -1.098      .259      .319     .00496     .0545
   3      .339      .250     -.925      .254      .297     .00410     .0944
   4      .385      .350     -.741      .243      .268     .00315     .1341
   5      .432      .450     -.561      .227      .235     .00223     .1714
   6      .479      .550     -.393      .206      .199     .00141     .2034
   7      .525      .650     -.238      .181      .163     .00075     .2266
   8      .572      .750     -.101      .152      .126     .00027     .2362
   9      .619      .850      .014      .116      .089    -.00003     .2236
  10      .659      .938      .103      .074      .052    -.00013     .1693
!
! CL asa    =  .208
! CDi asa   =  .00258
! e (%)     =  88.9
! CMc/4 asa =  .1339

My code:

%! alfa (graus) =  5.0
P = textscan(fid,'! alfa (graus) = %f','Delimiter',' ','MultipleDelimsAsOne',true,'headerLines',2,'CollectOutput',1);
alpha(1) = P{1};
%! CL asa    =  .208
P = textscan(fid,'! CL asa = %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerLines',4+n);
CL(1) = P{1};
%! CDi asa   =  .00258
P = textscan(fid,'! CDi asa   =  %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerlines',0);
CDi(1) = P{1};
%! CMc/4 asa =  .1339
P = textscan(fid,'! CMc/4 asa = %f','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'HeaderLines',2);
Cmc4(1) = P{1};