How do I read data after a specific string?

I have three sets of data: raw, average, and median. They are all in different .txt files and have 1024 data points with its x, y, z, etc.

Now I'm trying to get the data into the workspace so I can run some analysis on it but I don't know how to make a code work for the three different sets. They all start with machine and measurement specifications like date, voltage, scaling, etc. for 30-60 lines and then the data starts after three *.

Example median data_date_time
Comments and machine specifications:
... (for 30-60 lines)
Data: x, y, z, etc.
***
-8.710498,-0.039902,0.021362,-0.028009,-0.006365
-8.689081,-0.022736,0.006943,-0.027939,-0.006338
-8.667665,-0.014191,0.031357,-0.027870,-0.006311
-8.646248,-0.018768,0.033569,-0.027800,-0.006283
-8.624832,-0.029907,0.025787,-0.027731,-0.006256
-8.603415,-0.037689,0.003281,-0.027661,-0.006228
-8.581999,-0.048065,-0.009613,-0.027591,-0.006200
... (for 1024 points).

I've read some comparable questions in the forums and tried to format the answers to my problem but I've had no luck. What I do understand is that I have to read the file from the beginning, skip all 30-60 lines before *s and read the data after but I don't know how.

Answer 1

[Stephen23]

opt = {'Delimiter',','};
fmt = repmat('%f',1,5);
C = {};
fid = fopen('test.txt','rt');
while ~feof(fid)
    str = fgetl(fid);
    if strncmp('***',str,3)
        C{end+1} = textscan(fid,fmt,opt{:});
    end
end
fclose(fid);

Your data is in C.

[Jason Jeong]

This works but it's giving me the data as a 1024x1, in a 1x5 cell, in a 1x1 cell (C). How would I use specific columns? Just reference them as C{1,1}{1,X}? (X being the column I want?)

*I can just use C{1,1}{1,2} to get column 2. Thanks so much!

[Jason Jeong]

The code is shaping up nicely but I found out that for the raw signal data, the *s isn't in a line by itself and the code breaks down.

For the raw data, it looks like this:

Example median data_date_time
Comments and machine specifications:
... (for 30-60 lines)
Data: x, y, z, etc***
-8.710498,-0.039902,0.021362,-0.028009,-0.006365
-8.689081,-0.022736,0.006943,-0.027939,-0.006338
-8.667665,-0.014191,0.031357,-0.027870,-0.006311
-8.646248,-0.018768,0.033569,-0.027800,-0.006283
-8.624832,-0.029907,0.025787,-0.027731,-0.006256
-8.603415,-0.037689,0.003281,-0.027661,-0.006228
-8.581999,-0.048065,-0.009613,-0.027591,-0.006200
... (for 1024 points).

Is there an equivalent for strncmp that looks for the last n characters that I can use in an else statement? Something like:

while ~feof(fid)
    str = fgetl(fid);
    if strncmp('***',str,3)
        C{end+1} = textscan(fid,fmt,opt{:});
    else Opposite_of_strncmp('***',str,3)
        C{end+1} = textscan(fid,fmt,opt{:});
    end
end

[dpb]

l=fgetl(fid);
if strfind(l,'Data:')==1
  % read data here

From your initial description it seemed there was a dataset per file; are there more than one in a single file in actuality? How the loop is structured efficiently depends on the file structure.

Are cell arrays really worth the hassle later? What are you doing with each set of data?

[Jason Jeong]

There are three separate files that the machine spits out. Raw data, Median data (median of 12 raw data per set), and average data (average of 12). I originally thought that they were all formatted the same way (the one I originally posted). But I see now that the raw data is formatted slightly differently (the one I just put up).

I'm pretty sure I need the cell arrays (I need the first two columns to plot the data).

[dpb]

"I'm pretty sure I need the cell arrays (I need the first two columns to plot the data)."

Well, that's not a reason on its own for needing a cell array. The reason you might use a cell array is if there were multiple sets in a single file and you need to do something with more than one of these at a time but keeping them separate.

If they are in different files, there's really no reason to encapsulate the array into a cell other than that's what textscan does automgically; as I show, you may as well just cast to a simpler 2D array then and save the grief of the additional dereferencing later.

Answer 2

[dpb]

If you don't know the number of header lines or they're variable as I gather they are, two choices:

1. The easy route...see if importdata will "just work" and find the data you're looking for automagically doing the job for you for free, or
2. Scan the file line-by-line until you find the magic string, then read the rest. This isn't as hard as it sounds...

fid=fopen('yourfile','r');   % open the file; be sure to add the error-checking code
l=fgetl(fid);                % read first record
while ~strcmp(l,'***')       % loop until find the magic record
  l=fgetl(fid);
end
data=cell2mat(textscan(fid,'','collectoutput',1));  % read, convert cell to double array
fid=fclose(fid);             % done with file, go on...

Now do whatever you need with the array.

[Jason Jeong]

I tried it and it's telling me that there's too many input arguments for cell2mat

[dpb]

Ooops...left out the textscan root call...