extract useful info from text file filled with irrelevant info

The text files that I have are like this: useless info... * useful date (within useless info) useful info *useless info and like that until the end - if there a way of extracting the useful stuff rather than copy and pasting each individual one (as each file has like 200 things to copy into excel) ? Thanks

[Guillaume]

There is possibly a way using regexp but you haven't given us enough info.

How do you identify the useful date or the useful info? Unfortunately computers can't yet read our mind so you need to give them some criteria.

[Jamie Shelley]

I tried with regexp but I don't understand it enough to get it to work - But the document is basically filled with data from experiments, but there is data from two types (I just want one type) and the time (I meant time not date) is given within a paragraph with data about the following results (that are set out in six columns with their relative titles) The time is given like this for example - 14:06:15 (15 seconds past 6 minutes past 2)

[dpb]

We'll need to see a representative sample of the data file that illustrates all the conditions that arise and what, specifically, you wish to retrieve. As Guillame says, we can't know what that might be w/o being shown--as I just told the previous poster to whom I responded, the crystal ball is (yet again) in the shop for repair.

[per isakson]

"I tried with regexp but I don't understand it enough to get it to work." &nbsp This example illustrates the use of regexp

>> str = 'ven like this for example - 14:06:15 (15 seconds past 6 minutes past 2)';
>> xpr = '[0-2][0-9]:[0-6][0-9]:[0-6][0-9]';
>> regexp( str, xpr, 'match' )
ans = 
    '14:06:15'
>>

Avoiding false matches is a challenge when using regular expressions.

"But the document is basically filled with data from experiments" &nbsp Detailed info on both the "useless" and the "useful stuff" is needed to extract the useful part.

[Jamie Shelley]

Thanks, but I don't understand the xpr line, I just can't make sense of it in my head - what do the numbers represent please? And I'll try to get a sample next time I'm there, but it was literally like useless time and date 14:04:36 25/05/16 useless variable (x6 in column form) list of variable, tab delimeted, with respect to their variable title useless then repeat

[dpb]

xpr is a match string for a time; it says to match a numeral in the range [0-2], then one between [0-9] followed by : ... etc., etc., ... the ranges match what can be in a valid time in those locations on a 24-hr clock (altho I think the [0-6] should be [0-5] as minutes should roll over to 00 and next hour instead of showing 60 (altho I have seen a bad compiler return such in the past in the hour field of 24:00:[00-29] where it didn't roll over the day until 30 seconds of the first minute of the day had passed. It caused a memory overflow that crashed an application at midnight that had arrays dimensioned 0-23 by trying to write the 24 index--took me many long nights to track it down as being totally unexpected).

[Guillaume]

You still haven't answered the question on how to differentiate useless from useful. Until you do, we can't tell you what regular expression to use.

We need to know what are the exact criteria that makes a date useful, how can you tell the difference from a useless one. We need to know the exact criteria for deciding where the useful portion starts and ends.

[Jamie Shelley]

Thanks, I think I get the xpr thing now - and every other date is useful, because the program sends data from an experiment to a text file (an experiment that I don't need results for) before sending data for a subsequent experiment, from which I do need data - so the time changes every with every experiment, and is then given about the experiment it corresponds to - I don't have a copy of the actual data but I've made an example of what it's like (but it goes on for like 200 experiments or so)

[dpb]

OK, that's much different than the image was getting before; that's a pretty straightforward file structure to parse.

The key is whether there's a constant text string within the header for each experiment that can be used as a landmark for the search. If so, once you find that, then reading the data until the next header is trivial at which point you simply search for the next but one header before reading another section. If there were more than two 'spearmints and you want one of three or N you'd simply skip N-1 after finding the first of the ones you're interested in for a given run. If that were the 3rd of 5, say, you'd have the startup section of skipping two, read one, then the repeating pattern of "skip 4, read 1" until EOF.

[Jamie Shelley]

Okay, thank you, so am I going to need a loop for that please?

[dpb]

In one form or another, yes. "To repeat an operation, it is necessary to repeat an operation!" :)

It's quite possible you could read the whole file into memory and do string comparison to find the row indices of all the headers in "one swell foop" but then you would need to iterate over those altho again one could likely be able to delete large chunks between those locations leaving only the sections of interest to process, but still in the end one would needs must iterate, yes. Now, again, with Matlab you may be able to use things like cellfun or the like and mask the underlying looping at higher levels, but it reduces to a for...end loop under the hood eventually. Often it's far simpler at least initially to just use the "deadahead" approach and use the explicit loop and only after it's shown to be too slow worry excessively about vectorizing. And, often with these more complex cases the vectorized solution actually doesn't outperform the loop and is more difficult to write, debug and maintain.

[Jamie Shelley]

Okay thank you - I think I see what you mean, is there a way of finding the number of paragraphs/lines in the file please?

[dpb]

Sure, but you probably don't need to in order to extract the data of interest.

It would really, Really, REALLY help if you would include an actual file section if you want actual answers instead of continued generalities, but look at the code I provided another poster just recently parsing a similar kind of file to see how to use what's in the file to locate sections of interest... <import-from-data-from-messy-text-file>. That particular file had the nicety of having the number of cases as a parseable value early on in the file so could use a counted outside loop; if, as I gather, your file wouldn't have such just use a while ~feof(fid) loop or equivalent instead. Also, of course, you'd have to keep track/discern which is the one of interest initially and either skip one first or vice versa, depending on the order of whether it's the even/odd case you're interested in. But, as the above shows, it's really pretty simple concept, just takes some consideration of what it is that can be looked for.

Answer 1

[Guillaume]

Now that we finally know what useful and useless look like, we can finally answer the question (mostly).

Here is one way to extract the LPR section(s):

filecontent = fileread('example2.txt');
filesections = regexp(filecontent, 'File name.*?(?=(File name)|$)', 'match');  %match 'File name' and everything that follows up to the next 'File name' or the end of string.
testtypes = regexp(filesections, '(?<=Test type\s*)\S+', 'match', 'once'); %match non-blank characters after 'Test type'
wantedsections = filesections(strcmp(testtypes, 'LPR'));

edit: missing ) in first regex

[Jamie Shelley]

Thanks, but it comes up with 0x0 cells for the three variables created when I use that code, and I'm not sure why

[Jamie Shelley]

Also, I just realised I made a mistake in the file, the time I need is the one for the EIS, not the one for the LPR, as the results I need are the ones under EIS, so the file should have been like this: (I've also copy and pasted a couple more and changed the times incase that makes it easier to make it work), thanks

[Guillaume]

Sorry, it was missing the last closing bracket in the first regular expression.

Whichever section you need does not matter. Just put the correct name in the strcmp call.

[Jamie Shelley]

Thank you, that's a lot further than I managed to get - However, I still can't make it so it just gives me the specific parts I need (as in the time and the table of results, it still comes up with the excess info that comes with the EIS section if you see what I mean)

[Jamie Shelley]

I've managed to get the times I need, using a similar structure to the one you used to extract everything) but still can't extract the results for the EIS tests, as the each result is in one cell and so I don't know how to extract it from there, should I turn each cell into lots of cells (like one per word) or is there a better way of doing it? Thanks

[Guillaume]

You could extract the time with another regular expression:

sectiontimes = regexp(wantedsections, '(?<=Time and Date\s*)[^\n\r]*', 'match', 'once');
timevalues = datetime(sectiontimes, 'InputFormat', 'HH:mm:ss dd/MMM/yyyy'); %optional

Assuming the table you want is always surrounding by blank lines (which actually contain spaces or tabs), you could also isolate it with another regexp:

sectiontables = regexp(wantedsections, '(?<=\r\n[ \t]*\r\n).*?(?=($|\r\n[ \t]*\r\n))', 'match', 'once');

It's then a simple matter of using textscan:

tablevalues= cellfun(@(s) cell2mat(textscan(s, '%f %f %f %f %f', 'CollectOutput', true, 'HeaderLines', 1)), sectiontables, 'UniformOutput', false)

Alternatively, since the EIS sections all appear to have the same format, you could simply split them into lines and extract the lines of interest:

splitsections = regexp(wantedsections, '\r\n', 'split');
sectiostimes = cellfun(@(s) s{7}(15:34), splitsections, 'UniformOutput', false);
sectiontables = cellfun(@(s) strjoin(s(22:end), '\n'), splitsections, 'UniformOutput', false);

[Jamie Shelley]

Thanks, that extracts the tables, but is there a way of putting it into a format that could be sent to excel in multiple cells as opposed to one cell per table please? I've enclosed an example of what I mean, as I can't plot the graphs I need unless I can extract them from that format - Thanks so much for your pacience

[Jamie Shelley]

Just realised I didn't enclose it, here it is:

[Jamie Shelley]

I think that will be the last thing I need as I've produced the rest of the code using a sample I made manually, so it's just one small step from finishing but I can't work out how to get it into that format in an xlsx file

[Jamie Shelley]

In the excel file, the times at the bottom aren't neccesary as I programmed them in differently, I just need to get the test results into the layout shown in the example.xlsx file (but times there aren't neccessary), is there any way of extracting the tables from single cells into the format in the excel file please? If not, I'll start doing it all manually but I feel like it's so close to being finished that it would be a shame to start doing it manually now. Thanks

[Jamie Shelley]

This has been the longest weekend ever, but it's finally done (more or less). Thanks for the help with the really technical stuff, I had no idea how to use regexp and that stuff.

Answer 2

[Shameer Parmar]

use this command..

 Data = textread('FileName.txt', '%s', 'delimiter', '');

then apply the logic (FOR and IF loop) according to your requirement for reading and storing the data..

Please provide the data of your text file and about the required data so that I can help you for logic..

[Jamie Shelley]

Thanks, I tried that line of code but couldn't get it to work - I've enclosed a copy of the file (I've removed the irrelevant info, but I need to get the info from the EIS test and not the LPR ), I've been trying to work it out but I'm just having no luck at all with this one. Thanks

[Guillaume]

At last! We finally get an example of the data as dpb and I have been asking for ages. You still haven't given us the full picture, but can still make a start at answering the question.

[Jamie Shelley]

Sorry, I didn't have a copy of it on me, But I've got one now - it's just basically that but continuously for however many experiments were done - Thanks