How to extract numeric values form multiple text files into a single dynamic array?

I have a folder with numberous text files, each file have some numeric data of interst that I'm trying to extract, with the data from other files, into a single big array. Here's my approach:

dr = uigetdir()
d= dir([dr,'\*.txt'])
N = length(d)
A = zeros(1000,N)
for k = 1:N
    [fileLIST, FOLDER] = fopen(d(k).name, 'r');
    if fileLIST<0, error(['Failed fopen: ' d(k).name FOLDER]), end
    A(:,k)= textscan(fileLIST, '%f%f%*s%*s%*s%*s%*s%*s%*s%[^\n\r]', ...
        'Delimiter', '\t', 'headerlines', 3, 'ReturnOnError', false);
    fclose(fileLIST);
end

I have to use textscan with these parameters to get my data of interest from txt files, however I keep getting 'Conversion to double from cell is not possible'. I tried to wrap textscan inside a cell2mat, and it's not working either (All contents of the input cell array must be of the same data type). Can you please help me figure that out? Thanks!

Answer 1

[dpb]

It's always easier if we can see a piece of a file so know exactly what the form is, but...

Your error is that you're trying to assign cell array with ordinary array subscripting as written.

dr = uigetdir();
d= dir(fullfile(dr,'*.txt');
N = length(d);
A = cell(N,1);                                        % collect each file output in cell array
fmt=repmat('%f',1,2);  % ACTUAL FORMAT OF FILE ISN"T WHAT SHOWN ABOVE--DPB
for k = 1:N
  fid = fopen(fullfile(dr,d(k).name), 'r');           % fopen returns a file handle, not a string
  if fid<0, error(['Failed fopen: ' fullfile(dr,d(k).name)]), end
  A(k)= textscan(fid, fmt, 'Delimiter', '\t', ...
                           'headerlines', 2, ...
                           'CollectOutput', 1);
  fid=fclose(fid);
end
A=cell2mat(A);                                       % now convert the whole shebang

[dpb]

See the above edits to match the actual posted file format. I made two copies; ended up with

>> whos A
  Name         Size            Bytes  Class     Attributes

  A         2048x2             32768  double              

>> 

Answer 2

[Khaled Keshk]

Thank you so much, I appreciate your help.

I attached one of the files. I have 26 files like this, each one represents a 'single pixel' scan data. I need to extract the numerical values from each file (first 2 columns), and combine them in huge array (that keeps growing with more data from each file), and then, somehow, process it to get a single plot based on some other code I already have, representing or combining all these pixel data into a single image. Your approach still gets A as a 27x1 cell array, which even after conversion, still gets columns in a 27 row structure that does not grow to account for all the data points from each and every file.

Thanks again.

[dpb]

The file format doesn't match the template from your original posting of having

 '%f%f%*s%*s%*s%*s%*s%*s%*s%[^\n\r]'

that indicates it had three numeric columns followed by a number of unwanted nonnumeric columns. I wrote the format statement for such a file, NOT the file you have. Not surprisingly, it didn't work so well as it failed when it couldn't find anything else on the line after the first two floating point values.

I modified the format statement above to match this particular file structure -- if you change it yet again, you'll need to account for it. I also fixed the 'headerlines' value to 2 instead of three to match the actual file.

[dpb]

[KK's Answer moved to Comment -- dpb]

Thanks again for your insights, very helpful. Yes, you're absolutely right, this weird format I was already given has some unnecessary non-numeric data included; yours make much more sense. Now with all the data values captured, what I need to compute actually, is somehow to average all these data points from A matrix, and do some calculations (comparison) with regards to another B matrix, that has the data of a single reference file. Same file structure as the one I posted earlier no difference. An example would be a line like this:

temp=log(abs(fft(B(1))./abs(fft(A(1)))

This obviously yields a matrix dimensions error as B matrix represnts data from a single file as opposed to A matrix which has data from multiple files. Any idea of how to implement that properly? Can I break it down so that I include this line inside the loop such that B data get divided by same size (yet changing) A data with each file loop iteration?

Thanks a lot.

[dpb]

Sure...you can do anything you want inside the loop before you read the next file...

But to do what you're wanting you really don't need to concatenate the data at all; you simply compute (or read) B before starting the A loop and do the calculations each pass through.

To compute the average, just save the first into an accumulator array and then add each to that; then divide by N at the end...

...
fid = fopen(fullfile(dr,d(1).name), 'r');    % read first file
A(1)= textscan(fid, fmt, 'Delimiter', '\t', ...
                           'headerlines', 2, ...
                           'CollectOutput', 1);
fid=fclose(fid);
S=abs(fft(A{1}));                            % first spectrum
for k = 2:N                                  % now do rest
  fid = fopen(fullfile(dr,d(k).name), 'r');           % fopen returns a file handle, not a string
  if fid<0, error(['Failed fopen: ' fullfile(dr,d(k).name)]), end
  A(k)= textscan(fid, fmt, 'Delimiter', '\t', ...
                           'headerlines', 2, ...
                           'CollectOutput', 1);
  fid=fclose(fid);
  S=S+abs(fft(A{1}));                        % subsequent spectra
end
S=S/N;

Whatever else you want to do w/ B can be added too, of course...

[dpb]

[Answer followup Q? moved to Comment -- dpb]

Sounds good. Based on instructions I was recently given, I have to do my comparison with B matrix with each iteration of A. I tried to do just that and include my analysis code with each loop iteration, and I came across these issues:

When I include A = cell2mat(A) inside the previous loop to get my data into a matrix format for my first file scan, right after the textscan line, I get a "Conversion to double from cell is not possible" error. cell2mat works only when placed outside the for loop.

So based on that error, I figured that I don't need to change A into a matrix and just work with it as it is (cell array) and grab the 1st column for instance with each iteration, like this: timeS = A{:,1}, which should grab only the first column, however, it grabs both the first and second column! I believe it has to do with the textscan format, but couldn't figure out what it is. (A{:,2} comes out empty).

Any ideas?

Thanks a bunch!

[dpb]

The original script was written to accumulate all spectra in the loop in a cell array, A so it was defined to be a cell array at the beginning. Since it is a cell array, you can't convert the cell to an array via cell2mat.

A{:,1} is the first element of the cell array A that is the content returned by textscan so, yes, it does contain both columns.

If this is inside the loop, then A{:,2} is the second cell but hasn't been populated yet.

If you still need to keep all the data in the end, the way to dereference the cell array inside the loop is

A{i}(:,1)
A{i}(:,2)

for the two columns, respectively.

If you can simply do the processing "on the fly" and have the desired result without keeping all the intermediary results, then dispense with the initialization line

A=cell(N,1);

entirely and return the array from textscan as

A= cell2mat(textscan(fid, fmt, 'Delimiter', '\t', ...
                           'headerlines', 2, ...
                           'CollectOutput', 1));

Then the two columns are

A(:,1)                       
A(:,2)

and you can call fft() with whichever of those (or both) as needed, depending on which column(s) it is you're analyzing. I notice in going back it isn't clear just which it is that is to be analyzed; I had presumed both as fft operates on an array by column independently.

[dpb]

[K K's Answer moved to Comment -- dpb]

Now it works, thanks a lot. Getting the whole A matrix first and then derefernce each column separately. One more issue I've been trying to figure out since yesterday if you don't mind. Now I have for A matrix these 2 columns A(:,1), A(:,2) with each iteration. I also would have a similar structure for the reference B matrix B(:,1), B(:,2). I ran my analysis and got a plot. Is there a way to get some sort of 2d or 3d image in matalb not just a simple graph? Let's say I need to substract the min value from the max value for each column from each matrix; I would get a single number from each operation (since I have a total of 4 columns) that would fit something like a 2x2 structure or array. I need to get a 3d image out of it; something to represent the difference between max and min voltages for instance. Any thoughts?

Thank you so much for your help and the very useful tips.

[dpb]

Glad to hear... I knew you could do it! :)

I'm not sure just exactly what you're trying to describe but you might look at

doc waterfall

as a classic way to display time-varying spectra. The vector doesn't have to be a spectrum, of course, it could be your changes in amplitude by position.

Answer 3

[Khaled Keshk]

Good. I know waterfall but not sure how to force it on a matrix strucutre. I'll try some simple examples first.

Thanks again for your help, have a great weekend.