how to initialize an M×N array whose size is unknown in for-loop

Hi All,

I am trying to read 150 text files. Each contains the value of x and y. x is the input of an experiment and y is the measured value. Each experiment has different number of data. The number of data cannot be determined until I have read each file.

How should I use the arrays which store x and y in a for-loop efficiently to speed up the process of reading? How should I initialize them while each experiment has different number of elements? I tried to use a code similar to the following code.

x=zeros(??,150);
y=zeros(??,150);
for i=1:150
   fid=fopen(fname);
   data=textscan(fid,'%f %f');
   x(??,i)=data{1}
   y(??,i)=data{2}
end

By ?? I mean I don't exactly know what to use. Could someone please help me?

Thanks.

Answer 1

[Walter Roberson]

x = cell(150,1);
y = cell(150,1);
for K = 1 : 150
  fid=fopen(fname);
  data = textscan(fid, '%f %f');
  x{K} = data{1};
  y{K} = data{2};
end

You can always put them in to a matrix after you have collected them all. If you do that, you will need to decide how you want to pad the shorter columns.

Answer 2

[Naz]

One but not very intelligent way to do it is allocate more memory that you need. You probably have an idea of what what would be a maximum number of values in any of your txt. file. If you know that it should be less than a thousand, allocate a thousand rows (or 2000...). Also, you can fill the matrix with distinct value that you know does not exist in your data:

x=zeros(150,1000); % 1000 columns of x data
y=zeros(150,1000);
x(x==0)=pi;  % change your zeros to pi or (-1) to have it as a mark of the end of data (similar how they use 0 at the end of the string in C)
y(y==0)=pi;

then perform the same loop as you wrote above. When you read data, you know where to stop since there is a mark.

[Jonathan]

I use a similar technique often. I would initialize x as "x = nan(150, 1000);". This way, whenever the data is expected to contain no nans, there is no need to pick another value (such as pi or -1) as a default value. If using pi as the default value is desirable, I use "x = pi*ones(150,1000);".

Answer 3

[Bård Skaflestad]

As your data is all numeric, you can do something akin to the following

x = []; y = [];
for i = 1:150,
  % ... like before ...
  x(1:numel(data{1}), i) = data{1};
  y(1:numel(data{2}), i) = data{2};
end

This will reallocate x and y to appropriate sizes whilst reading the input. I'm not, however, advocating this solution due to the reallocation cost. Personally, I'd just use something like

x = cell([1, 150]); y = cell([1, 150]);
for i = 1:150,
  % ... like before ...
  x{i} = data{1};
  y{i} = data{2};
end

although the effectiveness depends greatly on what you intend to do to 'x' and 'y' once the data is in MATLAB.

Sincerely, Bård Skaflestad

Answer 4

[Jonathan]

A. Hill,

You can probably get a good estimate of the number of elements by examining the file sizes. Suppose the files are in the directory 'C:\files' and the extensions are '.dat'. You can use something like the following.

dirName = 'C:\files';
listing = dir(fullfile(dirName, '*.dat'));
sizeAllFiles = sum([listing.bytes]);
sizeOneEntry = 8; % or some more appropriate number.
numRowsAll = sizeAllFiles / sizeOneEntry;

x=zeros(numRowsAll,150);
y=zeros(numRowsAll,150);
iRow = 1;
for i=1:150
   fid  = fopen(fname);
   data = textscan(fid,'%f %f');
   numRowsOne = size(data{1},1);
   x(iRow:iRow+numRowsOne,i) = data{1};
   y(iRow:iRow+numRowsOne,i) = data{2};
   iRow = iRow + numRowsOne;
end
x = x(1:iRow-1);
y = y(1:iRow-1);

Let me know if this helps.

~Jonathan