Readtable For Loop for importing large quantities of csv files in to Matlab

Hello,

I am currently working on a project which requires me to import sequential data into matlab so that it can be processed and analysed as one data set. There is multiple csv files because the data logger creates a new file every 40mb or so.

I have put together the following code to import the data.

if true
  % code
end
clear all
close all
d = uigetdir();
filePattern = fullfile(d, '*.csv');
file = dir(filePattern);
x={};
for k = 1 : numel(file) 
      baseFileName = file(k).name;
      fullFileName = fullfile(d, baseFileName)
      x{k} = readtable(fullFileName);
end

There are 4 things I have not done yet.

1. Preallocate memory for x - I don't know how much to preallocate.

2. I am not appending the data in to one matrix. Rather, a cell array combining each set of data. For Example 3 sets of data creates a 1x3 cell array each including a table of the data.

3. Added a status print in cmd window i.e: "read file xyz"

4. Not added datetime format because the first column has cotents like this: "31/03/2017 07:58:31"

Here is an example of the output.

In this case what I actually need is a 64,803 x 191 table (Need to keep variable names - each row is a specific variable).

How do I achieve this?

For any given data set I will always have the same amount of columns but the number of rows may vary in each file.

Answer 1

[Guillaume]

1. Replace

x = {};

by

x = cell(1, numel(file));   %better name for that variable advised. How about: tables

Note that unless the number of files is significant, this won't have any noticeable impact as it only allocates memory for the cells, not for the tables that go into the cells. This is normal and how cell arrays work.

2. That's fine and is the proper way to do it

3. After the readtable:

fprintf('read file %s\n', fullFileName);

4. Assuming all the tables have the same variable names, to concatenate them all into one table, after the loop:

bigtable = vertcat(x{:});

[John Doe]

Thank you Guillaume. It all seems to be working now.

1. I am not sure what you consider big. I am currently running the script on 31 files (660mb). It's still running.

What would you consider to be significant? and what measures would you take to improve that for large data sets ( a few gb).

2. That's good!

3. Thank you works well

4. Tried vertcat previously I made an error in the syntax. Now works great (thank you!)

[Guillaume]

Preallocation is good practice so you should do it anyway. For the preallocation, the size of the content of the cell is irrelevant (the size of the tables), only the number of cells is (the number of files). So 31 files is small. This will result in an allocation of 31x8 = 248 bytes (+ some small overhead for storing the size and type of the cell array).

What is stored in the cell is just 8 bytes pointers to the actual data. The actual data will be allocated by readtable. You cannot preallocate memory for that (and there would be no point).

[John Doe]

I struggled with that concept previously. Thank you for clearing up.

[SUMAN]

if table doesn't have same variable name then how to concatenate them to one table