Strategy on how to save data from multiple runs of an experiment where variables changes

I need some guidance on the best way to structure and save data from experiments with many runs and chaning variables.

An example:

Let's say that I try to measure the light intesity from an LED after the light has passed an optical filter. I will do many experiments and change between different combinations of LEDs, filters and detectors, all with a traceable serial number. At every experiment i step the LED current between 0 and 100% in suitable steps and make 1000 samples at every current level so that I can analyze noise and drift at every level. Some combinantions of LED, filter and detector will be run several times. Every experiment that I perform will produce a text-file with the measured data inside.

How should I handle this data so that I easilly can make statistics from it in MATLAB?

Should I save everthing in tables, a large cell array or keep it in the text files? How do I map the LED, filter and detector ID:s to the data?

I would for example like to be able to easilly plot the STDEV of all LED:s with filter F3 and Detector D2 at a certain LED current. Or any other combination of equipment.

Any thoughts on how to save the data and keep mapping to the physical equipment would be much appreciated.

[Stephen23]

Use one table to conduct the statistical analyses.

That does not mean that all intermediate results and whatnot need to be stored there, but using one table has many benefits for the kind of processing that you seem to want to do:

• store (and use) numeric data as numeric, text data as text (or categorical), etc. This is much tricker with other container types like cell arrays.
• easily find groups based on one or more conditions, then apply functions or operations to those groups (e.g. exactly like your "plot the STDEV of all LED:s with filter F3 and Detector D2 at a certain LED current" example).
• can trivially include meta-data, eg. filenames, timestamps, or any other test parameters.

https://www.mathworks.com/help/matlab/matlab_prog/grouping-variables-for-splitting-data.html

https://www.mathworks.com/help/matlab/matlab_prog/split-data-into-groups-and-calculate-statistics.html

Answer 1

[the cyclist]

There's probably no single best answer here. But if the overall scale of the data is not too large, storing everything in one table is a very appealing way to go, and may be the most "modern" from a MATLAB point of view.

I would specifically recommend storing the data in what is known as the tidy format. Each row is a single observation, and the columns will track all the possible variables, which look to be

• LED type
• filter type
• detector type
• LED current
• sample number
• light intensity

Data are not implicitly coded into column names -- e.g. you do not want columns "Sample1", "Sample2", etc -- even though that sometimes seems more compact. The "long skinny" format is almost alway better for general analysis. That site I linked gives explanation. (It is written from the perspective of R, but the concept is agnostic to language.)

MATLAB operations on table are well designed to do summary and group statistics on data that are laid out like this.

[Quist]

Thanks!

I think I get the idea, but I realize that I will have a loooong table at the end. Probably some milions of rows... :/

[the cyclist]

Here's a bit of simple manipulation of a table that has 20 million rows and 12 columns. No problem.

If you don't store the data in one table, you have complications in joining the data, etc.

But, as I mentioned, there is probably no single best answer.

tic
tbl = array2table(randi(7,2e7,12)); % Majority of the time is spent making up these "data"
toc

Elapsed time is 5.685769 seconds.

tic
groupsummary(tbl,"Var1")

ans = 7×2 table
    Var1    GroupCount
    ____    __________

     1      2.8559e+06
     2      2.8591e+06
     3      2.8554e+06
     4      2.8595e+06
     5      2.8569e+06
     6      2.8578e+06
     7      2.8553e+06

toc

Elapsed time is 2.604186 seconds.

[Quist]

I'm rather new to MATLAB so some tips and tricks around how to work with the data in the table would be appreciated.

For example:

• Is it a good idea to make the variables Categorical?
• What is the best way to filer out data at some specific conditions?

T(T.Var1 == 'X' & T.Var2 == 'Y' & T.Var3 == 'Z',:).Data;

• In th example above how do I allow both X1 and X2 as Var1?
• What is the best way to calculate statistics on groups?

[Groups,Var1,Var2,Var3,Var4,Var5,Var6] = findgroups(T.Var1,T.Var1,T.Var1,T.Var1,T.Var1,T.Var1);
SomeAverage = splitapply(@mean,T.Data,Groups);
table(Var1,Var2,Var3,Var4,Var5,Var6,SomeAverage);

I almost would like to make a GUI where I can manipulate the settings...

[the cyclist]

This forum is not a very good way to teach you from scratch how to use MATLAB in general, or about tables as a data type. It is more suited to targeted questions when you are stuck on a particular task.

You need to immerse yourself in the excellent documentation provided, and its many examples. I would start with the main documentation about tables. From there, there are many specific articles about how to do statistics, deal with missing data, etc. Maybe just try to set up a few of your variables at first, instead of your whole dataset.

Thinking about setting up the data, you asked, "Is it a good idea to make the variables Categorical?". The answer to that is no. If you have measured current, that would not be categorical. But maybe you have a variable for "Detector Type", which would be. The data type should match the variable. (This is one of the advantages of using tables rather than a purely numeric data type, which could not store "Type X" and "Type Y".)

Regarding your second question ... There are often different methods for achieving the same result. (It's not possible to state "the best" way, without more context and insight into your problem.) The exact syntax will depend on the datatype, but if

T(T.Var1 == 'X' & T.Var2 == 'Y' & T.Var3 == 'Z',:).Data;

worked to filter how you wanted, then

T(ismember(T.Var1,{'X1','X2'}) & T.Var2 == 'Y' & T.Var3 == 'Z',:).Data;

should work.

Note that you can't do equality checks between character arrays of different sizes:

'X12'=='X2'

Arrays have incompatible sizes for this operation.

(That doesn't work because you are compareing a 1x3 array with a 1x2, and MATLAB doesn't know what you mean.) You need to use the strcmp function instead:

strcmp('X12','X2')

Alternatively, you could work with strings instead of character arrays:

"X12"=="X2"

I know, I know. It's a bit confusing and probably a bit overwhelming. Take it a step at a time, and post new, specific questions when you get stuck. Upload a small sample of your actual data/table if you can.