Reorganization of experimental data

Hello everybody,

I have a problem with the reorganization process of my experimental data.

Attached there is an example of a typical data set

              0              0        -93.964    -0.00976171
              1          0.001        -93.964    -0.00976627
              2          0.002        -93.965    -0.00991459
              3          0.003        -93.965    -0.00997991
              4          0.004        -93.963     -0.0100311
              5          0.005        -93.962     -0.0101206
              6          0.006        -93.964     -0.0101955
              7          0.007        -93.961     -0.0098519
              8          0.008        -93.958    -0.00946246
              9          0.009        -93.945    -0.00970383
             10           0.01        -93.921     -0.0105877
             11          0.011        -93.884     -0.0125595
             12          0.012        -93.829      -0.015714
             13          0.013        -93.755     -0.0194494


Results:
Sample:
  2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg


Curve Name:
  ]61[&2021_07_29_cool_variab_heat_fix_59603
Curve Values:
          Index              t             Ts          Value
                           [s]           [°C]           [mW]
              0              0        -93.205   -0.000370132
              1           0.01        -93.365    -0.00237034
              2           0.02        -93.578    -0.00514436
              3           0.03        -93.713    -0.00678095
              4           0.04        -93.800    -0.00771421
              5           0.05        -93.857    -0.00846012
              6           0.06        -93.893    -0.00883113
              7           0.07        -93.917    -0.00909422
              8           0.08        -93.932    -0.00928991
              9           0.09        -93.944    -0.00943325
             10            0.1        -93.949    -0.00949218
             11           0.11        -93.954    -0.00950072
             12           0.12        -93.957    -0.00963767
             13           0.13        -93.958     -0.0096308
             14           0.14        -93.960    -0.00962155
             15           0.15        -93.961    -0.00964608


Results:
Sample:
  2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg


________________________________________________________________________________
Not signed                                                        STARe SW 16.10

I want to remove all the written texts and empty lines and I want to reorganize the blocks of data in a way to invert their order.

Do you have any suggestion?

[Star Strider]

Attach a ‘typical data set’ as a .txt or .mat file.

Answer 1

[Star Strider]

The data file you attached does not appear to bear a strong resemblance to the data file you quoted.

I chose to use textscan here because I have experience using it with files with interrupted formats, however this file appears to have no interruptions:

fidi = fopen('typical_data_set.txt','rt');
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
fclose(fidi);
M = cell2mat(C);

Here ‘M’ is a (44006x4) double array.

A file with interruptions would simply require a while loop to loop through it, saving the individual sections. For an example, see: How to skip lines of Data in the middle of a text file . I believe that file was similar to the one you quoted.

EDIT — (1 Aug 2021 at 00:15)

Removed the generic code example. Nothing else changed.

.

[Daniele Sonaglioni]

Sorry, at first I attached an abridged version of my data file.

Your first code (attached below) works quite well but I also would like to reverse the order of the blocks in the txt file in a way that the first appearing block will become the last and so on.

fidi = fopen('typical_data_set.txt','rt');
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
fclose(fidi);
M = cell2mat(C);

How it is possible to improve your code?

[Star Strider]

I did not look at the entire file, however I did not detect any blocks. It just read as one entire matrix, with contiguous values of the first variable (that appears to be a counter of some sort).

Did you attach a different file this time?

.

[Daniele Sonaglioni]

No, I have not attached a different file. I was referring to the data between two lines of text. I want to invert their order.

[Star Strider]

I can only detect one block of data using:

fidi = fopen('typical_data_set.txt','rt');
k1 = 1;
while ~feof(fidi)
    C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
    M = cell2mat(C);
    if isempty(M)                                       % Empty Matrix Indicates End-Of-File
        fprintf(1,'Reading finished, %d Sections\n',k1-1)
        break
    end
    D{k1,:} = M;
    fseek(fidi, 0, 0);
    k1 = k1 + 1
end
fclose(fidi);
Out = cell2mat(D);

This code would detect a line of text that did not match the numeric format descriptors, stop, and re-start with the next block. (It then optionally combines the blocks into a single matrix if desired.) I have tested it on different files over the several years since I wrote it, and it consistently works. There is only one block of data in the posted file, and only one header line.

I have not gone through the file in detail by hand (it has 44006 rows), so I have no idea where the ‘two lines of text’ are. I would appreciate your displaying them and the rows and row numbers where they occur. I cannot detect their presence.

.

[Daniele Sonaglioni]

I am really sorry, I have attached the wrong file.

Here I attach the right file: you can see that through the file there are text lines (e.g. the first is at line number 3699).

I have tried to use your codes but they stop at the first block.

[Star Strider]

This turned out to be something of an adventure!

My code works, however it needs to be modified a bit for your file:

fidi = fopen('typical_data_set_2ndversion.txt','rt');
k1 = 1;
while ~feof(fidi)
    readline = fgetl(fidi);
    if strcmp(readline,'Results:')
        C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',9, 'CollectOutput',1);
    else
        C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',0, 'CollectOutput',1);
    end
    M = cell2mat(C);
    fprintf(1,'\tSection %2d: (%4d x %d)\n', k1, size(M))
    if isempty(M)                                           % Empty Matrix Indicates End-Of-File
        fprintf(1,'Reading finished, %d Sections\n',k1-1)
        break
    end
    D{k1,:} = M;
    fseek(fidi, 0, 0);
    k1 = k1 + 1;
end
fclose(fidi);
Dflip = flip(D);                                            % Reverse Section Order
% Out = cell2mat(D);                                          % Optional Concatenation

The end result is 56 segments of varying row length, all with 4 columns. (The section information is not retained, only the numeric information.) I did not go through all 56 results, however the few I sampled appear to be correct.

.

[Daniele Sonaglioni]

Your code works quite well but what I want to reverse is not the order of the columns in the file but the position of the whole block. Namely, the first block of data, after analyzed with your code, should become the last and so on. Is it possible?

[Star Strider]

Thank you.

If you check, that is exactly what my code does. The individual block contents aree not reordered in any way. Only their order in the ‘D’ cell array are changed, so that the last block becomes the first (for example ‘D{1}’ becomes ‘Dflip{56}’, ‘D{56}’ becomes ‘Dflip{1}’) and so for the rest. I checked that as well as selected individual blocks before I posted my code.

Adding these two lines after the ‘Dflip’ assignment:

Check1 = [D{1}(1:5,:); NaN(1,4); Dflip{end}(1:5,:)]
Check2 = [D{end}(1:5,:); NaN(1,4); Dflip{1}(1:5,:)]

produces:

Check1 =

    1.0000    0.0001   25.1940    0.0006
    2.0000    0.0002   25.1910    0.0006
    3.0000    0.0003   25.1900    0.0005
    4.0000    0.0004   25.1910    0.0004
    5.0000    0.0005   25.1820    0.0004
       NaN       NaN       NaN       NaN
    1.0000    0.0001   25.1940    0.0006
    2.0000    0.0002   25.1910    0.0006
    3.0000    0.0003   25.1900    0.0005
    4.0000    0.0004   25.1910    0.0004
    5.0000    0.0005   25.1820    0.0004

Check2 =

         0         0  -93.9600   -0.0104
    1.0000    0.0100  -93.9590   -0.0103
    2.0000    0.0200  -93.9580   -0.0104
    3.0000    0.0300  -93.9590   -0.0103
    4.0000    0.0400  -93.9590   -0.0103
       NaN       NaN       NaN       NaN
         0         0  -93.9600   -0.0104
    1.0000    0.0100  -93.9590   -0.0103
    2.0000    0.0200  -93.9580   -0.0104
    3.0000    0.0300  -93.9590   -0.0103
    4.0000    0.0400  -93.9590   -0.0103

The NaN lines are simply used to separate the matrix samples in order to make it easier to read.

Note that ‘D{end}’ is ‘D{56}’ in this file, and similarly for the ‘Dflip’ elements.

So, the order of each block is flipped as requested, not the contents. This additional bit of code (not necessary for the rest of the code) demonstrates that.

.

[Daniele Sonaglioni]

Thank you very much, you have literally saved my life!

I will accept your first answer because the site does not allow me to accept the intermediate one.

Thank you again!

[Star Strider]

As always, my pleasure!

(I am a physician. That is what we do!)

.

[Daniele Sonaglioni]

Wow fantastic, I am a physician too but, as you can see, I am not so good with coding.

What is your research field?

[Star Strider]

I am an M.D. Board Certified in Internal medicine, and am a fellowship-trained endocrinologist, although not B/C. I subsequently earned a M.S. in Biomedical Engineering, and have done research in diabetes and most notably neurophysiology that had no direct connection with endocrinology, and instrumentation. My areas of interest are system identification, parameter estimation, instrumentation, and signal processing. I am now retired.

My first exposure to computers was a short undergraduate course in FORTRAN. I earned my M.S. after doing my fellowship and becoming fascinated with homeostatic regulatory mechanisms, specifically with respect to endocrinology, however those signals are so difficult (and expensive) to gather that most of my research was in other areas, where that was not a problem.

And yours?

.

[Daniele Sonaglioni]

I have just earned my master degree in condensed matter physics and I am now focusing in soft matter physics, especially in the physics of proteins. In particular, I am specialized in differential scanning calorimetry.

Your field is very interesting: I also want to make some research in the medical field although you are more interested on "large scale" phenomena while I prefer smaller scales.

I think that coding is very useful in every field of study, not only physics and I am trying to lear as much as possible to make deeper data analysis to better understand what I am studying.

[Star Strider]

Interesting!

I wish you the best in your research!

Answer 2

[Daniele Sonaglioni]

Here you are: this is a typical data set of mine.