Organizing array of data skipping empty values

Hello,

I need to organize data (now stored in .txt files) in either a single .txt file or in a struct; my problem is that this txt files, that are coming from simulations, miss a variable number of lines each. What I thought to do is to load each file inside a structure named P, and to create a struct file named M with each substructure having the maximum number of rows and columns that the files should contain, filled with zeros.

Then I wanted to write every row contained in the substructures of P in the corresponding substructures of M, leaving a row of zeros where the data files are missing a line. The .txt files all have in the first column an increasing value going from -5 to 25 with steps of 0.1, with the exception of the missing lines, as I said before. Can anyone help me on this part? How could I do?

Below you can see what I have written until now:

path='C:\Users\iberi\Desktop\Università\Magistrale\Tesi\MATLAB\Grafici_xx07\Polars\Section1\';
Re={'Re0100' 'Re0110' 'Re0120' 'Re0130' 'Re0140' 'Re0150' 'Re0160' 'Re0170' 'Re0180' 'Re0190' 'Re0200' 'Re0210' 'Re0220' 'Re0230' 'Re0240' 'Re0250' 'Re0260' 'Re0270' 'Re0280' 'Re0290' 'Re0300' 'Re0310' 'Re0320' 'Re0330' 'Re0340' 'Re0350'};
for i=1:length(Re)
   
    M.(Re{i})=zeros(301,10);
    P.(Re{i})=load([path, 'CLARK_Y_W=0.02_TEGap0.75_Blend0.6_FOIL2CAD_smooth_300_T1_' Re{i} '_M0.00_N9.0.txt']);
    
    for j=1:301
        for z=1:10
        

        end
    end
    
end

[Mathieu NOE]

hello Marco

would you be able to share the data files ? pls make a zip if you have many

tx

[Marco Gualtieri]

Hello Mathieu,

of course, please find attached the zip file

Thank you for your reply

[Yazan]

I'm not sure I understood what you are after. Do you need to read txt files in which some lines are empty?

[Marco Gualtieri]

Hello Yazan, thank you for your reply,

the data files are the results of a simulation and contain data organized in rows: each row contains a set of data for a particular angle, increasing 0.1 each row, from -5 to 25. Now, since the simulation didn't converge in each point, some rows are missing; let's say, for example, that in some files you can jump from -3 to -2.8, skipping the row of the angle -2.9.

I need to organize those files in a way in which I can easily put on comment all the missing data rows; I have to do that for each file, so for example if in one file the row of the angle -2.9 is missing, I will have to comment that in all the other files, because I won't be able to use that data.

It's easier to see than to explain: if you want you can download the zip file that I uploaded in the reply before, you'll immediately get what I am saying

Answer 1

[Scott MacKenzie]

I don't think you need nested for-loops. Use the 1st column of the data read as an index into the M.Rexxxx structure to direct the data into the corresponding rows in the M.Rexxxx stucture, leaving the remaining rows as zeros. Also, I don't see any reason to read the data into an intermediate structure. Just read the data into a matrix (e.g., P). Here's the general idea:

for i=1:length(Re)    
    M.(Re{i})=eye(301,10);
    P=load([path, 'CLARK_Y_W=0.02_TEGap0.75_Blend0.6_FOIL2CAD_smooth_300_T1_' Re{i} '_M0.00_N9.0.txt']);
    
    % move data from P into rows in M.Rexxxx structure, using 1st column in P as index
    M.(Re{i})(P(:,1),:) = P;    
end

[Marco Gualtieri]

Hello Scott, thank you very much for your answer.

I think that yours is a very good idea, but how can I solve the problem that some of my indices are negative values?

[Scott MacKenzie]

Ah, yes, there's always one more detail. Hmmm, let's see... You need to translate the data in the first column in P, mapping -5, -4.9, -4.8, ... 25 to 1, 2, 3, ... 301. Try this

idx = P(:,1);
idx = rescale(idx,1,301);

[Marco Gualtieri]

Thank you again Scott, that seems to work. I made a small change, I think that maybe you typed that wrong. I wrote:

idx = P(:,1);

P(:,1) = rescale(idx,1,301);

You think that's correct?

[Scott MacKenzie]

Well, if that worked, then OK.

BTW, there is one potential problem in rescaling as proposed. If the source data are missing either the expected min or max value (-5 or 25), then rescaling won't give you a clean and correct set of integer indices. The fix is to add a fake min and max to the source data before rescaling, then remove those rescaled values afterward:

idx = [-5; 25; P(:,1)];
idx = rescale(idx, 1, 301);
idx = idx(3:end);
M.(Re{i})(idx,:) = P;    

Clearly, this wasn't an issue with your data. I guess every file has rows beginning with -5 and 25 and that the missing rows were somewhere between.

[Marco Gualtieri]

Actually that was a problem for a couple of files but I noticed it and added the last row; I just have to remember to put on comment all the last rows after I put the data togheter. But thank you again for everything!