Effacer les filtres
Effacer les filtres

Coalesce data from 1000 monte Carlo simulations with different data file lengths?

1 vue (au cours des 30 derniers jours)
Wesser
Wesser le 24 Oct 2022
Hi all,
I'm trying to coalesce data from 1000 monte Carlo simulations of contaminant infiltration into the ground. Each Monte Carlo run generated a file Obs_Node.txt. I want to skip the first 11 lines of this .txt file, then isolate each data column beneath that to save for future analysis. My challenge is that each Monte Carlo run has unique time convergence intervals...ie the length of the data columns are different for each MC iteration's Obs_Node.txt file. In other words, in one Path(i) (ie Monte Carlo iteration) there would be 123456 rows of data for each of the 5 columns and in another Path(i) there may be 65432 rows of data for each of the .txt's 5 data columns. So the ## changes with each Path(i).
My questions are:
1) how do I preallocate when the length of the data changes for each Path(i) at the top of the code?
2) how do I tell Matlab to scan the data when I don't know the length of the data colums ?
I attached an example of Obs_Node.txt for reference....there are only 5 rows in this reference file for simplicity.
num_sim = 1000; %number of moonte carlo simulations
%Obs_Node.out - preallocation of desired output data from each monte carlo run
TIME_INTERVAL = zeros(##,num_sim);
Node_CONC=zeros(##,num_sim);
THETA_T1 = zeros(##,num_sim);
FLUX_T1 = zeros(##,num_sim);
CONC_T1 = zeros(##,num_sim);
%~~~~~~~~~~Coalesce data from Obs_Node.out files~~~~~~~~~~~~
for i=1:num_sim
Obs_Node = fopen(["/Users/apple/Dropbox/Desktop/Simulations/MC_"+num2str(i)+'/Obs_Node.out']); % Open monte carlo output file in Path (i)
skip_lines=11; %skip all the lines until the output data of interest
for k=1:(skip_lines)
x=fgetl(Obs_Node);
end
temp1 = fscanf(Obs_Node,'%f',[5,##]); %scan the matrix of data (QUESTION 2 ABOVE)
TEMP1 = temp1'; % transpose data
time_interval = TEMP1(:,1); % Time intervals - 1st data column
TIME_INTERVAL(:,i) = time_interval(:); % saves each iteration's data
theta_T1 = TEMP1(:,3); % Hydraulic Conductivity - 3rd data column
THETA_T1(:,i) = theta_T1(:); % saves each iteration's data
fclose(Obs_Node);
end

Réponses (2)

Benjamin Thompson
Benjamin Thompson le 24 Oct 2022
With the data import tool you can generate a function like this. It needs some editing to handle some of the odd formatting of your file. Then you can pass [12 Inf] as the datalines argument. The "end" at the end of your file is difficult, so you can either remove the last line from the output after loading the file data, or have a separate tool or function that removes the last row with "end" from files before you read them into MATLAB.

David Hill
David Hill le 24 Oct 2022
opts = detectImportOptions('Obs_Node.txt');
opts.VariableTypes={'double','double','double'};
opts.Delimiter={'\t'};
opts.DataLines=[12 Inf];
t=readmatrix('Obs_Node.txt',opts);
t(:,1)=[];
t(isnan(t(:,1)),:)=[]
t = 5×5
0.0010 -25.0000 0.2827 -382.0000 0 0.0020 -25.0000 0.2827 -244.0000 0 0.0030 -25.0000 0.2827 -170.0000 0 0.0043 -25.0000 0.2827 -114.0000 0 0.0060 -25.0000 0.2827 -75.4000 0
  1 commentaire
Wesser
Wesser le 24 Oct 2022
This works conceptually but the file that is being read is actually "Obs_Node.out" and not "Obs_Node.txt". I had changed the file ending for the question above as I couldn't upload the example file without changing the file from .out to .txt. The opts route seems feasible, but how do I get matlab to read the .out file path? It doesn't seem that there is a name-value pair that can access the .out files.
Name-Value Pairs supported with Import Options OPTS:
% See also spreadsheetImportOptions, delimitedTextImportOptions, fixedWidthImportOptions,xmlImportOptions, readtable, readtimetable, readmatrix, readcell

Connectez-vous pour commenter.

Catégories

En savoir plus sur Logical dans Help Center et File Exchange

Tags

Produits


Version

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by