Data being scrambled during processing a sequence of files

3 views (last 30 days)
Morning, I have a problem with the processing of CSV files during the processing of a sequence of files from a simulation that I am running (image 1), the number on the folders representing the saved time steps occurring with a coupled open to show the files contained within. If the total number of folders containing the CSV's is ~< 150 (I have not determined the precise threshold) from the initial output the code performs perfectly and extracts and orders the CSV files for me to analyse. However, if the output from the simulation is ~>150 then the order of the data arrangement deteriorates (graph 1) and gets progressively worse with the increasing number of folders.
I must stress it is NOTthe output data of the simulation because I have tested it via two independent means. The first, directly from the UI of the programme itself (graph 2), which reads the same output data as image 1 to form the graph but also I have used a secondary code to sequence the same files as this code using a looped xlsread and the data arrangement extract the data (graph 3). Equally, I am running two types of analysis per simulation that export into two separate subfolders (transient and coupled analysis), the mis-arrangment exists for both.
From what I can determine, the problem may lie in one of these two lines or the compiler itself.
data_struct_Node = [mydata_NODE{:}]; %Compiles all the files into a structured cell array
D_mat_N = cat(3,; %concatenate the cell array - everything ordered based on data field
Attached are graphs and a set of folders which contains trainsent data for the first 30 folders of a total of 193. Does anyone have a solution to this problem?
Graph 1 - Primary code error (node 391).jpg
topLevelFolder = cd;
cd (topLevelFolder);
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
[singleSubFolder, remain] = strtok(remain, ';');
if isempty(singleSubFolder)
listOfFolderNames = [listOfFolderNames singleSubFolder];
numberOfFolders = length(listOfFolderNames);
fprintf('Searching folders for node.csv files...\n');
for k = 1 : numberOfFolders %timesteps
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
% Get desired file type
filePattern = sprintf('%s/*node.csv', thisFolder);
baseFileNames = dir(filePattern);
numberOfFiles = length(baseFileNames);
if numberOfFiles >= 1
% Go through all those files.
for f = 1 : numberOfFiles
fullFileName = fullfile(thisFolder, baseFileNames(f).name);
mydata_NODE{k} = importdata(fullFileName);
fprintf(' Folder %s has no files in it.\n', thisFolder);
fprintf('Processing NODE information...\n');
data_struct_Node = [mydata_NODE{:}]; % Compiles all the files into a structured cell array
D_mat_N = cat(3,; %concatenate the cell array - everything ordered based on data field.
PWP_check = D_mat_N (391,2,:);
RS_PWP_check = reshape(PWP_check,size(PWP_check,1),size(PWP_check,3)); %plot graph
fprintf('done \n');
Richard Rees
Richard Rees on 15 Nov 2019
Hi, I fixed the problem by using natsortfiles. Thank you very much for your time and help in this.

Sign in to comment.

Answers (0)


Find more on File Operations in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by