batch file processing (.txt files) for data analysis

Hi, I have more than 200 '.txt' files to process the data. Each file has same number of coloumns, i.e 10, but number of rows vary. Imported data to array variables and created a structure of each file. And further processing data of individual file for analysis.

Hy=importdata('hyde0710p1c1.15txt');

Ii=importdata('iisc0710p1c1.15txt');

save('Hy710.txt','Hy'); save('Ii710.txt','Ii');

hyde_org=struct('GPSTOW',Hy(:,1),'PRN',Hy(:,2),'EL',Hy(:,3),'AZ',Hy(:,4),'IPPMLat',Hy(:,5),'IPPGeoLong',Hy(:,6),'IPPGeoLat',Hy(:,7),'SlantCodeTEC',Hy(:,8),'SlantSmoothTEC',Hy(:,9),'VTEC',Hy(:,10));

iisc_org=struct('GPSTOW',Ii(:,1),'PRN',Ii(:,2),'EL',Ii(:,3),'AZ',Ii(:,4),'IPPMLat',Ii(:,5),'IPPGeoLong',Ii(:,6),'IPPGeoLat',Ii(:,7),'SlantCodeTEC',Ii(:,8),'SlantSmoothTEC',Ii(:,9),'VTEC',Ii(:,10));

But,now I need to append all the data(to append rows one after the other file's) of 200 files in ta single new file for processing.

Can anyone help please?

Answer 1

[Deepak]

Hi @Lakshmi Chodavarapu,

We can achieve the concatenation of data from multiple ".txt" files into a single structure in MATLAB by first listing all files using the "dir" function and initializing empty arrays for each data column. We then loop through each file, import the data using "importdata", and append the data to the respective arrays. After processing all files, we create a structure with these concatenated arrays and save it as a ".mat" file for further analysis.

Below is the sample MATLAB code to achieve the same:

% Directory containing the .txt files
folderPath = 'your/folder/path'; % Change this to your folder path
filePattern = fullfile(folderPath, '*.txt');
txtFiles = dir(filePattern);

% Initialize empty arrays for concatenation
GPSTOW = [];
PRN = [];
EL = [];
AZ = [];
IPPMLat = [];
IPPGeoLong = [];
IPPGeoLat = [];
SlantCodeTEC = [];
SlantSmoothTEC = [];
VTEC = [];

% Loop through each file
for k = 1:length(txtFiles)
    baseFileName = txtFiles(k).name;
    fullFileName = fullfile(folderPath, baseFileName);
    
    % Import data
    data = importdata(fullFileName);
    
    % Append data to arrays
    GPSTOW = [GPSTOW; data(:,1)];
    PRN = [PRN; data(:,2)];
    EL = [EL; data(:,3)];
    AZ = [AZ; data(:,4)];
    IPPMLat = [IPPMLat; data(:,5)];
    IPPGeoLong = [IPPGeoLong; data(:,6)];
    IPPGeoLat = [IPPGeoLat; data(:,7)];
    SlantCodeTEC = [SlantCodeTEC; data(:,8)];
    SlantSmoothTEC = [SlantSmoothTEC; data(:,9)];
    VTEC = [VTEC; data(:,10)];
end

% Create a single structure with all data
combinedData = struct('GPSTOW', GPSTOW, 'PRN', PRN, 'EL', EL, 'AZ', AZ, ...
                      'IPPMLat', IPPMLat, 'IPPGeoLong', IPPGeoLong, 'IPPGeoLat', IPPGeoLat, ...
                      'SlantCodeTEC', SlantCodeTEC, 'SlantSmoothTEC', SlantSmoothTEC, 'VTEC', VTEC);

% Save the combined structure to a .mat file
save('combinedData.mat', 'combinedData');

Please find attached the documentation of functions used for reference:

dir: www.mathworks.com/help/matlab/ref/dir.html

importdata: www.mathworks.com/help/matlab/ref/importdata.html

I hope this assists in resolving the issue.