How to import and read several large csv in matlab
12 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Ricardo Duarte
le 21 Juin 2022
Commenté : Sulaymon Eshkabilov
le 22 Juin 2022
Dear all,
I have a huge amont of large csv files (about 170Mb each).
I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).
This is what I did so far, however I found that it is not very efficient.
%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']);
K=length(x);
Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K
FILENAME = strcat(x(k).folder,'/', x(k).name);
TotalData=readtable(FILENAME);
table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
fprintf('File_%s: %s \n', num2str(k), FILENAME);
tabletotal=[table; tabletotal];
%Coluna 2= longitude
%Coluna 3= latitude
end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));
Thank you in advance
1 commentaire
Réponse acceptée
Sulaymon Eshkabilov
le 21 Juin 2022
If your data file names are sequential, then you may consider using this type of approach here:
FILE = fullfile('C:\Users\...', '*.csv'); % Directory where the files are residing.
LIST = dir(FILE);
N = length(LIST); % N = number of files
D = zeros(???, ???, N); % Memory allocation to speed up the process of data storing
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
end
... % Select and process the part of D that is necessary
An alternative way might be this one:
P=pwd; % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing.
S = dir(fullfile(P, '*.csv')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
D = D + readmatrix(F);
end
...
2 commentaires
Sulaymon Eshkabilov
le 22 Juin 2022
readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.
Plus de réponses (0)
Voir également
Catégories
En savoir plus sur Dialog Boxes dans Help Center et File Exchange
Produits
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!