How to import and read several large csv in matlab

12 vues (au cours des 30 derniers jours)
Ricardo Duarte
Ricardo Duarte le 21 Juin 2022
Dear all,
I have a huge amont of large csv files (about 170Mb each).
I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).
This is what I did so far, however I found that it is not very efficient.
%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']);
K=length(x);
Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K
FILENAME = strcat(x(k).folder,'/', x(k).name);
TotalData=readtable(FILENAME);
table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
fprintf('File_%s: %s \n', num2str(k), FILENAME);
tabletotal=[table; tabletotal];
%Coluna 2= longitude
%Coluna 3= latitude
end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));
Thank you in advance
  1 commentaire
M. A. Hopcroft
M. A. Hopcroft le 21 Juin 2022
Which part specifically is not very efficient?

Connectez-vous pour commenter.

Réponse acceptée

Sulaymon Eshkabilov
Sulaymon Eshkabilov le 21 Juin 2022
If your data file names are sequential, then you may consider using this type of approach here:
FILE = fullfile('C:\Users\...', '*.csv'); % Directory where the files are residing.
LIST = dir(FILE);
N = length(LIST); % N = number of files
D = zeros(???, ???, N); % Memory allocation to speed up the process of data storing
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
end
... % Select and process the part of D that is necessary
An alternative way might be this one:
P=pwd; % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing.
S = dir(fullfile(P, '*.csv')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
D = D + readmatrix(F);
end
...
  2 commentaires
Ricardo Duarte
Ricardo Duarte le 22 Juin 2022
Thank you @Sulaymon Eshkabilov, but is there any equivalent function of readmatrix?
Sulaymon Eshkabilov
Sulaymon Eshkabilov le 22 Juin 2022
readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.

Connectez-vous pour commenter.

Plus de réponses (0)

Catégories

En savoir plus sur Dialog Boxes dans Help Center et File Exchange

Produits

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by