Effacer les filtres
Effacer les filtres

Fast code to open and write separate 366 .bin files for each (row,col)

1 vue (au cours des 30 derniers jours)
nlm
nlm le 24 Jan 2021
Commenté : nlm le 25 Jan 2021
I am trying to speed up my code. I have a 18000*36000*366 matrix. I split the matrix into 18 bands of each 1000* 36000 .bin files located in 366 folders. I plan to run the same code for 18 times. The problem is it takes 3.5hrs to process 1*36000 like I wanted. It is super slow, any help is appreciated. Below is my code,
for lat = 1:length(lat_band) % length of lat_band = 1000
for lon = 1:36000
if (mask(lat,lon)~=9) % if ocean then skip
data_1km = data; % matrix of size [366,1];
else
data_1km = NaN(366,1);
end
tmp = 'path';
tic
for day = 1:366
fwrite(fopen(sprintf('%s%d', tmp, day, '/band_1.bin'), 'a'), data_1km(day),'double');
end
fclose('all');
clear data_1km day
toc
end
end

Réponse acceptée

Walter Roberson
Walter Roberson le 24 Jan 2021
That code fopen()'s 366 files inside the loop, which can consume all of the file handles. Better would be
for day = 1:366
fid = fopen(sprintf('%s%d', tmp, day, '/band_1.bin');
fwrite(fid, 'a'), data_1km(day),'double');
fclose(fid)
end
If your system can handle and your process is authorized to have 366 files, then fopen() all of them before for lat and index into the list of handles in the loop. You are doing a lot of fopen() of the same file, and that is expensive.
  11 commentaires
Walter Roberson
Walter Roberson le 24 Jan 2021
I will run this code for 18 lat_bands.
What is the file name to be used for band #4 day #7 ? And could you confirm that the variable lat is the one that stores the current lat band number?
nlm
nlm le 25 Jan 2021
Can you tell me why running a code as simple as 0.012 secs. That is a long time isn't it ?
data = NaN(366,1);
tic
for day = 1:ND
fwrite(fids(day), data(day), 'double');
end
toc

Connectez-vous pour commenter.

Plus de réponses (0)

Catégories

En savoir plus sur Low-Level File I/O dans Help Center et File Exchange

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by