Writing a tall array / datastore to a .dat
Afficher commentaires plus anciens
Hello, I'm having trouble writing a large .dat file. It's composed of 64 x 1GB files. I attempted to follow the method here, however I'm having trouble writing the output (a 64 x 407297536 matrix) to a .dat which will end up being ~64GB.
Is there an easy way to write a giant .dat?
Thanks very much.
%% User inputs
channels = 1:64;
demed_flag = 1;
store_flag = 1;
% Choose a directory to store the files
outDir = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg';
writeDir = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg\write';
%% Setup
n_channels = length(channels);
% Check how many samplesa are in a single channel
[Sample_check, Header] = Nlx2MatCSC(['CSC' num2str(channels(1),'%02.f') '.ncs'],...
[0 0 0 0 1], 1, 1, [] );
temp_a = reshape(Sample_check,[],1)';
n_samples = length(temp_a);
clear Header Sample_check temp_a
%% Import .ncs data files from the channels list to individual .mat files
fprintf('*Importing data*\n')
for i=1:n_channels
disp(['Importing channel ' num2str(i) ' of ' num2str(n_channels) ' (' ...
num2str(i/n_channels*100,2) '%)'])
%fprintf('i = %.0f\n', i )
[Samples, Header] = Nlx2MatCSC(['CSC' num2str(channels(i),'%02.f') '.ncs'],...
[0 0 0 0 1], 1, 1, [] );
data = reshape(Samples,[],1)';
% Choose a file name - ensure these progress in order
fname = fullfile(outDir, sprintf('data_%05d.mat', channels(i)));
% Save the data and increment counters
save(fname, 'data', '-v7.3');
clear Samples Header data fname
end
clear i
fprintf('*Importing data complete*\n');
%% Create a datastore from the files
% Read the data back in as a tall array. First create a datastore ...
fprintf('*Creating a datastore*\n');
ds = fileDatastore(fullfile(outDir, '*.mat'), ...
'ReadFcn', @(fname) getfield(load(fname), 'data'), ...
'UniformRead', true);
fprintf('*Creating a datastore complete*\n');
% ... and then a tall array
fprintf('*Storing the datastore in a tall array*\n');
tdata = tall(ds);
fprintf('*Storing the datastore in a tall array complete*\n');
%% Write to file for KS2
if store_flag == 1
fprintf('*Writing data*\n');
fid = fopen('myNewFile_zeros.dat', 'w');
fwrite(fid,tdata, 'int16'); % Code breaks here********************
fclose(fid);
fprintf('*Writing data complete*\n');
beep
end
1 commentaire
Divya Gaddipati
le 13 Mai 2021
Could you also mention the error you get?
Réponses (0)
Catégories
En savoir plus sur Audio and Video Data dans Centre d'aide et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!