Datastore for reading large binary files versus incremental fread

I am trying to understand why should I be using Datastore to read 100Gb binary file (other than exceeds memory capacity of RAM)?

For one filedatastore reads the full file before splitting up so that does not help me at all (Matlab 2018version). I cannot even read the full data set because it is huge. So why was it even introduced in the first place? I thought the whole point was to resolve the memory issues.

I could create a custom datastore, but then how is that different from using fseek and fread in a for-loop?

Any examples for reading binary large data in help or documentation? If I do create a customized data store as given in the documentation - do I have to run it in a for-loop? Its not clear to me from reading it here

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html

For example this code runs too slow for an 8Gb file

1. Do I have to open and close the file everytime? Can I just leave it open?

Why do I need fds object when I can simply do the same with fread and fseek in the for-loop below?

sz = 128*1e6*1.024;

fds = fileDatastore(location,'ReadFcn',@(filename) MyReadFcn(filename,offset,sz));

Dat =[];%should pre-allocate this 
for kf = 1:10
   offset = sz;
   while hasdata(ds)
     data = read(fds);
     Dat=cat(1,data,Dat);
   end
   reset(ds)
   
end

function data = MyReadFcn(filename,offset,sz)
fid=fopen(filename,'rb');
% seek to the offset
fseek(fid,offset,'bof');

% read amount of data
[data] = fread(fid,sz,'*uint8');
fclose(fid);
end

Thank you

Answer 1

[Walter Roberson]

If you use https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html with ReadMode 'byte' or 'partialfile' then the datastore facility will ask to read only a portion of the file at a time, and will automatically keep track of where you are in the file.

Can you just leave the file open? Yes.

If your datastore has more than one file, then you can

persistent fid previous_filename
if isempty(fid) || ~strcmp(filename, previous_filename)
   fclose(fid);
   fid = fopen(filename);
   previous_filename = filename;
end

You could also detect end-of-file, in which case you would

fclose(fid);
fid = [];

Answer 2

[dpb]

Basically, the datastore objects are higher-level abstractions for more general file collections. If you can read a portion of the file directly as in your example, you can indeed cut out the middle-man and handle all the details yourself.

It's an attempt at a more user-friendly route -- but, "there is no free lunch!" so there is an overhead to be paid.

I've not had much occasion to play with; but it was my impression that the need to read the full file first was no limitation--if that were the case, it would indeed seem to negate the whole point.

An alternative you might consider for such a file would be memmapfile

[gujax]

Thanks dpb. I think some options in v2018b are lacking. I may have to upgrade and retry.