MappedTensor disk-mapped variables and lazy loading

version 21.08 (190 KB) by Emmanuel Farhi
A class that represents data remaining on disk with a very low memory foot-print, but behaves as a normal numeric array.

18 Downloads

Updated 17 Aug 2021

View License

This is a class that creates a "numeric variable" representing some data remaining on disk. This way you can transparently treat data larger than the available memory, aka lazy loading and memory-mapped disk data.
You can:
  • create an array of ANY size
  • populate objects with huge data (stored on disk)
  • use many math operators (131 methods)
  • import from: EDF, HDF5, Analyze, ADSC Detector, MAR CCD, MAT 7.3, MRC CCP4, NifTI, NumPy NPY NRRD, Uncompressed TIFF and tomography VOL/PAR.
  • export to: HDF5, Analyze, NumPy NPY, NRRD and NifTI.
Example:
m1 = load(MappedTensor, 'hugefile1.nrrd');
m2 = MappedTensor('hugefile2', [1000 1000 1000],'Format','double');
m3 = m1+m2; % sizes must match
save(m3, 'hugefile_sum.hdf5:/Data'); % creates a 'Data' entry
This work was initiated by D. Muir at http://dylan-muir.com/articles/mapped_tensor/ and is now highly improved (operators, load, save). It benefited from the HDF5 mem mapped object, and more generally is inspired by the memmapfile Matlab object, but in a much more advanced way.
Warning : of course, operators will be slower than when done in memory. Expect an order of magnitude slower.

Cite As

Emmanuel Farhi (2022). MappedTensor disk-mapped variables and lazy loading (https://www.mathworks.com/matlabcentral/fileexchange/93625-mappedtensor-disk-mapped-variables-and-lazy-loading), MATLAB Central File Exchange. Retrieved .

MATLAB Release Compatibility
Created with R2012b
Compatible with R2012b and later releases
Platform Compatibility
Windows macOS Linux

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

MappedTensor-v21.08/@MappedTensor

MappedTensor-v21.08/@MappedTensor/private