Loading binary data into custom array

Hi,

I'm trying to load in a binary data file with a unique data structure {uint32, uint32, double, double, double,.., double, uint32}. After doing a little bit of research, folks on the forums have recommended using the function "memmapfile()" to read in binary data files with custom arrays. The problem with this is my data is all in big-endian format and memmapfile() outputs the data in little endian format.

I'm looking for help as to whether or not memmapfile() can be produce a big-endian output without me using functions like swapbytes() or typecast() as it increases the loading/processing of the data significantly.

Any help is deeply appreciated,

MP

[Walter Roberson]

I notice that https://github.com/DylanMuir/MappedTensor allows big-Endian

[MPan]

Thanks Walter, I'm going to give this a shot. This seems promising.

[MPan]

Walter,

I had the chance to review this code today and it appears that the MappedTensor does not provide the ability to read data into custom data structures based on the issue that was opened in Github. The author suggested loading the data into different binary files to then be stored in the various fields of the struct but that doesn't really seem to be any more efficient than using fread().

https://github.com/DylanMuir/MappedTensor/issues/16

Regards,

Mithul

Answer 1

[weikang zhao]

The “fopen” function may be more flexible. It allows you to access the data in the binary file in any way, of course, including any data type and big-endian. The disadvantage is that it cannot be implemented with a simple function, and a script needs to be written.

[MPan]

Hi Weikang,

That was also a function I used during my first attempt; however, reading in binary files and assigning them to my custom struct one field via a "for-loop" was very time consuming. I also have several different structs that have variations in what order the fields and data types are in so I'm looking to create a generic script that can handle this for me.

Thanks for the suggestion.