You want to cast your data to right type. Default numeric type in MATLAB in double. You may want to use uint8() on your fread() calls so that you only store byte-sized chunks into your variable.
If you do this consistently you should get a 8x reduction in the data size.
Also as others have commented, it is always good to share your code to provide context to your problem!