Saving output values into a and archive .mat file which grows with time

I am currently writing a lab applicaiton that calculates certain values by analyzing IR spectra. I would like to store the numerical results in an archive file which both contains the calculated values and some text related to each sample (the spectra themselves, which usually contains largers numbers of data, are optional). This should happen every time I finish my analysis and end the session, and the archive file is supposed to grow gradually with time. Now, here is a general question: which file format is the best format for saving the numerical and string output? Is .mat a suitable format?

Answer 1

[Walter Roberson]

You might want to consider using separate variables, and using a -v7.3 .mat using the matfile facilities, which can allow you to grow arrays.

With matfile you could, yes, also use something like a nonscalar struct array, which might perhaps make more sense from how you are thinking about your data. However, the challenge with that is that when you use -v7.3 files, that homogenous data structures (mixed types) have a lot more overhead than is the case for -v7.0 files and so can be much slower.

There are alternatives. For example with a traditional -v7 .mat file, you can save -append to append new variables to a file -- which would require that you use a different variable name each time (-append is not for appending data to an existing variable.)

[Saeid]

The archive will not be colossal, but sizeable enought so that the speed of accessing the data might become a minor issue in the long term. Plus, I still need to get used to understanding all the different variations of .mat files for archiving of data. This may take some time, but your answer was very helpful, in that at least I know this is the right format for the purpose. Thanks a lot!

Cheers,

Saeid

[Walter Roberson]

If you were doing a lot of querying then using a Database might potentially be of value.

Beyond that:

There is a difference in efficiency between write-once systems and allow-update systems, at least when it comes to text (or changing the number of datapoints, or anything that changes the size of an entry in bytes). When updates are permitted and are not exactly the same number of bytes as the original, then you have to either have free-space management (and possibly garbage collection) or else you have to rewrite the file (at least from that point onwards.)

.mat files already have some level of free-space management -- though for -v7 files it tends to lose access to space (append new stuff at the end but not great on filling in the holes left by data no longer in use.)