Strategies to store/load 'configuration' data

Hi, we often have the need to store/load configuration data, for hundreds of cases. Current practice is to store it in an M-file GET function like:

function out = get_config(caseString)

    out = [];
    switch lower(caseString)
     case 'foo'
      out.name = 'asdf'
      out.machine = 'big_red'
      out.gain = 23;
      out.offset = 0.3;
     case 'bar'
      out.name = 'jklm'
      out.machine = 'eagle'
      out.gain = 19;
      out.offset = -0.1;
  ...

The things that I like about this strategy are:

1. It's in a human-readable, non-binary format.
2. The parameter data is stored in the same file as the GET function.
3. It's fast (only executes the one valid case statement, doesn't have to load a file each time).

Things that I don't particularly like about this strategy:

1. There's a lot of code repetition in typing/copy-pasting the structures
2. Adding a new field to the structure is painful
3. The field names are duplicated in each case, making changing them a 'find/replace & hope' operation (unless you use dynamic field names I suppose)

So far I've played around with strategies of a) using containers.Map and b) storing the data in a separate CSV or XLS file. I haven't yet come up with a way to do a) that helps out much with the 'negatives' of the current strategy. I've considered doing b), which can eliminate many of the current negatives by using a header row, but would require keeping track of an extra file and potentially being slower. Is there any way to embed a 'data grid' into an M-file? That could be the best of both worlds.

Thanks for your thoughts and suggestions! Eric

[Walter Roberson]

I'm not sure what a "data grid" is ?

[Eric Sampson]

Walter, when I said 'data grid' I was thinking about something like the way that the MATLAB Variable Editor displays an array, but having it show up in the Editor inside an M-file inline with the rest of the code. Something like Yair might be able to figure out with his Java magic :)

[Stephane]

Since 2022b, there is a dictionary object that can help https://au.mathworks.com/help/matlab/ref/dictionary.html

From the help file: A dictionary is an object that maps unique keys to values. A dictionary is useful for fast lookup of values in a larger set. A dictionary is a map that stores data as values, which can be accessed using corresponding unique keys. Each pair of keys and values is an entry.

Answer 1

[Walter Roberson]

For each class of similar data, I used a cell array of strings for the field names, and a cell array of data for the contents, and a small bit of boiler-plate code that converted the combination into a struct array.

For program configuration parameters, I used a cell array of data, with the first field being a parameter name, and each of the other fields having a consistent function. I then had routines to fetch and store those rows by parameter name. One of the fields was a type indicator; for numeric values, other fields had min and max values to allow validation, and I also had a field to handle enumerated allowed values. Each of the rows had a post-set callback field as well. Minor auxillary routines were added to validate proposed new values.

[Eric Sampson]

Thanks Walter, but I'm not quite able to picture the scenario you describe. When you have a minute, perhaps you could reply back with how you write the code for my earlier example:

     case 'foo'
      out.name = 'asdf'
      out.machine = 'big_red'
      out.gain = 23;
      out.offset = 0.3;
     case 'bar'
      out.name = 'jklm'
      out.machine = 'eagle'
      out.gain = 19;
      out.offset = -0.1;

[Walter Roberson]

fnames = {'key', 'name', 'machine', 'gain', 'offset'};
content = {'foo', 'asdf', 'big_red', 23, 0.3; ...
           'bar', 'jklm', 'eagle', 19, -0.1; ....
          }
as_struct = cell2struct(content, fnames, 2);

After which you can index as_struct by constant indices or you can

logical_idx = strcmp(TheKey, {as_struct.key});
this_record = as_struct(logical_idx);

[Eric Sampson]

Thanks Walter, I forgot to come back and accept your answer!

[Eric Sampson]

Walter, I think you could also use Maps if you want to avoid using constant indices or constructing adhoc logical indexes:

as_struct = cell2struct(content(:,2:end), fnames(2:end), 2);
as_struct_in_cells = arrayfun(@(x) x, as_struct', 'UniformOutput', false);

mymap = containers.Map(content(:,1), as_struct_in_cells);
mymap('foo').machine  % ans = big_red

Answer 2

[Image Analyst]

I usually have a structure called UserSettings and a GUI for setting up values for members of that structure. For example UserSettings.gain, UserSettings.Exposure, UserSettings.cameraModel, etc. Then I save the settings (configuration) in a .mat file. For multiple configurations I could have an array of such structures to save in a single file, or save the multiple structures in multiple .mat files. The opening code for a program will open the appropriate .mat file, and the program will also save the mat file at appropriate times after changes have been made to the structure. That's how I do it. So it's like your "out" structure except I have a GUI to allow the user to change the settings instead of hard coding it like you do, and I save it in a binary .mat file instead of a text m-file like you do.

[Walter Roberson]

I used setpref() / getpref() to store user program preferences (e.g., font size). I would run the auxillary validation routines (from the configuration table in the source) on the stored preferences; if the preference was missing or not one of the allowed values, then the preference would get updated from the default value configured in the source.

I handled program preferences completely separately from user values associated with a particular run or data-set. For example "last file loaded" and related information I saved to a .mat file, but matters such as which color to associate with "training set" seldom changes and went through the configuration table / setpref / getpref arrangement.

For the program preferences coded in through the configuration table, the configuration table contained enough information about type and allowed values that when the user asked to edit the preference, I could create an appropriate configuration window on the fly. However, the GUI items associated with choosing how to process any particular runs usually had too much variability to use configuration-table type code for them.