Load Big Data for Simulations

Simulating models with many time steps and signals can use and create data that is too large to fit into working memory on your computer. When your simulation input data does not fit into memory, you can choose one of several strategies to use that data as simulation input with root-level Inport blocks. These strategies work for loading data stored in Dataset format in a Version 7.3 MAT-file, including data logged from another simulation.

  1. When individual input signals are too large to fit into memory, you can use a matlab.io.datastore.SimulationDatastore object to access the signal data. The data from the SimulationDatastore object loads into the simulation incrementally in chunks that fit into memory.

  2. When your simulation inputs are specified by a Simulink.SimulationData.Dataset object in a file that is too large to load into memory, you can stream the entire contents of the Dataset object into your model using a Simulink.SimulationData.DatasetRef object.

  3. When the signals fit into memory and are stored in a file that is too large to load into memory, you can load individual signals from the file into memory using a Simulink.SimulationData.DatasetRef object.

All big data loading strategies are for the special case when your data does not fit into memory and can require extra steps. These examples use data that fits fully into memory to illustrate the steps required for big data loading. When your simulation inputs fit into memory, consider using other loading techniques.

Stream Individual Signals Using SimulationDatastore Objects

When individual signals in your input data are too large to fit into memory, you can create matlab.io.datastore.SimulationDatastore objects for those signals and stream them into your model. To create a SimulationDatastore object for a signal you want to stream into your model, first create a Simulink.SimulationData.DatasetRef object to reference the Dataset object that contains your signal of interest. For example, create a DatasetRef for logged data from a simulation of the slexAircraftExample model.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can create a SimulationDatastore object for your desired signal by indexing into the DatasetRef object with curly braces or using the getAsDatastore method for the DatasetRef object. In both cases, the SimulationDatastore object exists in the Values property of the returned Simulink.SimulationData.Signal object.

When you know the index of the signal within the Dataset object, you can index into the DatasetRef object with curly braces to create a SimulationDatastore for your signal.

alphaRad_ds = logsout_DSR{4}
alphaRad_ds = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'alpha, rad'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 4
            Values: [1x1 matlab.io.datastore.SimulationDatastore]


  Methods, Superclasses

To create a SimulationDatastore object for a signal using the signal index, name, or block path, use the getAsDatastore method. For example, create a SimulationDatastore object for the Stick signal.

stick_ds = logsout_DSR.getAsDatastore('Stick')
stick_ds = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'Stick'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 1
            Values: [1x1 matlab.io.datastore.SimulationDatastore]


  Methods, Superclasses

Because the Values properties of the stick_ds and alphaRad_ds Simulink.SimulationData.Signal objects are SimulationDatastores, the signal data streams into your model. You can include a SimulationDatastore backed Signal object as an element in a Dataset object or as an item in the Input parameter comma-separated list.

Stream an Entire Dataset Using a DatasetRef Object

When your simulation inputs are specified in a Dataset in a file that is too large to load into memory, you can create a Simulink.SimulationData.DatasetRef object to stream your simulation inputs into your model. When you specify a DatasetRef object for the Input parameter on the Data Import/Export pane, all the signals in the Dataset object used to create the DatasetRef stream into your model. Use one of the other big data loading techniques to load or stream individual signals from a Dataset object.

When the file where your simulation input Dataset is stored contains other Datasets and data, you can use the Simulink.SimulationData.DatasetRef.getDatasetVariableNames function to view a list of the Dataset objects contained in the file. Previewing the variable names in the Dataset object is particularly useful when the file contents do not fit into memory.

datasetNames = Simulink.SimulationData.DatasetRef.getDatasetVariableNames('aircraftData.mat')
datasetNames = 1x3 cell array
    {'logsout'}    {'xout'}    {'yout'}

Create a DatasetRef object for logsout.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can load logsout_DSR using the Input parameter the same way you would load a Simulink.SimulationData.Dataset object. Each signal in the Dataset object used to create the DatasetRef streams into the model in chunks that fit into memory.

Load Individual Signals from a DatasetRef Object

When your simulation input signals individually fit into memory and are stored in a Dataset object in a file that does not fit into memory, use a Simulink.SimulationData.DatasetRef object to load each signal of interest into memory. Then, you can load the signals as simulation inputs for your model.

First, create the DatasetRef object to reference the Dataset object in the file that contains the signals you want to load. For example, create a DatasetRef object for data logged to file from a simulation of the slexAircraftExample model.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can use the get or getElement methods to load individual signals into memory with the DatasetRef object. Both methods load the specified element into memory, using the same syntax. You can specify the signal you want to load into memory using its index within the Dataset object or its name. If you don't know the name of the signal you want to load, use the getElementNames method to see the names of the elements in the Dataset object referenced by the DatasetRef object.

elNames = logsout_DSR.getElementNames
elNames = 15x1 cell array
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }
    {'alpha, rad'}
    {'q, rad/sec'}
    {0x0 char    }
    {'qGust'     }
    {'wGust'     }
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }
    {'Stick'     }
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }

Load the qGust signal into memory using its name.

qGust = logsout_DSR.getElement('qGust')
qGust = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'qGust'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 2
            Values: [1x1 timeseries]


  Methods, Superclasses

You can add the qGust signal to a Dataset object of simulation input signals to load to the root-level Inport blocks in your model, or you can specify qGust as an item in the Input parameter comma-separated list.

See Also

| |

Related Topics