Work with Big Data for Simulations

Simulation of models with many time steps and signals can involve big data that is too large to fit into the RAM of your computer. Such situations include:

• Logging simulation data (signal logging, output port logging, and state logging)

• Loading input signal data for simulating a model

• Running multiple or parallel simulations

To work with big data for simulations, store the data to persistent storage in a MAT file. Using big data techniques for simulations requires additional steps beyond what you do when the data is small enough to fit in workspace memory. As you develop a model, consider logging and loading simulation data without using persistent storage unless you discover that your model has big data requirements that overload memory.

Big Data Workflow

This example is a high-level workflow for handling big data that one simulation produces and that another simulation uses as input. For more detailed information about the major workflow tasks, see:

• Log Data to Persistent Storage

• Load Big Data for Simulations

• Analyze Big Data from a Simulation

Tip

This example uses a SimulationDatastore object for streaming data into a model. Alternatively, you can stream a DatasetRef object directly into a model.

1. Configure two models to log several signals.

2. Simulate the models, logging the data to persistent storage for each model.

   sim(mdl1,'LoggingToFile','on','LoggingFileName','data1.mat');
   sim(mdl2,'LoggingToFile','on','LoggingFileName','data2.mat');

   Logging that involves big data requires saving the data to persistent storage as a v7.3 MAT file. Only the data logged in Dataset format is saved to the file. Data logged in other formats, such as Structure with time, is saved in memory, in the base workspace.

   The data that you log to persistent storage is streamed during the simulation in small chunks, to minimize memory requirements. The data is stored in a file that contains Dataset objects for each set of logged data (for example, logsout and xout).

3. Create DatasetRef objects (dsr1 and dsr2) for specific sets of logged signals. Then create SimulationDatastore objects (dst1 and dst2) for values of elements of the DatasetRef objects. This example code creates a SimulationDatastore for the 12th element of logsout for the first simulation. For the second simulation, the example code creates a signal with values being a SimulationDatastore object for the seventh element of logsout. You can use curly braces for indexing.

   dsr1 = Simulink.SimulationData.DatasetRef('data1.mat','logsout');
   dsr2 = Simulink.SimulationData.DatasetRef('data2.mat','logsout');
   dst1 = dsr1{12};
   dst2 = dsr2{7};

4. Use SimulationDatastore objects as an external input for another simulation. To load the SimulationDatastore data, include it in a Dataset object. The datastore input is incrementally loaded from the MAT file. The third input is a timeseries object, which is loaded into memory as a whole, not incrementally.

   input = Simulink.SimulationData.Dataset;
   input{1} = dst1; 
   input{2} = dst2;
   ts = timeseries(rand(5,1),1,'Name','RandomSignals');
   input{3} = ts;
   sim(mdl3,'ExternalInput','input');

5. Use MATLAB^® big data analysis to work with the SimulationDatastore objects. Create a timetable object by reading the values of a SimulationDatastore object. The read function reads a portion of the data. The readall function reads all the data.

   tt = dst1.Values.read;

6. Set the MATLAB session as the global execution environment (mapreducer) for working with the tall timetable. Create a tall timetable from a SimulationDatastore object and read a timetable object with in-memory data.

   mapreducer(0);
   ttt = tall(dst1.Values);
   

Tip

For another example showing how to work with big simulation data, see Working with Big Data.