Experiment Manager

Design and run experiments to train and compare machine learning models

Since R2023a

Description

You can use the Experiment Manager app to create machine learning experiments to train models under multiple initial conditions and compare the results. For example, you can use Experiment Manager to:

Try a range of hyperparameter values using Bayesian optimization.
Compare the results of using different data sets, preprocessing steps, or metrics.

To set up your experiment quickly, start by training a model in Classification Learner or Regression Learner. Then, export the model to Experiment Manager.

Experiment Manager provides visualizations, filters, and annotations to help you manage your experiment results and record your observations. To improve reproducibility, Experiment Manager stores a copy of the experiment definition every time that you run an experiment. You can access past experiment definitions to keep track of the combinations of hyperparameters that produce each of your results.

Experiment Manager organizes your experiments and results in projects.

You can store several experiments in the same project.
Each experiment contains a set of results for each time that you run the experiment.
Each set of results consists of one or more trials that correspond to a different combination of hyperparameters.

The Experiment Browser pane displays the hierarchy of experiments and results in the project. For example, this project has three experiments, each of which has several sets of results.

The orange round-bottom flask indicates a general-purpose experiment that you can run in MATLAB^® without a Statistics and Machine Learning Toolbox™ license. The blue Erlenmeyer flask indicates a built-in training experiment for deep learning. The green beaker indicates a custom training experiment for deep learning or machine learning. For more information about general-purpose experiments, see Manage Experiments (MATLAB). For more information about experiments for deep learning workflows, see Manage Experiments (Deep Learning Toolbox).

By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox™, you can run multiple trials at the same time or run a single trial on multiple GPUs, on a cluster, or in the cloud. If you have MATLAB Parallel Server™, you can also offload experiments as batch jobs in a remote cluster so that you can continue working or close your MATLAB session while your experiment runs. For more information, see Run Experiments in Parallel and Offload Experiments as Batch Jobs to a Cluster.

Required Products

Deep Learning Toolbox™ to run built-in or custom training experiments for deep learning and to view confusion matrices for these experiments
Statistics and Machine Learning Toolbox to run custom training experiments for machine learning and experiments that use Bayesian optimization
Parallel Computing Toolbox to run multiple trials at the same time or a single trial at a time on multiple GPUs, on a cluster, or in the cloud
MATLAB Parallel Server to offload experiments as batch jobs in a remote cluster

Open the Experiment Manager App

MATLAB Toolstrip: On the Apps tab, under MATLAB, click the Experiment Manager icon (since R2023b).
MATLAB command prompt: Enter experimentManager.

Examples

Tips

To navigate Experiment Manager when using a mouse is not an option, use shortcut keyboards. For more information, see Keyboard Shortcuts for Experiment Manager.
To reduce the size of your experiments, discard the results and visualizations of any trial that is no longer relevant. In the Actions column of the results table, click the Discard button for the trial.

Version History

Introduced in R2023a

expand all

R2023b: App available in MATLAB

You can now use Experiment Manager in MATLAB, with or without Statistics and Machine Learning Toolbox. When you share your experiments with colleagues who do not have a Statistics and Machine Learning Toolbox license, they can open your experiments and access your results. Experiment Manager requires:

Deep Learning Toolbox to run built-in or custom training experiments for deep learning and to view confusion matrices for these experiments
Statistics and Machine Learning Toolbox to run custom training experiments for machine learning and experiments that use Bayesian optimization
Parallel Computing Toolbox to run multiple trials at the same time or a single trial at a time on multiple GPUs, on a cluster, or in the cloud
MATLAB Parallel Server to offload experiments as batch jobs in a remote cluster

For more information on general-purpose experiments that you can run in MATLAB, see Manage Experiments.

R2023b: Delete multiple experiments and results

Use the Experiment Browser to delete multiple experiments or multiple results from a project in a single operation. Select the experiments or results you want to delete, then right-click and select Delete. Your selection must contain only experiments or only results. If you delete an experiment, Experiment Manager also deletes the results contained in the experiment.

R2023a: App available in Statistics and Machine Learning Toolbox

You can now use Experiment Manager when you have a Statistics and Machine Learning Toolbox license. In previous releases, you needed a Deep Learning Toolbox license to use the Experiment Manager app. For more information on how to use the app for deep learning workflows, see Manage Experiments (Deep Learning Toolbox).