Main Content

Experiment Manager

Design and run experiments to train and compare machine learning models

Since R2023a

Description

You can use the Experiment Manager app to create machine learning experiments to train models under multiple initial conditions and compare the results. For example, you can use Experiment Manager to:

  • Try a range of hyperparameter values using Bayesian optimization.

  • Compare the results of using different data sets, preprocessing steps, or metrics.

To set up your experiment quickly, start by training a model in Classification Learner or Regression Learner. Then, export the model to Experiment Manager.

Experiment Manager provides visualizations, filters, and annotations to help you manage your experiment results and record your observations. To improve reproducibility, Experiment Manager stores a copy of the experiment definition every time that you run an experiment. You can access past experiment definitions to keep track of the combinations of hyperparameters that produce each of your results.

Experiment Manager organizes your experiments and results in projects.

  • You can store several experiments in the same project.

  • Each experiment contains a set of results for each time that you run the experiment.

  • Each set of results consists of one or more trials that correspond to a different combination of hyperparameters.

The Experiment Browser pane displays the hierarchy of experiments and results in the project. For example, this project has three experiments, each of which has several sets of results.

Experiment Browser showing three experiments. Experiment1 is a general-purpose example with three results. Experiment 2 is a built-in training experiment for deep learning with two results. Experiment3 is a custom training experiment for deep learning or machine learning with two results.

The orange round-bottom flask indicates a general-purpose experiment that you can run in MATLAB® without a Statistics and Machine Learning Toolbox™ license. The blue Erlenmeyer flask indicates a built-in training experiment for deep learning. The green beaker indicates a custom training experiment for deep learning or machine learning. For more information about general-purpose experiments, see Manage Experiments (MATLAB). For more information about experiments for deep learning workflows, see Manage Experiments (Deep Learning Toolbox).

By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox™, you can run multiple trials at the same time or run a single trial on multiple GPUs, on a cluster, or in the cloud. If you have MATLAB Parallel Server™, you can also offload experiments as batch jobs in a remote cluster so that you can continue working or close your MATLAB session while your experiment runs. For more information, see Run Experiments in Parallel and Offload Experiments as Batch Jobs to a Cluster.

Required Products

  • Deep Learning Toolbox™ to run built-in or custom training experiments for deep learning and to view confusion matrices for these experiments

  • Statistics and Machine Learning Toolbox to run custom training experiments for machine learning and experiments that use Bayesian optimization

  • Parallel Computing Toolbox to run multiple trials at the same time or a single trial at a time on multiple GPUs, on a cluster, or in the cloud

  • MATLAB Parallel Server to offload experiments as batch jobs in a remote cluster

Experiment Manager app

Open the Experiment Manager App

  • MATLAB Toolstrip: On the Apps tab, under MATLAB, click the Experiment Manager icon (since R2023b).

  • MATLAB command prompt: Enter experimentManager.

Tips

  • To navigate Experiment Manager when using a mouse is not an option, use shortcut keyboards. For more information, see Keyboard Shortcuts for Experiment Manager.

  • To reduce the size of your experiments, discard the results and visualizations of any trial that is no longer relevant. In the Actions column of the results table, click the Discard button for the trial.

Version History

Introduced in R2023a

expand all