How to export the overview table from the Experiment Manager App

I started using the Experiment Manager App to explore a Neural Network performance on a calssification task and how it depends on the hyperparameters of the model.

https://ch.mathworks.com/help/deeplearning/ref/experimentmanager-app.html

The overall design of the App is quite appealing for a first look but the options for exporting the results for further analysis are too limited.

After a hyperparameter sweep the app displays a table of results that displays the accuracy and loss for each trial together with the choosen hyperparameters.

It would be extreemly useful to export this table e.g. to plot validation and test accuracy as function of the hyperparameters to gain more insights. However I could not find such an export option. It is only possible to export the parameters of individual trained Networks. Please let me know if this option exists?

If not, It would be great if such an export option could be added to the app. Or if a function could be provided to extract the results parameters from the results subfolders.

Thanks and Kind Regards

Sebastian Landwehr

Answer 1

[Anshika Chaurasia]

Hi Sebastian,

You can try the following script which will give a table containing validation and training loss/accuracy for all trials:

% sweep intialLearnRate hyperparameter 
myInitialLearnRate = [0.0025 0.0050 0.0075];
trialdetail = cell(numel(myInitialLearnRate),1);
for i = 1:3
    % params struct containing value of hyperparameter
    params = struct('myInitialLearnRate',myInitialLearnRate(i));
    % passing hyperparameter value to experiment setup function
    [imds,layers,options] = Experiment1_setup1(params);
    % train the network 
    [net,info] = trainNetwork(imds,layers,options);
    % storing the Training information such as 
    % Training loss, Training Accuracy, Validation Loss, 
    % Validation Acc for a trial
    trialdetail{i,:} = info;
    
    % storing information for table
    Training_acc(i) = info.TrainingAccuracy(end);
    Training_loss(i) = info.TrainingLoss(end);
    Validation_acc(i) = info.FinalValidationAccuracy;
    Validation_loss(i) = info.FinalValidationLoss;
end
Trial = 1:3;
% Table containing Training informations for all trials
Table = table(Trial(:),Training_acc(:),Training_loss(:),Validation_acc(:),Validation_loss(:),...
    'VariableNames',{'Trial','Training Acc','Training Loss','Validation Acc','Validation Loss'});

function [imds,layers,options] = Experiment1_setup1(params)
% This template function constructs an image data store as a training data set,
% a layer graph, and training options. Outputs of this setup function are used 
% to call the trainNetwork function. 

% Modify this function as a template to set up your own experiment.
digitDatasetPath = fullfile(matlabroot,'toolbox','nnet', ...
    'nndemos','nndatasets','DigitDataset');
imds = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');

numTrainingFiles = 750;
[imds,imdsValidation] = splitEachLabel(imds,numTrainingFiles);
inputSize = [28 28 1];
numClasses = 10;
layers = [
    imageInputLayer(inputSize)
    convolution2dLayer(5,20)
    batchNormalizationLayer
    reluLayer
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];
options = trainingOptions('sgdm', ...
    'MaxEpochs',5, ... 
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'InitialLearnRate',params.myInitialLearnRate);
end

Refer to this documentation for more information on Experiment1_setup1 function.

I have brought your idea of export option to the notice of our developers. They may investigate the matter further.

[Sebastian Johannes Heinz-Josef Landwehr]

Hi Anshika,

thanks for sharing this simple working example. I see now how the info can be extracted.

Thanks for forwarding the matter to the developers. I am sure this would be a valuable addition.

Cheers

Sebastian

[Vishnu Keyen]

Hi Anshika,

Thanks for the code. I have understood how the info is being extracted from trainNetwork()

I am doing a similar experiment, However, I am using a custom metric for my evaluation. I have specified it in as a seperate mlx file under its section in experiment manager.

can u help me extract that metric (which i have specified) in the above code?

I understand that, since I can get the trained network, I can calculate my custom metric to find it out.

But I am asking, is there a simpler way like what you have shared here.

Thanks in advance

[Payman Goodarzi]

Although is no export function for an experiment table, it is possible to extract its variables from the Results folder, the structure is understandable.

My problem is that I can not find the "custom metric", where is it stored?

[Vishnu Keyen]

I am sorry for not posting the code with it.

The custom metric is specified as a function in the experiment manager.