dlquantizer

Quantize a deep neural network to 8-bit scaled integer data types

Description

Use the dlquantizer object to reduce the memory requirement of a deep neural network by quantizing weights, biases, and activations to 8-bit scaled integer data types.

Creation

Description

example

quantObj = dlquantizer(net) creates a dlquantizer object for the specified network.

Quantization of a neural network requires a GPU, the GPU Coder™ Interface for Deep Learning Libraries support package, and the Deep Learning Toolbox™ Model Quantization Library support package. If this support package is not installed, then the function provides a download link. Using a GPU requires a CUDA® enabled NVIDIA® GPU with compute capability 6.1 or higher.

Input Arguments

expand all

Pretrained neural network, specified as a DAGNetwork, SeriesNetwork, or a yolov2ObjectDetector object.

Properties

expand all

Pretrained neural network, specified as a DAGNetwork, SeriesNetwork, or a yolov2ObjectDetector object.

Object Functions

calibrateSimulate and collect ranges of a deep neural network
validateQuantize and validate a deep neural network

Examples

collapse all

This example shows how to quantize learnable parameters in the convolution layers of a neural network, and explore the behavior of the quantized network. In this example, you quantize the squeezenet neural network after retraining the network to classify new images according to the Train Deep Learning Network to Classify New Images example. In this example, the memory required for the network is reduced approximately 75% through quantization while the accuracy of the network is not affected.

Load the pretrained network.

net
net = 

  DAGNetwork with properties:

         Layers: [68x1 nnet.cnn.layer.Layer]
    Connections: [75x2 table]
     InputNames: {'data'}
    OutputNames: {'new_classoutput'}

Define calibration and validation data to use for quantization.

The calibration data is used to collect the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. For the best quantization results, the calibration data must be representative of inputs to the network.

The validation data is used to test the network after quantization to understand the effects of the limited range and precision of the quantized convolution layers in the network.

In this example, use the images in the MerchData data set. Define an augmentedImageDatastore object to resize the data for the network. Then, split the data into calibration and validation data sets.

unzip('MerchData.zip');
imds = imageDatastore('MerchData', ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');
[calData, valData] = splitEachLabel(imds, 0.7, 'randomized');
aug_calData = augmentedImageDatastore([227 227], calData);
aug_valData = augmentedImageDatastore([227 227], valData);

Create a dlquantizer object and specify the network to quantize.

quantObj = dlquantizer(net);

Define a metric function to use to compare the behavior of the network before and after quantization. Save this function in a local file.

function accuracy = hComputeModelAccuracy(predictionScores, net, dataStore)
%% Computes model-level accuracy statistics
    
    % Load ground truth
    tmp = readall(dataStore);
    groundTruth = tmp.response;
    
    % Compare with predicted label with actual ground truth 
    predictionError = {};
    for idx=1:numel(groundTruth)
        [~, idy] = max(predictionScores(idx,:)); 
        yActual = net.Layers(end).Classes(idy);
        predictionError{end+1} = (yActual == groundTruth(idx)); %#ok
    end
    
    % Sum all prediction errors.
    predictionError = [predictionError{:}];
    accuracy = sum(predictionError)/numel(predictionError);
end

Specify the metric function in a dlquantizationOptions object.

quantOpts = dlquantizationOptions('MetricFcn', ...
    {@(x)hComputeModelAccuracy(x, net, aug_valData)});

Use the calibrate function to exercise the network with sample inputs and collect range information. The calibrate function exercises the network and collects the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. The function returns a table. Each row of the table contains range information for a learnable parameter of the optimized network.

calResults = calibrate(quantObj, aug_calData)
calResults =

  95x5 table

                   Optimized Layer Name                      Network Layer Name        Learnables / Activations     MinValue      MaxValue  
    __________________________________________________    _________________________    ________________________    __________    ___________

    {'conv1_relu_conv1_Weights'                      }    {'relu_conv1'           }         "Weights"                -0.91985        0.88489
    {'conv1_relu_conv1_Bias'                         }    {'relu_conv1'           }         "Bias"                   -0.07925        0.26343
    {'fire2-squeeze1x1_fire2-relu_squeeze1x1_Weights'}    {'fire2-relu_squeeze1x1'}         "Weights"                   -1.38         1.2477
    {'fire2-squeeze1x1_fire2-relu_squeeze1x1_Bias'   }    {'fire2-relu_squeeze1x1'}         "Bias"                   -0.11641        0.24273
    {'fire2-expand1x1_fire2-relu_expand1x1_Weights'  }    {'fire2-relu_expand1x1' }         "Weights"                 -0.7406        0.90982
    {'fire2-expand1x1_fire2-relu_expand1x1_Bias'     }    {'fire2-relu_expand1x1' }         "Bias"                  -0.060056        0.14602
    {'fire2-expand3x3_fire2-relu_expand3x3_Weights'  }    {'fire2-relu_expand3x3' }         "Weights"                -0.74397        0.66905
    {'fire2-expand3x3_fire2-relu_expand3x3_Bias'     }    {'fire2-relu_expand3x3' }         "Bias"                  -0.051778       0.074239
    {'fire3-squeeze1x1_fire3-relu_squeeze1x1_Weights'}    {'fire3-relu_squeeze1x1'}         "Weights"                -0.77263        0.68897
    {'fire3-squeeze1x1_fire3-relu_squeeze1x1_Bias'   }    {'fire3-relu_squeeze1x1'}         "Bias"                   -0.10141        0.32678
    {'fire3-expand1x1_fire3-relu_expand1x1_Weights'  }    {'fire3-relu_expand1x1' }         "Weights"                -0.72131        0.97287
    {'fire3-expand1x1_fire3-relu_expand1x1_Bias'     }    {'fire3-relu_expand1x1' }         "Bias"                  -0.067043        0.30424
    {'fire3-expand3x3_fire3-relu_expand3x3_Weights'  }    {'fire3-relu_expand3x3' }         "Weights"                -0.61196        0.77431
    {'fire3-expand3x3_fire3-relu_expand3x3_Bias'     }    {'fire3-relu_expand3x3' }         "Bias"                  -0.053612        0.10329
    {'fire4-squeeze1x1_fire4-relu_squeeze1x1_Weights'}    {'fire4-relu_squeeze1x1'}         "Weights"                -0.74145         1.0888
    {'fire4-squeeze1x1_fire4-relu_squeeze1x1_Bias'   }    {'fire4-relu_squeeze1x1'}         "Bias"                   -0.10886        0.13882
...

Use the validate function to quantize the learnable parameters in the convolution layers of the network and exercise the network. The function uses the metric function defined in the dlquantizationOptions object to compare the results of the network before and after quantization.

valResults = validate(quantObj, aug_valData, quantOpts)
valResults = 

  struct with fields:

       NumSamples: 20
    MetricResults: [1x1 struct]

Examine the MetricResults.Result field of the validation output to see the performance of the quantized network.

valResults.MetricResults.Result
ans =

  2x3 table

    NetworkImplementation    MetricOutput    LearnableParameterMemory(bytes)
    _____________________    ____________    _______________________________

     {'Floating-Point'}           1                    2.9003e+06           
     {'Quantized'     }           1                    7.3393e+05           

In this example, the memory required for the network was reduced approximately 75% through quantization. The accuracy of the network is not affected.

The weights, biases, and activations of the convolution layers of the network specified in the dlquantizer object now use scaled 8-bit integer data types.

Introduced in R2020a