neuronPCA

Principal component analysis of neuron activations

Since R2022b

Description

The compressNetworkUsingProjection function uses principal component analysis (PCA) to identify the subspace of learnable parameters that result in the highest variance in neuron activations by analyzing the network activations using a data set representative of the training data. The PCA step can be computationally intensive. If you expect to compress the same network multiple times (for example, when exploring different levels of compression), then you can perform the PCA step up front using a neuronPCA object.

Forward passes of a projected deep neural network are typically faster when you deploy the network to embedded hardware using library-free C or C++ code generation.

If you prune or quantize your network, then use compression using projection after pruning and before quantization. Network compression using projection supports projecting LSTM layers only.

Creation

Syntax

npca = neuronPCA(net,mbq)

npca = neuronPCA(net,X1,...,XN)

npca = neuronPCA(___,LayerNames=names)

npca = neuronPCA(___,VerbosityLevel=level)

Description

npca = neuronPCA(net,mbq) computes the principal component analysis of the neuron activations in net using the data in the mini-batch queue mbq. The software analyzes the input and output activations of the projectable layers in net. This function requires the Deep Learning Toolbox™ Model Quantization Library support package. This support package is a free add-on that you can download using the Add-On Explorer. Alternatively, see Deep Learning Toolbox Model Quantization Library.

npca = neuronPCA(net,X1,...,XN) computes the principal component analysis of the neuron activations using the data in the dlarray objects X1,...,XN, where N is the number of network inputs.

npca = neuronPCA(___,LayerNames=names) specifies which layers to analyze and sets the LayerNames property.

npca = neuronPCA(___,VerbosityLevel=level) specifies the amount of information displayed in the command window.

Input Arguments

net — Neural network

initialized dlnetwork object

Neural network, specified as an initialized dlnetwork object.

mbq — Mini-batch queue

minibatchqueue object

Mini-batch queue that outputs data for each input of the network, specified as a minibatchqueue object.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Do not pad sequence as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

X1,...,XN — Input data

formatted dlarray

Input data, specified as a formatted dlarray.

For more information about dlarray formats, see the fmt input argument of dlarray.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Do not pad sequence as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

level — Verbosity level

"summary" (default) | "steps" | "iterations" | "off"

Verbosity level, specified as one of these values:

• "summary" — Display a summary of the neuron PCA algorithm.

• "steps" — Display information about the steps of the neuron PCA algorithm.

• "iterations" — Display information about the iterations of the neuron PCA algorithm.

• "off" — Do not display information.

Properties

LayerNames — Names of layers to analyze

string array | cell array of character vectors | character vector

Names of layers to analyze, specified as a string array, cell array of character vectors, or a character vector containing a single layer name.

The software, by default, analyzes all the layers in the network that support projection. Network compression using projection supports projecting LSTM layers only.

Data Types: string | cell

InputEigenvalues — Eigenvalues of activations input to projectable layers

cell array

Eigenvalues of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvalues for the corresponding projectable layer in LayerNames.

Data Types: cell

InputEigenvectors — Eigenvectors of activations input to projectable layers

cell array

Eigenvectors of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvectors for the corresponding projectable layer in LayerNames.

Data Types: cell

OutputEigenvalues — Eigenvalues of activations output by projectable layers

cell array

Eigenvalues of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvalues for the corresponding projectable layer in LayerNames.

Data Types: cell

OutputEigenvectors — Eigenvectors of activations output by projectable layers

cell array

Eigenvectors of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvectors for the corresponding projectable layer in LayerNames.

Data Types: cell

Examples

Analyze Neuron Activations for Compression Using Projection

Load the pretrained network in dlnetJapaneseVowels and the training data.

load dlnetJapaneseVowels
XTrain = japaneseVowelsTrainData;

Create a mini-batch queue containing the training data. To create a mini-batch queue from in-memory data, convert the sequences to an array datastore.

adsXTrain = arrayDatastore(XTrain,OutputType="same");

Create the minibatchqueue object.

• Specify a mini-batch size of 16.

• Preprocess the mini-batches using the preprocessMiniBatchPredictors function, listed in the Mini-Batch Predictors Preprocessing Function section of the example.

• Specify that the output data has format "CTB" (channel, time, batch).

miniBatchSize = 16;

mbqTrain = minibatchqueue(adsXTrain, ...
    MiniBatchSize=miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatchPredictors, ...
    MiniBatchFormat="CTB");

Create the neuronPCA object.

npca = neuronPCA(net,mbqTrain);

neuronPCA analyzed 1 layers: "lstm"

View the properties of the neuronPCA object.

npca

npca = 
  neuronPCA with properties:

            LayerNames: "lstm"
      InputEigenvalues: {[12×1 double]}
     InputEigenvectors: {[12×12 double]}
     OutputEigenvalues: {[100×1 double]}
    OutputEigenvectors: {[100×100 double]}

Mini-Batch Predictors Preprocessing Function

The preprocessMiniBatchPredictors function preprocesses a mini-batch of predictors by extracting the sequence data from the input cell array and truncating them along the second dimension so that they have the same length.

Note: Do not pad sequence data when doing the PCA step for projection as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

function X = preprocessMiniBatchPredictors(dataX)

X = padsequences(dataX,2,Length="shortest");

end

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

Usage notes and limitations:

• This function runs on the GPU if any of these conditions are met:

  • Any of the values of the network learnable parameters inside net.Learnables.Value are dlarray objects with underlying data of type gpuArray

  • The input argument mbq outputs dlarray objects with underlying data of type gpuArray

  • The input arguments X1,...,XN are dlarray objects with underlying data of type gpuArray

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2022b