1D-Convolution Layer not supported by calibrate function

Good morning,

I am trying to follow this example: https://it.mathworks.com/help/coder/ug/generate-code-for-quantized-lstm-network-and-deploy-on-cortex-m-target.html on how to generate an Int8 Code for an implementation in a STM32.

My network is composed by the following layers:

6×1 Layer array with layers:

1 'input' Sequence Input Sequence input with 1 dimensions

2 'conv1' 1-D Convolution 10 8×1 convolutions with stride 1 and padding 'same'

3 'batchnorm1' Batch Normalization Batch normalization with 10 channels

4 'relu1' ReLU ReLU

5 'gru1' Projected Layer Projected GRU with 32 hidden units

6 'output' Projected Layer Projected fully connected layer with output size 1

When I try to calibrate the network as described in the example, I have the following error showing that the 1D-convolutional layer is not supported in the CPU environment: "Code generation for conv1 is not supported for target library 'mkldnn'. See documentation for a list of supported layers with each target library."

Can I solve this problem without having to change the 1D-convolutional layer?

Thank you in advance,

Silvia

[Jayanti]

Hi @Silvia

As per my understanding you are using custom network instead of the model used in example. If so,can you provide the code of your custom network, so that I can reproduce the issue?

[Silvia]

Hello @Jayanti, thank you for your help.

Yes, I am trying to adapt the example to my situation, in which I have a custom network.

I cannot provide you the data set, but I can still show you the pipeline I followed. The initial trained network was then compressed with the projection technique.

%% Load Training - Validation Set

load trainingset.mat trainingset;
datasetSize = size(trainingset,2);
trainsetSize = 0.9 * datasetSize; % leaving 10% for Validation

folder      = "---";

% Folder del dataset
dataDir = string(folder + "\dataset");

% For Training
if ~exist(fullfile(dataDir,"train"),'dir')
    mkdir(fullfile(dataDir,"train"))
    cnt = 0;
    % Training set generation
    for idx = 1:trainsetSize
        cnt = cnt + 1;
        cleanSignal = trainingset(idx).cleansignal;   % it has to be in the form: number of samples x 1 !
        noisySignal = trainingset(idx).noisysignal;
    
        save(fullfile(dataDir,"train", ...
                "data_" + num2str(cnt) + ".mat"), ...
                "cleanSignal","noisySignal");
    end
    % Prepare Datastores to Consume Data
    ds_Train = signalDatastore(fullfile(dataDir,"train"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
else
    % Directly prepare Datastores to Consume Data (without training
    % generation)
    ds_Train = signalDatastore(fullfile(dataDir,"train"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
end

% For Validation
cnt = 0;
if ~exist(fullfile(dataDir,"validate"),'dir')
    mkdir(fullfile(dataDir,"validate"))
    % Validation set generation
    for idx = trainsetSize+1 : datasetSize
        cnt = cnt + 1;
        cleanSignal = trainingset(idx).cleansignal;
        noisySignal = trainingset(idx).noisysignal;
    
        save(fullfile(dataDir,"validate", ...
                "data_" + num2str(cnt) + ".mat"), ...
                "cleanSignal","noisySignal");
    end
    % Prepare Datastores to Consume Data
    ds_Validate = signalDatastore(fullfile(dataDir,"validate"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
else
    % Directly prepare Datastores to Consume Data (without valisation set
    % generation)
    ds_Validate = signalDatastore(fullfile(dataDir,"validate"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
end

data = preview(ds_Train);

%% Define the Neural Network Architecture

numHiddenUnits = 32;

% Hybrid Approach ()

layers = [
    sequenceInputLayer(1, 'Name', 'input')
    convolution1dLayer(8, 10, 'Padding', 'same', 'Name', 'conv1')
    batchNormalizationLayer('Name', 'batchnorm1')
    reluLayer('Name', 'relu1')
    gruLayer(numHiddenUnits, 'OutputMode', 'sequence', 'Name', 'gru1')
    fullyConnectedLayer(1, 'Name', 'output')
];
  
%% Specify Training Options

miniBatchSize = 64;

options = trainingOptions('adam', ...            % Use the Adam optimizer
    'MaxEpochs', 30, ...                         % Max number of epochs
    'InitialLearnRate', 1e-4, ...                % Initial learning rate
    'MiniBatchSize', miniBatchSize, ...          % Size of the batches
    'Shuffle', 'every-epoch', ...                % Shuffle the data every epoch
    'ValidationData', ds_Validate, ...           % Validation data
    'Plots', 'training-progress', ...            % Plot training progress
    'Verbose', false, ...                        % Suppress verbose output 
    'ValidationFrequency', floor(trainsetSize/miniBatchSize), ...
    'Metrics', 'rmse');                          % Evaluate using 'rmse'

%% Train the Network

% Ask the user to select an option
userChoice = input('Enter "Train" to train the model or "Download" to download pre-trained model: ', 's');

% Define network name or load network
if userChoice == "Train"
    % Ask the user for the new network name
    netName = input('Enter the name to save the trained network as (e.g., "LSTM01"): ', 's');
    netFile = [netName, '.mat'];
elseif userChoice == "Download"
    % Open file selection dialog to choose the network file
    [fileName, filePath] = uigetfile('*.mat', 'Select the pre-trained network file');
    if isequal(fileName, 0)
        disp('No file selected. Exiting...');
        return
    else
        netFile = fullfile(filePath, fileName);
    end
else
    error('Invalid choice. Please enter either "Train" or "Download".');
end

% If train is selected, train the network
if userChoice == "Train"
    % Train the model and save it with the specified name
    disp("Training the network...");
    trainedNet = trainnet(ds_Train, layers, 'mse', options);
    save(netFile, 'trainedNet');
elseif userChoice == "Download"
    % Load the specified pre-trained network
    if isfile(netFile)
        disp("Loading the pre-trained network...");
        load(netFile, 'trainedNet');
    else
        error(['The specified network file "' netFile '" does not exist.']);
    end
else
    error('Invalid choice. Please enter either "Train" or "Download".');
end


netOriginal = trainedNet;


%% Compressed network via PCA (reduction of 55%)
% Create a mini-batch queue object 
numSignals = length(trainingset);
XTrain = cell(numSignals, 1);

for i = 1:numSignals
    noisySignal = trainingset(i).noisysignal';
    XTrain{i} = noisySignal;
end

% Input must be a datastore
adsXTrain = arrayDatastore(XTrain, 'OutputType', 'same');

miniBatchSize = 16;

mbqTrain = minibatchqueue(adsXTrain, ...
    MiniBatchSize=miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatchPredictors, ...
    MiniBatchFormat="CTB");

% Create the neuronPCA object
npca = neuronPCA(netOriginal,mbqTrain,VerbosityLevel="steps");
npca

LearnablesReductionValue = 0.55;

% Thus we will set the LearnablesReduction to 55%
netProjected = compressNetworkUsingProjection(netOriginal,npca,...
    LearnablesReduction=LearnablesReductionValue,...
    VerbosityLevel="off");

%% Preparing data for quantization
load calibrationset.mat calibrationset;

calibrationsetSize = size(calibrationset,2);

if ~exist(fullfile(dataDir,"calibration"),'dir')
    mkdir(fullfile(dataDir,"calibration"))
    cnt = 0;
    % Calibration set generation
    for idx = 1:calibrationsetSize
        cnt = cnt + 1;
        cleanSignal = calibrationset(idx).cleansignal;
        noisySignal = calibrationset(idx).noisysignal;
    
        save(fullfile(dataDir,"calibration", ...
                "data_" + num2str(cnt) + ".mat"), ...
                "cleanSignal","noisySignal");
    end
    % Prepare Datastores to Consume Data
    ds_Cal = signalDatastore(fullfile(dataDir,"calibration"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
else
    % Directly prepare Datastores to Consume Data (without training
    % generation)
    ds_Cal = signalDatastore(fullfile(dataDir,"calibration"), ...
    SignalVariableNames=["noisySignal","cleanSignal"], ...
    ReadOutputOrientation="row");
end

% Adapt Calibration Set to the mini-batch queue specification

numSignals = length(calibrationset);
XCal = cell(numSignals, 1);

for i = 1:numSignals
    noisySignal = calibrationset(i).noisysignal';
    XCal{i} = noisySignal;
end

%disp(size(XCal));       % Should be [1000, 1]
%disp(size(XCal{1}));    % Should be [1, Nsamples]

% Input must be a datastore
adsXCal = arrayDatastore(XCal, 'OutputType', 'same');

mbqCal = minibatchqueue(adsXCal, ...
    MiniBatchSize=miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatchPredictors, ...
    MiniBatchFormat="CTB");

% Quantization
quantObj = dlquantizer(netProjected, 'ExecutionEnvironment', 'CPU');
quantObj.calibrate(adsXCal)

[Hariprasad Ravishankar]

Hi @Silvia,

The Deep Learning Toolbox does not support quantizing convolution1dLayer or generating code for quantized convolution1dLayer as of R2024b.

However, you can still deploy your unquantized (single-precision compute) network to Cortex-M by generating plain C code and customizing it using the Cortex-M Code replacement library option.

You can specify 'None' as the TargetLibrary when creating a DeepLearningConfig to specify no 3p deep learning libraries (Generate plain C) as follows:

cfg.DeepLearningConfig = coder.DeepLearningConfig('TargetLibrary', 'none');

You can specify the Code Replacement Library for Cortex-M as follows:

cfg.CodeReplacementLibrary = 'ARM Cortex-M (CMSIS)';

Please take a look at this example for more details:

https://www.mathworks.com/help/coder/ug/generate-code-for-lstm-network-and-deploy-on-cortex-m.html

[Silvia]

Thank you @Hariprasad Ravishankar and sorry for the late reply.

I am trying to follow the first possible approach in the example (Generate PIL Executable That Accepts a Single Observation of Variable Sequence Length).

% Create a Code Configuration Object
cfg = coder.config('lib','ecoder',true);

% Configure Object for PIL (processor-in-the-loop) Execution
cfg.VerificationMode = 'PIL';

% Specify 'None' as the TargetLibrary when creating a DeepLearningConfig to
% specify no third-parties deep learning libraries (Generate plain C):
cfg.DeepLearningConfig = coder.DeepLearningConfig('TargetLibrary', 'none');

% Specify Target Hardware
cfg.Hardware = coder.hardware('STM32 Nucleo F401RE'); 

% Set PIL Communication Interfance (a serial PIL communication interface)
cfg.Hardware.PILInterface = 'Serial';

% Determine the COM port for serial communication
cfg.Hardware.PILCOMPort = 'COM2';

% Limit stack size because the default stack size is much larger than the
% available memory on the hardware. Set to a smaller value (try with 512)
cfg.StackUsageMax = 512;

% View the log
cfg.Verbose = 1;

% Specify the Code Replacement Library for Cortex-M
cfg.CodeReplacementLibrary = 'ARM Cortex-M (CMSIS)';

%% Generate PIL Executable that accepts a single observation of variable sequence length
% Type function loads the network (in .mat file) into a persistent variable
% The function reuses this persistent object on subsequent prediction calls
type('FinalFineTuned_predict.m')

% Specify input type and size of the input argument tp the codegen command
% by using the coder.typeof function
noisyInputType = coder.newtype('double', [Inf 1], [1 0]);

% Run the codegen command to generate code and PIL executable
codegen -config cfg FinalFineTuned_predict -args {noisyInputType} -report

And my FinalFineTuned_predict.m function is the following:

function out = FinalFineTuned_predict(in) %#codegen

% A persistent object mynet is used to load the series network object.
% At the first call to this function, the persistent object is constructed and
% setup. When the function is called subsequent times, the same object is reused 
% to call predict on inputs, thus avoiding reconstructing and reloading the
% network object.

% Copyright 2019-2021 The MathWorks, Inc. 

persistent mynet;

if isempty(mynet)
    mynet = coder.loadDeepLearningNetwork('FinalFineTuned.mat');
end

% pass in input   
out = predict(mynet, in);

When I run the code, I have the following error:

"### Compiling function(s) FinalFineTuned_predict ...

### Generating compilation report ...

Input data argument to predict must be dlarray type.

Error in ==> FinalFineTuned_predict Line: 18 Column: 7

Code generation failed: View Error Report"

To decide the input type, I used the coderTypeEditor giving as input a variable NoisySignal (7716x1 double) and specifying that I don't want to fix the length of the signal (thus [Inf 1] instead of [7716 1]). If I normally use the predict function in a MATLAB script with this NoisySignal (CleanSignal = predict(netFineTuned, NoisySignal)) no errors occur. Do you know what I am missing?

Thank you in advance,

Silvia

[Hariprasad Ravishankar]

Hi @Silvia,

For code generation we expect the input to predict to be a dlarray. Please try modifying your function as follows:

function out = FinalFineTuned_predict(in) %#codegen
% A persistent object mynet is used to load the series network object.
% At the first call to this function, the persistent object is constructed and
% setup. When the function is called subsequent times, the same object is reused 
% to call predict on inputs, thus avoiding reconstructing and reloading the
% network object.
% Copyright 2019-2021 The MathWorks, Inc. 
persistent mynet;
if isempty(mynet)
    mynet = coder.loadDeepLearningNetwork('FinalFineTuned.mat');
end
% pass in input 
% We first cast the 'double' input to 'single' as code-generation only supports 'single' precision compute for dlnetwork.
% We specify the format of input as 'TC' to indicate that the first
% dimension is 'Time' and second dimension is 'channel'.
outDlarray = predict(mynet, dlarray(single(in), 'TC'); 

% We extract data from the dlarray to get back the result in 'single'
% datatype.
out = extractdata(outDlarray);
end

[Silvia]

Thank you very much, this solved my error!

Now I am encountering another error: "ProjectedLayer objects not supported for code generation. Unpack the network first using unpackProjectedLayers."

Since I would need to compare the inference times of my models (Original, Fine-Tuned and Projected), if I now unpack the Projected layer, I cannot compare the three models?