InputSize definition for sequenceInputLayer for multidimensional time sequences

Hi,

I am trying to input data to a LSTM network using a sequenceInputLayer. The input data is a cell array with dimensions 6766x1, in which, each cell is a 3D tensor of dimensions 1024 x numOfTimeSteps x 2. Each rows x depth slice is a time step, and the numOfTimeSteps is variable between each cell. It is not clear how the property InputSize should be set in this case, as I am dealing with a vector sequence but one that is multidimensional and of variable length between cells.

Any thoughts?

Thanks,

Jacopo

Answer 1

[Paras Gupta]

Hi Jacopo,

I understand that you are trying to input data to an LSTM network using a "sequenceInputLayer". Your input data is a cell array with dimensions 6766x1, where each cell is a 3D tensor of dimensions 1024 x numOfTimeSteps x 2.

For the "sequenceInputLayer", the "InputSize" should be set to the size of each time step. In your case, each time step is a 2D slice of size [1024, 2], which will be the input size for the "sequenceInputLayer". Additionally, while training and inference, you need to consider the 'STCB' (Sequence, Time, Channel, Batch) data format for input data.

The following example code illustrates the use of "sequenceInputLayer" for your use-case:

% Define parameters
numSequences = 6766;
maxNumTimeSteps = 50;
inputSize = [1024, 2];
numHiddenUnits = 100;
numResponses = inputSize(2);

% Generate dummy data
data = cell(numSequences, 1);
taregt = cell(numSequences, 1);
for i = 1:numSequences
    numTimeSteps = randi([1, maxNumTimeSteps]); % Random number of time steps
    data{i} = randn(inputSize(1), numTimeSteps, inputSize(2)); % Random 3D tensor
    target{i} = rand(numResponses, numTimeSteps) % random target data
end

% Define the network
layers = [
    sequenceInputLayer(inputSize)
    lstmLayer(128)
    fullyConnectedLayer(numResponses)];

% Define training options
options = trainingOptions("adam", ...
    InputDataFormats="STCB",...
    TargetDataFormats="CTB",...
    MaxEpochs=1, ...
    Plots="training-progress");

% Train the network
net = trainnet(data, target, layers, 'mse', options);

% Pass input and make prediction
pred = predict(net, ones(1024, 40, 2), InputDataFormats="STCB", OutputDataFormats="CTB");

Please refer to the following documentation links for more information on the functions and data formats use in the code above:

• trainnet - https://www.mathworks.com/help/deeplearning/ref/trainnet.html
• predict - https://www.mathworks.com/help/deeplearning/ref/dlnetwork.predict.html
• training options - https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html
• Deep Learning data formats - https://www.mathworks.com/help/deeplearning/ug/deep-learning-data-formats.html
• sequenceInputLayer - https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.sequenceinputlayer.html

Hope this helps you move forward with your work.

Answer 2

[Jacopo Biasetti]

Hi Paras,

Many thanks for your reply.

I am following the example https://se.mathworks.com/help/deeplearning/ug/classify-videos-using-deep-learning-with-custom-training-loop.html with just the input data being modified to be 3D tensors and not 2D tensors as explained in my first message. I modified the input data format as you suggested, but despite that, when I run the code, I get the following error:

Error using dlnetwork/forward (line 580)
Layer 'sequence': Invalid input data. Invalid size of channel dimension. Layer expects
input with channel dimension size 2 but received input with size 392.

Error in classify_video_sequences_full_network_v1>modelGradients (line 472)
[dlYPred,state] = forward(dlnet,dlX);

Error in deep.internal.dlfeval (line 17)
[varargout{1:nargout}] = fun(x{:});

Error in deep.internal.dlfevalWithNestingCheck (line 19)
[varargout{1:nargout}] = deep.internal.dlfeval(fun,varargin{:});

Error in dlfeval (line 31)
[varargout{1:nargout}] = deep.internal.dlfevalWithNestingCheck(fun,varargin{:});

Error in classify_video_sequences_full_network_v1 (line 207)
[gradients,state,loss] = dlfeval(@modelGradients,dlnet,dlX,dlY);

The whole training code is exactly the same as in the tutorial I linked above, and reported below (only the relevant portions):

layersLSTM = [
    sequenceInputLayer([1024 2],'Name','sequence')
    bilstmLayer(2000,'OutputMode','last','Name','bilstm')
    dropoutLayer(0.5,'Name','drop')
    fullyConnectedLayer(numClasses,'Name','fc')
    softmaxLayer('Name','softmax')
    ];
dlnet = dlnetwork(layersLSTM); %#ok<*NASGU>

numEpochs = 15;
miniBatchSize = 128;
initialLearnRate = 1e-4;
decay = 0.001;
gradDecay = 0.9;
sqGradDecay = 0.999;
plots = "training-progress";

mbq = minibatchqueue(dsTrain,...
    'MiniBatchSize',miniBatchSize,...
    'MiniBatchFcn', @preprocessLabeledSequences,...
    'MiniBatchFormat',{'SCTB',''});

if plots == "training-progress"
    figure
    lineLossTrain = animatedline('Color',[0.85 0.325 0.098]);
    ylim([0 inf])
    xlabel("Iteration")
    ylabel("Loss")
    grid on
end

averageGrad = [];
averageSqGrad = [];

iteration = 0;
start = tic;

% Loop over epochs.
for epoch = 1:numEpochs
    % Shuffle data.
    shuffle(mbq);

    % Loop over mini-batches.
    while hasdata(mbq)
        iteration = iteration + 1;

        % Read mini-batch of data.
        [dlX, dlY] = next(mbq);

        % Evaluate the model gradients, state, and loss using dlfeval and the
        % modelGradients function.
        [gradients,state,loss] = dlfeval(@modelGradients,dlnet,dlX,dlY);

        % Determine learning rate for time-based decay learning rate schedule.
        learnRate = initialLearnRate/(1 + decay*iteration);

        % Update the network parameters using the Adam optimizer.
        [dlnet,averageGrad,averageSqGrad] = adamupdate(dlnet,gradients,averageGrad,averageSqGrad, ...
            iteration,learnRate,gradDecay,sqGradDecay);

        % Display the training progress.
        if plots == "training-progress"
            D = duration(0,0,toc(start),'Format','hh:mm:ss');
            addpoints(lineLossTrain,iteration,double(gather(extractdata(loss))))
            title("Epoch: " + epoch + " of " + numEpochs + ", Elapsed: " + string(D))
            drawnow
        end
    end
end

function [gradients,state,loss] = modelGradients(dlnet,dlX,Y)

    [dlYPred,state] = forward(dlnet,dlX);
    
    loss = crossentropy(dlYPred,Y);
    gradients = dlgradient(loss,dlnet.Learnables);

end

Hope you can provide an answer as to why this error pops us.

Many thanks,

Jacopo