selfAttentionLayer can't process sequence-to-label problem?

selfAttentionLayer why can't handle the following simple sequence classification problem, already through the flattenLayer into one-dimensional data, on the contrary, lstm specify "outputMode" as "last" will pass.

% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));

% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
    flattenLayer;

    selfAttentionLayer(6,48);
    % lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
    layerNormalizationLayer;

    fullyConnectedLayer(numClasses);
    softmaxLayer];

net = dlnetwork(layers);

% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
    MaxEpochs=1, ...
    InitialLearnRate=0.01,...
    Shuffle="every-epoch", ...
    GradientThreshold=1, ...
    Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);

Error using trainnet
Number of observations in predictors (1000) and targets (1) must match. Check that the data and network are consistent.

Answer 1

[xingxingcui]

In terms of the output feature map dimensions, there is a time "T" dimension that has to be eliminated in order to match the output dimensions, which can usually be done by indexing1dLayer. So the layers array is added before the fullyConnectedLayer.

% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
    flattenLayer;
    selfAttentionLayer(6,48);
    % lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
    layerNormalizationLayer;
    
    indexing1dLayer; % Add this!!!
    fullyConnectedLayer(numClasses);
    softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
    MaxEpochs=1, ...
    InitialLearnRate=0.01,...
    Shuffle="every-epoch", ...
    GradientThreshold=1, ...
    Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);

    Iteration    Epoch    TimeElapsed    LearnRate    TrainingLoss
    _________    _____    ___________    _________    ____________
            1        1       00:00:02         0.01          1.5374
            7        1       00:00:06         0.01          1.5272
Training stopped: Max epochs completed

-------------------------Off-topic interlude-------------------------------

I am currently looking for a job in the field of CV algorithm development, based in Shenzhen, Guangdong, China. I would be very grateful if anyone is willing to offer me a job or make a recommendation. My preliminary resume can be found at: https://cuixing158.github.io/about/ . Thank you!

Email: [e-mail]

[Imrana]

Hello Cui,

Can you ellobrate it little more, I am stucked in same kind of a problem but is time series forecasting. I have to use lstm to irregular time series data. How to use tspan and put mask on them in self attention layer?

Best Regards,

[xingxingcui]

Hi,@Imrana

you can use analyzeNetwork function to visual network.

Look at the part I've circled in red. The T-dimension is eliminated before you can pick up the next layers.

[Imrana]

Hi @cui,xingxing,

Thank You.

[DGM]

Posted as a comment-as-flag by chang gao:

Useful answer.

[jingwen]

Your answer helps me! Thank you