Matlab bug and deadly error while trying to visualize FasterRCNN network feature maps from softmax layer

I want to visualize the feature maps extracted from the softmax layer of a trained FasterRCNN network. I am aware that this can be done for the RCNN networks as in the DeepLearningRCNNObjectDetectionExample.m with the below code.

% Extract the activations from the softmax layer. These are the classification scores
% produced by the network as it scans the image.
featureMap = activations(rcnn.Network, testImage, 'softmax', 'OutputAs', 'channels');

% The softmax activations are stored in a 3-D array.
size(featureMap)

The 3rd dimension in featureMap corresponds to the object classes

rcnn.ClassNames

The stop sign feature map is stored in the first channel

stopSignMap = featureMap(:, :, 1);

The size of the activations output is smaller than the input image due to the downsampling operations in the network. To generate a nicer visualization, resize stopSignMap to the size of the input image. This is a very crude approximation that maps activations to image pixels and should only be used for illustrative purposes.

% Resize stopSignMap for visualization
[height, width, ~] = size(testImage);
stopSignMap = imresize(stopSignMap, [height, width]);

% Visualize the feature map superimposed on the test image. 
featureMapOnImage = imfuse(testImage, stopSignMap); 

figure
imshow(featureMapOnImage)

But when I do the same of my fasterRCNNObjectDetector it gives an error as:

Error using vision.cnn.FastRCNN/activations (line 207)
Expected layer to match one of these values:

'imageinput', 'conv_1', 'relu_1', 'conv_2', 'relu_2', 'roi pooling layer', 'fc_1', 'relu_3', 'fc_2', 'softmax', 'classoutput'

The input, 'OutputAs', did not match any of the valid values.

Please read below for the update; I discovered a bug and a deadly error while trying to visualize the fasterRCNN feature maps.

[Matlab_user]

Update:

I tried to look into the activations() function that is provided in the faster rcnn class and it reads data as below:

output = activations(this, data, roi, outputLayer, inputLayer)

where outputLayer and inputLayer need to be scalars and roi I belive should be the bounding boxes for data. Then I proceed to call this function as below:

featureMap = activations(detector.Network, I, bboxes, 11, 9);

Where my softmax layer is located in layer 10 and so I specified the outputLayer = 11, inputLayer = 9. But then I get a deadly error as below, and then matlab refuses to work with GPU, and I need to restart matlab to be able to call gpuDevice again.

Error using nnet.internal.cnngpu.reluForward
An error occurred during PTX compilation of <image>.
The information log was:
<No information>
The error log was:
<No information>
The CUDA error code was: CUDA_ERROR_ILLEGAL_ADDRESS.

Error in nnet.internal.cnn.layer.util.ReLUGPUStrategy/forward (line 8)
            Z = nnet.internal.cnngpu.reluForward(X);

Error in nnet.internal.cnn.layer.ReLU/forward (line 39)
            [Z, memory] = this.ExecutionStrategy.forward(X);

Error in vision.internal.cnn.internalFastRCNNSeriesNetwork/activations (line 87)
                    clsOutput = this.Layers{currentLayer}.forward( clsOutput );

Error in vision.cnn.FastRCNN/activations (line 250)
                YChannelFormat = predictNetwork.activations(X, roi, layerID, inputLayerID);

There is something weird going on here, and I would be thankful if you could assist me with this issue.

Answer 1

[Joss Knight]

The activations function of the vision.cnn.FastRCNN is undocumented and Hidden and so not officially supported. Its inputs are not carefully parsed to ensure they are valid and so yes, you can make things crash if you use it.

For what it's worth, the fifth argument ' inputLayer ' is for internal use and you shouldn't use it, Just let it default to 1.

outputLayer is supposed to be the layer who's output you want to inspect, not the layer after.

[Matlab_user]

Thank you for your response. I tried to leave the inputLayer empty as:

featureMap = activations(detector.Network, I , bboxes, 11);

but then I get the following error.

Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS 
> In vision.cnn.FastRCNN/activations (line 200) 
Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

Error in nnet.internal.cnn.layer.learnable.PredictionLearnableParameter/populateCacheIfEmpty (line 80)
              this.CacheHandle.Value = gpuArray(this.HostValue);

Error in nnet.internal.cnn.layer.learnable.PredictionLearnableParameter/get.Value (line 61)
              this.populateCacheIfEmpty();

Error in nnet.internal.cnn.layer.FullyConnected/forward (line 76)
              X, this.Weights.Value, this.Bias.Value);

Error in vision.internal.cnn.internalFastRCNNSeriesNetwork/activations (line 87)
                  clsOutput = this.Layers{currentLayer}.forward( clsOutput );

Error in vision.cnn.FastRCNN/activations (line 250)
              YChannelFormat = predictNetwork.activations(X, roi, layerID, inputLayerID);

and then again gpuDevice becomes inaccessible.

[Joss Knight]

I can't really comment I'm afraid. Clearly there are many undocumented limitations of this undocumented undisclosed function. If you like I can take your question as a request to make this function public.