Class Activation Mapping gives different scores than classify function?

Hi!

I tried to make a Class Activation Mapping (CAM) code so that I can see the regions on which the classification of my image is based. However, the scores of the CAM don't seem to be in line with the scores I get when I use the classify function. The network I trained has three possible classes: Absent, OnlyMCs, PresentMass.

The code I used, I got from the link below, and I modified it a bit:

https://www.mathworks.com/help/deeplearning/ug/investigate-network-predictions-using-class-activation-mapping.html

See my code here:

load  ResNet_3class

dataDir = 'Images';
 imds = imageDatastore(dataDir,...
    'IncludeSubfolders',true,...
    'FileExtensions','.tiff', ... 
    'LabelSource','foldernames');
inputSize = [512 512 3];
imdsTest_augm = augmentedImageDatastore([inputSize(1) inputSize(2)], imds, 'ColorPreProcessing', 'gray2rgb');

net = ResNet_3class

[LesionPred,scores] = classify(net,imdsTest_augm)
LesionTrue = imds.Labels;
figure(1)
cm=confusionchart(LesionTrue,LesionPred)

%% CAM
classes = net.Layers(end).Classes;
layerName =  'res5b_relu' ; 
ImageNumber = 1050;

im = imread(imdsTest_augm.Files{ImageNumber});
im = imresize(im,[inputSize(1),inputSize(1)]);
im = gray2rgb(im);
imageActivations = activations(net,im,layerName);

scores_1 = squeeze(mean(imageActivations,[1 2])); %imAct from size [16 16 512] to [512 1] %= same as GAP layer!!

fcWeights = net.Layers(end-2).Weights;   % 'fc1000' layer weights
fcBias = net.Layers(end-2).Bias;
scores_2 = fcWeights*scores_1 + fcBias; 

[~, classIds] = maxk(scores_2, 3);

weightVector = shiftdim(fcWeights(classIds(1), :), -1);

classActivationMap = sum(imageActivations.*weightVector, 3); 

scores_3 = exp(scores_2)/sum(exp(scores_2));
maxScores = scores_3(classIds);
labels = classes(classIds);

LesionPred(ImageNumber, 1)
scores(ImageNumber,:)

figure(2)
subplot(1, 2, 1)
imshow(im)
TrueLabel = string(LesionTrue(ImageNumber, 1));
title('True Class:' + " " + TrueLabel);

subplot(1, 2, 2)
CAMshow(im, classActivationMap)
title(string(labels) + ", " + string(maxScores));

drawnow

So, for example for a certain image I got with [LesionPred,scores] = classify(net,imdsTest_augm) the scores

0.0021934 0.7136005 0.2842060 . Therefore, the image was classified as an 'OnlyMCs' image by the network. However, the CAM gave me the following outcome:

So the scores 0.99872 0.0012439 4.0788e-05 . Which are very different. Also, the whole image 'heats up', which I don't understand.

I'm really new with deep learning, so maybe it's just a small mistake I'm making. Hopefully someone can help me on my way. Thanks!

(I can't use the GradCAM function because I don't have the license for the newer version of Matlab).

Answer 1

[aditi bagora]

Hi Noa,

I see that you are attempting to generate class activation mapping for your prediction, but the results are not as expected.

Upon reviewing your code, it appears that you have used ResNet for classification into three categories. During my initial review, I noticed that you have created separate variables for scores, and it seems that you are using the incorrect score variable to compute the “classIds”.

To address this issue, I suggest making the following change:

Ensure that the classes are generated before updating the score variable. In your case, the “classIds" should be calculated based on “score_1" rather than "score_2”.

imageActivations = activations(net,im,layerName);
scores_1 = squeeze(mean(imageActivations,[1 2])); %imAct from size [16 16 512] to [512 1] %= same as GAP layer!!
[~, classIds] = maxk(scores_1, 3); % classIds should be calculated from score_1 and not score_2

fcWeights = net.Layers(end-2).Weights;   % 'fc1000' layer weights
fcBias = net.Layers(end-2).Bias;
scores_2 = fcWeights*scores_1 + fcBias; 

weightVector = shiftdim(fcWeights(classIds(1), :), -1);
classActivationMap = sum(imageActivations.*weightVector, 3);

Please refer to the following link for more information and correct codeflow:

Deep Learning Visualizations: CAM Visualization: https://blogs.mathworks.com/deep-learning/2019/01/31/deep-learning-visualizations-cam-visualization/

Hope this helps and solves your query.

Regards,

Aditi