Accuracy of ANN confusion matrix is not correct

Hi, i been working on ANN to design a prediction model.This is my coding.But after i run the coding the accuracy in confusion matrix is not same as the final accuracy using formula

clear, close all

clc

% help nndatasets

load ('ann.mat');

RandStream.setGlobalStream(RandStream ('mt19937ar','seed',1));

x = [aiparameter.mc1,aiparameter.torque1];

y = aiparameter.output;

a = isnan(y);

[u,~] = find(a==0);

x = x(u,:); y = y(u);

x = x'; y = y';

% x=rows2vars(m);

% y=rows2vars(n);

%[y,n]=vec2ind(y1);%

%y(y == 2) = 0;

len_ = length(y);

temp = rand(1,length(y));

[~, ind] = sort(temp);

x = x(:, ind);

y = y(:, ind);

y_store = categorical(y);

b = 1:4;

for ii = 1:len_

yy(:,ii) = (b == y(ii));

end

y = single(yy);

net = patternnet;

% Divide data into k-folds

kfold = 5; %10

fold = cvpartition(len_, 'kfold', kfold);

% NN train for every fold

for k = 1 : kfold

% Call index of training & testing sets

trainIdx = fold.training(k); testIdx = fold.test(k);

% Call training & testing features and labels

xtrain = x(:,trainIdx);

ytrain = y(:,trainIdx);

xtest = x(:,testIdx);

ytest = y(:,testIdx);

% configure network

net = configure(net,x,y);

trInd = find(trainIdx); tstInd = find(testIdx);

net.divideFcn = 'divideind';

net.divideParam.trainInd = trInd;

net.divideParam.testInd = tstInd;

% Initialize network

net.layers{1}.name='Hidden Layer 1';

net.layers{2}.name='Output Layer';

net.layers{1}.size = 12; % number of hidden neuron (1-10)

net.layers{1}.transferFcn = 'satlin';

net.trainFcn = 'trainlm'; % training algorithm (trainlm, trainscg, etc)

net.performFcn = 'mse';

net.trainParam.epochs = 10000;

% Train the Network

[net, tr] = train(net, x, y);

% Test the Network

yhat = net(x);

% adjust threshold

thr = 0.5;

% for ii = 1:1:len_

% if yhat(:,ii) >= thr

% yhat(:,ii) = 1;

% else

% yhat(:,ii) = 0;

% end

% end

clear Ypred

for ii = 1:len_

[~,Ypred(ii)] = max(yhat(:,ii));

end

Ypred = categorical(Ypred);

% validate trained network

e = gsubtract(y, yhat);

performance = perform(net, y, yhat);

perf_fold(k)=performance;

% Calculate Training and Testing Performance (Confusion Matrix)

x_trn = x(:,tr.trainInd);

x_tst = x(:,tr.testInd);

y_trn = y_store(:,tr.trainInd);

y_tst = y_store(:,tr.testInd);

yhat_trn = Ypred(:,tr.trainInd);

yhat_tst = Ypred(:,tr.testInd);

figure, plotconfusion(y_trn, yhat_trn, 'Training');

figure, plotconfusion(y_tst, yhat_tst, 'Testing');

% for ii = 1:10%length(y_tst)

% [y_class(ii),~] = find(y_tst(:,ii) == 1);

% [yhat_class(ii),~] = find(yhat_tst(:,ii) == 1);

% end

C=confusionmat(y_tst,yhat_tst)

tp(k)=C(1,1);

fn(k)=C(1,2);

tn(k)=C(2,2);

fp(k)=C(2,1);

end

TP=round(mean(tp));

FN=round(mean(fn));

TN=round(mean(tn));

FP=round(mean(fp));

accuracy=(TP+TN)/(TP+FN+TN+FP)*100

sensitivity=TP/(TP+FN)*100

specificity=TN/(TN+FP)*100

[Poorna]

Can you provide the "ann.mat" file?

Answer 1

[Arnav]

Hi @Keshasni,

I understand that you are trying to perform k-fold cross-validation training of the ANN model. Here, there are 5 folds used. Also, the model is used to classify between 4 classes. You are trying to find the values of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) for each fold and then trying to find out the accuracy, sensitivity and specificity.

There are a few issues in the code provided below:

    tp(k)=C(1,1);
    fn(k)=C(1,2);
    tn(k)=C(2,2);
    fp(k)=C(2,1); 

Since there are 4 classes, the confusion matrix for each fold will be a 4X4 matrix and not a 2X2 matrix. Hence, the value of TP (and other values) cannot be simply taken from the Confusion Matrix by indexing.

For a multi-class classification problem, there are 2 ways in which the estimates are defined.

• Firstly, it is necessary to calculate TP, TN, FP and FN for each of the classes. It should be noted that these values are vectors holding one value for each of the classes. This can be done as shown below:

    % b is the vector of classes
    for i = b
        TP(i) = C(i, i);
        FN(i) = sum(C(i, :)) - TP(i);
        FP(i) = sum(C(:, i)) - TP(i);
        TN(i) = sum(C(:)) - (TP(i) + FN(i) + FP(i));
    end

• Macro Averaging: This method calculates the metric for each class and then averages them. For instance, the macro averaged specificity can be calculated as:

    macroSpecificity = mean(TN ./ (TN + FP));

• Mirco Averaging: This method sums the confusion matrices for all the classes then it computes the metric. For instance, the micro averaged specificity can be calculated as:

    microSpecificity = sum(TN) / (sum(TN) + sum(FP));

Generally, macro-averaging is preferred as it gives equal weight to all the classes as opposed to micro-averaging which gives more weight to majority classes.

It should be noted that the above calculations only make sense for each of the folds separately. An ensemble of the models trained for each fold should be tested separately to find out its performance metrics.

Hope it helped you!