Do the "sigmoidLayer" and "softmaxLayer" in the Deep Learning Toolbox give the same result for a binary classification problem?

I want to create a classifier which classifies my data into two classes: 0, 1. Do the "sigmoidLayer" and "softmaxLayer" give identical results for this purpose, and if not which layer should I choose for my binary classification problem?

Answer 1

[MathWorks Support Team]

 For a network with one single class output which represents a binary classification problem, the "softmaxLayer" and the "sigmoidLayer" will not give the same result.

Below is an example where 'softmaxLayer' and 'sigmoidLayer' do not have identical results:

netSoftmax = dlnetwork([featureInputLayer(1), softmaxLayer]);
netSigmoid = dlnetwork([featureInputLayer(1), sigmoidLayer]);
X = dlarray(3, "C");
YSigmoid = predict(netSigmoid,X)
YSoftMax = predict(netSoftmax,X)

YSigmoid =   1(C) × 1(U) dlarray     0.9526 YSoftMax =   1(C) × 1(U) dlarray      1

As we can see for a single input of "3", the "sigmoidLayer" will return "0.9526" and the "softmaxLayer" will return "1". This is because the formula for the softmax function guarantees to always return a probability distribution, which means that the sum of all the entries will add up to 1. So for a single input, the results of the softmax function will always be 1. Therefore, the softmax function should never be used for binary classification with one output neuron.

On the other hand, the sigmoid function will always return values in the range [0,1] but the sum of all the entries does not need to add up to 1.

To use the "softmaxLayer" for binary classification, the network has to be modified to have two output neurons, and even in that case the results will not be numerically the same compared to the "sigmoidLayer".

In summary, it is recommend to use "sigmoidLayer" when the network has a single output neuron and "softmaxLayer" for two or more output neurons.