What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning?
The softmax function generalizes the logistic sigmoid to multiple mutually exclusive classes. It converts a vector of class scores, also called logits, into probabilities that sum to one. The multinomial logistic loss then measures the negative log-probability assigned to the correct class.
For a sample with feature vector \(\mathbf{x}\), multinomial logistic regression computes one score per class:
\[z_i = \mathbf{w}_i^T\mathbf{x} + b_i.\]Here, \(\mathbf{w}_i\) contains the weights for class \(i\), and \(b_i\) is its bias. For \(M\) classes, softmax computes
\[p_i = P(y=i\mid\mathbf{x}) = \frac{e^{z_i}}{\sum_{j=1}^{M} e^{z_j}}.\]In words, we exponentiate each class score and divide it by the sum of all exponentiated scores. Every probability is positive, and their sum is one. Larger scores receive larger probabilities.
For binary classification, the logistic sigmoid maps a single score to the probability of the positive class:
\[\sigma(z) = \frac{1}{1+e^{-z}}.\]Two-class softmax gives the same probability when the sigmoid receives the difference between the two class scores, \(p_1=\sigma(z_1-z_2)\).
Multinomial logistic loss
For a sample whose correct class is \(y\), the loss is
\[L(\mathbf{z},y) = -\log p_y.\]Equivalently, using one-hot targets \(t_i\), we can write the cross-entropy as
\[L(\mathbf{z},\mathbf{t}) = -\sum_{i=1}^{M} t_i\log p_i.\]Only the correct class has a target value of one, so only its log-probability contributes to the loss. A correct-class probability of 0.8 gives a loss of about 0.22; a probability of 0.1 gives about 2.30. Training usually minimizes the mean loss over the training examples.
For numerical stability, softmax can subtract the largest logit before exponentiation without changing the probabilities. Loss implementations typically combine log-softmax and negative log-likelihood directly. For example, PyTorch’s CrossEntropyLoss expects raw logits, so we should not apply softmax before passing the scores to that loss.