Cross-entropy loss and the softmax gradient

The concept

A classifier with softmax outputs produces a probability for each class. The loss that fits this setting is cross-entropy: for the correct class c, the loss is −log p_c. It is small when the model assigns a high probability to the right class and grows without bound as that probability approaches zero. Minimizing it is equivalent to maximum-likelihood training of the model.

Combined with softmax, cross-entropy has a very clean gradient with respect to the scores: ∂L/∂zᵢ = pᵢ − yᵢ, the predicted probability minus the one-hot target. The program verifies this against a finite-difference estimate for each score. The first loss is lower for class 0 (the highest-scoring class) than for class 2, which the model thinks is unlikely.

The last two numbers show why cross-entropy is preferred with sigmoid or softmax outputs. Suppose a sigmoid unit is confidently wrong: its score is −6 and the target is 1. The gradient of squared error with respect to the score is multiplied by the sigmoid's slope, which is almost zero in the saturated region, so it is only −0.0025 and learning barely moves. The cross-entropy gradient is y − t = −0.9975, large exactly when the error is large.

Code example

import math


def softmax(values):
    peak = max(values)
    exps = [math.exp(v - peak) for v in values]
    total = sum(exps)
    return [e / total for e in exps]


def cross_entropy(scores, target):
    return -math.log(softmax(scores)[target])


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


scores = [2.0, 1.0, 0.1]
target = 0
p = softmax(scores)
analytic = [p[i] - (1 if i == target else 0) for i in range(3)]
numeric = []
for i in range(3):
    up = scores[:]
    up[i] += 1e-6
    down = scores[:]
    down[i] -= 1e-6
    numeric.append((cross_entropy(up, target) - cross_entropy(down, target)) / 2e-6)
print(all(abs(a - n) < 1e-6 for a, n in zip(analytic, numeric)))
print(round(cross_entropy(scores, 0), 4), round(cross_entropy(scores, 2), 4))
y = sigmoid(-6)
print(round((y - 1) * y * (1 - y), 4), round(y - 1, 4))

True
0.417 2.317
-0.0025 -0.9975

Follow the code

Open the walkthrough to inspect the example one step at a time.

—

True

True
0.417 2.317

True
0.417 2.317
-0.0025 -0.9975

Behind the scenes

Cost and common mistakes

Evaluating softmax cross-entropy for k classes is O(k). Its gradient is also O(k).

• Computing log(softmax(z)) in two separate steps can underflow to log(0). Libraries combine them as a log-sum-exp for stability.
• Do not apply softmax twice: loss functions that take raw scores (logits) already include it.
• Cross-entropy is for classification. For predicting real numbers, mean squared error is the usual choice.

Try it yourself

What is the loss if the model gives the correct class probability 0.5?

Further reading

• Deep Learning (Goodfellow, Bengio, Courville) (external)
• Stanford CS231n: Deep Learning for Computer Vision (external)
• Dive into Deep Learning (external)