Loss: how wrong is the network?

What you will be able to do

Measure how wrong the ticket network is with one number that training can make smaller: the loss. Accuracy only counts right answers, so it cannot tell a lucky guess from a sure answer, and a network cannot learn from it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It computes softmax by hand, the loss of the untrained ticket network, the loss and the accuracy side by side, and the softmax-twice mistake. Its setup cells download the tickets and the course’s code, so it needs no local setup. About 25 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The forward pass of the ticket network: word counts → 64 hidden neurons with ReLU → 5 scores, one per team, from The forward pass.
• The log loss, −ln p, from Logarithms of probabilities in the mathematics course.
• The logistic function, which turns a score into a number from 0 to 1, from Classification intuition in the machine learning course.

What you need on your computer

• Nothing for the notebook route.
• For the local route: PyTorch and the folder ticket-router with data/ and ticketnet.py, from Set up PyTorch. The tickets as numbers below gives the setup lines that every code file of this module starts with. If import torch fails, set up the environment again as that lesson shows.

Tested with Python 3.12 and PyTorch 2.14.1 on a CPU.

Two weather forecasts

Two weather apps make a forecast for tomorrow. The first says “90% chance of rain”. The second says “55% chance of rain”. It rains. Both apps said that rain was more likely than not, so if you only count right and wrong, both were right.

But you trust the first app more. It was sure, and it was right. If it had not rained, you would also blame the first app more: it was sure, and it was wrong. A fair score for a forecast looks at the probability that the forecast gave to what really happened, not only at the winner.

The problem: scores are not probabilities

The ticket network of The forward pass ends with five numbers, one for each team. For the first validation ticket, the untrained network gives these:

Team	delivery	returns	payment	warranty	account
Score	−0.129	0.023	0.107	0.024	0.065

These numbers are not probabilities. One is negative, and they do not add up to 1. So you cannot say “the network is 10% sure of payment”.

The right team of this ticket is payment, and payment has the highest score. So if you count right and wrong, the network got this ticket right. But the network has random weights and has learned nothing. It was lucky, and accuracy cannot see that. Training needs a number that sees how sure the network was of the right team.

Softmax: from scores to probabilities

Softmax turns any list of scores into probabilities in two steps. Take three made-up scores for three teams: 2.0, 1.0 and 0.1.

1. Make every score positive: compute e to the power of the score (exp), where e ≈ 2.718. You get 7.389, 2.718 and 1.105. A larger score still gives a larger number.
2. Divide each result by their sum, 11.212. You get 0.659, 0.242 and 0.099, which add up to 1.

The first score is twice the second, but its probability is 2.7 times as large. Softmax looks at the differences between scores: one point more always means e ≈ 2.718 times the probability. So adding the same number to every score changes nothing.

The name says what it does. The plain “max” of the scores picks one winner and gives it everything. Softmax is a soft version: the winner gets the largest share, and the others keep a smaller share. With only two teams, softmax is the logistic function of the machine learning course’s logistic regression.

Cross-entropy: the surprise at the right answer

Now the forecast score. Take the probability p that the network gave to the right team, and compute −ln p. This is the loss of one example: a number that says how wrong the network is. It is the log loss that you met in the mathematics course.

Read the curve from right to left. A sure, right answer (p close to 1) costs almost nothing. An even guess between five teams (p = 0.2) costs 1.609. A sure, wrong answer (p close to 0) costs a lot, and the cost has no upper limit.

The loss of one example, −ln p, against the probability p that the network gave to the right team. It is 0 when p = 1 and grows without limit as p falls to 0.

Text description of the chart

The curve starts high on the left: at p = 0.01 the loss is 4.6, at p = 0.02 it is 3.9. It falls quickly, then slowly: 1.609 at p = 0.2 (an even guess between five teams) and 0.693 at 0.5. Then 0.417 at 0.659, 0.105 at 0.9, and 0 at p = 1.

In the three-team example, if the right team is the first one, the loss is −ln 0.659 = 0.417. If the right team is the third one, the loss is −ln 0.099 = 2.317: the network was sure of the wrong team, so it is very surprised.

This loss is called cross-entropy. “Entropy” comes from information theory, where −ln p measures the surprise of an event with probability p. “Cross” means that it compares two things: the network’s probabilities and the right answer. For a whole set of tickets, the loss is the mean of the per-ticket losses. In a formula, with s the scores and “right” the right team:

$$p_i = \frac{e^{s_i}}{\sum_j e^{s_j}} \qquad \text{loss} = -\ln p_{\text{right}}$$

The tickets as numbers

The code of this module starts from the tickets as word counts, as in The forward pass. This time, the course’s ticketnet.py builds them, with the same code that you wrote there. In the notebook, the setup cells run these lines for you. On your own computer, put them at the top of each code file of this module, in the folder ticket-router:

loss.py

import torch
from torch import nn
from ticketnet import *        # the course's code: TEAMS, load_split, Vocab, bag_of_words, labels, BagOfWordsNet, set_seed, evaluate, train ...

train_rows, valid_rows = load_split("data/train.csv"), load_split("data/valid.csv")
vocab = Vocab([r["text"] for r in train_rows])                 # the vocabulary: training tickets only
x_train, y_train = bag_of_words(train_rows, vocab), labels(train_rows)
x_valid, y_valid = bag_of_words(valid_rows, vocab), labels(valid_rows)
loss_fn = nn.CrossEntropyLoss()                                # the loss of this lesson
print("x_train", tuple(x_train.shape), " x_valid", tuple(x_valid.shape), " vocabulary", len(vocab))

The output:

x_train (4696, 2864)  x_valid (600, 2864)  vocabulary 2864

labels() turns each ticket’s team into its position in TEAMS, from 0 (delivery) to 4 (account). BagOfWordsNet is the network of The forward pass, and set_seed(0) gives it the same starting weights.

The untrained ticket network

Now the real ticket. The diagram follows the four steps for the first validation ticket, with the untrained network’s real numbers.

From the five scores of the untrained network to the loss of one ticket: exp, divide by the sum, then −ln of the right team's probability.

Text description of the diagram

The first validation ticket goes through the untrained network. Step 1, the five scores: delivery −0.129, returns 0.023, payment 0.107, warranty 0.024, account 0.065. Step 2, exp of each score: 0.879, 1.023, 1.113, 1.024 and 1.067; their sum is 5.107. Step 3, each value divided by the sum gives the probabilities 0.172, 0.200, 0.218, 0.201 and 0.209, which add up to 1. Step 4, the right team is payment, so p = 0.218 and the loss is −ln 0.218 = 1.523.

In PyTorch, torch.softmax(scores, dim=1) computes the probabilities along dimension 1, the five teams of each ticket. nn.CrossEntropyLoss() computes the loss. It takes the scores, not the probabilities, and the index of the right team; it does the softmax itself. Priya runs both on the first validation ticket:

loss.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
scores = model(x_valid[:1])
print("scores:       ", scores)
print("probabilities:", torch.softmax(scores, dim=1))
print("right team:   ", TEAMS[y_valid[0]])
print("loss:         ", loss_fn(scores, y_valid[:1]))

The output:

scores:        tensor([[-0.1288,  0.0228,  0.1072,  0.0242,  0.0647]],
       grad_fn=<AddmmBackward0>)
probabilities: tensor([[0.1722, 0.2003, 0.2180, 0.2006, 0.2089]], grad_fn=<SoftmaxBackward0>)
right team:    payment
loss:          tensor(1.5234, grad_fn=<NllLossBackward0>)

The grad_fn=… part is a note that PyTorch keeps for training; Backpropagation and autograd explains it. The loss is 1.523, the same as by hand. Now the whole validation set. evaluate(), from the course’s ticketnet.py, returns the mean loss and the accuracy:

loss.py

import math

valid_loss, valid_acc = evaluate(model, x_valid, y_valid)
print("validation loss:", round(valid_loss, 4), "| ln 5 =", round(math.log(5), 4))
print("validation accuracy:", round(valid_acc, 4))

The output:

validation loss: 1.624 | ln 5 = 1.6094
validation accuracy: 0.1817

A network that gives exactly 0.2 to every team has a loss of −ln 0.2 = ln 5 = 1.609. The untrained network is at 1.624, very close: it guesses about evenly. Its accuracy, 18.2%, is a chance result too; even “always delivery” gets 27.5%. Training must push the loss down. After training in this module, the same network reaches a validation loss of about 0.06.

Why we train on the loss but report accuracy

Omar asks: “Customers care about right answers. Why not train the network on accuracy?” Priya shows him one experiment. She changes one number of the untrained network, the bias of the payment score, from −3 to 3. For each value, she measures the loss and the accuracy on the 4,696 training tickets.

The loss of the untrained network on the training tickets, while one number changes: the bias of the payment score. The curve is smooth, so every small change shows.

Text description of the chart

The loss is a smooth valley. At a bias of −3 it is 2.06, at 0 it is 1.624, its lowest point, and at 3 it is 2.54. The network’s real bias, 0.087, is close to the bottom.

The accuracy of the same network over the same range. It is flat almost everywhere, with one jump.

Text description of the chart

The accuracy is a flat line at 14.7% from a bias of −3 to about 0. Between about 0 and 0.3 it jumps up, to 21.3%, and then it is flat again up to 3. At 21.3%, the network answers “payment” for every ticket.

The loss answers the question “is this small change a little better or a little worse?” everywhere. The accuracy is flat: on most of the range, a small change does not move it at all, so it cannot say which way to go. Priya shows the same in code. She makes every ticket’s payment score lower by 0.3, 0.2 and 0.1, which is the same as a smaller payment bias:

loss.py

scores = model(x_train)
for change in [-0.3, -0.2, -0.1]:
    shifted = scores.clone()
    shifted[:, 2] += change          # every ticket's payment score, a little lower
    loss = loss_fn(shifted, y_train).item()
    accuracy = (shifted.argmax(dim=1) == y_train).float().mean().item()
    print(f"change {change:.2f}   loss {loss:.5f}   accuracy {accuracy:.4f}")

The output:

change -0.30   loss 1.63139   accuracy 0.1469
change -0.20   loss 1.62730   accuracy 0.1469
change -0.10   loss 1.62467   accuracy 0.1469

scores.clone() makes a copy, so the original scores stay. .item() turns a one-number tensor into a Python number. The loss says that −0.1 is better than −0.2; the accuracy cannot tell the three apart. The next lesson uses exactly this: training follows the slope of the loss, and a flat line has no slope.

The loss also sees what accuracy hides. After two epochs of training (passes through all the training tickets), the course’s network has a validation accuracy of 98.2% and a loss of 0.206. After twelve epochs, the accuracy is almost the same, 98.3%, but the loss is 0.063. The network became much more sure of its right answers, and only the loss shows it. So:

• Train on the loss. It changes smoothly with every weight, and it rewards sure, right answers.
• Report accuracy (and the other metrics of the machine learning course) to people. “98 tickets in 100 reach the right team” means something to Grace; “a loss of 0.063” does not.

When to use cross-entropy, and what it costs

Use cross-entropy when the network chooses one class from several, as the router does: one right team per ticket. For a number, such as minutes or a price, use a loss for numbers, such as the mean squared error of the mathematics course.

What it costs: the loss is hard to explain to people, so always report a metric next to it. And because a sure, wrong answer costs so much, a few bad examples can weigh a lot. One ticket with p = 0.001 for its given team costs 6.9, as much as 66 tickets at p = 0.9. The training data has 70 tickets with a wrong team on purpose, and Debug the experiment uses this effect to find them.

Guided practice: the loss by hand

1. Use the probabilities of the first validation ticket: delivery 0.172, returns 0.200, payment 0.218, warranty 0.201, account 0.209.
2. Suppose the right team were account. Write down p for the right team.
3. Compute −ln p on a calculator (the ln key) or in Python with -math.log(p). You should see about 1.565.
4. Now suppose the right team were delivery. Compute the loss again. You should see about 1.760.
5. In the notebook, complete loss_for() in section 6 and run its check cell, which uses delivery as the right team.

Show the answer

For account, p = 0.209 and the loss is −ln 0.209 = 1.565. For delivery, p = 0.172 and the loss is −ln 0.172 = 1.760 (the notebook gives 1.759, from the unrounded probability 0.1722). Delivery has the lowest probability, so it has the highest loss: the network would be most surprised if the ticket were for delivery.

Your turn: a sure, wrong answer

Go back to the three made-up scores 2.0, 1.0 and 0.1. Before you compute, predict: is the loss larger when the right team is the second one or the third one? Then compute both, by hand and with loss_fn(torch.tensor([[2.0, 1.0, 0.1]]), torch.tensor([1])) (and [2] for the third).

Show the answer

The second team has p = 0.242, so the loss is −ln 0.242 = 1.417. The third team has p = 0.099, so the loss is 2.317. The third is larger: the network gave the right team the smallest share, so it is the most surprised.

Common mistake

Softmax twice

Symptom: Tomás adds torch.softmax() at the end of the network “so that it gives probabilities”, and keeps nn.CrossEntropyLoss(). Training runs, and the accuracy looks good. But the loss stays high. In Priya’s run of his version, after 3 epochs the validation accuracy is 98.8% and the loss is still 0.96.

Cause: nn.CrossEntropyLoss() applies softmax itself. With Tomás’s change, the probabilities go through softmax a second time, and the loss is wrong. For the three-team example, the loss of the scores is 0.417; the loss of the probabilities is 0.802. Even a perfect answer, 1.0 for the right team of five, gets a loss of 0.905 this way. So the loss can never fall below 0.905, and every number that uses it is wrong.

Fix: let the network return the raw scores, and give those to nn.CrossEntropyLoss(). Apply torch.softmax() only when you want to read or show the probabilities.

What it means for a real AI service: read the documentation of a loss function before you use it. Check what input it expects (scores or probabilities) and what it does itself. A model can look fine on accuracy while every loss in the training log is wrong.

Knowledge check

Check your understanding

In your own words: what does the cross-entropy loss of one ticket measure, and why is it large when the network is sure of the wrong team?

Show the answer

It measures how surprised the network is by the right answer: −ln p, where p is the probability that the network gave to the right team. If the network is sure of a wrong team, the right team gets a small p, for example 0.01. Then −ln p is large (4.6), because the logarithm of a small number is a large negative number. A sure, right answer (p near 1) costs almost nothing.

Check your understanding

Predict: a network gives the scores 3, 3, 3, 3, 3 for the five teams. What probabilities does softmax give, and what is the loss? Then add 10 to every score: what changes?

Show the answer

All five scores are equal, so softmax gives each team 0.2. The loss is −ln 0.2 = ln 5 = 1.609, the loss of an even guess. Adding 10 to every score changes nothing: softmax looks only at the differences between scores, and the differences are still 0. The probabilities stay 0.2 and the loss stays 1.609.

Check your understanding

Omar wants the network trained directly on accuracy, because accuracy is what customers feel. Give one reason why training uses the loss instead.

Show the answer

Accuracy changes only when an answer flips to another team, so for most small changes of a weight it does not move at all. A flat number gives no direction: training cannot tell if a small change is better or worse. The loss changes smoothly with every weight, so it always shows which way is better. It also rewards a network for being more sure of right answers, which accuracy cannot see. We still report accuracy to people, because it is easy to understand.

Recap

• Softmax turns the network’s scores into probabilities: exp of each score, divided by their sum. Only the differences between scores matter.
• The loss of one example is −ln p of the right team: the cross-entropy. It is small for a sure, right answer and large for a sure, wrong one. The loss of a set is the mean.
• The untrained ticket network has a validation loss of 1.624, about ln 5 = 1.609, the loss of an even guess, and an accuracy of 18.2%.
• Train on the loss, report accuracy. The loss changes smoothly with every weight; accuracy is flat between its jumps.
• nn.CrossEntropyLoss() takes scores, not probabilities. Softmax twice makes the loss wrong.

Next, make the loss smaller: change the weights in small steps, always downhill.

Next: Gradient descent