Backpropagation and autograd

What you will be able to do

Get the slope of the loss for every weight of the network from one backward pass, instead of nudging 183,685 weights one by one. This is what makes training a network possible at all, and PyTorch does it for you with one call.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It runs one neuron forward and backward and checks the result by nudging. It also checks one gradient of the ticket network by hand, prints the shapes of all the gradients and times the backward pass. About 25 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Softmax and the loss −ln p, from Loss: how wrong is the network?
• The slope of the loss, the update rule and the four-ticket network with one neuron, from Gradient descent.
• The layers of the ticket network and their shapes, from The forward pass.

What you need on your computer

• Nothing for the notebook route.
• For the local route: PyTorch and the folder ticket-router, from Set up PyTorch, and the setup lines of The tickets as numbers at the top of each code file.

Tested with Python 3.12 and PyTorch 2.14.1 on a CPU.

A complaint on a factory line

A biscuit factory has a line of three stations: mixing, baking and packing. A customer complains: the biscuits are too dark. Whose fault is it, and how much?

The complaint arrives at the end of the line, so you start there and walk backwards. The packers say: “Darker biscuits from the oven give more complaints: 2 more complaints for each shade darker.” The bakers say: “Each extra minute in the oven makes the biscuits 3 shades darker.” So each extra minute of baking gives 2 × 3 = 6 more complaints. Each station only knows its own effect on the next station. Multiply the effects as you walk back, and every station learns its share of the blame.

The problem: one slope per weight is too slow to measure

Gradient descent needs the slope of the loss for every parameter, at every step. In the last lesson, you measured a slope by nudging: change one parameter a tiny bit, run the network again, and see how the loss changes. That costs one extra forward pass per parameter.

The ticket network has 183,685 parameters. In the notebook, one forward pass on all 4,696 training tickets took 0.016 seconds. Nudging every parameter once costs about 183,685 × 0.016 s ≈ 48 minutes, for one step. Training the network in this module takes 1,480 steps, so nudging would take about 49 days. Training needs all the slopes at the cost of about one pass.

The idea: pass the blame backwards

Backpropagation is the factory line for a network. The forward pass computes the loss from the inputs, one step at a time, and PyTorch keeps a record of every step: the computation graph. Then the backward pass walks the graph from the loss back to the inputs. At each step, it multiplies the blame that arrives by the step’s own local slope, and it passes the result on.

The rule behind it is the chain rule: when one change causes another, their slopes multiply, as the 2 × 3 = 6 of the biscuits. The name “backpropagation” says what happens: the error is sent back (propagated) through the network. Autograd, short for automatic gradients, is PyTorch’s tool that does it.

Each kind of step has a known local slope, and PyTorch knows all of them. This lesson needs only four:

• −ln p has the local slope −1/p. At p = 0.5: −2.
• sigmoid has the local slope p × (1 − p). At p = 0.5: 0.25.
• w × x changes by x for each unit of w (and by w for each unit of x).
• adding b changes by 1 for each unit of b.

One neuron, forward and backward

Take the four-ticket network of Gradient descent and one ticket, C: it says “refund” twice (x = 2), and it is a payment ticket. Take w = 0.5 and b = −1. The diagram shows the forward pass on top and the backward pass below, with every real number.

One neuron on ticket C. Forward (solid, values): z = 0.5 × 2 − 1 = 0, p = 0.5, loss = 0.693. Backward (dashed, gradients): each step multiplies by its local slope, down to grad w = −1.0 and grad b = −0.5.

Text description of the diagram

Forward pass, left to right. The input x = 2 and the weight w = 0.5 go into a multiply node: w × x = 1.0. The add node adds the bias b = −1: z = 0.0. The sigmoid node gives p = 0.5. The minus-ln node gives the loss, 0.693.

Backward pass, right to left. It starts with 1 at the loss. The minus-ln node multiplies by its local slope −1/p = −2, so −2.0 goes back to p. The sigmoid node multiplies by p(1 − p) = 0.25, so −0.5 goes back to z. The add node multiplies by 1, so −0.5 goes to the multiply node, and grad b = −0.5. The multiply node multiplies by x = 2, so grad w = −1.0.

Read the backward row from the right:

1. Start at the loss with 1: the loss changes 1 per unit of itself.
2. The −ln step: 1 × (−2) = −2.0. If p grows a little, the loss falls twice as much.
3. The sigmoid step: −2 × 0.25 = −0.5. If z grows a little, the loss falls half as much.
4. The add step passes −0.5 on to both of its inputs, so grad b = −0.5.
5. The multiply step multiplies by x = 2: grad w = −1.0.

Both gradients are negative, so gradient descent will make w and b larger. That raises p, the probability of payment, for a payment ticket: the right direction.

Autograd does the same

In PyTorch, you mark each parameter with requires_grad=True. It means: “keep a record of every step that uses this number, so that you can compute its gradient later.” Then loss.backward() walks the record backwards, and each parameter’s gradient appears in its .grad.

one_neuron.py

import torch

x, y = 2.0, 1.0                                # ticket C: "refund" twice, a payment ticket
w = torch.tensor(0.5, requires_grad=True)      # the weight: keep a record for its gradient
b = torch.tensor(-1.0, requires_grad=True)     # the bias: the same
z = w * x + b
p = torch.sigmoid(z)
loss = -torch.log(p)
print("z =", z)
print("p =", p)
print("loss =", loss)
loss.backward()
print("grad w =", w.grad)
print("grad b =", b.grad)

The output:

z = tensor(0., grad_fn=<AddBackward0>)
p = tensor(0.5000, grad_fn=<SigmoidBackward0>)
loss = tensor(0.6931, grad_fn=<NegBackward0>)
grad w = tensor(-1.)
grad b = tensor(-0.5000)

Now the grad_fn notes of the loss lesson make sense. Each one names the step that made the number: an addition, a sigmoid, a negation. Together they are the computation graph. The gradients are exactly the numbers of the diagram. The notebook also nudges w by 0.001 to each side, as in the last lesson, and gets −0.99996: the same slope, up to rounding.

The ticket network: one gradient checked by hand

The ticket network is a longer line with the same idea. The backward pass starts at the loss and goes through the output layer, the ReLU and the first layer. On the way, it leaves a gradient for every parameter.

One backward pass through the ticket network leaves a gradient for every parameter, with the same shape as the parameter.

Text description of the diagram

Forward, left to right: the word counts of one ticket, shape (1, 2864), go into the first layer, with weight (64, 2864) and bias (64). Then come ReLU and the output layer, with weight (5, 64) and bias (5). The output layer gives 5 scores, shape (1, 5), and then the loss, one number. Backward, right to left, from the loss: the output layer gets a gradient for its weight, shape (5, 64), and its bias, shape (5). The first layer gets a gradient for its weight, shape (64, 2864), and its bias, shape (64).

You can check one of these gradients by hand. For softmax with cross-entropy, the gradient of each score has a simple form. It is the probability minus 1 for the right team, and the probability itself for every other team. The output bias of a team is added to that team’s score, so its gradient is the same number. It is the pattern of the neuron above too: p − 1 = 0.5 − 1 = −0.5 at z.

The untrained network gives the first validation ticket the probabilities 0.172, 0.200, 0.218, 0.201 and 0.209, and the right team is payment. So the five gradients of the output bias should be 0.172, 0.200, 0.218 − 1 = −0.782, 0.201 and 0.209. Priya asks PyTorch:

check_gradient.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
scores = model(x_valid[:1])
loss = loss_fn(scores, y_valid[:1])
loss.backward()
print("probabilities:      ", torch.softmax(scores, dim=1).detach())
print("grad of output bias:", model.layers[3].bias.grad)
for name, weight in model.named_parameters():
    print(f"{name:16} weight {str(tuple(weight.shape)):12} grad {tuple(weight.grad.shape)}")

The output:

probabilities:       tensor([[0.1722, 0.2003, 0.2180, 0.2006, 0.2089]])
grad of output bias: tensor([ 0.1722,  0.2003, -0.7820,  0.2006,  0.2089])
layers.0.weight  weight (64, 2864)   grad (64, 2864)
layers.0.bias    weight (64,)        grad (64,)
layers.3.weight  weight (5, 64)      grad (5, 64)
layers.3.bias    weight (5,)         grad (5,)

.detach() only hides the grad_fn note in the print. model.layers[3] is the output layer: layer 0 is the first nn.Linear, then ReLU (1), dropout (2, switched off in this network) and the output layer (3). named_parameters() lists every parameter with its name.

The hand check agrees with PyTorch to four decimals. The payment bias has a negative gradient, so gradient descent will raise it, and every other bias will fall. The network moves towards payment for this ticket. And every parameter has a gradient of its own shape: 183,685 slopes from one backward pass.

How long did that take? In the notebook, one forward pass on all the training tickets took 0.016 seconds, and a forward and backward pass took 0.017 seconds. The backward pass costs about as much as one more forward pass, however many parameters there are. Nudging every parameter would take about 48 minutes.

When to use autograd, and what it costs

Always use it for training. Writing gradients by hand is slow and error-prone, and nudging is far too slow for a real network. Every layer of torch.nn keeps its record and has its local slopes ready.

What it costs:

• Memory. The record keeps every value of the forward pass until the backward pass has used it. For a large model this is a lot of memory. It is one reason why training needs much more memory than using a model; Inference costs counts it for a real model.
• Gradients add up. backward() adds each new gradient to the number already in .grad. Run the neuron’s forward and backward pass twice, and w.grad is −2.0, not −1.0. That is useful in some cases, but in a training loop you must clear the gradients before each step. The next lesson shows what happens if you forget.
• No record when you do not need one. To use a network without training it, you switch the record off; Batches, epochs and validation shows how and why.

Guided practice: ticket D by hand

Ticket D also says “refund” twice (x = 2), but it goes to returns. For a ticket that is not payment, the loss is −ln(1 − p), and its local slope is 1 ÷ (1 − p).

1. Use w = 0.5 and b = −1. Compute the forward pass: z, p and the loss. You should see z = 0, p = 0.5 and a loss of 0.693.
2. Compute the local slope of the loss for p: 1 ÷ (1 − 0.5). You should see 2.
3. Multiply by the sigmoid’s local slope, p(1 − p) = 0.25. You should see 0.5: the blame at z.
4. Compute grad b (× 1) and grad w (× x = 2). You should see grad b = 0.5 and grad w = 1.0.
5. Check in PyTorch: make w and b with requires_grad=True, compute loss = -torch.log(1 - torch.sigmoid(w * 2.0 + b)), call loss.backward() and print w.grad and b.grad.

Show the answer

PyTorch prints tensor(1.) for w and tensor(0.5000) for b, as by hand. The gradients are positive this time, so gradient descent makes w and b smaller. That lowers p, the probability of payment, for a returns ticket: the right direction. Tickets C and D pull w in opposite directions with the same strength, which is why the best w for the four tickets is a compromise.

Your turn: ticket B

Ticket B says “refund” once (x = 1) and is a payment ticket. With w = 0.5 and b = −1, predict the sign of grad w first. Then compute it with autograd, as in section 9 of the notebook, and by hand with the pattern “p − 1 at z”.

Show the answer

z = 0.5 × 1 − 1 = −0.5, so p = sigmoid(−0.5) = 0.378, and the loss is −ln 0.378 = 0.974. The blame at z is p − 1 = −0.622. Then grad b = −0.622 × 1 and grad w = −0.622 × x = −0.622 × 1. PyTorch gives −0.6225 for both. The sign is negative: raise w and b to make payment more likely for this payment ticket.

Common mistake

No record, no gradient

Symptom: Tomás makes the weight and the bias as plain tensors, w = torch.tensor(0.5), and the forward pass runs. At loss.backward(), PyTorch stops with: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.

Cause: without requires_grad=True, PyTorch keeps no record of the steps, so there is no computation graph to walk back. If only some parameters have it, there is no error at all: the others simply keep .grad as None, and they never learn.

Fix: make every parameter with requires_grad=True, or use the layers of torch.nn (such as nn.Linear), which do it for you. To check, print [name for name, p in model.named_parameters() if p.grad is None] after a backward pass: the list should be empty.

What it means for a real AI service: a network can train without error and still have frozen parts. When a model learns less than you expect, check that every parameter you meant to train has a gradient.

Knowledge check

Check your understanding

In your own words: what does the backward pass do, and why is it so much faster than nudging each weight?

Show the answer

The backward pass walks the record of the forward pass from the loss back to the inputs. At each step it multiplies the blame that arrives by the step’s local slope (the chain rule) and passes the result on. So every parameter gets its gradient on the way. It is fast because it reuses the work: one backward pass costs about one more forward pass, however many parameters there are. Nudging needs one extra forward pass per parameter: 183,685 for the ticket network.

Check your understanding

Predict: in the neuron of this lesson, the input x changes from 2 to 3, with the same w and b. Does grad w change? Explain with the chain rule, without computing every number.

Show the answer

Yes. The last step of the backward pass multiplies the blame at z by x, so a larger x gives a larger grad w. But z changes too: z = 0.5 × 3 − 1 = 0.5, so p = sigmoid(0.5) = 0.622 and the blame at z is p − 1 = −0.378. So grad w = −0.378 × 3 = −1.13, a little larger in size than −1.0. A parameter that meets a larger input gets more of the blame.

Check your understanding

After loss.backward(), the gradient of the payment output bias is −0.782 and the others are positive. What will gradient descent do to the five output biases, and what does that change for this ticket?

Show the answer

Gradient descent moves each bias against its gradient. The payment bias has a negative gradient, so it goes up; the other four have positive gradients, so they go down. The payment score rises and the other scores fall, so softmax gives payment, the right team, a larger probability, and the loss of this ticket falls.

Recap

• The forward pass keeps a record of every step: the computation graph. The backward pass walks it from the loss back, multiplying local slopes (the chain rule), and leaves a gradient for every parameter: backpropagation.
• In PyTorch, requires_grad=True starts the record, loss.backward() runs the backward pass, and .grad holds each gradient. This is autograd.
• One neuron on ticket C: loss 0.693, grad w = −1.0, grad b = −0.5, by hand and by PyTorch. On the ticket network, the output bias gradients are the probabilities minus 1 for the right team: −0.782 for payment.
• One backward pass costs about one forward pass; nudging all 183,685 parameters would take about 48 minutes per step.
• Gradients add up in .grad, so a training loop must clear them before each step.

Next: put the forward pass, the loss, the backward pass and the update together into a training loop, and train the ticket router.

Next: The training loop