Gradient descent

What you will be able to do

Make the loss of a network smaller by changing its weights in small steps, each one downhill. This is how every network in this course learns, from one weight to the 183,685 weights of the ticket router.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It builds the four-ticket network in plain Python, measures the slope by nudging and runs gradient descent by hand. Then it adds a second parameter and runs five steps on the real ticket network. About 25 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The loss, −ln p of the right team, and why training uses it, from Loss: how wrong is the network?
• The slope of a loss curve and the update rule, from Loss and gradients in the mathematics course. This lesson refreshes both in two sentences.
• The logistic function (here called sigmoid), from Classification intuition in the machine learning course.

What you need on your computer

• Nothing for the notebook route or for the explorers on this page.
• For the local route: PyTorch and the folder ticket-router, from Set up PyTorch. The four-ticket example uses plain Python only.

Tested with Python 3.12 and PyTorch 2.14.1 on a CPU.

Walking down a hill in fog

You are on a hillside in thick fog, and you want to reach the bottom of the valley. You cannot see the valley. But you can feel the ground under your feet: it slopes down to the left, or to the right. So you take a step downhill, feel again, and take another step. Where the ground is steep, you take a big step; where it is almost flat, a small one. When the ground is flat in every direction, you are at the bottom.

The problem: 183,685 random numbers

The untrained ticket network has 183,685 parameters: its weights and biases, all set to small random numbers. Its validation loss is 1.624, the loss of an even guess. Training must find better values for all of them. But which way should each one move, and how far?

One idea is to try values and keep the best. Count what that costs. If each parameter may take only 100 values, one parameter needs 100 tries, and two parameters need 100 × 100 = 10,000 tries. The network needs 100^183,685 tries: a number with 367,371 digits. No computer can do that, now or ever. Training needs a way that uses the ground under its feet: the slope.

The idea: step against the slope

The mathematics course showed the two parts. The slope of the loss says how much the loss changes when a parameter grows a little. The update rule moves the parameter against its slope: new value = value − learning rate × slope. (Loss and gradients has the details, with a one-parameter model of handling times.)

For a network, the list of all the slopes, one per parameter, is the gradient. Repeating the update rule until the loss stops falling is gradient descent: “descent” because each step goes down the loss. The learning rate is the number that sets the size of each step. It is a setting that you choose; it is not learned.

A network with one weight

Start with the smallest network possible: one neuron, one input, four made-up tickets. The neuron answers one question: “Is this ticket for the payment team?” Its input x is how many times the ticket says “refund”.

Ticket	Text	x	Payment?
A	My parcel has not arrived yet.	0	no (delivery)
B	You confirmed my refund, but the money is not on my card.	1	yes
C	You confirmed my refund two weeks ago. Where is the refund?	2	yes
D	I sent the lamp back. Can I get a refund, or is there no refund on sale items?	2	no (returns)

Ticket D is the hard one. It says “refund” twice, but it goes to returns, because the shop has not confirmed the return yet (Grace’s routing rules say so). So “more refunds means payment” is only partly true, and the best weight is a compromise.

The neuron computes p = sigmoid(w·x + b), the probability of payment. The bias b stays at −1 for now, so the network has one parameter to learn: the weight w. The loss is the cross-entropy of the last lesson, the mean over the four tickets.

At w = 0, every ticket gets w·x + b = −1, so p = sigmoid(−1) = 0.269. Tickets B and C are payment tickets, so each costs −ln 0.269 = 1.313. Tickets A and D are not. For them, the right answer is “not payment”, whose probability is 1 − p, so each costs −ln(1 − 0.269) = 0.313. The mean is (0.313 + 1.313 + 1.313 + 0.313) ÷ 4 = 0.813.

The slope, by nudging

How does the loss change when w grows a little? Nudge w a tiny bit to each side and compare. Priya writes the loss in plain Python, so that every number is visible:

one_weight.py

import math

refunds = [0, 1, 2, 2]      # how many times each ticket says "refund"
is_payment = [0, 1, 1, 0]   # 1 = the payment team, 0 = another team

def loss(w, b=-1.0):
    """The mean loss of the one-neuron network on the four tickets."""
    total = 0.0
    for x, y in zip(refunds, is_payment):
        p = 1 / (1 + math.exp(-(w * x + b)))       # the neuron: sigmoid(w·x + b)
        total += -math.log(p) if y == 1 else -math.log(1 - p)
    return total / len(refunds)

def slope(w, nudge=1e-6):
    """The slope of the loss at w: nudge w both ways and compare."""
    return (loss(w + nudge) - loss(w - nudge)) / (2 * nudge)

print("slope at w = 0:", round(slope(0), 4))
print("slope at w = 2:", round(slope(2), 4))

The output:

slope at w = 0: -0.4138
slope at w = 2: 0.3853

At w = 0 the slope is negative: the loss falls when w grows, so downhill is to the right. At w = 2 the slope is positive, so downhill is to the left. Either way, the step goes against the slope.

Three steps by hand

Now the update rule, with a learning rate of 1, from w = 0:

Step	w	Loss	Slope	Move = −1 × slope
0	0.000	0.813	−0.414	+0.414
1	0.414	0.684	−0.204	+0.204
2	0.617	0.654	−0.090	+0.090
3	0.708	0.648	−0.041

Each step goes downhill, and each step is smaller than the last, without any change to the learning rate. The steps shrink because the slope gets flatter near the bottom. The lowest loss of this curve is 0.6465, at w ≈ 0.78. In the explorer below, every step is a real PyTorch update of the same network. Its slopes are exact, and the nudged slopes above agree with them to four decimals.

Explore

Gradient descent on one weight

Step

3

Weight w

0.708

Loss

0.648

Slope

−0.041

Next move = −rate × slope

+0.041

Moving downhill. Each step is smaller than the last, because the slope gets smaller near the bottom.

The loss of the one-neuron network on the four tickets, for each value of its weight w (the bias stays at −1). Each step is a real PyTorch update: w ← w − learning rate × slope. The thin black line at the ball shows the slope; the dashed line marks the lowest point.

Text description of the explorer

The explorer shows the loss curve of one weight w, from −1.5 to 6. The curve is a valley: the loss is 1.38 at w = −1, 0.813 at w = 0, lowest (0.6465) at about w = 0.78, 0.93 at w = 2 and 1.36 at w = 3. The learning rate can be 0.1, 0.3, 1, 3 or 10. Each step moves w by −learning rate × slope, from w = 0.

With 0.1, the steps are tiny: w is 0.04, 0.08, 0.12 after three steps, and after 30 steps the loss is still above the bottom. With 0.3, w reaches 0.32 after three steps. With 1: 0.414, 0.617, 0.708, then 0.748, 0.767 … the loss is 0.6466 after five steps. With 3, w jumps past the bottom and back: 1.242, 0.626, 0.882, 0.732, 0.815 … a zig-zag that settles. With 10, the first step jumps to w = 4.14 and the loss rises from 0.813 to 1.908. Then w jumps to −0.75, 5.62, 0.65, 1.39, −1.15 and so on. The loss jumps between 0.65 and 2.7 and never settles.

The learning rate: too small, good, too large

The explorer shows the three kinds of learning rate. Too small (0.1): every step goes downhill, but the steps are so small that 30 steps do not reach the bottom. Good (1): a few steps reach the bottom. Too large (10): the first step jumps far past the bottom, to a point where the loss is higher than at the start, and the steps never settle. In between (3), the steps jump over the bottom and back, but the jumps shrink, so the run still settles.

The optimization lesson of the mathematics course shows the same three cases on a different curve. Learning rate and optimizers shows them on the real ticket network, and how to choose a rate.

Two parameters: the weight and the bias

Now let the bias b learn too. The loss now depends on two numbers, so it is a landscape, not a curve. A contour map draws it from above, like a hiking map: each line joins the points with the same loss. Gradient descent works in the same way. Each parameter gets its own slope (nudge one, keep the other fixed), and each step moves both parameters at once.

Explore

Gradient descent on two parameters

Step

3

Weight w

−0.4

Bias b

0.908

Loss

0.777

Slopes (w, b)

−0.067, 0.098

Moving downhill, but slowly: with this learning rate, each step is small.

The loss of the one-neuron network over its weight w and its bias b, as a contour map: each line joins points with the same loss. The cross marks the lowest point. Each step is a real PyTorch update of both parameters, from w = −1, b = 1.

Text description of the explorer

The map shows w from −1.5 to 3 across and b from −3.5 to 1.5 up. The contour lines are long ovals around the lowest point, at w = 0.77 and b = −0.97 (loss 0.6465). The ovals lie on a slant, so the valley runs from the upper left to the lower right. The start is at w = −1, b = 1, with a loss of 0.908.

With a learning rate of 1, the path goes down the steep side first, then turns and follows the valley floor. The loss is 0.666 after 20 steps, 0.649 after 40 and 0.647 after 60, at w = 0.71 and b = −0.87. That is close to the cross, but not on it, because the valley floor is almost flat. With 0.3, the path is the same shape but slower: after 60 steps the loss is still 0.671. With 3, the path zig-zags across the valley; the loss is 0.687 after 60 steps. With 10, the steps jump across the whole map, the loss reaches 4.2, and the path does not settle.

The path first runs straight down the steep side, at a right angle to the contour lines: that is the steepest way down. Then it turns and walks along the valley floor, where the slope is small, so the steps are small too. After 60 steps the loss is 0.647, close to the lowest 0.6465 but not there. A long, flat valley like this one is common in real networks. It is one reason why the course’s networks use a better optimizer than plain gradient descent, from The training loop onward.

The ticket network: every weight at once

The ticket network works in the same way, with 183,685 parameters instead of two. Its gradient is a list of 183,685 slopes, and each step moves every parameter against its own slope. Priya runs 30 steps of plain gradient descent with a learning rate of 0.5 on all 4,696 training tickets. The code uses a PyTorch tool from the next lesson, which computes all the slopes at once. You write this loop yourself in The training loop.

Plain gradient descent on all 183,685 parameters of the ticket network, learning rate 0.5, all training tickets in every step. Real run, PyTorch 2.14.1 on a CPU.

Text description of the chart

The loss starts at 1.624 and falls at every step for 27 steps: 1.599, 1.578, 1.558, 1.538 in the first steps, faster in the middle (1.389 at step 10, 0.967 at step 20), down to 0.708 at step 27. Then it jumps up: 0.793 at step 28 and 1.143 at step 29.

The loss falls at every step for 27 steps, from 1.624 to 0.708. Then it jumps up, to 1.143. Near that point, a step of 0.5 was too large for this network, as the learning rate of 10 was for the four tickets. Plain gradient descent with one learning rate for every weight is fragile; Learning rate and optimizers shows the fixes.

When to use gradient descent, and what it costs

Neural networks always learn with gradient descent or a variant of it. No other method scales to thousands or billions of parameters. Once you have the slopes, a step costs about the same, however many parameters there are.

What it costs:

• A learning rate to choose. Too small wastes time; too large jumps past the bottom. You find a good one by trying a few and comparing the loss curves.
• No promise of the lowest point. Gradient descent stops where the ground is flat. On a curve with several valleys, that can be a higher valley, a local minimum (the mathematics course shows one). For large networks this matters less than you might expect, but the start and the random seed do change the result a little.
• The slopes themselves. Nudging each parameter to measure its slope needs one extra loss computation per parameter: 183,686 computations for one step of the ticket network. The next lesson shows how PyTorch gets all the slopes from one backward pass instead.

Guided practice: a learning rate of 3

1. In the first explorer, make your prediction, then select I have made my prediction.
2. Select the learning rate 3.
3. Select Step three times. Write down w after each step. You should see 1.242, 0.626 and 0.882.
4. Look at the slope after each step. Write down its sign: + or −.
5. Explain in one sentence why the steps go right, left, right.

Show an example answer

The slopes after the three steps are +0.205, −0.085 and +0.050: the sign changes at every step. Each step is so large that it jumps over the bottom (w ≈ 0.78), so the slope on the other side points back. The jumps get smaller each time, because the slope near the bottom is small, so this run still settles: w is 0.767 after six steps.

Your turn: start on the other side

Start at w = 2 instead of 0, with a learning rate of 1. The slope at w = 2 is +0.385 (from the code above). Predict first: does w grow or shrink? Then compute one step by hand, and check the loss after it with loss(...) in the notebook.

Show the answer

The slope is positive, so w shrinks: new w = 2 − 1 × 0.385 = 1.615. The loss falls from 0.931 at w = 2 to 0.795 at w = 1.615. The rule works from both sides, because the sign of the slope always points uphill and the step goes the other way.

Common mistake

A step uphill

Symptom: Tomás writes the update rule as w = w + rate * slope(w). The loss grows at every step: 0.813, 1.018, 1.369, 1.837. A smaller learning rate makes it grow more slowly, but it still grows.

Cause: a plus sign moves with the slope, so every step goes uphill. The slope points in the direction where the loss grows.

Fix: subtract: w = w - rate * slope(w). In PyTorch, the optimizers of The training loop do the subtraction for you, which removes this mistake.

What it means for a real AI service: a loss that grows steadily from the first step is a bug, not bad luck. Check the update before you change the data or the network.

Knowledge check

Check your understanding

In your own words: what does one step of gradient descent do, and what sets the size of the step?

Show the answer

It measures the slope of the loss for each parameter, then moves each parameter against its slope: new value = value − learning rate × slope. The size of the step is the learning rate times the slope. So a steep slope gives a big step and a flat slope a small one, and the learning rate, which you choose, scales all the steps.

Check your understanding

Predict: at w = 1.5, the slope of the four-ticket loss is +0.286. With a learning rate of 1, what is the next w? With a learning rate of 10?

Show the answer

With a learning rate of 1, the step is −1 × 0.286 = −0.286, so the next w is 1.214. That is closer to the bottom at about 0.78, and the loss falls from 0.760 to 0.691. With a learning rate of 10, the step is −2.86, so the next w is −1.36. That is far past the bottom, on the other side, where the loss is about 1.6: higher than at the start. That is what a learning rate that is too large does.

Check your understanding

Tomás says: 'Why use slopes at all? Just try every value for every weight and keep the best.' What do you tell him?

Show the answer

Count the tries. With only 100 values per parameter, two parameters need 10,000 tries. The ticket network’s 183,685 parameters need 100 to the power 183,685: a number with 367,371 digits. No computer can do that. Gradient descent needs only the slopes at the current point, and one step moves all the parameters at once.

Recap

• Gradient descent changes each parameter against its slope: new value = value − learning rate × slope. The gradient is the list of all the slopes.
• The steps shrink near the bottom on their own, because the slope gets flatter.
• A learning rate that is too small is slow. One that is too large jumps past the bottom, and the loss can get worse than at the start (0.813 → 1.908 for the four tickets).
• With two parameters, the loss is a landscape; gradient descent goes down the steepest way, then along the valley floor.
• On the ticket network, one step moves all 183,685 parameters. Plain gradient descent with a rate of 0.5 brought the loss from 1.624 to 0.708 in 27 steps, then jumped up.

Next: how PyTorch finds all 183,685 slopes at once, without nudging each weight.

Next: Backpropagation and autograd