Loss and gradients

What you will be able to do

Measure how wrong a model is with one number, and find out in which direction to change the model to make that number smaller. These two ideas, the loss and the gradient, are how every neural network learns.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It calculates the loss and the gradient of the ticket model for any w, and draws the loss curve as text bars. It makes steps of gradient descent with a text trace, and measures two partial derivatives. The guided practice and the independent variation have checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The ticket model minutes = w × length, and the four tickets T-3101 to T-3104, from Functions and change: A model is a function.
• The slope with units, and the derivative as the slope of a tangent, from Functions and change: The slope at one point.
• The mean, and the variance as the mean of squared distances, from Describe a dataset: Spread. The loss in this lesson also squares differences.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file, with the heading ## How models learn. You added the heading in the “Before you start” box of Functions and change. If you cannot find the file, make a new empty file with that name and add the heading: this lesson does not use earlier answers.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

• Training logs. A training script prints lines such as step 200 loss 0.418. The loss is the number that training tries to make small. If it does not go down, the model does not learn.
• Choosing a loss. A library asks you to choose a loss function. Mean squared error (MSE) is the usual choice when the model predicts a number, such as minutes or a price. A different loss, cross-entropy, is usual for classes, such as the team of a ticket.
• Gradients in frameworks. In PyTorch, one line, loss.backward(), computes the gradient for every parameter of the model. Then the optimizer changes each parameter a little. This lesson shows what that line computes, on a model with one parameter.

How wrong is w = 6? The loss

In the last lesson, Tomás’s model was minutes = w × length. The line with w = 6 looked close to the four tickets, but “close” was not a number. Priya makes it a number.

For each ticket, the error is the actual minutes minus the predicted minutes. Then she squares each error, and takes the mean of the squares:

Ticket	Length x	Actual minutes y	Prediction 6 × x	Error y − prediction	Squared error
T-3101	1	7	6	1	1
T-3102	2	13	12	1	1
T-3103	3	17	18	−1	1
T-3104	4	26	24	2	4
				Sum	7

The mean of the squared errors is 7 ÷ 4 = 1.75. This number is the mean squared error (MSE).

A loss is one number that measures how wrong a model is on a set of examples. Lower is better, and 0 means no error at all. Here the loss is the MSE, in squared minutes. The square has two jobs:

• It removes the sign. An error of −1 and an error of +1 are equally wrong. Without the square, they would cancel.
• It makes large errors count more. T-3104 is wrong by 2 minutes, but adds 4 to the sum. One error of 4 minutes would add 16, as much as sixteen errors of 1 minute.

$$L(w) = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - w x_i \right)^2$$

Read it as “the loss at w is the mean, over all n tickets, of the squared difference between the actual minutes and the prediction”. You met the sum symbol in Module 1.

Priya does the same calculation for other values of w:

w	Predictions	Errors	Squared errors	Sum	Loss (MSE)
2	2, 4, 6, 8	5, 9, 11, 18	25, 81, 121, 324	551	137.75
5	5, 10, 15, 20	2, 3, 2, 6	4, 9, 4, 36	53	13.25
6	6, 12, 18, 24	1, 1, −1, 2	1, 1, 1, 4	7	1.75
7	7, 14, 21, 28	0, −1, −4, −2	0, 1, 16, 4	21	5.25

The loss curve

The loss is a function of w: each value of w gives one loss. Draw it for many values, and you get the loss curve:

The loss of the model for each value of w. The curve is a bowl with its bottom at w ≈ 6.27.

Text description of the chart

The horizontal axis is w, from 2 to 10.5. The vertical axis is the loss. The curve is a bowl. At w = 2, the loss is 137.75. It falls to 13.25 at w = 5, 1.75 at w = 6, and its lowest value, about 1.22, at w ≈ 6.27. Then it rises again: 5.25 at w = 7, and about 135.6 at w = 10.5.

The lowest point is at w ≈ 6.27, with a loss of about 1.22. The loss never reaches 0, because no straight line goes through all four dots. For this tiny model, you could find the bottom from the chart. A real model has millions of parameters, so nobody can draw its loss curve. Training must find the bottom without seeing the whole curve. It only knows the slope where it stands.

The gradient: the slope of the loss

Stand at one value of w on the loss curve. The slope of the curve there tells you two things:

• The sign gives the direction. A negative slope means the loss falls to the right, so a larger w is better. A positive slope means a smaller w is better.
• The size gives the steepness. A large slope means the loss changes fast here, so you are probably far from the bottom.

This slope is the gradient: the derivative of the loss with respect to a parameter. With one parameter, the gradient is one number. Its unit is “loss per unit of w”, here squared minutes per (minute per hundred words). In practice, people seldom write that unit, but the meaning stays: change in loss per unit change of w.

Explore

Read the gradient at any w

• Curve
• Tangent at the point

w

2

Loss

137.75

Slope (gradient)

−64

To lower the loss

increase w

The slope is −64: negative, so the loss falls to the right. A larger w lowers the loss.

The loss curve of the ticket model, with the tangent at the chosen w. The readout shows the slope and which way lowers the loss.

Text description of the explorer

The chart shows the loss curve for w from 1 to 11.5. At the start, the point is at w = 2, with a loss of 137.75. The tangent there falls steeply: the slope is −64, so a larger w lowers the loss. The slider, a drag or the arrow keys move the point between w = 2 and w = 10.5. At w = 5, the slope is −19. At w = 6, it is −4. At w ≈ 6.27, it is 0: the bottom of the curve. At w = 7, it is +11, so a smaller w lowers the loss.

For this model, the gradient has a short formula. You do not need to derive it:

$$\frac{dL}{dw} = 15w - 94$$

Read it as “the derivative of the loss with respect to w equals 15 times w, minus 94”. At w = 6, it is 90 − 94 = −4. At w = 2, it is 30 − 94 = −64.

Where the gradient comes from

The gradient is a mean over the tickets, like the loss. Each ticket adds its own part: 2 × (prediction − actual) × length. At w = 6:

Ticket	Prediction − actual	× length	× 2 = this ticket’s part
T-3101	6 − 7 = −1	−1	−2
T-3102	12 − 13 = −1	−2	−4
T-3103	18 − 17 = 1	3	6
T-3104	24 − 26 = −2	−8	−16
		Mean	−16 ÷ 4 = −4

Each ticket pulls w its own way. T-3103 says “make w smaller”, because the model predicts too much for it. The other three say “make w larger”, and T-3104, the longest ticket, pulls hardest. The gradient is the mean of all the pulls. Remember this table: in the next lesson, it explains why real training curves are noisy.

The update rule: step downhill

Now Priya can improve w without seeing the curve. She uses the update rule:

$$w_{\text{new}} = w - \eta \times \frac{dL}{dw}$$

Read it as “the new w is the old w minus the learning rate times the gradient”. The Greek letter η (eta) is the learning rate: a small number, chosen by a person, that sets how far each change goes. One use of the rule is a step. This method is gradient descent: walk down the loss curve, one step at a time.

Two details make the rule work:

• The minus sign turns the direction around. A negative gradient (loss falls to the right) makes w larger.
• The learning rate keeps the step small. At w = 2, the gradient is −64. Without a learning rate, w would jump to 66, far past the bottom.

Worked example: two steps by hand

Priya starts at w = 2 with a learning rate of 0.05. For each step, she computes the gradient (15w − 94), the move (−0.05 × gradient) and the new w:

Step	w before	Gradient 15w − 94	Move −0.05 × gradient	w after	Loss after
1	2	−64	+3.2	5.2	9.75
2	5.2	−16	+0.8	6.0	1.75
3	6.0	−4	+0.2	6.2	1.25

The loss falls from 137.75 to 9.75, then 1.75, then 1.25. The steps get smaller by themselves: the learning rate stays the same, but the gradient shrinks near the bottom, where the curve is flatter. After three steps, w = 6.2 is close to the best value, 6.27.

Grace asks what that means for her. The model now says: each hundred words of a ticket adds about 6.2 minutes of work.

Explore

Gradient descent on the ticket model

Step

8

w

6.267

Loss

1.217

Slope (gradient)

−0.001

Next move = −rate × slope

0

Converged after 8 steps: the slope is about zero, so the steps are about zero too.

The loss curve of the ticket model. Each step moves w by −learning rate × gradient. The lower chart shows the loss after each step, as a training log does.

Text description of the explorer

The upper chart is the loss curve of the ticket model. It is a bowl with its bottom at w ≈ 6.27 and a loss of about 1.217. The ball starts at w = 2. The learning-rate slider offers 13 values from 0.005 to 0.16; it starts at 0.05. Step makes one update, Play repeats it, and Reset goes back to w = 2. The readout shows the step number, w, the loss, the gradient and the next move. With learning rate 0.05, w goes 2, 5.2, 6.0, 6.2, 6.25, and the loss reads 1.217 after five steps. Without JavaScript, the chart shows the first eight steps at 0.05. The lower chart shows the loss after each step: it drops from 137.75 to 9.75 after the first step, then flattens.

Try a few other learning rates in the explorer before you continue. The next lesson explains what you see.

Two parameters: partial derivatives

Real models have more than one parameter. Tomás adds a second one, a fixed number of minutes for every ticket, b:

$$\text{minutes} = w \times \text{length} + b$$

The parameter b is a bias: it moves the whole line up or down. Now the loss depends on two knobs, w and b. To find the direction for each knob, Priya nudges one knob at a time and keeps the other one fixed. She starts at w = 6, b = 0, where the loss is 1.75:

Change	w	b	Loss	Change in loss	Change in loss ÷ nudge	Exact slope (very small nudge)
Start	6	0	1.75
Nudge w by 0.1	6.1	0	1.425	−0.325	−3.25	−4
Nudge b by 0.1	6	0.1	1.61	−0.14	−1.4	−1.5

Each exact slope is a partial derivative: the slope of the loss for one parameter, while all other parameters stay fixed. Mathematicians write it with a curly d, ∂:

$$\frac{\partial L}{\partial w} = -4, \qquad \frac{\partial L}{\partial b} = -1.5$$

Read them as “the partial derivative of the loss with respect to w is −4” and “with respect to b is −1.5”. The gradient is the list of all partial derivatives: here (−4, −1.5). Both are negative, so the update rule makes both w and b a little larger. The same rule runs for each parameter, with its own partial derivative.

A large language model has billions of parameters. Its gradient is a list of billions of partial derivatives, one per parameter. A method called backpropagation computes them all at once, and the update rule changes every parameter a little, at every step. The idea is the same as in the table above.

Guided practice: the loss at w = 7

Check Priya’s table yourself. Write the results in c03-notes.md.

1. For each ticket, calculate the prediction 7 × length. Expected: 7, 14, 21, 28.
2. Calculate each error, actual − prediction. Expected: 0, −1, −4, −2.
3. Square each error. Expected: 0, 1, 16, 4.
4. Add the squares and divide by 4. Expected: 21 ÷ 4 = 5.25.
5. Calculate the gradient at w = 7 with 15w − 94. Expected: 105 − 94 = 11.
6. Write which way the update rule moves w, and why.

Show an example answer

The loss at w = 7 is 5.25, higher than 1.75 at w = 6. The gradient is +11: positive, so the loss rises to the right. The update rule subtracts the learning rate times 11, so w becomes smaller. With a learning rate of 0.05, the move is −0.55, and the new w is 6.45.

Check: one ticket, T-3103, gives 16 of the 21. One large error controls most of the loss.

Independent variation: start on the other side

Start at w = 8 with a learning rate of 0.05.

1. Calculate the loss at w = 8.
2. Make two steps with the update rule. For each step, write the gradient, the move and the new w.
3. Calculate the loss after the second step. Compare the direction of the moves with Priya’s example.

Show an example answer

1. Predictions 8, 16, 24, 32. Errors −1, −3, −7, −6. Squares 1, 9, 49, 36. Sum 95. Loss 95 ÷ 4 = 23.75.
2. Step 1: gradient 15 × 8 − 94 = 26. Move −0.05 × 26 = −1.3. New w = 6.7. Step 2: gradient 15 × 6.7 − 94 = 6.5. Move −0.325. New w = 6.375.
3. Predictions 6.375, 12.75, 19.125, 25.5. Errors 0.625, 0.25, −2.125, 0.5. Squares 0.390625, 0.0625, 4.515625, 0.25. Sum 5.21875. Loss about 1.30.

The moves are negative here, because the gradient is positive: the model predicts too many minutes, so w gets smaller. In Priya’s example, the moves were positive. The update rule finds the direction by itself.

Run Python in your browser

Optional: check the loss and the gradient

This box computes the loss and the gradient for any w. Run it, then add 8 and 6.375 to the list and check your answers. In professional work, a library computes the loss for you, for example mean_squared_error in scikit-learn, or MSELoss in PyTorch.

Python code (you can change it)lengths = [1, 2, 3, 4] # hundreds of words minutes = [7, 13, 17, 26] # actual handling minutes def loss(w): """Mean squared error of the model minutes = w * length.""" squared = [(w * x - y) ** 2 for x, y in zip(lengths, minutes)] return sum(squared) / len(squared) def gradient(w): """Slope of the loss at w: the mean of 2 * (prediction - actual) * length.""" parts = [2 * (w * x - y) * x for x, y in zip(lengths, minutes)] return sum(parts) / len(parts) for w in [5, 6, 7]: print(f"w = {w}: loss {loss(w):.2f}, gradient {gradient(w):.2f}")

Common mistake

Measuring with plain errors: the errors cancel

Symptom: a report says: “The model with w = 6.3 has a mean error of 0 minutes. It is perfect.” But T-3103 is wrong by almost 2 minutes.

Cause: the report takes the mean of the errors without squaring them. At w = 6.3, the errors are 0.7, 0.4, −1.9 and 0.8. They add up to 0, so the mean error is 0, although every prediction is wrong.

Fix: square the errors (MSE), or take their absolute values (mean absolute error), before you take the mean. The MSE at w = 6.3 is 1.225, not 0. A mean error near 0 only shows that the model is not too high or too low overall.

Check your understanding

Why does the loss use squared errors and not the errors themselves? Give both reasons.

Show the answer

First, the square removes the sign, so errors above and below the actual value cannot cancel. Second, it makes large errors count more: an error of 4 adds 16, but an error of 1 adds only 1. A model with one very bad prediction gets a high loss.

Check your understanding

At w = 7, the gradient is +11. Predict: does the update rule make w larger or smaller? With a learning rate of 0.05, what is the new w?

Show the answer

Smaller. The gradient is positive, so the loss rises to the right, and the minus sign in the rule moves w to the left. The move is −0.05 × 11 = −0.55, so the new w is 7 − 0.55 = 6.45.

Check your understanding

A model has two parameters. At the current values, the partial derivatives are −4 for w and +2 for b. What does the update rule do to each parameter, and why?

Show the answer

It makes w larger, because its partial derivative is negative: the loss falls when w grows. It makes b smaller, because its partial derivative is positive: the loss rises when b grows. Each parameter moves against its own partial derivative.

Recap

• The loss is one number that measures how wrong a model is. The mean squared error squares each error, so errors cannot cancel and large errors count more.
• The loss curve shows the loss for each value of a parameter. Training looks for its lowest point.
• The gradient is the slope of the loss. Its sign gives the direction, and its size gives the steepness.
• The update rule, w − learning rate × gradient, makes one step downhill. Steps get smaller near the bottom by themselves.
• With many parameters, each one has a partial derivative. The gradient is the list of them, and the same rule updates every parameter.

Next, you see what happens when the learning rate is too small or too large, and why real training curves are noisy.

Next: Optimization behaviour