Learning rate and optimizers

What you will be able to do

Read a training curve and say whether the learning rate is too small, too large or about right. Then you do not waste an afternoon on a run that cannot work. Then choose an optimizer for the ticket router with a reason, and check the choice with two extra runs.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code block, task and check of this lesson. It trains the bag-of-words router with five learning rates and three optimizers, and plots the curves. Its setup cells download the tickets and the course’s ticketnet.py, so it needs no local setup. About 30 minutes; the training runs take about 1 minute on a free Colab CPU.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The loss, and why an untrained router has a loss near ln 5 = 1.609, from Loss.
• Gradient descent: a step is the learning rate times the slope, from Gradient descent.
• The five steps of the training loop and the course’s train() function, from The training loop.
• Mini-batches, epochs, and training and validation curves, from Batches, epochs and validation.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ticket-router with data/ and ticketnet.py, and its virtual environment with PyTorch, from Set up PyTorch. If python -c "import torch" fails, activate the environment again. If a data file is missing or changed, delete the folder data and download it again as in that lesson.

Every number on this page comes from real runs, PyTorch 2.14.1 on a CPU.

Walking down a hill with different steps

You are on a hillside in thick fog, and you want to reach the bottom of the valley. You cannot see the valley. You can only feel which way the ground slopes under your feet. That was gradient descent in Gradient descent. This lesson is about one question that the fog story left open: how long is each step?

With tiny steps, you always go downhill, but night comes before you arrive. With huge leaps, you jump over the bottom of the valley and land high on the other side. The next leap takes you back across, and perhaps even higher. Only a step size in between gets you down quickly and safely.

The problem: the same network, five different results

Priya trains the bag-of-words router of Module 3 five times. Everything is the same: the tickets, the network, the starting weights (seed 0) and 15 epochs. Only the learning rate changes. She uses plain gradient descent on mini-batches, the optimizer "sgd" of train().

First she loads the tickets and turns them into bag-of-words tensors, as in Module 3:

learning_rates.py

import torch
from torch import nn
from ticketnet import *       # the course's code: Vocab, bag_of_words, BagOfWordsNet, train() ...

train_rows, valid_rows = load_split("data/train.csv"), load_split("data/valid.csv")
vocab = Vocab([r["text"] for r in train_rows])          # words from the training tickets only
x_train, x_valid = bag_of_words(train_rows, vocab), bag_of_words(valid_rows, vocab)
y_train, y_valid = labels(train_rows), labels(valid_rows)
print("x_train", tuple(x_train.shape), " x_valid", tuple(x_valid.shape))

The output:

x_train (4696, 2864)  x_valid (600, 2864)

Then the five runs. set_seed(0) before each network gives every run the same starting weights, so the learning rate is the only difference.

learning_rates.py

runs = {}
for learning_rate in [0.00001, 0.001, 0.1, 1, 10]:
    set_seed(0)                                      # the same starting weights for every run
    model = BagOfWordsNet(len(vocab))
    runs[learning_rate] = train(model, x_train, y_train, x_valid, y_valid, epochs=15,
                                lr=learning_rate, optimizer="sgd", log=None)
    first, last = runs[learning_rate][0], runs[learning_rate][-1]
    print(f"lr {learning_rate:<6g} train loss {first['train_loss']:.3f} -> {last['train_loss']:.3f}"
          f"   valid accuracy {last['valid_acc']:.1%}")

The output:

lr 1e-05  train loss 1.624 -> 1.623   valid accuracy 18.3%
lr 0.001  train loss 1.619 -> 1.569   valid accuracy 39.7%
lr 0.1    train loss 1.188 -> 0.097   valid accuracy 98.0%
lr 1      train loss 1.051 -> 0.024   valid accuracy 97.8%
lr 10     train loss 1.579 -> 2.509   valid accuracy 27.5%

The results go from 18.3% to 98.0% right. Nothing crashed, and no line printed an error. With 0.00001, the loss stays at 1.623, close to ln 5: the router still guesses. With 10, the loss ends higher than it started, and the router gets 27.5% right. That is exactly the share of delivery tickets in the validation set, so it probably answers delivery for almost every ticket. A wrong learning rate is a silent failure. You see it only in the numbers, so you must learn to read them.

The idea: the step size decides the path

In Module 3 you met the rule for one step: the new weight is the old weight minus the learning rate times the slope. The slope says which way is downhill. The learning rate says how far to go. It is a number that you choose before training; the network does not learn it. It is called a rate because it scales every step that the slope asks for.

Take the tiniest possible example: one weight w, and the loss w × w. The bottom of the valley is at w = 0. At any point, the slope of w × w is 2 × w. Start at w = 3 and take four steps with four learning rates:

tiny_descent.py

def descend(learning_rate, w=3.0, steps=4):
    """Walk downhill on the loss w * w, starting at w = 3."""
    path = [w]
    for _ in range(steps):
        slope = 2 * w                     # the slope of w * w at this w
        w = w - learning_rate * slope     # one step: learning rate x slope
        path.append(round(w, 3))
    return path

for learning_rate in [0.1, 0.4, 0.9, 1.1]:
    print(f"learning rate {learning_rate}: w = {descend(learning_rate)}")

The output:

learning rate 0.1: w = [3.0, 2.4, 1.92, 1.536, 1.229]
learning rate 0.4: w = [3.0, 0.6, 0.12, 0.024, 0.005]
learning rate 0.9: w = [3.0, -2.4, 1.92, -1.536, 1.229]
learning rate 1.1: w = [3.0, -3.6, 4.32, -5.184, 6.221]

Check the first step of each by hand. At w = 3, the slope is 6. With learning rate 0.1, the step is 0.6, so w goes to 2.4. With 0.4, the step is 2.4, so w goes to 0.6. With 0.9, the step is 5.4: it jumps over 0 to −2.4. With 1.1, the step is 6.6, so w lands at −3.6, further from 0 than where it started.

• learning rate 0.1
• learning rate 0.4
• learning rate 0.9
• learning rate 1.1

The weight after each of four steps on the loss w × w, from w = 3. Small steps creep towards 0; a good rate gets there; a large one swings across 0; a too-large one swings further out every step.

Text description of the chart

A line chart with steps 0 to 4 across and the weight w up the side, from −6 to 7. All four lines start at 3. Learning rate 0.1 falls slowly: 2.4, 1.92, 1.536, 1.229. Learning rate 0.4 falls fast: 0.6, 0.12, 0.024, 0.005. Learning rate 0.9 zigzags across 0 with shrinking swings: −2.4, 1.92, −1.536, 1.229. Learning rate 1.1 zigzags with growing swings: −3.6, 4.32, −5.184, 6.221.

Four behaviours appear, and you will see all of them on the tickets:

• Too small (0.1 here): every step goes downhill, but it takes many steps to arrive.
• Good (0.4): a few steps reach the bottom.
• Unstable (0.9): each step overshoots. The swings shrink here, but slowly, and the path zigzags.
• Too large (1.1): each step overshoots by more than the last one, so the weight moves away from the bottom. This is called divergence: the training moves away from a solution instead of towards it.

Why is 1 the edge for this loss? Each step multiplies w by 1 − 2 × learning rate. With 0.1 that factor is 0.8; with 1.1 it is −1.2, so the size of w grows by 20% every step. A real network has thousands of weights and a curved, bumpy valley, so nobody can compute its edge in this way. You find it from the curves.

Read the curves: three shapes

Back to the tickets. The training curve and the validation curve of a run tell you what went wrong, in the same way as the zigzags above. Here are three shapes from real runs of the router. Learn to recognise them, because you will meet them in every project.

• Training loss
• Validation loss

Too small, so the network underfits: SGD with learning rate 0.001. Both losses fall a little every epoch and stay far above zero.

Text description of the chart

Both lines start near 1.62 and fall in a straight, gentle slope to 1.569 after 15 epochs. They lie on top of each other. The router gets 39.7% of the validation tickets right.

Underfitting from a learning rate that is too small. The two losses lie on top of each other, and both are high. The network has not learned enough yet, so it is no better on the tickets it learned from than on new ones. Underfitting means that the model has not learned the pattern that is there. The machine learning course showed it with a decision tree that asks one question. Here the network is big enough. It is the steps that are too short. What to do: raise the learning rate by 10, or train much longer.

• Training loss
• Validation loss

Unstable: SGD with learning rate 1. The loss falls fast, then jumps up again, several times.

Text description of the chart

Both lines fall from about 1.05 to about 0.15 in three epochs. Then they jump: up to about 0.35 at epoch 4, down to about 0.09 at epoch 5, and up to about 0.25 at epoch 6. At epoch 7 they reach 1.43, almost back to guessing. Then they fall again, with smaller jumps, and end at 0.024 (training) and 0.087 (validation) at epoch 15.

Unstable from a learning rate that is too large. The loss falls quickly, and then it jumps up again. The epoch-by-epoch numbers show how bad a jump can be:

for row in runs[1][:8]:                               # SGD with learning rate 1, first 8 epochs
    print(f"epoch {row['epoch']}: train loss {row['train_loss']:.3f}, valid accuracy {row['valid_acc']:.1%}")

The output:

epoch 1: train loss 1.051, valid accuracy 50.8%
epoch 2: train loss 0.363, valid accuracy 88.2%
epoch 3: train loss 0.160, valid accuracy 96.2%
epoch 4: train loss 0.323, valid accuracy 85.5%
epoch 5: train loss 0.085, valid accuracy 97.7%
epoch 6: train loss 0.236, valid accuracy 88.8%
epoch 7: train loss 1.433, valid accuracy 28.3%
epoch 8: train loss 0.216, valid accuracy 94.2%

At epoch 7, the router falls from 88.8% to 28.3% right, and one epoch later it is back at 94.2%. This run happens to end well, at 97.8%. But if you had stopped at epoch 7, you would have saved a broken router. A run like this is luck, not a result. What to do: lower the learning rate by 3 to 10 times.

• Training loss
• Validation loss

Overfitting: a router with 512 hidden neurons and Adam's default learning rate, 40 epochs. The training loss falls towards 0; the validation loss falls until epoch 7 and then rises.

Text description of the chart

The training loss falls from 0.159 at epoch 1 to 0.0001 at epoch 40. The validation loss falls from 0.144 to its lowest value, 0.063, at epoch 7. Then it rises slowly and steadily to 0.106 at epoch 40. The gap between the two lines grows every epoch after epoch 7.

Overfitting. The training loss keeps falling, but the validation loss turns and rises. The two curves move apart. The learning rate is fine here: the network learns its training tickets too well, including their accidents. You met this picture in Batches, epochs and validation. The next lesson, Overfitting controls, is about this shape.

So read the two curves together. Both high and flat: the steps are too small. Jumps: the steps are too large. Both falling, then apart: overfitting.

Explore the learning rate

The explorer has all five learning rates of Priya’s SGD runs. Every curve is a real run, recorded during the course’s fact runs. Predict first.

Explore

The learning rate of SGD against the real curves

Curve shape

Good

Valid accuracy, epoch 15

98.0%

Lowest valid loss

0.103 (epoch 14)

Training loss, epoch 1 → 15

1.188 → 0.097

SGD, learning rate 0.1: a good learning rate. The loss falls fast in the first epochs, then slowly. 98.0% of the validation tickets are right after 15 epochs.

• Training loss
• Validation loss

The bag-of-words router trained for 15 epochs with SGD, mini-batches of 64, seed 0. Real training and validation losses of each run (PyTorch 2.14.1, CPU). Losses above 2.5 are drawn at the top edge.

Text description of the explorer

The explorer has one control: the learning rate, with five buttons: 0.00001, 0.001, 0.1, 1 and 10. It shows the curve shape and the validation accuracy after 15 epochs. It also shows the lowest validation loss with its epoch, and the training loss at epochs 1 and 15. The chart shows the training loss and the validation loss for epochs 1 to 15. A dashed line at 1.61 marks the loss of guessing.

0.00001: too small. Training loss 1.624 to 1.623, validation accuracy 18.3%. Both lines stay on the guessing line. 0.001: too small. Training loss 1.619 to 1.569, validation accuracy 39.7%. Both lines fall a little. 0.1: good. Training loss 1.188 to 0.097, validation accuracy 98.0%, lowest validation loss 0.103 at epoch 14. 1: unstable. Training loss 1.051 to 0.024, validation accuracy 97.8%, with jumps at epochs 4, 6 and 7 (up to 1.433). 10: too large. Training loss 1.579 to 2.509, validation accuracy 27.5%. The loss goes up and down above the guessing line and reaches 3.496.

Ten times larger than 0.1 is unstable, and 100 times larger diverges. Ten times smaller hardly moves in 15 epochs. The good range for this network and SGD is narrow: about one power of ten wide.

Optimizers: three ways to take a step

The optimizer is the part of the training loop that turns the gradients into weight changes: step 5, optimizer.step(), in The training loop. Until now you used Adam without comparing it with the others. There are many optimizers. You need three of them, because each one adds one idea to the one before.

SGD: one step from one batch

SGD is the plain rule: step = learning rate × slope. The name means stochastic gradient descent, and “stochastic” means “with chance in it”. Each step uses the slope of one random mini-batch of 64 tickets, not of all 4,696, so each slope is a noisy estimate. Plain SGD has two weaknesses. On a long, gentle slope it creeps, because the slope is small. And one learning rate must suit every weight: the ones with steep slopes and the ones with flat slopes.

Momentum: a ball that keeps rolling

Roll a ball down a long, gentle slope. It does not stop and restart at every bump. It keeps its speed and picks up more. Momentum gives the steps this memory. The optimizer keeps a running sum of the recent steps, the velocity. Each new velocity is 90% of the last velocity plus the new slope. Here it is on a slope that stays at 1, with learning rate 0.1:

momentum_tiny.py

learning_rate, slope = 0.1, 1.0       # a long, gentle slope: the same slope at every step
velocity, sgd_total, momentum_total = 0.0, 0.0, 0.0
for step in range(1, 5):
    velocity = 0.9 * velocity + slope           # momentum keeps 90% of the last push
    sgd_total += learning_rate * slope
    momentum_total += learning_rate * velocity
    print(f"step {step}: SGD moved {sgd_total:.3f}, momentum moved {momentum_total:.3f}")

The output:

step 1: SGD moved 0.100, momentum moved 0.100
step 2: SGD moved 0.200, momentum moved 0.290
step 3: SGD moved 0.300, momentum moved 0.561
step 4: SGD moved 0.400, momentum moved 0.905

After four steps, momentum has gone more than twice as far. On a steady slope, the velocity grows towards 1 ÷ (1 − 0.9) = 10. So momentum can move up to 10 times faster than SGD with the same learning rate. That helps on gentle slopes. It also means that a learning rate that was already large for SGD is far too large with momentum. In train(), optimizer="momentum" is SGD with this 0.9.

Adam: a step size for each weight

Now two weights. The loss changes 1,000 times faster with w[0] than with w[1]: one slope is 10, the other 0.01. Watch the first step of SGD and of Adam, both with learning rate 0.01:

adam_step.py

import torch

for name in ["SGD", "Adam"]:
    w = torch.tensor([1.0, 1.0], requires_grad=True)       # two weights, both start at 1
    optimizer = torch.optim.SGD([w], lr=0.01) if name == "SGD" else torch.optim.Adam([w], lr=0.01)
    loss = 10 * w[0] + 0.01 * w[1]      # w[0] has a steep slope (10), w[1] a flat one (0.01)
    loss.backward()
    optimizer.step()
    steps = (1 - w.detach()).tolist()
    print(f"{name:<4} step of w[0]: {steps[0]:.5f}   step of w[1]: {steps[1]:.5f}")

The output:

SGD  step of w[0]: 0.10000   step of w[1]: 0.00010
Adam step of w[0]: 0.01000   step of w[1]: 0.01000

SGD moves the steep weight 1,000 times more than the flat one. Adam moves both by 0.01, the learning rate. Adam divides each weight’s slope by the typical size of that weight’s recent slopes. So each weight gets its own step size, and the learning rate becomes roughly “how far any weight may move in one step”. Adam also keeps a momentum, like the ball. Its name comes from adaptive moment estimation: it adapts each step to running averages of the slopes, which statisticians call moments.

For the ticket router this matters. A word like “refund” appears in hundreds of tickets, so its weights get a slope in almost every batch. A word like “hammock” appears in a few tickets, so its weights rarely get one. Adam gives the rare words full-size steps when they do appear.

The three optimizers on the tickets

Priya runs each optimizer with two learning rates:

optimizers.py

for optimizer in ["sgd", "momentum", "adam"]:
    for learning_rate in [0.001, 0.1]:
        set_seed(0)
        model = BagOfWordsNet(len(vocab))
        history = train(model, x_train, y_train, x_valid, y_valid, epochs=15,
                        lr=learning_rate, optimizer=optimizer, log=None)
        print(f"{optimizer:<8} lr {learning_rate:<5g} valid accuracy after 1 epoch {history[0]['valid_acc']:.1%},"
              f" after 15 {history[-1]['valid_acc']:.1%}")

The output:

sgd      lr 0.001 valid accuracy after 1 epoch 20.7%, after 15 39.7%
sgd      lr 0.1   valid accuracy after 1 epoch 77.3%, after 15 98.0%
momentum lr 0.001 valid accuracy after 1 epoch 29.0%, after 15 87.7%
momentum lr 0.1   valid accuracy after 1 epoch 92.7%, after 15 97.5%
adam     lr 0.001 valid accuracy after 1 epoch 97.2%, after 15 97.7%
adam     lr 0.1   valid accuracy after 1 epoch 96.2%, after 15 94.5%

At 0.001, SGD is far too slow, momentum is faster, and Adam is right after one epoch. At 0.1, SGD and momentum are good, and Adam is worse than at 0.001. So the same learning rate means a different step size for each optimizer. Each optimizer has its own good range, and the best results of all three are close: 98.0%, 97.5% and 97.7%. The optimizer changes how fast and how safely you get there, more than where you arrive.

Now use the second control of the explorer to compare all fifteen runs.

Explore

Learning rate and optimizer together

Curve shape

Good

Valid accuracy, epoch 15

97.7%

Lowest valid loss

0.063 (epoch 12)

Training loss, epoch 1 → 15

0.534 → 0.011

Adam, learning rate 0.001: a good learning rate. The loss falls fast in the first epochs, then slowly. 97.7% of the validation tickets are right after 15 epochs.

• Training loss
• Validation loss

The same router and the same 15 epochs, with SGD, SGD with momentum 0.9, or Adam, at five learning rates: 15 real runs. Losses above 2.5 are drawn at the top edge.

Text description of the explorer

The explorer has two controls: the learning rate (0.00001, 0.001, 0.1, 1, 10) and the optimizer (SGD, momentum, Adam). It starts at Adam with 0.001. The readouts and the chart are as in the first explorer. Validation accuracy after 15 epochs, and the curve shape:

SGD. At 0.00001: 18.3%, too small. At 0.001: 39.7%, too small. At 0.1: 98.0%, good. At 1: 97.8%, unstable. At 10: 27.5%, too large. Momentum. At 0.00001: 22.3%, too small. At 0.001: 87.7%, too small. At 0.1: 97.5%, good. At 1: 31.7%, too large. At 10: 14.7%, too large; the loss reaches 10.1. Adam. At 0.00001: 80.3%, too small. At 0.001: 97.7%, good, with the lowest validation loss 0.063 at epoch 12. At 0.1: 94.5%, unstable. At 1: 21.2%, too large. At 10: 30.2%, too large; the loss reaches 103.5.

The default, and how to check it

You do not need a search over every optimizer and learning rate. Start with a well-known default and check it. For a new network, the usual default is Adam with learning rate 0.001. It is the default of train() and of most libraries, because it works for many networks without changes.

Then check it with two more runs: one learning rate 10 times smaller and one 10 times larger. Compare the lowest validation loss of each run:

check_default.py

for learning_rate in [0.0001, 0.001, 0.01]:      # the default, 10 times smaller, 10 times larger
    set_seed(0)
    model = BagOfWordsNet(len(vocab))
    history = train(model, x_train, y_train, x_valid, y_valid, epochs=15, lr=learning_rate, log=None)
    best = min(history, key=lambda row: row["valid_loss"])
    print(f"Adam lr {learning_rate:<6g} best valid loss {best['valid_loss']:.3f} at epoch {best['epoch']:>2},"
          f" accuracy then {best['valid_acc']:.1%}")

The output:

Adam lr 0.0001 best valid loss 0.286 at epoch 15, accuracy then 98.5%
Adam lr 0.001  best valid loss 0.063 at epoch 12, accuracy then 98.3%
Adam lr 0.01   best valid loss 0.058 at epoch  2, accuracy then 98.8%

Read the epochs first. At 0.0001, the best epoch is the last one, so the loss was still falling: too slow for 15 epochs. At 0.01, the best epoch is the second one, so the network learns very fast and then starts to overfit. The default sits between them. The lowest losses at 0.001 and 0.01 differ by 0.005. Their accuracies differ by 0.5 points, which is 3 of 600 tickets. That is not a real difference, so Priya keeps the default. A learning rate is good when it is inside the good range, not when it wins by three tickets.

The default is not always right. The course’s average-embedding network in Tokens and embeddings learns better with 0.01. The two extra runs find such cases quickly.

When to choose which. Use Adam with 0.001 first, for any new network. Use SGD with momentum when a well-tested recipe for your kind of network says so. Image networks often use it, with a learning rate that falls during training. Plain SGD is mainly for teaching and for tiny problems.

What it costs. Adam keeps two extra numbers for every weight: the running average of its slopes and of their squares. So the optimizer needs twice the memory of the weights, in addition to the weights and the gradients. For the router, with 183,685 weights, that is nothing. For a large language model it is a real cost, which Module 7 measures in Inference costs.

Guided practice: read the explorer

1. Make your prediction in the first explorer, then select I have made my prediction.
2. Select 0.1. You should see the shape “Good” and 98.0%.
3. Select 1. You should see “Unstable” and three jumps in the curves.
4. Select 10. You should see “Too large”, 27.5%, and the note “off the chart” above the plot.
5. Select 0.00001. You should see “Too small” and two flat lines on the guessing line.
6. In the second explorer, select Adam and 0.1. Write one sentence: which shape is it, and what would you change?

Show an example answer

“Adam with 0.1 is unstable. The training loss jumps from 0.030 at epoch 5 up to 0.133 at epoch 12, and the validation loss climbs from 0.104 to 0.443. The run ends at 94.5% instead of 97.7%. I would lower the learning rate to 0.001, the usual default for Adam, and check 0.01 and 0.0001 next to it.”

Your turn: momentum at learning rate 1

SGD with learning rate 1 is unstable, but it ends at 97.8%. Predict: what happens with momentum at the same learning rate? Use what you learned about the velocity. Then check it in the second explorer.

Show the answer

Momentum at 1 diverges: the shape is “Too large”, and the router ends at 31.7%. On a steady slope, momentum makes the steps up to 10 times longer than SGD with the same learning rate. A learning rate that was already on the edge for SGD is far beyond it with momentum. For momentum, 0.1 is good (97.5%).

Common mistake

A learning rate carried over to another optimizer

Symptom: Tomás changes Priya’s script from SGD to Adam, “because Adam is better”, and keeps her learning rate of 0.1. The router gets worse: 94.5% instead of 98.0%. The validation loss climbs from 0.104 at epoch 2 to 0.443 at epoch 13. He concludes that Adam is worse on tickets.

Cause: a learning rate belongs to an optimizer. For SGD, 0.1 means “a step of 0.1 times the slope”. For Adam, it means “a step of up to about 0.1 for every weight”, which is huge. The curves have the unstable shape.

Fix: when you change the optimizer, start again from that optimizer’s default, 0.001 for Adam. Check it with one run 10 times smaller and one 10 times larger. Adam at 0.001 reaches 97.7%. Never compare two optimizers at one shared learning rate; compare each one at its own good learning rate.

What it means for a real AI service: a “better” optimizer, model or setting can look worse only because another setting did not change with it. Before you drop an option, check that it ran in its own good range.

Knowledge check

Check your understanding

You see a training curve where the training loss and the validation loss lie on top of each other and fall only a little in 15 epochs, from 1.62 to 1.57. What is the most likely cause, and what do you try first?

Show the answer

The learning rate is too small, so the network underfits: it has not learned enough yet. A loss near 1.61 means that the router still almost guesses among five teams. Raise the learning rate by 10 and look at the curve again, or train for many more epochs. A larger network would not help, because the problem is the step size, not the size of the network.

Check your understanding

Predict: on the loss w × w, starting at w = 3, what is w after one step with learning rate 0.5? Why is that special?

Show the answer

The slope at 3 is 2 × 3 = 6, and the step is 0.5 × 6 = 3. So w = 3 − 3 = 0: the bottom, in one step. Each step multiplies w by 1 − 2 × 0.5 = 0. Real networks never have such a simple valley, so no learning rate does this there. It shows that the best step size depends on the shape of the loss.

Check your understanding

Omar asks: Adam reached 97.7% and SGD 98.0%. Is SGD the better optimizer for the router? Answer him in two or three sentences.

Show the answer

No. The difference is 0.3 points, which is 2 of 600 validation tickets, so it is not a real difference. Each optimizer reached about the same accuracy at its own good learning rate. Adam got there after one epoch and needed no search for the learning rate, which is why it is the usual first choice.

Recap

• The learning rate sets the step size: step = learning rate × slope. A wrong one fails silently: SGD gave 18.3% at 0.00001 and 27.5% at 10, against 98.0% at 0.1.
• Three curve shapes. Both losses high and flat: too small, underfitting. Jumps up and down: too large, unstable or divergence. Training falls while validation rises: overfitting, the next lesson.
• The optimizer turns gradients into steps. SGD uses the slope of each mini-batch. Momentum keeps a velocity, so steps grow on long slopes. Adam gives each weight its own step size.
• Each optimizer has its own good learning-rate range. Start with Adam at 0.001, then check 10 times smaller and 10 times larger. Differences of a few tickets are not worth tuning.

Next, the third shape: a router that learns its training tickets too well, and four ways to stop it.

Next: Overfitting controls