Overfitting controls

Lesson 13
65 min
Module 4: Diagnose training problems

Lesson 13 of 23 · Deep Learning and Transformers ExplainedOverfitting controls

Course overview

1. What a neural network is

2. Tensors and PyTorch

3. How neural networks learn

4. Diagnose training problems

5. Represent sequences and meaning

6. Assemble a transformer

7. Use pretrained models

8. Under the hood: the mathematicsOptional

9. Case studies

What you will be able to do

Stop the router from learning its training tickets by heart. You will see overfitting in the validation loss even when the accuracy hides it. You will control it with early stopping, dropout and weight decay, and decide when new training examples made from old ones are safe.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code block, task and check of this lesson. It trains the large router with and without each control, keeps the best epoch, and repeats Tomás’s measurement with dropout left on. Its setup cells download the tickets and ticketnet.py, so it needs no local setup. About 35 minutes; the training runs take about 1 minute on a free Colab CPU.

Open in Colab Open in Kaggle View on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 65 minutes

What you need to know

Overfitting and underfitting, from the three students of Control complexity in the machine learning course.
Regularization as a penalty on large weights, from Other limits: leaf size and regularization.
Why each look at the validation set uses it up a little, from Train, validation and test sets.
model.eval() and torch.no_grad(), and the best epoch, from Batches, epochs and validation.
The three curve shapes and Adam’s default learning rate, from Learning rate and optimizers.

What you need on your computer (only for the local route; the notebook needs none of it)

The folder ticket-router with data/, ticketnet.py and PyTorch, from Set up PyTorch, and the tensors x_train, y_train, x_valid, y_valid and vocab from the first code block of Learning rate and optimizers. Run that block again if you start a new session.

Every number on this page comes from real runs, PyTorch 2.14.1 on a CPU. The five runs of 40 epochs take about 40 seconds in total.

The student who learns the answer key

You know this story from the machine learning course. One student learns the methods. Another learns every practice exercise by heart, with its answer, and is lost when the exam asks new questions. A model that does the second thing overfits: it learns the details and accidents of its training examples, and it works worse on new ones. The machine learning course controlled it by limiting a tree’s depth and by regularization, a penalty on large weights.

A neural network can be a very good “answer-key student”, because it has so many weights. Its controls work a little differently, and one of them, dropout, has no match in the machine learning course. First, the problem on the tickets.

The problem: a bigger router that gets worse while it trains

Tomás thinks that a bigger network must be a better one. He gives the router 512 hidden neurons instead of 64, and trains it for 40 epochs with Adam’s default learning rate. He uses "adamw", the version of Adam that weight decay needs later in this lesson; without weight decay, it is the same as Adam.

set_seed(0)
big = BagOfWordsNet(len(vocab), hidden=512)            # 8 times more hidden neurons than before
print("parameters:", count_parameters(big))
history = train(big, x_train, y_train, x_valid, y_valid, epochs=40, optimizer="adamw", log=None)
for row in history:
    if row["epoch"] in (1, 2, 4, 7, 10, 20, 30, 40):
        print(f"epoch {row['epoch']:>2}: train loss {row['train_loss']:.4f}, accuracy {row['train_acc']:.2%}"
              f" | valid loss {row['valid_loss']:.4f}, accuracy {row['valid_acc']:.1%}")

The output:

parameters: 1469445
epoch  1: train loss 0.1588, accuracy 97.47% | valid loss 0.1437, accuracy 98.3%
epoch  2: train loss 0.0815, accuracy 98.51% | valid loss 0.0909, accuracy 99.2%
epoch  4: train loss 0.0293, accuracy 99.53% | valid loss 0.0730, accuracy 98.0%
epoch  7: train loss 0.0080, accuracy 99.96% | valid loss 0.0628, accuracy 97.8%
epoch 10: train loss 0.0034, accuracy 99.98% | valid loss 0.0714, accuracy 97.8%
epoch 20: train loss 0.0006, accuracy 100.00% | valid loss 0.0874, accuracy 97.5%
epoch 30: train loss 0.0002, accuracy 100.00% | valid loss 0.0971, accuracy 97.5%
epoch 40: train loss 0.0001, accuracy 100.00% | valid loss 0.1059, accuracy 97.3%

The network has 1,469,445 parameters for 4,696 training tickets: more than 300 parameters per ticket. It first gets every training ticket right at epoch 9, and from epoch 11 on it always does, including the 70 tickets with a wrong label in the file. Its training loss falls to 0.0001. That is memorising: the only way to give a wrong label a high probability is to remember that one ticket.

Accuracy hides it; the loss shows it

Now look at the validation accuracy. It is highest at epoch 2 (99.2%). From epoch 4 on, it stays between 97.3% and 98.3%. Nothing dramatic happens, so a person who watches only the accuracy sees no problem.

The validation accuracy of the large router over 40 epochs. From epoch 4 on, it stays between 97.3% and 98.3%: the chart hides the overfitting.

Text description of the chart

A line chart from epoch 1 to 40, with the accuracy from 90% to 100%. The line starts at 98.3%, peaks at 99.2% at epoch 2 and is 98.8% at epoch 3. From epoch 4 on, it stays almost flat between 97.3% and 98.3%, and drifts down to 97.3% at epoch 40.

The validation loss tells another story. It falls to 0.063 at epoch 7, then rises, to 0.106 at epoch 40: 70% higher. Why does the loss move while the accuracy does not? The loss measures how sure the router is of the right team, not only whether its top team is right. Priya counts the wrong validation tickets, and how much probability the router gives its wrong answers, at epoch 7 and at epoch 40:

def wrong_and_sure(model):
    """How many validation tickets are wrong, and how sure the network is of those wrong answers."""
    model.eval()
    with torch.no_grad():
        probs = model(x_valid).softmax(1)
    wrong = probs.argmax(1) != y_valid
    return int(wrong.sum()), round(float(probs.max(1).values[wrong].mean()), 3)

set_seed(0)
at_epoch_7 = BagOfWordsNet(len(vocab), hidden=512)
train(at_epoch_7, x_train, y_train, x_valid, y_valid, epochs=7, optimizer="adamw", log=None)
print("epoch 7:  wrong tickets and mean probability of the wrong answer", wrong_and_sure(at_epoch_7))
print("epoch 40: wrong tickets and mean probability of the wrong answer", wrong_and_sure(big))

The output:

epoch 7:  wrong tickets and mean probability of the wrong answer (13, 0.681)
epoch 40: wrong tickets and mean probability of the wrong answer (16, 0.816)

Three more tickets are wrong at epoch 40, but the bigger change is in the probabilities. At epoch 7, the router gives its wrong answers 0.681 on average: it is unsure where it is wrong. At epoch 40, it gives them 0.816. The overfitted router becomes more and more sure of everything, also when it is wrong.

Omar asks what this means for customers. “A router that knows when it is unsure can send those tickets to a person,” says Priya. “The overfitted one is sure of its mistakes, so they go straight to the wrong queue.” That is why you watch the validation loss for overfitting, and report accuracy to people.

How to detect overfitting

Go back to the student with the answer key. Her teacher cannot look inside her head. The teacher can only compare two scores: one on the exercises she practised, and one on a mock exam with new questions. A network gives you the same two scores after every epoch: the training loss and the validation loss. You met them in Reading the two curves. Here you read them on Tomás’s large run. You also add two signals: the gap between the curves, and how sure the router is when it is wrong.

The gap between the two losses

The loss that matters is the loss on tickets that customers will write tomorrow. Nobody can measure it, because those tickets do not exist yet. It has a name, the expected loss, and two losses that you can measure stand in for it:

R(f) = \mathbb{E}_{(x,\,y)}\big[\,\ell(f(x), y)\,\big]

\hat{R}_{\text{train}}(f) = \frac{1}{n}\sum_{i=1}^{n} \ell(f(x_i), y_i)

\hat{R}_{\text{valid}}(f) = \frac{1}{m}\sum_{j=1}^{m} \ell(f(x_j), y_j)

Here $f$ is the router with its weights, and $\ell$ is the loss of one ticket $x$ with its right team $y$ : the cross-entropy of Loss. $R(f)$ is the mean loss over every ticket that the shop could receive. The $\mathbb{E}$ means “the average over all of them”. The hat on $\hat{R}$ means “estimated from a sample”: the mean over the $n = 4{,}696$ training tickets, or over the $m = 600$ validation tickets.

The generalization gap is $R(f) - \hat{R}_{\text{train}}(f)$ : how much worse the router does on new tickets than on the tickets it learned from. The validation loss takes the place of $R(f)$ , so the gap you can measure is:

\text{gap} = \hat{R}_{\text{valid}}(f) - \hat{R}_{\text{train}}(f)

Check it with the numbers of the run. At epoch 7, the gap is 0.0628 − 0.0080 = 0.0548. At epoch 40, it is 0.1059 − 0.0001 = 0.1058. The validation loss is a fair estimate only while the validation set does not choose the router. Choosing the epoch with it uses it a little, so the test set stays untouched.

Priya trains the router again and records five signals after every epoch. train() calls the function that you give as log after each epoch, so she measures there:

def watch(epochs=40):
    """Train the large router again and record five signals after every epoch."""
    set_seed(0)
    model = BagOfWordsNet(len(vocab), hidden=512)
    rows = []

    def after_epoch(row):                                    # train() calls this after each epoch
        train_loss, _ = evaluate(model, x_train, y_train)    # not rounded, unlike row["train_loss"]
        valid_loss, _ = evaluate(model, x_valid, y_valid)
        wrong, sure = wrong_and_sure(model)                  # from the code block above
        with torch.no_grad():
            worst = nn.functional.cross_entropy(model(x_valid), y_valid, reduction="none").max().item()
        rows.append({**row, "gap": valid_loss - train_loss, "sure_when_wrong": sure, "worst": worst})

    train(model, x_train, y_train, x_valid, y_valid, epochs=epochs, optimizer="adamw", log=after_epoch)
    return rows

rows = watch()
for row in rows:
    if row["epoch"] in (1, 2, 5, 7, 9, 20, 40):
        print(f"epoch {row['epoch']:>2}: train {row['train_loss']:.4f}  valid {row['valid_loss']:.4f}"
              f"  gap {row['gap']:+.4f}  accuracy {row['valid_acc']:.2%}"
              f"  sure when wrong {row['sure_when_wrong']:.2f}  worst ticket {row['worst']:.2f}")

The output:

epoch  1: train 0.1588  valid 0.1437  gap -0.0151  accuracy 98.33%  sure when wrong 0.55  worst ticket 3.29
epoch  2: train 0.0815  valid 0.0909  gap +0.0093  accuracy 99.17%  sure when wrong 0.53  worst ticket 2.86
epoch  5: train 0.0173  valid 0.0661  gap +0.0488  accuracy 98.33%  sure when wrong 0.69  worst ticket 2.83
epoch  7: train 0.0080  valid 0.0628  gap +0.0548  accuracy 97.83%  sure when wrong 0.68  worst ticket 4.08
epoch  9: train 0.0042  valid 0.0735  gap +0.0692  accuracy 97.67%  sure when wrong 0.74  worst ticket 6.27
epoch 20: train 0.0006  valid 0.0874  gap +0.0868  accuracy 97.50%  sure when wrong 0.79  worst ticket 9.96
epoch 40: train 0.0001  valid 0.1059  gap +0.1058  accuracy 97.33%  sure when wrong 0.82  worst ticket 12.79

Explore

Watch the large router train, epoch by epoch

Epoch: 40
Training loss: 0.0001
Validation loss: 0.1059
Gap: +0.1058
Validation accuracy: 97.33%
Sure when wrong: 0.82
Worst ticket's loss: 12.79

Epoch 40: the validation loss (0.1059) has been above its lowest (0.0628 at epoch 7) for 33 epochs. With a patience of 5, early stopping stopped after epoch 12 and kept the weights of epoch 7. Accuracy hardly moves: 97.33%, 3 more wrong tickets than at epoch 7. But the router is surer of its wrong answers: 0.82 against 0.68.

Training loss (dashed)
Validation loss
Gap between them

The real run of the large router without a control: 512 hidden neurons, AdamW, seed 0. The shaded area between the curves is the gap; the grey band after the lowest point is where the validation loss is higher than its best.

Text description of the explorer

The explorer has one control, a slider for the epoch from 1 to 40. With JavaScript, it starts at epoch 1, and the chart shows the curves only up to the chosen epoch. Without JavaScript, it shows all 40 epochs. The top chart shows the training loss (dashed) and the validation loss from 0 to 0.16, with the area between them shaded. A black dot marks the lowest validation loss so far. The bottom chart shows the gap, from −0.02 to 0.12. Seven readouts show the values at the chosen epoch. They are the epoch, the two losses, the gap and the validation accuracy. The last two are the mean confidence on wrong validation answers and the worst ticket’s loss.

At epoch 1, the training loss is 0.1588 and the validation loss 0.1437, so the gap (−0.0151) is negative. From epoch 2 to 7, both losses fall and the gap grows (0.0093 to 0.0548). Every one of these epochs sets a new lowest validation loss. Epoch 7 has the lowest of the run (0.0628). After epoch 7, a grey band starts: the validation loss is higher than 0.0628 at every later epoch. With a patience of 5, early stopping stops after epoch 12. At epoch 40, the training loss is 0.0001, the validation loss is 0.1059 and the gap is 0.1058 in the readouts. The validation accuracy goes from 97.83% at epoch 7 to 97.33% at epoch 40. The confidence on wrong answers goes from 0.68 to 0.82, and the worst ticket’s loss from 4.08 to 12.79.

The gap grows before the problem starts

A growing gap looks like the alarm. Priya counts how the two signals move:

def rises(values):
    """How many times a value is higher than the one before it."""
    return sum(b > a for a, b in zip(values, values[1:]))

gaps = [row["gap"] for row in rows]
best = min(rows, key=lambda row: row["valid_loss"])
after_best = [row["valid_loss"] for row in rows[best["epoch"] - 1:]]
print("gap positive from epoch", next(row["epoch"] for row in rows if row["gap"] > 0))
print("gap rises in", rises(gaps), "of", len(gaps) - 1, "steps")
print("lowest validation loss:", best["valid_loss"], "at epoch", best["epoch"])
print("after it, the validation loss rises in", rises(after_best), "of", len(after_best) - 1, "steps")

The output:

gap positive from epoch 2
gap rises in 34 of 39 steps
lowest validation loss: 0.0628 at epoch 7
after it, the validation loss rises in 28 of 33 steps

The gap grows in 34 of 39 steps, from epoch 2 on. But from epoch 2 to epoch 7, the validation loss still falls: the router still gets better on new tickets. The gap grows because the router learns its training tickets faster than it learns the general pattern. Almost every network that learns does this. So a growing gap alone is not the signal to stop.

The signal is the validation loss turning upwards. It is lowest at epoch 7, and higher at every later epoch: it rises in 28 of the 33 steps after it. That is exactly what early stopping watches. The size of the gap still tells you something: it shows how much room a control has. The dropout run later in this lesson has a gap of 0.031 at its best epoch, against 0.055 here.

Why the loss shows it and the accuracy hides it

You saw the confidence on wrong answers at two epochs above. The next chart shows it at every epoch. It rises while the accuracy hardly moves.

On wrong validation answers
On right validation answers

The mean confidence of the large router on its wrong and right validation answers, by epoch. After the best epoch, it becomes surer of its mistakes.

Text description of the chart

A line chart from epoch 1 to 40, confidence from 0.5 to 1. The confidence on right answers starts at 0.90 and climbs to 0.98 at epoch 7 and 0.99 at epoch 40. The confidence on wrong answers starts at 0.55 and is 0.53 at epoch 2. It is 0.68 at epoch 7, 0.79 at epoch 20 and 0.82 at epoch 40. It rises with small ups and downs, because only 10 to 16 tickets are wrong at each epoch.

Why does a sure mistake move the loss so much? The loss of one ticket is minus the logarithm of the probability of its right team:

\ell = -\ln p_{\text{right}}

A probability of 0.5 costs −ln 0.5 = 0.69. The worst validation ticket at epoch 7 has a loss of 4.08: its right team gets a probability of $e^{-4.08} \approx 0.017$ . At epoch 40, the worst ticket’s loss is 12.79: a probability of about 0.000003. The loss has no ceiling, so a sure mistake costs more and more. That one ticket adds 12.79 ÷ 600 = 0.021 to the mean validation loss: a third of the best epoch’s whole loss. The accuracy counts it as one wrong ticket of 600, at epoch 7 and at epoch 40.

A checklist: how to detect overfitting

Watch the validation loss after every epoch, not only at the end. Keep the epoch where it is lowest; early stopping does this for you.
Compare the training loss with the validation loss. Both high: underfitting. A gap that grows while the validation loss falls: the router still learns. The validation loss rises while the training loss falls: overfitting.
Look at the confidence on errors, and at the worst tickets’ losses, not only at the accuracy. A router that becomes sure of its mistakes is memorising.
Check with several seeds before you conclude. This section is one seed and one split. With 600 validation tickets, one ticket is 0.17 percentage points, so differences of 1 to 3 tickets are noise. Train again with two or three other seeds. If the validation loss turns upwards near the same epoch each time, the pattern is real.

Early stopping: keep the best epoch

Bread in an oven is ready at one moment. Take it out too early, and it is raw; leave it in until the timer ends, and it burns. A careful baker watches the bread, not the timer.

Early stopping does the same with training. After each epoch, measure the validation loss. Keep a copy of the weights of the best epoch so far. If the loss has not improved for a few epochs, stop, and use the copy. The number of epochs you wait is called the patience. You need patience because one bad epoch can be chance.

Here is the rule on the real validation losses of epochs 5 to 12 of Tomás’s run, with a patience of 5:

Epoch	5	6	7	8	9	10	11	12
Validation loss	0.0661	0.0644	0.0628	0.0728	0.0735	0.0714	0.0746	0.0757
Epochs without a new best	0	0	0	1	2	3	4	5: stop

Epoch 7 is the best. Epochs 8 to 12 are all worse, so training stops after epoch 12. It keeps the weights of epoch 7: validation loss 0.063 and 97.8% right. It saves 28 epochs of work and gives the best router of the run. You met the best epoch in Batches, epochs and validation; early stopping makes it automatic.

What it costs: the validation set now chooses the epoch, so it is used up a little more, as the machine learning course explained. That is one more reason to keep the test set for the end.

Dropout: nobody can rely on one colleague

Think of a team in which any member may be absent on any day. Nobody can leave a task to one colleague, because that colleague may not come in. So everyone learns a bit of each task, and the team works on any day. Dropout trains a network in the same way.

The problem it solves: with 512 hidden neurons, a few neurons can specialise in single training tickets. One neuron can learn “this exact mix of words is warranty”. That is true for one ticket in the file and useful for no new ticket. The other neurons then rely on it.

The idea: during training, switch off each hidden neuron at random, with a chance p, for each batch. A switched-off neuron outputs 0 for that batch, so the others cannot rely on it. The neurons that stay on are scaled up by 1 ÷ (1 − p), so the total signal stays about the same. The name says what happens: neurons drop out of the network for one step. At evaluation, every neuron is on and nothing is scaled.

nn.Dropout does this. Here it is on the outputs of four hidden neurons, with p = 0.5:

torch.manual_seed(3)
drop = nn.Dropout(p=0.5)                         # switch off each neuron with chance 0.5
h = torch.tensor([0.8, 0.2, 0.5, 0.6])           # the outputs of 4 hidden neurons
drop.train()                                     # training mode: dropout is on
print("training, pass 1:", [round(v, 2) for v in drop(h).tolist()])
print("training, pass 2:", [round(v, 2) for v in drop(h).tolist()])
drop.eval()                                      # evaluation mode: dropout is off
print("evaluation:      ", [round(v, 2) for v in drop(h).tolist()])

The output:

training, pass 1: [1.6, 0.4, 0.0, 1.2]
training, pass 2: [0.0, 0.0, 1.0, 1.2]
evaluation:       [0.8, 0.2, 0.5, 0.6]

In pass 1, the third neuron is off, and the other three are doubled (1 ÷ (1 − 0.5) = 2): 0.8 becomes 1.6. In pass 2, the first two are off. In evaluation mode, the outputs pass through unchanged. The course’s BagOfWordsNet has an nn.Dropout after its ReLU, and dropout=0.5 switches it on.

Three small networks with 3 inputs, 5 hidden neurons and 2 outputs. Training, batch 1: hidden neurons 2 and 4 are dashed circles with a cross and have no connections. Training, batch 2: hidden neurons 1, 3 and 5 are crossed out instead. Evaluation with model.eval(): all 5 hidden neurons are on and connected. — Dropout switches off a different random set of hidden neurons for every training batch. At evaluation, all neurons work together.

What it costs: each training step uses only part of the network, so training needs more epochs to reach the same training loss. And you must switch dropout off when you measure or use the network. The failure case below shows what happens when you forget.

Weight decay: a small cost for every large weight

The machine learning course’s logistic regression had regularization: a penalty on large weights, set with C. Networks use the same idea, under the name weight decay. At every step, the optimizer also shrinks every weight a little towards 0. A weight stays large only if the data keeps pushing it back up. So the network uses large weights only where many tickets agree, and not for one ticket.

A tiny example shows the shrinking. AdamW multiplies each weight by 1 − learning rate × weight decay at each step. Take a weight of 4, a learning rate of 0.1 and a weight decay of 0.5. Each step multiplies the weight by 1 − 0.05 = 0.95. With no push from the data, the weight shrinks: 4, then 3.8, then 3.61, then 3.43. It decays, which gives the method its name. With the real settings (learning rate 0.001, weight decay 0.05), the factor is 0.99995 per step. That is very gentle, but it acts at all 74 steps of every epoch.

In PyTorch, it is one argument: weight_decay=0.05, with the optimizer AdamW. Use AdamW, not Adam, for weight decay. Plain Adam adds the decay to the slope and then divides by each weight’s step size, so the decay becomes uneven.

Compare the controls on the router

Priya trains the large router three times for 40 epochs. The first run has no control, the second has dropout 0.5, and the third has weight decay 0.05. Everything else is the same.

controls = {"no control": dict(dropout=0.0, weight_decay=0.0),
            "dropout 0.5": dict(dropout=0.5, weight_decay=0.0),
            "weight decay 0.05": dict(dropout=0.0, weight_decay=0.05)}
results = {}
for name, c in controls.items():
    set_seed(0)
    model = BagOfWordsNet(len(vocab), hidden=512, dropout=c["dropout"])
    results[name] = train(model, x_train, y_train, x_valid, y_valid, epochs=40, optimizer="adamw",
                          weight_decay=c["weight_decay"], log=None)
    best = min(results[name], key=lambda row: row["valid_loss"])
    last = results[name][-1]
    print(f"{name:<18} best epoch {best['epoch']:>2}: loss {best['valid_loss']:.3f}, accuracy {best['valid_acc']:.1%}"
          f" | epoch 40: loss {last['valid_loss']:.3f}, accuracy {last['valid_acc']:.1%}")

The output:

no control         best epoch  7: loss 0.063, accuracy 97.8% | epoch 40: loss 0.106, accuracy 97.3%
dropout 0.5        best epoch  6: loss 0.056, accuracy 98.7% | epoch 40: loss 0.104, accuracy 97.0%
weight decay 0.05  best epoch  7: loss 0.063, accuracy 97.8% | epoch 40: loss 0.098, accuracy 97.2%

No control
Dropout 0.5
Weight decay 0.05

The validation loss of the large router with no control, with dropout 0.5 and with weight decay 0.05. Dropout gives the lowest point; all three rise after epoch 6 or 7.

Text description of the chart

A line chart from epoch 1 to 40, validation loss from 0 to 0.2. All three lines fall fast in the first epochs. Dropout starts higher (0.193) but reaches the lowest point of the chart, 0.056 at epoch 6. No control and weight decay both reach 0.063 at epoch 7. These two lines lie almost on top of each other. After epoch 7, all three lines rise slowly. At epoch 40: no control 0.106, dropout 0.104, weight decay 0.098.

Read it honestly. Dropout helped a little: the lowest loss (0.056 against 0.063) and 98.7% right, about 5 more tickets of 600. Weight decay 0.05 changed almost nothing at the best epoch. It only slowed the rise a little: 0.098 against 0.106 at epoch 40. And no control stopped the rise: every run overfits after epoch 6 or 7. The control that keeps the good router is early stopping.

So Priya combines dropout with early stopping. She finds the best epoch of the dropout run and trains the router again for exactly that many epochs. train() starts from the same seed, so the second run repeats the first one, epoch by epoch, and stops at the best one:

best = min(results["dropout 0.5"], key=lambda row: row["valid_loss"])
print("best epoch:", best["epoch"])
set_seed(0)
router = BagOfWordsNet(len(vocab), hidden=512, dropout=0.5)
train(router, x_train, y_train, x_valid, y_valid, epochs=best["epoch"], optimizer="adamw", log=None)
loss, acc = evaluate(router, x_valid, y_valid)
print(f"stopped at epoch {best['epoch']}: valid loss {loss:.3f}, accuracy {acc:.1%}")

The output:

best epoch: 6
stopped at epoch 6: valid loss 0.056, accuracy 98.7%

On larger networks, people save the weights at each new best epoch instead of training twice; Save and inspect results shows how to save weights.

Keep it in proportion. The best router of this lesson reaches 98.7% on validation. The machine learning course’s tools, TF-IDF and logistic regression, reached 98.8% on the same tickets, with no epochs to watch. The controls make the network trustworthy; they do not make it better than a simpler model on this data.

When to use which. Use early stopping always: it is free, and it needs only the validation loss that you already compute. Add dropout (0.1 to 0.5) when a large network overfits early. Dropout is also part of most transformer blocks, as Module 6 shows. Add weight decay (0.01 to 0.1 with AdamW) as a mild extra; strong weight decay pushes the network towards underfitting. None of them replaces more and better data.

Data augmentation: only where the label stays true

A child learns the word “cat” from a few pictures. Show her the same picture in a mirror, and it is still a cat, so a mirror picture is a free extra example. Data augmentation makes new training examples from old ones, with changes that keep the right answer. More examples make memorising harder.

For images, many changes keep the label: a shirt flipped from left to right is still a shirt. The case study Classify clothing images uses this flip and measures what it does.

Text is different, because one word can change the meaning. A common text augmentation replaces a word with another word of the same kind. Here is one replacement, and the router’s real answers:

def route(text):
    router.eval()
    with torch.no_grad():
        probs = router(bag_of_words([{"text": text}], vocab)).softmax(1)[0]
    return TEAMS[int(probs.argmax())], round(float(probs.max()), 3)

for text in ["Where is my parcel? It is three days late.",
             "Where is my refund? It is three days late."]:
    print(route(text), text)

The output:

('delivery', 0.934) Where is my parcel? It is three days late.
('delivery', 0.875) Where is my refund? It is three days late.

The first ticket is a delivery ticket. The second one is not: by Grace’s policy, a missing refund goes to returns or payment, never to delivery. An augmentation that turns “parcel” into “refund” and keeps the label delivery writes a wrong label into the training data. Look at the router’s answer, too: it already says delivery (0.875) for the refund ticket, probably because “late” and “days” are delivery words. Augmented copies with the wrong label would teach it more of this mistake.

So for text, use only changes that a person who knows the policy has checked, such as a different order number or a typo. And check a sample of the new examples by hand, as Grace would check a new routing rule.

Common mistake

Measuring with dropout still on

Symptom: Tomás makes the mistake of Batches, epochs and validation, now with the code. He checks the router on one payment ticket three times and gets three different probabilities for payment. “The network is random,” he says. He trained a small router with dropout 0.5 for 8 epochs, and forgot model.eval():

set_seed(0)
small = BagOfWordsNet(len(vocab), dropout=0.5)       # 64 hidden neurons, dropout 0.5
train(small, x_train, y_train, x_valid, y_valid, epochs=8, log=None)
ticket = bag_of_words(valid_rows[:1], vocab)         # one payment ticket, shape (1, 2864)
small.train()                                        # Tomás's mistake: dropout is still on
with torch.no_grad():
    for run in range(3):
        print("training mode,   payment:", round(float(small(ticket).softmax(1)[0, 2]), 3))
small.eval()                                         # the right way to measure
with torch.no_grad():
    for run in range(3):
        print("evaluation mode, payment:", round(float(small(ticket).softmax(1)[0, 2]), 3))

The output:

training mode,   payment: 0.996
training mode,   payment: 1.0
training mode,   payment: 0.983
evaluation mode, payment: 0.999
evaluation mode, payment: 0.999
evaluation mode, payment: 0.999

Cause: in training mode, dropout switches off a different half of the hidden neurons in every pass. So the same ticket gets a different answer each time. train() leaves the network in training mode. On the whole validation set the effect is small but real. At epoch 40, the large dropout router of this lesson scored 97.17% in training mode and 97.0% in evaluation mode. A score that changes when you run it again cannot be compared with anything.

Fix: call model.eval() before you measure or use the network, and model.train() before you train it again. Measure inside torch.no_grad(). The course’s evaluate() does both. Answers that change between runs on the same input are the symptom to look for.

What it means for a real AI service: a deployed router in training mode gives customers different answers for the same ticket. Its measured accuracy does not match what it does. Put model.eval() in the code that loads the model, not in a person’s memory.

Guided practice: choose the stopping epoch

Run the code block of the three controls. You should see the three lines of the output above, with best epochs 7, 6 and 7.
Print the validation losses of the dropout run for epochs 5 to 12: [row["valid_loss"] for row in results["dropout 0.5"][4:12]]. You should see [0.0611, 0.0561, 0.0565, 0.0575, 0.0576, 0.0615, 0.0585, 0.0591].
Apply early stopping with a patience of 5 by hand. Write down the best epoch and the epoch where training stops.
Run the early-stopping code block. You should see stopped at epoch 6: valid loss 0.056, accuracy 98.7%.

Show an example answer

The best validation loss is 0.0561 at epoch 6. Epochs 7 to 11 are all higher: 0.0565, 0.0575, 0.0576, 0.0615 and 0.0585. After epoch 11, there have been 5 epochs without a new best, so training stops. The router keeps the weights of epoch 6: validation loss 0.056 and 98.7% right.

Your turn: much stronger weight decay

Weight decay 0.05 hardly changed anything. Predict what happens with 0.5 and with 5: does the best validation loss go down, and does the best epoch move? Then run it:

for weight_decay in [0.5, 5.0]:                    # 10 and 100 times stronger than 0.05
    set_seed(0)
    model = BagOfWordsNet(len(vocab), hidden=512)
    history = train(model, x_train, y_train, x_valid, y_valid, epochs=40, optimizer="adamw",
                    weight_decay=weight_decay, log=None)
    best, last = min(history, key=lambda row: row["valid_loss"]), history[-1]
    print(f"weight decay {weight_decay:<4} best epoch {best['epoch']:>2}: loss {best['valid_loss']:.3f},"
          f" accuracy {best['valid_acc']:.1%} | epoch 40: loss {last['valid_loss']:.3f},"
          f" train accuracy {last['train_acc']:.1%}")

Show the output and the answer

weight decay 0.5  best epoch  7: loss 0.065, accuracy 98.2% | epoch 40: loss 0.085, train accuracy 100.0%
weight decay 5.0  best epoch 29: loss 0.098, accuracy 98.8% | epoch 40: loss 0.108, train accuracy 99.4%

With 0.5, the rise after the best epoch is smaller: 0.085 at epoch 40, against 0.106 with no control. But the best loss is no better. With 5, the network can no longer memorise: the training accuracy stops at 99.4%, and the best epoch moves to 29. But its best loss, 0.098, is much worse than 0.063. The weights are kept so small that the router is less sure of its right answers too, even though it gets 98.8% right. Strong weight decay trades overfitting for underfitting, and the validation loss shows the trade.

Knowledge check

Check your understanding

Between epoch 7 and epoch 40, the large router's validation accuracy falls only from 97.8% to 97.3%, but its validation loss rises from 0.063 to 0.106. Explain how both can be true.

Show the answer

Accuracy counts only whether the top team is right. The loss also measures how much probability the router gives the right team. After epoch 7, the router becomes more and more sure of every answer, also of its wrong ones. The mean probability of its wrong answers grows from 0.681 to 0.816. Confident wrong answers make the loss grow fast, while the number of wrong answers changes by only three tickets. So watch the loss for overfitting.

Check your understanding

Predict: the validation losses of epochs 1 to 8 are 0.20, 0.12, 0.09, 0.10, 0.08, 0.09, 0.11, 0.12. With a patience of 2, at which epoch does training stop, and which epoch's weights do you keep?

Show the answer

The best loss is 0.09 at epoch 3, then 0.08 at epoch 5. Epoch 4 (0.10) was one epoch without a new best, but epoch 5 set a new best, so the count starts again. Epochs 6 (0.09) and 7 (0.11) are two epochs without a new best. So training stops after epoch 7, and you keep the weights of epoch 5.

Check your understanding

Grace wants more training tickets for the warranty team. Tomás suggests copying each warranty ticket and replacing the product name with another product, for example kettle with hammock. Is that a safe augmentation? What would you check?

Show the answer

Probably safe: by the policy, the team depends on what happened (a fault after some use), not on which product it is. So “the kettle stopped working after a month” and “the hammock stopped working after a month” are both warranty. But check a sample by hand: some replacements make a ticket strange, for example a hammock that “stops heating”. Never replace words that decide the team, such as “arrived broken” against “broke after use”, or “parcel” against “refund”.

Recap

A large network can memorise its training tickets. With 1,469,445 parameters, it reached 100% on training, including the 70 wrong labels. Meanwhile, the validation loss rose from 0.063 (epoch 7) to 0.106 (epoch 40).
Accuracy hides overfitting; the loss shows it. The overfitted router is sure of its wrong answers (0.816 against 0.681). One sure mistake can cost a loss of 12.79 (−ln p).
Detect it from evidence: the gap (validation loss − training loss) grows from epoch 2, while the validation loss still falls until epoch 7. Stop on the validation loss turning upwards, not on the gap; check with more than one seed.
Early stopping keeps the epoch with the lowest validation loss; a patience of a few epochs ignores one bad epoch. Dropout switches off random hidden neurons in training, so that no neuron can be relied on (best: 98.7%). Weight decay shrinks every weight a little at each step; 0.05 changed little here.
Data augmentation is safe only where the label stays true: a flipped shirt is still a shirt, but “parcel” to “refund” changes the team.
Measure with model.eval() and torch.no_grad(): with dropout left on, the same ticket gets a different answer every time.

Next, a checklist for when a training run fails and you do not know why.

Next: Debug the experiment