Batches, epochs and validation

What you will be able to do

Train the ticket router as real networks are trained: many small steps on shuffled batches, pass after pass. After each pass, measure on the validation tickets. Then read the two curves to choose the best moment to stop.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson and the module practice. It covers batches, the loop with validation, sorted tickets without shuffling, the full 20-epoch run with its curves, and dropout left on. Each run takes a few seconds on a CPU. About 35 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• The five steps of a training step and the optimizer, from The training loop.
• Training, validation and test sets, and overfitting, from Train, validation and test sets in the machine learning course.
• Why small random batches make a loss curve noisy, from Stochastic updates in the mathematics course (optional).

What you need on your computer

• Nothing for the notebook route.
• For the local route: PyTorch and the folder ticket-router, from Set up PyTorch, and the setup lines of The tickets as numbers at the top of each code file.

Tested with Python 3.12 and PyTorch 2.14.1 on a CPU.

Tasting the soup

A cook makes a large pot of soup. To check the salt, she does not eat the whole pot. She stirs it, takes one spoonful and tastes it. A spoonful from a stirred pot tells her almost as much as the whole pot. And she can taste, add a little salt, stir and taste again many times. If she forgot to stir, every spoonful would come from the top, and she would salt the soup for the wrong layer.

Before she serves it, she asks a friend who did not cook it to taste. The friend only tastes. She does not add salt.

The problem: full batches are slow to learn and large in memory

The last lesson’s loop used all 4,696 training tickets in every step: full-batch training. Each step reads every ticket once and makes one update. Priya measured what one, two, five and twenty such passes over the data give on the validation tickets:

Passes over the training tickets	1	2	5	20
Full batch (one step per pass): validation accuracy	33.7%	47.5%	67.5%	88.0%
Batches of 64 (74 steps per pass): validation accuracy	97.2%	98.2%	98.5%	97.3%

With the same reading of the data, small batches give 74 updates instead of one, and the network is at 97.2% after the first pass. Full batches also need memory for every ticket at once: fine for 4,696 short tickets, impossible for millions of images.

The idea: shuffled mini-batches, epochs and a validation pass

A mini-batch, or batch, is a small group of training examples, here 64 tickets. Each training step uses one batch: the five steps on 64 tickets, then the next batch. The loss of one batch is not exactly the loss of all the tickets, so each step is a little noisy, as a spoonful is not exactly the pot. But it points downhill on average, and there are many more steps.

An epoch is one pass through all the training batches, so through every training ticket once. With batches of 64, one epoch of 4,696 tickets is 74 training steps: 73 full batches and a last batch of 24. Shuffling puts the tickets in a new random order at the start of every epoch: the cook stirs the pot.

One epoch: shuffle the 4,696 training tickets, then one training step (one update) per batch of 64. After the epoch, measure on the validation tickets without any update.

Text description of the diagram

The 4,696 training tickets are shuffled into a new order at the start of every epoch. Then they are cut into batches of 64 tickets: batch 1, batch 2, and so on, to batch 74, which has the last 24 tickets. Each batch gives one update of the weights. So one epoch is 74 updates. After the epoch, the network is measured on the validation tickets, and that measurement makes no update.

Why does shuffling matter? Priya sorted the training tickets by team and trained one epoch without shuffling, so every batch held one team only. Each run of one-team batches pulled the network towards that team, and the next team pulled it away. The validation accuracy after one epoch was 40.5%, against 97.0% for the same sorted tickets with shuffling.

Measure without changing: evaluation mode

After each epoch, the network is measured on the validation tickets: the friend tastes, but adds no salt. This evaluation pass must not change the network, and it must use the network as it will be used later. Two PyTorch calls make sure of that:

• model.eval() switches the network to evaluation mode. Some layers act differently during training. A dropout layer, for example, switches off a random part of the neurons at every training step (Overfitting controls in Module 4 explains why that helps). In evaluation mode, dropout is off, and the same ticket always gets the same answer. model.train() switches back.
• with torch.no_grad(): tells PyTorch to keep no record for gradients inside the block. That saves memory and time, and without a record there can be no backward() and no update by accident.

A training step changes the weights; an evaluation pass only measures them. The difference is in the mode, the record and the last two steps.

Text description of the diagram

Left, a training step, which is an update: model.train(), optimizer.zero_grad(), scores = model(batch), loss = loss_fn(scores, labels), then loss.backward(), which computes the gradients, and optimizer.step(), which changes the weights. Right, an evaluation pass, which makes no update: model.eval() switches dropout off, with torch.no_grad(): keeps no gradients, scores = model(valid), loss = loss_fn(scores, labels), and the loss and accuracy are numbers to read. There is no backward pass and no step, so the weights stay the same.

Worked example: the loop with batches

Priya adds the two changes to the training loop: an inner loop over the shuffled batches, and an evaluation pass after each epoch.

train_batches.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
batch_size = 64

for epoch in range(1, 4):
    model.train()                                    # training mode
    order = torch.randperm(len(x_train))             # shuffle: a new order every epoch
    for start in range(0, len(order), batch_size):
        batch = order[start:start + batch_size]      # the row numbers of one batch
        optimizer.zero_grad()
        loss = loss_fn(model(x_train[batch]), y_train[batch])
        loss.backward()
        optimizer.step()
    model.eval()                                     # evaluation mode: dropout off
    with torch.no_grad():                            # no record, no gradients
        scores = model(x_valid)
        valid_loss = loss_fn(scores, y_valid).item()
        valid_acc = (scores.argmax(dim=1) == y_valid).float().mean().item()
    print(f"epoch {epoch}   validation loss {valid_loss:.4f}   accuracy {valid_acc:.4f}")

The output:

epoch 1   validation loss 0.5486   accuracy 0.9717
epoch 2   validation loss 0.2065   accuracy 0.9867
epoch 3   validation loss 0.1374   accuracy 0.9850

torch.randperm(n) gives the numbers 0 to n − 1 in a random order, and x_train[batch] picks those rows: shape (64, 2864) for a full batch. argmax(dim=1) takes the team with the highest score for each ticket. After one epoch of 74 small steps, the network is at 97.2%; the full-batch loop needed 100 steps for 97.8%.

The course’s ticketnet.py has the same loop as train(): the five steps per batch, with the same comments. After every epoch, it runs evaluate() on the training and validation tickets. It returns one row per epoch. Priya trains for 20 epochs:

train_batches.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
history = train(model, x_train, y_train, x_valid, y_valid, epochs=20, log=None)
best = min(history, key=lambda row: row["valid_loss"])
print(history[0])
print(best)
print(history[-1])

The output:

{'epoch': 1, 'train_loss': 0.5345, 'train_acc': 0.9593, 'valid_loss': 0.5527, 'valid_acc': 0.9717}
{'epoch': 12, 'train_loss': 0.0179, 'train_acc': 0.9977, 'valid_loss': 0.0633, 'valid_acc': 0.9833}
{'epoch': 20, 'train_loss': 0.0059, 'train_acc': 0.9996, 'valid_loss': 0.0723, 'valid_acc': 0.9733}

The first epoch’s numbers differ a little from the loop above, because train() sets the seed again before it shuffles, so its batches are different ones. The notebook prints all 20 rows.

Reading the two curves

• Training loss
• Validation loss

Training and validation loss after each epoch of the course's run: bag-of-words network, Adam, learning rate 0.001, batches of 64, seed 0. The validation loss is lowest at epoch 12.

Text description of the chart

Both losses start near 0.55 after epoch 1 and fall fast: about 0.21 after epoch 2 and 0.10 after epoch 4. The training loss keeps falling at every epoch, to 0.018 at epoch 12 and 0.006 at epoch 20. The validation loss falls more slowly after epoch 6, reaches its lowest value, 0.063, at epoch 12, and then rises a little, to 0.072 at epoch 20.

The two losses fall together at first: what the network learns from the training tickets also works on the validation tickets. After about epoch 6, they separate. The training loss keeps falling towards zero, but the validation loss stops falling, reaches its lowest point at epoch 12 (0.063), and then rises.

• Training accuracy
• Validation accuracy

Training and validation accuracy of the same run. The training accuracy climbs to 99.96%; the validation accuracy peaks early and then drifts down.

Text description of the chart

The training accuracy climbs from 95.9% after epoch 1 to 99.8% at epoch 12 and 99.96% from epoch 16 on: almost every training ticket. The validation accuracy is 97.2% after epoch 1, highest at epoch 4 (99.2%), 98.3% at epoch 12, and 97.3% at epoch 20.

By epoch 20, the network gets 99.96% of the training tickets right. But 70 training tickets have a wrong team on purpose, about 1.5%, so a network that understood the task could not reach that. It learns the 70 wrong labels by heart. This is overfitting, the problem of the machine learning course: the network learns details of its training data that do not hold for new tickets. The validation loss shows it first; the validation accuracy moves only a little.

Two habits follow:

• Keep the best epoch, chosen by the validation loss: here epoch 12, with a validation loss of 0.063 and an accuracy of 98.3%. Stopping at the best epoch is called early stopping; Overfitting controls makes it a method, with dropout and weight decay.
• Never choose on the test set. The validation tickets choose the epoch, so their result is now a little optimistic. The 600 test tickets stay sealed until the end of the course.

When to use batches, and what they cost

Always train with shuffled mini-batches. Sizes from 16 to 512 are common; 32 to 128 is a good start on a CPU. Measure on the validation set after every epoch, in evaluation mode.

What they cost: each step sees only a part of the data, so the training loss of each batch jumps up and down. The mathematics course shows why; the trend is what counts. The batch size is one more setting to choose: smaller batches give more, noisier steps; larger batches need more memory. And each evaluation pass takes time, which is small here (600 tickets) and worth it.

Guided practice: one epoch by hand

1. Compute how many batches of 64 make one epoch of 4,696 tickets: divide and round up. You should get 74.
2. Compute the size of the last batch: 4,696 − 73 × 64. You should get 24.
3. In the notebook, run the cell of section 1. You should see batches per epoch: 74 and the shapes (64, 2864) and (24, 2864).
4. Run the loop of section 2. You should see 97.17% after epoch 1.
5. Count the updates: how many training steps did the three epochs make?

Show the answer

4,696 ÷ 64 = 73.4, so 74 batches: 73 full ones (4,672 tickets) and a last batch of 24. Three epochs make 3 × 74 = 222 training steps, and three evaluation passes, which make no update. The full-batch loop of the last lesson made 100 steps, but read the training data 100 times; this loop read it 3 times.

Your turn: when would you stop?

Use the 20 rows of the course’s run (the notebook prints them). Omar asks: “Why not just stop at epoch 4? It has the highest validation accuracy, 99.2%.” Before you answer, look at the validation loss at epoch 4 and at epoch 12.

Show the answer

At epoch 4 the validation loss is 0.105; at epoch 12 it is 0.063. Epoch 4 has 595 of 600 validation tickets right, epoch 12 has 590: a difference of five tickets, small enough to be luck on 600 tickets. The loss looks at every ticket and at how sure the network is, so it is a steadier guide. That is why the course chooses the epoch by the validation loss. Either choice is defensible if you say how you chose, and you choose before you look at the test set.

Common mistake

Measuring with dropout on

Symptom: Tomás trains a network with a dropout layer (dropout 0.5) and then asks it about one validation ticket, a payment ticket, three times. He forgets model.eval(). The three probabilities of payment differ: 0.996, 1.000 and 0.983. He runs the whole validation set twice and gets two different accuracies. Answers that change between runs are the symptom.

Cause: the network is still in training mode, so dropout switches off a different random half of the hidden neurons at every pass. Each pass is a slightly different network. In evaluation mode, the same ticket gets 0.999 every time.

Fix: call model.eval() before you measure or use a network, and model.train() before the next training epoch. Use with torch.no_grad(): for the measurement too. The course’s evaluate() does both.

What it means for a real AI service: training-only behaviour left on in a service gives customers answers that change from one request to the next. And nobody can repeat its measured accuracy. Check the mode wherever a model is loaded to make predictions.

See it end to endDeep network or gradient boosting?Case study

The case study trains a network with this loop, ten seeds and early stopping, and compares it fairly with gradient boosting.

Knowledge check

Check your understanding

What are a batch and an epoch? With batches of 32, how many training steps make one epoch of 4,696 tickets?

Show the answer

A batch is a small group of training examples that one training step uses. An epoch is one pass through all the training examples, batch by batch. With batches of 32: 4,696 ÷ 32 = 146.75, so 147 steps per epoch (146 full batches and a last batch of 24 tickets).

Check your understanding

Predict: Priya trains 40 epochs instead of 20, with everything else the same. What will the training loss and the validation loss do after epoch 20, and which epoch should she keep?

Show the answer

The training loss will keep falling towards zero, because the network keeps learning the training tickets, including the wrong labels, by heart. The validation loss will most likely keep rising slowly, as it did from epoch 12 to 20. She should still keep the epoch with the lowest validation loss, which will be epoch 12 or near it, not the last one. More epochs do not help once the validation loss rises.

Module practice

Guided practice → independent task. You have seen Priya train the 64-neuron ticket network with logged losses and metrics. Now train a smaller network and log it the same way.

1. In section 6 of the notebook, make BagOfWordsNet(len(vocab), hidden=32) after set_seed(0), and print its number of parameters.
2. Train it for 10 epochs with train(), and keep the 10 returned rows in practice.
3. Print the rows as a log: epoch, training loss, validation loss, validation accuracy.
4. Find the best epoch by validation loss, put the row in practice_best, and run the check cell.
5. Write three sentences: the best epoch and its numbers; how this network compares with the 64-neuron one at the same epoch; and whether more epochs might help.

Show a model answer

The network has 91,845 parameters, half of the 183,685 of the 64-neuron network. Its log:

Epoch	1	2	4	6	8	10
Training loss	0.813	0.316	0.146	0.092	0.064	0.045
Validation loss	0.837	0.327	0.144	0.105	0.091	0.077
Validation accuracy	95.5%	98.0%	98.5%	98.5%	98.3%	98.7%

“The best epoch is the last one, epoch 10: validation loss 0.077, accuracy 98.7%. At epoch 10, the 64-neuron network had a validation loss of 0.066 and an accuracy of 98.5%. So the smaller network learns a little more slowly, but it reaches about the same accuracy. Its validation loss was still falling at epoch 10, so a few more epochs may help. I would train to 20 and keep the epoch with the lowest validation loss.”

Module check

Check your understanding

Retrieval: name the five steps of a training step in order, and the two calls that make a measurement on the validation tickets safe.

Show the answer

The five steps: optimizer.zero_grad(), the forward pass model(x), the loss loss_fn(scores, y), loss.backward() and optimizer.step(). For a measurement: model.eval(), which switches off training-only behaviour such as dropout, and with torch.no_grad():, which keeps no record for gradients.

Check your understanding

Explain or predict: the untrained ticket network has a validation loss of 1.624. Why is that number close to ln 5, and what would a loss of 0.063 mean for a typical validation ticket?

Show the answer

The untrained network gives each of the five teams about 0.2, and the loss of one ticket is −ln p of the right team, so about −ln 0.2 = ln 5 = 1.609. A mean loss of 0.063 is the loss of a probability of about e to the power −0.063 = 0.94 for the right team on a typical ticket: the network is sure and mostly right. (The mean hides spread: most tickets have a loss near 0 and a few wrong ones have large losses.)

Check your understanding

Completion check: here are two pieces of code that both run the network on tickets. For each, say whether it is a parameter update or an evaluation-only pass, and which lines decide it. (A) model.train(); optimizer.zero_grad(); loss = loss_fn(model(x_batch), y_batch); loss.backward(); optimizer.step(). (B) model.eval(); with torch.no_grad(): loss = loss_fn(model(x_valid), y_valid).

Show the answer

(A) is a parameter update, a training step. loss.backward() computes a gradient for every parameter and optimizer.step() changes the weights; model.train() keeps training-only behaviour such as dropout on, and zero_grad() clears the old gradients. (B) is an evaluation-only pass. It computes a loss to read, but it has no backward() and no step(), so no weight changes. torch.no_grad() keeps no record, so a backward pass is not even possible. model.eval() switches dropout off, so the answers are the ones the model will give in use. A forward pass alone never changes a network: only step() does.

Recap

This lesson: train in shuffled mini-batches (74 steps of 64 tickets per epoch), measure after each epoch in evaluation mode (model.eval() and torch.no_grad()), and keep the epoch with the lowest validation loss. The course’s run: 97.2% after one epoch, best at epoch 12 (validation loss 0.063, accuracy 98.3%), then overfitting: training accuracy 99.96%, with the 70 wrong labels learned by heart.

This module: a network learns by repeating one cycle. The loss (cross-entropy, −ln p of the right team) says how wrong it is. Backpropagation gives a gradient for every weight from one backward pass. Gradient descent moves every weight against its gradient, scaled by the learning rate. The training loop repeats the five steps, batch after batch. The validation pass checks the result without changing it. You trained the ticket router from 18.2% to 98.3%.

Reset procedure: if a notebook gives other numbers, restart the kernel and run all the cells from the top. Every run starts with set_seed(0), so the numbers repeat. If the setup cell reports a wrong checksum, delete the folder ticket-router and run the setup cell again. On your own computer, check that the virtual environment from Set up PyTorch is active and that python -c "import torch; print(torch.__version__)" prints a 2.x version.

Next: Module 4 diagnoses training problems, starting with the learning rate.

Next: Learning rate and optimizers