Debug the experiment

What you will be able to do

A training run can give a bad router and no error message. Find the cause in a fixed order, from the cheapest check to the most technical one. Then repair it and show the evidence, from the curves or the gradients, that the repair worked.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code block, task and check of this lesson, and the module practice. It writes Tomás’s broken training script into your folder, so that you can run it, find the two bugs and repair them. Its setup cells download the tickets and ticketnet.py, so it needs no local setup. About 45 minutes; every run takes seconds on a free Colab CPU.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Backpropagation as passing the blame backwards, and .grad, from Backpropagation and autograd.
• The five steps of the training loop, from The training loop.
• The three curve shapes and the learning rate, from Learning rate and optimizers, and overfitting and the validation loss, from Overfitting controls.
• Cross-validation and folds, from Train, validation and test sets, and leakage, from Fit transformations correctly in the SQL and data preparation course.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ticket-router with data/, ticketnet.py and PyTorch, from Set up PyTorch, and the tensors of the first code block of Learning rate and optimizers.
• data/split_manifest.json, the list of the planted wrong labels. If your data folder does not have it, download it from the labs repository into ticket-router/data.
• For the module practice: Tomás’s script broken_train.py, saved in ticket-router.

Every number on this page comes from real runs, PyTorch 2.14.1 on a CPU.

A bicycle that does not ride well

Your bicycle is hard to ride. You could take the gears apart first, but a careful mechanic starts with the tyres. They are quick to check, and flat tyres are common. Then the chain, then the brakes rubbing on the wheel, and only then the gears. Each check is cheap, and each one rules out a part. The order saves time.

Training a network needs the same habit. When a run fails, it rarely shows an error. The loss is too high, or the accuracy is near chance, or a good run suddenly becomes bad. The same symptom can have many causes: wrong labels, broken inputs, a bug in the code, steps that are too large. Tomás’s way is to change the learning rate, add a layer, and try again. Sometimes it works, and nobody learns why. Priya uses a checklist.

The problem: a symptom with many causes

Here is a run that looks like overfitting at first. Priya gives each training ticket the team of another, random training ticket, and trains the router for 10 epochs. This is a test: it shows what a run with useless labels looks like.

debug.py

g = torch.Generator().manual_seed(0)
shuffled_labels = y_train[torch.randperm(len(y_train), generator=g)]   # each ticket gets a random ticket's team
set_seed(0)
model = BagOfWordsNet(len(vocab))
history = train(model, x_train, shuffled_labels, x_valid, y_valid, epochs=10, log=None)
for row in history[::3]:
    print(f"epoch {row['epoch']:>2}: train accuracy {row['train_acc']:.1%}, valid accuracy {row['valid_acc']:.1%}")

The output:

epoch  1: train accuracy 28.4%, valid accuracy 25.5%
epoch  4: train accuracy 52.0%, valid accuracy 24.8%
epoch  7: train accuracy 68.7%, valid accuracy 20.7%
epoch 10: train accuracy 80.0%, valid accuracy 19.2%

• Training accuracy
• Validation accuracy

The router trained on shuffled labels. It learns 80% of the meaningless training labels by heart, while the validation accuracy stays at the level of chance.

Text description of the chart

A line chart over 10 epochs, accuracy from 0% to 100%. The training accuracy climbs steadily from 28.4% to 80.0%. The validation accuracy starts at 25.5% and slowly falls to 19.2%, about the 20% of guessing among five teams.

The network learns 80% of the meaningless labels by heart, while the validation accuracy stays at about 20%, the level of guessing. There is nothing to learn, so it memorises. Remember this picture: training goes up, validation stays at chance. Overfitting looks similar, but its validation accuracy starts high. When validation never leaves chance, suspect the labels or the code before you add dropout.

The checklist

Priya checks in this order. The early checks are cheap and find the most common bugs; the later ones need more knowledge.

The debugging checklist: four checks in order, each with what a failure looks like and what to repair. Tune only after all four pass.

Text description of the diagram

The checklist starts at “A run fails and you do not know why”. Check 1: look at examples and labels; print a few tickets with their teams. If the labels are wrong or shifted, repair the labels or the code that pairs them. Check 2: check the inputs: shapes, value ranges, unknown words, and a vocabulary built from the training tickets only. If there is a leak, a wrong shape or huge values, repair the preparation. Check 3: overfit a tiny batch: can the network reach 100% on 20 tickets? If not, the bug is in the code. Check 4: check the gradients: the gradient norm of each layer. If they are huge or zero, lower the learning rate; if they are near zero in the early layers, change the activation. When all four checks pass, tune the learning rate and the overfitting controls.

1. Look at examples and labels

Print a few training tickets next to their labels, and read them. It takes one minute, and it finds the worst bugs: labels that belong to other tickets, empty texts, a wrong column.

The file also has a quieter problem. Like a real help desk, 1.5% of the training tickets have a wrong team: 70 tickets. You cannot read 4,696 tickets to find them. Can the network point to them? Tomás has an idea: “Train the network, then look at the training tickets that it gets wrong. Those are the suspicious labels.”

labels.py

import json
noisy = set(json.load(open("data/split_manifest.json"))["noisy_train_ids"])   # the 70 planted wrong labels
set_seed(0)
full = BagOfWordsNet(len(vocab))
train(full, x_train, y_train, x_valid, y_valid, epochs=20, log=None)   # trained on all training rows
full.eval()
with torch.no_grad():
    wrong = (full(x_train).argmax(1) != y_train).nonzero().flatten().tolist()
print("training tickets the network gets wrong:", len(wrong))
print("of these, planted wrong labels:", sum(train_rows[i]["ticket_id"] in noisy for i in wrong))

For this course, the file split_manifest.json lists the IDs of the 70 planted wrong labels, so you can check any method. A real project has no such list.

The output:

training tickets the network gets wrong: 2
of these, planted wrong labels: 2

The network gets only 2 training tickets wrong, so it finds only 2 of the 70. It learned the other 68 wrong labels by heart, as the last lesson showed. A network cannot tell you which of its training labels are wrong if it has memorised them.

The fix is an idea from the machine learning course: cross-validation. Cut the training tickets into 5 folds. Train 5 networks, each on 4 folds, and let each one score the fold that it did not see. Every ticket then gets a probability from a network that never learned its label. These are out-of-fold predictions. A label that such a network finds very unlikely is a suspect.

labels.py

order = torch.randperm(len(train_rows), generator=torch.Generator().manual_seed(0))
probs = torch.zeros(len(train_rows), len(TEAMS))
for k in range(5):                                   # 5 folds
    held_out = order[k::5]                            # every 5th ticket: this fold
    kept = torch.tensor(sorted(set(order.tolist()) - set(held_out.tolist())))
    set_seed(k)
    model = BagOfWordsNet(len(vocab))
    train(model, x_train[kept], y_train[kept], x_train[held_out], y_train[held_out], epochs=8, log=None)
    model.eval()
    with torch.no_grad():
        probs[held_out] = model(x_train[held_out]).softmax(1)   # scored by a model that never saw them
p_given = probs[torch.arange(len(train_rows)), y_train]       # the probability of the label in the file
suspects = p_given.argsort()[:50].tolist()                    # the 50 least likely labels
for i in suspects[:3]:
    r = train_rows[i]
    print(r["ticket_id"], "label:", r["team"], " network:", TEAMS[int(probs[i].argmax())],
          f" p(label) {float(p_given[i]):.3f} ", r["text"][:60] + "...")

The output:

T-60623 label: warranty  network: delivery  p(label) 0.000  I had a very helpful chat with my neighbour last week, and s...
T-60552 label: payment  network: returns  p(label) 0.000  I got the wrong item for my log store order LK-160935. The b...
T-62277 label: warranty  network: delivery  p(label) 0.000  Hi there! So I was really looking forward to getting my hamm...

The first suspect is about a parcel that the courier left at the side gate: delivery, not warranty. The second is a wrong item to send back: returns, not payment. How many of the 50 suspects are planted wrong labels?

labels.py

print("planted wrong labels:", len(noisy))
print("among the 50 suspects:", sum(train_rows[i]["ticket_id"] in noisy for i in suspects))
print("among the 100 lowest:", sum(train_rows[i]["ticket_id"] in noisy for i in p_given.argsort()[:100].tolist()))

The output:

planted wrong labels: 70
among the 50 suspects: 49
among the 100 lowest: 69

Two ways to look for the 70 wrong training labels. Full network: the training errors of the network trained on all rows (2). Out-of-fold: the 50 least likely labels from out-of-fold predictions (49).

Text description of the chart

A column chart with two columns, on an axis from 0 to 70. The training errors of the network trained on all training tickets contain 2 of the 70 planted wrong labels. The 50 labels with the lowest out-of-fold probability contain 49 of them.

49 of the 50 suspects are planted wrong labels, and the 100 lowest contain 69 of the 70. The same network, used in a different way, went from 2 to 49. The rule behind it: judge a label only with a model that did not learn it.

What should Priya do with the suspects? Not delete them automatically. Grace reads them and corrects the labels that are wrong. The out-of-fold networks disagree with 180 training labels in total. Most of those are hard tickets, not wrong labels, so a person must decide.

2. Check the inputs

Next, look at what the network receives. Print the shape and the range of the values:

inputs.py

print("input shape:", tuple(x_train.shape), " values from", int(x_train.min()), "to", int(x_train.max()))
print("after log1p, the network sees values up to", round(float(torch.log1p(x_train).max()), 2))

The output:

input shape: (4696, 2864)  values from 0 to 16
after log1p, the network sees values up to 2.83

One ticket has a word 16 times. The course’s BagOfWordsNet first takes log1p of the counts: the logarithm of 1 + count. So 16 becomes 2.83, and a repeated word does not shout 16 times louder than the others. A network gets unstable when some inputs are hundreds of times larger than others, so check the range whenever you prepare a new input.

Then check for leakage: information from the validation or test tickets that reached training. Fit transformations correctly in the SQL and data preparation course measured how leakage makes a score too good. In a text network, the vocabulary is a step that learns from data. Tomás builds it from the training and validation tickets, “to have more words”:

inputs.py

def unknown_share(rows, vocab):
    ids = [t for r in rows for t in vocab.encode(r["text"])]
    return sum(t == 1 for t in ids) / len(ids)            # ID 1 is <unk>

leaky = Vocab([r["text"] for r in train_rows + valid_rows])  # the mistake: valid words in the vocabulary
print("train only:   ", len(vocab), "words,", f"{unknown_share(valid_rows, vocab):.1%} of valid tokens unknown")
print("train + valid:", len(leaky), "words,", f"{unknown_share(valid_rows, leaky):.1%} of valid tokens unknown")

The output:

train only:    2864 words, 1.2% of valid tokens unknown
train + valid: 2987 words, 0.8% of valid tokens unknown

vocab.encode() turns a text into the positions of its tokens in the vocabulary, as stoi did in The forward pass. The leaky vocabulary has 123 words that only the validation tickets brought in, so fewer validation words are unknown. New tickets tomorrow will not get that help. Here the effect on the score is small, but it is still a leak, and no curve shows it. You find it only by checking how each input was built: every step that learns from data uses the training tickets only.

3. Overfit a tiny batch

Before a long trip, you drive the car around the block. If it does not get around the block, the trip will not work either. The same holds for a network. Take 20 training tickets and train on them alone, many times. A working network and training loop must learn 20 tickets by heart:

tiny_batch.py

set_seed(0)
tiny_model = BagOfWordsNet(len(vocab))
tiny = torch.arange(20)                               # the first 20 training tickets
tiny_history = train(tiny_model, x_train[tiny], y_train[tiny], x_train[tiny], y_train[tiny],
                     epochs=60, batch_size=20, lr=0.01, log=None)
for row in tiny_history[::10] + [tiny_history[-1]]:
    print(f"step {row['epoch']:>2}: loss {row['train_loss']:.4f}, accuracy {row['train_acc']:.0%}")

The output:

step  1: loss 1.3483, accuracy 80%
step 11: loss 0.0256, accuracy 100%
step 21: loss 0.0007, accuracy 100%
step 31: loss 0.0001, accuracy 100%
step 41: loss 0.0000, accuracy 100%
step 51: loss 0.0000, accuracy 100%
step 60: loss 0.0000, accuracy 100%

The batch is the whole set of 20, so each epoch is one step. The output shows every tenth step; from step 5 on, the network gets all 20 right, and the loss falls to 0.0000. The code can learn. If this check fails, no amount of data or tuning will help. Look for the bug in the model, the loss, the optimizer, or the code that pairs tickets with labels.

Memorising is the bad result in the last lesson, and the good result here. On 20 tickets, memorising proves that the network and the training loop work.

4. Check the gradients

The gradients carry the blame backwards through the network, as in Backpropagation and autograd. They should have a sensible size in every layer. Measure each layer’s gradient norm: one number for the size of all the gradients of a layer. It is the square root of the sum of their squares. Think of it as the length of the arrow that says “move this way”.

Exploding gradients. With a learning rate that is far too large, one step throws the weights very far. Then the next loss is enormous, and so are the next gradients. Here is Adam with learning rate 10, 10,000 times its default, for four steps:

gradients.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
optimizer = torch.optim.Adam(model.parameters(), lr=10)     # 10 000 times the default
loss_fn = nn.CrossEntropyLoss()
for step in range(4):
    batch = torch.arange(step * 64, step * 64 + 64)
    optimizer.zero_grad()
    loss = loss_fn(model(x_train[batch]), y_train[batch])
    loss.backward()
    print(f"step {step}: loss {loss.item():10.3f}   gradient norm, first layer {model.layers[0].weight.grad.norm().item():8.3f}")
    optimizer.step()

The output:

step 0: loss      1.616   gradient norm, first layer    0.354
step 1: loss  33688.914   gradient norm, first layer  209.409
step 2: loss  24560.604   gradient norm, first layer  121.913
step 3: loss  19077.770   gradient norm, first layer  101.325

After one step, the loss jumps from 1.616 to 33,689, and the gradient norm grows almost 600 times. In some networks the numbers grow until they become nan (“not a number”); here they stay huge. The repair: lower the learning rate. Some training code also caps the gradient norm, with torch.nn.utils.clip_grad_norm_, so that one bad batch cannot throw the weights far.

Vanishing gradients. The opposite problem is gradients that become tiny in the early layers, so those layers hardly learn. Priya builds a deep stack of 10 linear layers, with an activation after each of the first 9. She measures each layer’s gradient norm after one backward pass. She tries the sigmoid of the machine learning course and the ReLU of this course:

gradients.py

def gradient_norms(activation):
    """A stack of 10 linear layers; one backward pass; the gradient size of each layer."""
    set_seed(0)
    layers = [nn.Linear(len(vocab), 64), activation()]
    for _ in range(8):
        layers += [nn.Linear(64, 64), activation()]
    stack = nn.Sequential(*layers, nn.Linear(64, 5))
    loss = nn.CrossEntropyLoss()(stack(torch.log1p(x_train[:256])), y_train[:256])
    loss.backward()
    return [layer.weight.grad.norm().item() for layer in stack if isinstance(layer, nn.Linear)]

for name, activation in [("sigmoid", nn.Sigmoid), ("ReLU", nn.ReLU)]:
    norms = gradient_norms(activation)
    print(f"{name:<8} first layer {norms[0]:.2e}   last layer {norms[-1]:.2e}")

The output:

sigmoid  first layer 1.45e-08   last layer 6.24e-01
ReLU     first layer 1.31e-04   last layer 2.05e-02

1.45e-08 means 1.45 × 10⁻⁸, or 0.0000000145. With sigmoid, the first layer’s gradient is about 40 million times smaller than the last layer’s. The chart shows every layer:

• Sigmoid
• ReLU

The gradient norm of each of the 10 layers after one backward pass, on a scale where each step up is 10 times larger. With sigmoid, the gradient shrinks about 7 times in every layer on its way back.

Text description of the chart

A line chart of the gradient norm of layers 1 to 10, on a logarithmic scale from 10⁻⁸ to 1. The sigmoid line falls in an almost straight line, from 0.62 at layer 10 to 0.0000000145 at layer 1. That is about 7 times smaller per layer. The ReLU line falls from 0.021 at layer 10 to about 0.00002 at layer 2, and is 0.00013 at layer 1. It also shrinks, but the first layer’s gradient is about 9,000 times larger than with sigmoid.

Why does sigmoid do this? Its slope is never more than 0.25. Backpropagation multiplies by that slope at every layer on the way back, so after 9 sigmoids the blame has shrunk to almost nothing. The first layers then hardly move. ReLU has a slope of 1 for every positive input, so it loses much less. This is one reason why deep networks use ReLU-like activations. Module 6 adds two more defences: residual connections and normalization, inside every transformer block.

When to look at gradients: when the loss jumps or becomes nan, or when the loss hardly moves although checks 1 to 3 passed. Print the norm of every layer for a few steps. A value that is huge, zero, or millions of times smaller than its neighbours points to the problem.

Common mistake

Tuning before checking

Symptom: Tomás’s router stays near 20% on validation. He tries five learning rates, adds dropout and a second hidden layer, and spends a day. Nothing helps.

Cause: the labels in his training tensor belonged to other tickets. With labels like that, no learning rate and no architecture can work. The run looks like the shuffled-labels picture above: training accuracy rises, and validation accuracy stays at chance.

Fix: follow the checklist in order before you tune anything. Print five tickets with their labels (check 1). Then check the inputs, overfit 20 tickets, and look at the gradients. Each check takes minutes.

What it means for a real AI service: a model that is “hard to train” often has a data or code bug. Tuning hides the bug for a while, and the bug ships with the model.

Module practice

Now repair a real failure, with the checklist. The worked examples above showed each check with its intermediate results; here you use them yourself.

Tomás wrote broken_train.py. It trains the bag-of-words router for 8 epochs with Adam. After each epoch, it prints the losses, the accuracies and the gradient norm of the first layer. It runs without an error, but the router is bad. The script has two bugs.

1. Get the script. In the notebook, run the cell that writes broken_train.py. Locally, save broken_train.py in your ticket-router folder.
2. Run it: python broken_train.py (in the notebook: %run broken_train.py). It takes about 5 seconds.
3. Read the output with the curve shapes and the gradient sizes of this module. Write down the symptom, and which check of the checklist it points to.
4. Repair one bug, run the script again, and compare the new output with the old one.
5. Repeat steps 3 and 4 until the router reaches about 98% on validation.
6. Write a diagnosis record: for each bug, the symptom, the evidence (numbers from the output), the cause, the repair, and the evidence after the repair.

You should see this output in step 2:

epoch 1: train loss   77.472  acc  21.4% | valid loss   21.518  acc  21.3% | gradient norm    8.843
epoch 2: train loss   40.332  acc  17.2% | valid loss   13.111  acc  17.0% | gradient norm    9.303
epoch 3: train loss    2.642  acc  27.4% | valid loss    2.148  acc  27.5% | gradient norm    0.000
epoch 4: train loss    1.866  acc  27.4% | valid loss    2.065  acc  27.5% | gradient norm    0.000
epoch 5: train loss    1.735  acc  14.5% | valid loss    1.938  acc  14.7% | gradient norm    0.000
epoch 6: train loss    1.605  acc  27.4% | valid loss    1.811  acc  27.5% | gradient norm    0.000
epoch 7: train loss    1.726  acc  21.4% | valid loss    1.933  acc  21.0% | gradient norm    0.000
epoch 8: train loss    1.893  acc  20.1% | valid loss    2.094  acc  20.0% | gradient norm    0.000

Show the model answer

Bug 1, found first: the learning rate. The first output has a loss of 77.5 after one epoch, far above 1.61, the loss of guessing. Then the gradient norm is 0.000 from epoch 3 on: nothing can learn any more. From epoch 3 on, the validation accuracy is always the share of one team: 27.5% delivery, 14.7% account, 21.0% payment, 20.0% returns. So the router answers one team for every ticket. The script sets LEARNING_RATE = 10 for Adam: 10,000 times the default. The huge first steps pushed 56 of the 64 hidden neurons into a state where ReLU outputs 0 for every training ticket. Those neurons get no gradient at all, and the first layer’s gradient norm rounds to 0.000. Repair: LEARNING_RATE = 0.001. The output after this repair:

epoch 1: train loss    1.559  acc  27.6% | valid loss    1.587  acc  27.0% | gradient norm    0.363
epoch 2: train loss    1.501  acc  35.1% | valid loss    1.590  acc  23.0% | gradient norm    0.578
epoch 3: train loss    1.410  acc  39.9% | valid loss    1.617  acc  23.8% | gradient norm    0.658
epoch 4: train loss    1.296  acc  54.0% | valid loss    1.662  acc  21.2% | gradient norm    1.110
epoch 5: train loss    1.176  acc  60.1% | valid loss    1.705  acc  20.2% | gradient norm    1.017
epoch 6: train loss    1.071  acc  64.7% | valid loss    1.794  acc  20.5% | gradient norm    1.213
epoch 7: train loss    0.957  acc  69.4% | valid loss    1.829  acc  20.8% | gradient norm    1.461
epoch 8: train loss    0.866  acc  73.6% | valid loss    1.914  acc  20.7% | gradient norm    1.513

The gradients now have normal sizes (0.36 to 1.5), and the training loss falls. That repair worked. But the router is still bad.

Bug 2: the labels do not belong to their tickets. The training accuracy climbs to 73.6%. Meanwhile, the validation accuracy stays near 20%, and the validation loss rises from 1.587 to 1.914. That is the shuffled-labels picture: the network memorises labels that carry no meaning. Check 1 says: look at examples and labels. Add these lines after the mix: they print the first five tickets with their labels.

for i in range(5):                       # look at the first examples and their labels
    print(TEAMS[int(y_train[i])], "|", train_rows[int(mix[i])]["text"][:70])

The output (add TEAMS to the import line first):

payment | Hi there, I sent my barbecue grill back a while ago and the return was
returns | Hi there, I bought the garden hose reel about four months ago and it w
warranty | Hi, I tried the discount code for my compost bin order LK-308764 but t
delivery | Did you get my pressure washer return?
account | Hi, I'm really anxious about this. I sent my barbecue grill back last

A hose reel bought four months ago is warranty, not returns. A discount code is payment, not warranty. “Did you get my return?” is returns, not delivery. The script mixes the tickets with x_train = x_train[mix], but not the labels. Repair: add y_train = y_train[mix] on the next line, so that each label moves with its ticket. The output after both repairs:

epoch 1: train loss    0.536  acc  96.4% | valid loss    0.559  acc  97.7% | gradient norm    0.601
epoch 2: train loss    0.209  acc  97.4% | valid loss    0.212  acc  98.2% | gradient norm    0.812
epoch 3: train loss    0.136  acc  98.0% | valid loss    0.139  acc  98.5% | gradient norm    0.202
epoch 4: train loss    0.100  acc  98.4% | valid loss    0.110  acc  98.5% | gradient norm    0.170
epoch 5: train loss    0.078  acc  98.7% | valid loss    0.095  acc  98.2% | gradient norm    0.108
epoch 6: train loss    0.061  acc  98.8% | valid loss    0.082  acc  98.3% | gradient norm    0.391
epoch 7: train loss    0.048  acc  99.2% | valid loss    0.076  acc  98.8% | gradient norm    0.437
epoch 8: train loss    0.040  acc  99.3% | valid loss    0.083  acc  98.0% | gradient norm    0.085

Both losses fall together to about 0.08. The validation accuracy is 97.7% after one epoch and 98.0% after eight, and the gradients stay between 0.09 and 0.81. The lowest validation loss is at epoch 7, so early stopping would keep epoch 7 (98.8%).

The diagnosis record:

	Bug 1	Bug 2
Symptom	Loss 77.5, accuracy at the share of one team	Training accuracy rises, validation stays near 20%
Evidence	Gradient norm 8.8, then 0.000 from epoch 3	Validation loss rises 1.587 to 1.914; printed labels belong to other tickets
Cause	LEARNING_RATE = 10 for Adam	Tickets mixed with mix, labels not
Repair	LEARNING_RATE = 0.001	y_train = y_train[mix]
Evidence after	Gradient norms 0.36 to 1.5; training loss falls	Validation loss 0.559 to 0.076; 98.0% after 8 epochs

The repaired script is fixed_train.py. You may have found the bugs in the other order; that is fine if your evidence supports each repair.

Module check

See it end to endDeep network or gradient boosting?Case study

In the case study, a network fed raw, unscaled columns falls to chance: the input check of this lesson finds why.

Check your understanding

Name the four checks of the debugging checklist in their order, and say why the tiny-batch check comes before the gradient check.

Show the answer

1. Look at examples and labels. 2. Check the inputs (shapes, ranges, leakage). 3. Overfit a tiny batch. 4. Check the gradients. The order goes from cheap checks that find common bugs to more technical ones. The tiny-batch check is one short run that tests the network, the loss, the optimizer and the label pairing at once. If the network cannot learn 20 tickets, the bug is in the code. Gradient details matter only after that.

Check your understanding

Predict: a router's training accuracy rises from 30% to 85% in 10 epochs, but its validation accuracy stays between 18% and 22%. Is this overfitting? What do you check first?

Show the answer

Probably not ordinary overfitting: an overfitted network has a good validation score first, which then gets worse. Here, validation stays at the level of chance (20% for five teams) from the start, while training rises. That means the network memorises labels that do not match the inputs. Check 1 first: print a few training tickets with their labels. The labels are wrong or shifted, for example after a shuffle of the inputs without the labels.

Check your understanding

Completion check: Tomás repaired broken_train.py by changing only the learning rate from 10 to 0.001, and says it is fixed because the loss falls now. Use evidence from the curves or the gradients to say whether he is right, and what repair is still needed.

Show the answer

He repaired one bug, and the gradients show it. At learning rate 10, the gradient norm was 0.000 from epoch 3; now it is between 0.36 and 1.5, so the network learns. But the falling training loss is not enough. The validation curve shows the second bug. The validation accuracy stays near 20% (20.7% at epoch 8), and the validation loss rises from 1.587 to 1.914. Meanwhile, the training accuracy climbs to 73.6%. That is the picture of labels that do not match their tickets. The labels must move with the tickets: y_train = y_train[mix]. After that repair, the validation loss falls to 0.076 and the accuracy reaches 98.0%, which is the evidence that the repair worked.

Recap

• When a run fails silently, check in order: examples and labels, inputs, a tiny batch, gradients. Tune only after all four pass.
• Judge a label only with a model that did not learn it. The network trained on all rows found 2 of the 70 wrong labels. Out-of-fold predictions put 49 of them among the 50 suspects.
• Training accuracy rising while validation stays at chance means meaningless labels: shuffled, shifted or wrong.
• A vocabulary built with validation tickets is leakage; no curve shows it, so check how each input was built.
• If the network cannot learn 20 tickets by heart, the bug is in the code.
• Exploding gradients (a loss of 33,689 at learning rate 10) call for a smaller learning rate. Vanishing gradients (1.45 × 10⁻⁸ in the first layer of a deep sigmoid stack) call for ReLU and the defences of Module 6.

This module, in short: you can now read training curves and choose a learning rate and an optimizer. You can control overfitting, and find the cause of a failed run with evidence.

Reset. To start the practice again, get a fresh copy of broken_train.py. Run the cell that writes it in the notebook again, or download it again from the labs repository. If ticketnet.py or a data file changed by mistake, delete the folder ticket-router and run the notebook’s setup cell again. Locally, download them again as in Set up PyTorch. In the notebook, use Runtime › Restart session (Colab) or Run › Restart & clear cell outputs (Kaggle) before you run all cells again.

Next, Module 5. The bag of words cannot see word order, so the next module gives every word a learned meaning and then lets words look at each other.

Next: Tokens and embeddings