Monitor and recover

What you will be able to do

Watch a run while it trains, decide when it should stop, and continue a stopped run without losing work. A training loss that keeps falling does not tell you that the adapter is getting better.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The run’s plan, its configuration file and what a checkpoint is, from Plan and configure the run.
• Overfitting and early stopping, from Overfitting controls in Deep Learning and Transformers Explained.

What you need on your computer

• The project after Plan and configure the run: python -m adapt plan configs/lora-r8.toml shows 226 steps, and the model is downloaded.
• For the optional training: about 7 minutes and 2 GB of free memory. Without it, you read the course’s recorded runs.

Tested on macOS (Apple silicon) with Python 3.12.13, PyTorch 2.14.1, transformers 5.19.0 and peft 0.21.2. The Windows and Linux commands are written, not tested.

Save points in a game

In a long computer game, you save before a hard level. If the power goes off, you lose only the minutes since the last save. Good players also keep the save from their strongest moment, and go back to it if a later choice makes things worse.

A training run needs the same three things. It saves to continue after a stop. It keeps the best version. And it shows the right number to tell which version is best.

The problem: the number that looks perfect

The course has a run that overfits on purpose, configs/overfit.toml: 48 records, 20 epochs, a high learning rate (0.001) and rank 16. At its end, the training loss is 0.0003, almost zero. Watch only this number, and the adapter looks perfect.

It is not. On the evaluation course’s dev cases, it chooses the right team for 69 of 100 tickets; the main run chooses it for 81. The training loss only measures the adapter’s own training records. And if the laptop closes at step 200 of a 20-minute run, a run without saved state starts again from step 0.

Read the two curves

Two numbers answer two different questions:

• The training loss uses the batch that the adapter just learned from: “How well does it fit what it practises?” A batch has only 8 records, so the value jumps from step to step. Read its mean over 25 steps.
• The validation loss uses the same 100 validation records every time. The adapter never trains on them: “How well does it do on records that it has not seen?”

In this project, both losses count only the answer’s tokens: the JSON object and the end mark. Here is the course’s first run, lora-r8:

• Training loss (mean of 25 steps)
• Validation loss

The first run: both losses fall together for 226 steps. The dark filled dots are the checkpoints (steps 50, 100, 150, 200 and 226). The validation loss is still falling at the last step.

Text description of the chart

The chart shows the two losses of the first run over its 226 steps. The training loss, as a mean of each 25 steps, falls from 0.58 (steps 1 to 25) to 0.06 (steps 201 to 225). The validation loss falls from 0.2598 at step 25 to 0.0995 at step 100 and 0.0595 at step 200. It is 0.0597 at step 225 and 0.0584 at step 226, the lowest value. Filled dots on the validation line mark the five checkpoints, at steps 50, 100, 150, 200 and 226.

Read the curves in three patterns:

1. Both fall: the adapter still learns things that work on new records. Training longer may help.
2. The validation loss stops falling: more training does not help. Stop.
3. The training loss falls and the validation loss rises: the adapter learns details of its own records that do not work on others. That is overfitting. Stop, and keep the version with the lowest validation loss.

The first run is pattern 1, as the chart shows: the run was not finished.

A fourth pattern is an unstable run: the loss jumps up and stays high, or becomes “nan”. The usual cause is a learning rate that is too high (Learning rate and optimizers). The course’s runs show only noise: single steps jump, but the 25-step mean keeps falling.

Overfitting, and the best adapter

The overfitting run validates once per epoch. Here are epochs 4 to 8 (python -m adapt curves overfit --every 6):

Epoch	Training loss	Validation loss
4	0.0842	0.1007
5	0.0494	0.0889
6	0.0307	0.0851
7	0.0163	0.0949
8	0.0074	0.1137

After epoch 6, the training loss keeps falling, and the validation loss rises. The chart shows all 20 epochs:

• Training loss (mean of the epoch)
• Validation loss

The run made to overfit: the training loss falls to almost zero, while the validation loss is lowest at epoch 6 and then rises to 0.1267.

Text description of the chart

The chart shows the run made to overfit, per epoch, over 20 epochs of 6 steps. The training loss falls from 0.507 in epoch 1 to 0.0307 in epoch 6. It reaches 0.0024 in epoch 10 and 0.0003 in epoch 20. The validation loss falls from 0.216 in epoch 1 to its lowest value, 0.0851, in epoch 6. Then it rises to 0.1137 in epoch 8 and ends at 0.1267 in epoch 20. The gap between the two curves grows after epoch 6.

The training code keeps two adapters. best/ is saved at each new lowest validation loss: here, step 36 (epoch 6). adapter/ is the adapter after the last step. Keeping the version with the lowest validation loss is early stopping. It costs a validation pass every few steps, and it uses the validation set to choose. So the final judgement needs cases that played no part in any choice: the held-out cases, in Module 5.

Checkpoints and resume

A checkpoint must hold everything that the next step needs, not only the adapter:

A checkpoint holds the adapter, the optimizer's state, the random state and the best loss so far, so a new process continues exactly where the run stopped.

Text description of the diagram

The diagram shows a run in five boxes. First, steps 1 to 100 run, and each step adds a line to the log, log.jsonl. At step 100, the run writes the checkpoint step-00100. It holds the adapter, the optimizer’s state, the random state and the best validation loss so far. Then the run stops: on purpose with --stop-after 100, or because the laptop closed or the power went off. Later, --resume starts a new process. It loads step-00100 and keeps the log up to step 100. Last, steps 101 to 226 run, with the same losses as a run that never stopped.

The optimizer’s state holds AdamW’s running averages for every trained number; without it, the next steps differ. The random state decides the next dropout pattern. The batch order comes from the seed and the epoch, so it needs no saving. The run keeps only the newest 2 checkpoints (keep_checkpoints = 2).

The course measured it twice: the overfitting run, stopped after step 45, and the first run, stopped after step 100 (the guided practice below). Both gave every loss of the run that never stopped. This holds on the same computer with the same versions; elsewhere, the last digits can differ.

Discuss: train longer?

The first run has finished, with its validation loss still falling. Priya scores it on the dev cases (Compare baselines shows how). It chooses the right team for 81 of 100 tickets. But none of the 23 tickets that need a person goes to a person.

• Tomás: “The loss is still falling. Train longer, and it will learn when a ticket needs a person.”
• Priya: “One change only: 6 epochs instead of 2, same data, same seed. We decide on the validation loss and the dev cases. The holdout cases stay closed.”
• Kwame: “The longer run takes about 20 minutes on the CPU. That’s cheap. But what if it does not help?”
• Grace: “I don’t care about the loss. A customer who reports a burning smell from a product must get a person.”

Before you read on, answer these:

1. Is a second run with 6 epochs a fair experiment? What makes it fair?
2. Which result would show that more training fixed the problem?
3. If the validation loss falls a lot, does Grace get what she needs?

The course’s answer. The second run, lora-r8-long, is fair: it changes only epochs = 6. With the same seed and batches, its first 225 steps are the first run’s.

• Run 2: validation loss (6 epochs)
• Run 1: validation loss (2 epochs)

The second run is the first run continued: the same first 225 steps, then a validation loss that falls to 0.0193 at step 678.

Text description of the chart

The chart compares the validation loss of the two runs. Run 1 has 226 steps and ends at 0.0584. Run 2 has 678 steps. Up to step 225, its values are the same as run 1’s. After that, it keeps falling: 0.0402 at step 300, 0.0288 at step 400 and 0.0231 at step 500. At step 678 it reaches 0.0193, its lowest value. It goes up a little at some validations, for example from 0.0266 at step 425 to 0.0305 at step 475. The trend still falls.

The validation loss fell from 0.0584 to 0.0193, and routing improved from 81 to 89 of 100 dev tickets. But the second run also sends 0 of 23 tickets that need a person to a person. A lower loss is not the fix.

Why not? The loss is a mean over all the answer’s tokens, and needs_human is one token of each answer. Only 78 of the 900 training records need a person, so “false” is right for 91% of them. All 78 follow one rule (H1, safety); no record shows rules H2 to H5, or a French ticket. More epochs repeat the same records: they cannot teach what the records do not contain. Module 5 measures what the adapter learned about this one token.

So the course keeps the second run as its adapter for Module 5, because it routes better. It also records the honest result: more training did not fix the safety decision.

What would change the answer: training records that need a person under every rule, and French tickets: a change of data, not of epochs.

Guided practice: train, stop and resume

Steps 4 to 7 train: they are optional, and take about 7 minutes and 1.9 GB of memory. Without them, read their outputs, and do step 8 with the recorded runs.

1. Copy this lesson’s files from the snapshot. The recorded runs come with them.

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m04
   cp $S/adapt/train.py adapt/
   cp $S/configs/lora-r8-long.toml $S/configs/overfit.toml configs/
   cp $S/tests/test_config.py $S/tests/test_train.py $S/tests/tiny.py tests/
   cp -R $S/runs/lora-r8 $S/runs/lora-r8-long $S/runs/overfit runs/
   cp $S/README.md .

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m04"
   Copy-Item $S\adapt\train.py adapt\
   Copy-Item $S\configs\lora-r8-long.toml, $S\configs\overfit.toml configs\
   Copy-Item $S\tests\test_config.py, $S\tests\test_train.py, $S\tests\tiny.py tests\
   Copy-Item -Recurse $S\runs\lora-r8, $S\runs\lora-r8-long, $S\runs\overfit runs\
   Copy-Item $S\README.md .

2. Run the tests: python -m pytest. The last line says 27 passed, 1 deselected, 1 warning. One new test stops and resumes a tiny random model.

3. Run the test that needs the model: python -m pytest -m model. You see 1 passed, 27 deselected.

4. Keep the recorded run under another name, so that you can compare with it later:

   macOS and Linux

   mv runs/lora-r8 runs/lora-r8-recorded

   Windows

   Rename-Item runs\lora-r8 lora-r8-recorded

5. Train, and stop after step 100: python -m adapt train configs/lora-r8.toml --stop-after 100. It takes about 3 minutes. Warnings about use_cache, tensor.detach() and a missing config file are normal. You see:

   Training lora-r8: 900 records, 226 steps, 460,800 of 134,975,808 parameters trainable (0.34%).
   step   25/226  train loss 0.3257  validation loss 0.2598
   step   50/226  train loss 0.1513  validation loss 0.1491
   step   75/226  train loss 0.1258  validation loss 0.1139
   step  100/226  train loss 0.0846  validation loss 0.0995
   Stopped after step 100 (as asked). Resume with --resume.

6. Resume in a new process: python -m adapt train configs/lora-r8.toml --resume. It takes about 4 minutes. You see:

   Resuming lora-r8 from step-00100 (step 100 of 226).
   Training lora-r8: 900 records, 226 steps, 460,800 of 134,975,808 parameters trainable (0.34%).
   step  125/226  train loss 0.1024  validation loss 0.0880
   step  150/226  train loss 0.1258  validation loss 0.0800
   step  175/226  train loss 0.0571  validation loss 0.0693
   step  200/226  train loss 0.0501  validation loss 0.0595
   step  225/226  train loss 0.0275  validation loss 0.0597
   step  226/226  train loss 0.0350  validation loss 0.0584
   Done: best validation loss 0.0584 at step 226; adapter 1.9 MB in …/ticket-adapter/runs/lora-r8/adapter

   Every loss equals the recorded run’s, and the new adapter’s weights file is identical to the recorded one, byte for byte. Your path in the last line is different.

7. Look at the folder: runs/lora-r8 has adapter, best, checkpoints, log.jsonl and summary.json. checkpoints keeps the newest two, step-00200 and step-00226.

8. Read the curves: python -m adapt curves lora-r8.

    step  epoch  train loss  validation
      25   0.22      0.5818      0.2598
      50   0.44      0.2084      0.1491
      75   0.66      0.1377      0.1139
     100   0.89      0.1182      0.0995
     125   1.11      0.0972      0.0880
     150   1.33      0.0911      0.0800
     175   1.55      0.0723      0.0693
     200   1.77      0.0607      0.0595
     225   1.99      0.0595      0.0597
     226   2.00      0.0350      0.0584  <- lowest validation loss

   With the recorded runs, also run python -m adapt curves overfit --every 6. Find the arrow <- lowest validation loss at step 36.

Common mistake

A restart that would overwrite a run

Symptom: after the stop in step 5, Sam runs the same command again, without --resume. It stops at once:

…/ticket-adapter/runs/lora-r8 exists. Use --resume to continue it, or delete it to start again.

Cause: train() refuses to start a run whose folder exists, unless you ask to resume it. Otherwise, the new run would write over the stopped run’s log and checkpoints.

Fix: add --resume to continue. To start again from step 0 on purpose, delete the run’s folder first.

Consequence in AI work: a run that silently restarts mixes two runs in one log, and its checkpoints and best/ can come from different runs. A clear refusal costs one more command.

Module practice: inspect the second run

Independent task. Analyse the second run, lora-r8-long, as if you had to approve it. Use the recorded run, or train it again first (15 to 20 minutes; move its recorded folder away as in step 4).

1. Run python -m adapt plan configs/lora-r8-long.toml and python -m adapt curves lora-r8-long --every 50.
2. Write a short run record: the exact configuration, the model’s revision, the library versions, the computer and the time.
3. Say at which step you would stop this run, and why. Use the curve.
4. Say what the curve cannot tell you.

Show a model answer

plan shows 678 steps (113 per epoch, 6 epochs), 27 validations and 13 checkpoints:

lora-r8-long: 900 training records, 100 validation records
  batch 8, 6 epochs = 113 steps per epoch, 678 steps (5424 examples seen)
  validation loss every 25 steps (27 times), checkpoint every 50 steps (13)
  LoRA rank 8, alpha 16, dropout 0.05, on q_proj, v_proj; learning rate 0.0002; at most 320 tokens per example; 2 threads

The curve, every 50 steps:

 step  epoch  train loss  validation
   50   0.44      0.3951      0.1491
  100   0.89      0.1279      0.0995
  150   1.33      0.0942      0.0800
  200   1.77      0.0665      0.0595
  250   2.21      0.0572      0.0535
  300   2.65      0.0429      0.0402
  350   3.10      0.0363      0.0381
  400   3.54      0.0307      0.0288
  450   3.98      0.0266      0.0297
  500   4.42      0.0213      0.0231
  550   4.87      0.0217      0.0228
  600   5.31      0.0178      0.0225
  650   5.75      0.0164      0.0194
  678   6.00      0.0171      0.0193  <- lowest validation loss
train loss = the mean over the steps of each line; validation = at that step

Run record. Configuration: configs/lora-r8-long.toml, identical to lora-r8.toml except epochs = 6 (seed 12, rank 8, alpha 16, dropout 0.05 on q_proj and v_proj, learning rate 2e-4, batch 8, at most 320 tokens, 2 threads). Data: data/prepared/train.jsonl (900 records, manifest 73f29c94b022) and val.jsonl (100, f0b7e6da2e1a). Model: HuggingFaceTB/SmolLM2-135M-Instruct at revision 12fd25f77366fa6b3b4b768ec3050bf629380bac. Libraries: Python 3.12.13, PyTorch 2.14.1, transformers 5.19.0, peft 0.21.2 (from runs/lora-r8-long/summary.json). Computer: Apple M4, 2 threads; 1,156 seconds (about 19 minutes), while other jobs also ran. Peak memory: 1,885 MB.

When to stop. The validation loss falls to 0.0193 at step 678, the last step and the lowest. It still went down from 0.0225 at step 600, so stopping earlier would lose a little. After step 500, it falls slowly (0.0231 at step 500, 0.0193 at step 678) and moves up and down between validations. A third run with more epochs would gain little on this measure. Keep best/, which here is the same as adapter/.

What the curve cannot tell. It says nothing about the tickets that need a person, French tickets or the rules H2 to H5. The validation records have no French ticket and no H2 to H5 case either. Only the comparison on the evaluation course’s held-out cases, in Module 5, can show that.

Knowledge check

Check your understanding

During a run, the training loss falls from 0.09 to 0.01 while the validation loss rises from 0.08 to 0.12. What is happening, what do you do, and which adapter do you keep?

Show the answer

The adapter overfits: it learns details of its training records that do not help on new records. Stop the run, or stop paying attention to the later steps. Keep the adapter with the lowest validation loss, which this project saves in best/, not the last one. Then judge it on held-out cases that played no part in the choice.

Check your understanding

Why does a checkpoint save the optimizer's state and the random state, and not only the adapter? What would go wrong without them?

Show the answer

AdamW keeps running averages for every trained number, and they decide the size and direction of the next steps. Without them, the resumed run takes different steps from a run that never stopped. The random state decides the next dropout pattern, so without it the noise differs too. The resumed run would still train, but it would not be the run that you planned. Its log would mix two different runs. With both saved, the overfitting run’s 140 log lines were identical after a resume.

Check your understanding

Completion check: explain when the course's first run should stop, and write down what someone needs to repeat it exactly. Then say which evidence would make you change your mind about its stopping point.

Show the answer

When to stop. Stop when the validation loss stops falling or starts to rise, and keep the version with the lowest validation loss. The first run never reached that point. Its validation loss fell from 0.2598 to 0.0584 and was still falling at its last step, 226. So its stopping point was set by the plan (2 epochs), not by the curve. That was the reason for the second run, with one change: 6 epochs. Its lowest validation loss, 0.0193, is also at its last step.

What to record. The configuration file configs/lora-r8.toml (seed 12, rank 8, alpha 16, dropout 0.05 on q_proj and v_proj, learning rate 2e-4, batch 8, 2 epochs, at most 320 tokens, validation every 25 steps, checkpoint every 50, 2 threads). The data files and their manifests (73f29c94b022, f0b7e6da2e1a). The model and its revision (12fd25f77366…), the tokenizer files in model/, and the library versions (PyTorch 2.14.1, transformers 5.19.0, peft 0.21.2). The computer: Apple M4, 2 threads, 367 seconds, peak 1,882 MB.

What would change my mind. A validation loss that rises before the last step: then stop earlier and keep best/. Results on the held-out cases that get worse while the validation loss falls: then the validation records are not like real tickets. And a measured result that matters more than the loss, such as the 0 of 23 tickets sent to a person. It shows that no stopping point fixes a gap in the data.

Recap

• The training loss says how well the adapter fits what it practises. The validation loss, on 100 records it never trains on, says how it does on new records. Read both.
• Both falling: still learning. Validation flat: stop. Training falling while validation rises: overfitting. Keep best/, the adapter with the lowest validation loss (epoch 6 of the overfitting run, 0.0851).
• A checkpoint holds the adapter, the optimizer’s state, the random state and the best loss so far. A resumed run gave the same losses at every step as a run that never stopped.
• The first run’s validation loss was still falling, so a second run changed one thing: 6 epochs. The validation loss fell to 0.0193 and routing improved (81 to 89 of 100). But the adapter still sent 0 of 23 tickets that need a person to a person. A lower loss did not fix a gap in the data.

This module, in short: you can plan a run from one configuration file, and pin its model and libraries. You can measure its memory, read its curves, keep the best adapter and resume a stopped run.

Reset. If you trained in steps 4 to 7, delete runs/lora-r8 and rename runs/lora-r8-recorded to runs/lora-r8: both hold the same losses. If your project is broken, copy the snapshot end-of-m04 as in Use a snapshot. It has the three recorded runs, so you do not need to train. Make the virtual environment again and install the libraries as in step 3 of Plan and configure the run. Delete the checkpoints of your own runs when you no longer need them.

Next, Module 5 compares the adapter with the hosted assistant, the base model and a simple classifier on the same held-out cases.

Next: Compare baselines