Save and inspect results

What you will be able to do

Make a result that someone else can repeat. You save the ticket router so that a fresh program gives exactly the same predictions, and you see what else must travel with the weights. Then you write an experiment record that states its settings, its hardware and its limits.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook trains the router, saves it, reloads it in a fresh Python process and compares the predictions. It shows the two vocabulary failures, checks the data checksums, and runs the module practice with SmolLM2-135M. About 40 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Training with validation and the best epoch, from Batches, epochs and validation.
• The vocabulary and the bag of words, from The forward pass.
• Seeds, from Set up PyTorch.
• SmolLM2’s generation settings (greedy, temperature, seed), from Training versus inference.

What you need on your computer

• Nothing for the notebook route.
• For the local route: the folder ticket-router with data/ and ticketnet.py, from Set up PyTorch, and transformers for the module practice, from Training versus inference.

Tested with Python 3.12, PyTorch 2.14.1 and transformers 5.19.0 on a Mac mini CPU. We ran the notebook in Jupyter on macOS, not in Colab or Kaggle.

A cake and its recipe

A baker makes a cake that everyone loves. Two things let her make it again. She can freeze a slice, which keeps the cake but not the method. Or she can write the recipe. A useful recipe says more than “flour, sugar, eggs”. It names the flour, the weights, the oven temperature, the time and the tin. It even notes “my oven runs hot”.

A trained network has the same two parts. The weights are the frozen slice. The record of how they were made is the recipe.

The problem: a result that lives only in one notebook

Priya’s router reached 98.3% on the validation tickets at epoch 12. That result lives in the memory of her notebook. If the kernel restarts, the weights are gone, and retraining takes time. Amira needs the router in the help-desk app, on another computer, in a program that never saw Priya’s notebook. And in three months, someone will ask: “Was it 98.3% on which data, with which code?” Tomás has a quick answer: “I’ll just send you the file.”

The idea: a checkpoint, plus everything it depends on

A checkpoint is a saved copy of a model’s learned numbers, from which training or inference can continue. The name comes from a race or a game: a point that you can return to.

In PyTorch, model.state_dict() returns the learned numbers as a dictionary: the name of each weight tensor, and the tensor. Priya trains the bag-of-words router of Batches, epochs and validation again, for 12 epochs, its best epoch. vocab, x_train, y_train, x_valid and y_valid are made as in that lesson; the notebook’s first cells make them, with the imports.

save_router.py

set_seed(0)
model = BagOfWordsNet(len(vocab))
history = train(model, x_train, y_train, x_valid, y_valid, epochs=12, log=None)
print(history[-1])
for name, tensor in model.state_dict().items():
    print(f"{name:16} {tuple(tensor.shape)}")

The output:

{'epoch': 12, 'train_loss': 0.0179, 'train_acc': 0.9977, 'valid_loss': 0.0633, 'valid_acc': 0.9833}
layers.0.weight  (64, 2864)
layers.0.bias    (64,)
layers.3.weight  (5, 64)
layers.3.bias    (5,)

Four tensors: the first layer (2,864 → 64) and the output layer (64 → 5). Positions 1 and 2 of layers are ReLU and dropout, which have no weights. The names come from the attributes of the class, so only code with the same attributes can load them.

The weights alone cannot route a ticket. Column 1,000 of the input means one particular word, so the vocabulary, in its order, must travel with the weights. And whoever rebuilds the network needs its configuration: the vocabulary size, the hidden size, the dropout. Priya saves all three:

save_router.py

Path("checkpoint").mkdir(exist_ok=True)
torch.save(model.state_dict(), "checkpoint/router.pt")                     # the weights
Path("checkpoint/vocab.json").write_text(json.dumps(vocab.itos))           # the tokens, in column order
Path("checkpoint/config.json").write_text(json.dumps({"vocab_size": len(vocab), "hidden": 64, "dropout": 0.0}))

torch.save() writes tensors to a file; vocab.itos is the list of tokens, where the position is the column number. The files are small: router.pt 737,107 bytes, vocab.json 28,486 bytes, config.json 50 bytes.

Prove it in a fresh process

A notebook remembers everything you ran, so a reload inside the same notebook proves little. The real test is a new Python process that has only the files. Before she saved the model, Priya also saved its validation probabilities in checkpoint/valid_probs.pt, to compare.

Save in one process, reload in another: the files must be enough to rebuild the network and give the same predictions.

Text description of the diagram

On the left, process 1 is Priya’s notebook. It trains for 12 epochs, to 98.3% on the validation tickets, and saves the weights with torch.save(model.state_dict()). In the middle are three files: router.pt, vocab.json and config.json. On the right, process 2 is a fresh Python. It builds BagOfWordsNet from config.json, loads the weights with load_state_dict(), calls model.eval(), predicts the 600 validation tickets and compares: 600 of 600 are the same.

reload_router.py

import json, torch
from ticketnet import BagOfWordsNet, Vocab, bag_of_words, labels, load_split

config = json.load(open("checkpoint/config.json"))
vocab = Vocab([])                                        # an empty vocabulary ...
vocab.itos = json.load(open("checkpoint/vocab.json"))    # ... filled with the saved tokens, in order
vocab.stoi = {token: i for i, token in enumerate(vocab.itos)}

model = BagOfWordsNet(config["vocab_size"], hidden=config["hidden"], dropout=config["dropout"])
model.load_state_dict(torch.load("checkpoint/router.pt"))
model.eval()                                             # inference: dropout off

valid = load_split("data/valid.csv")
with torch.no_grad():
    probs = torch.softmax(model(bag_of_words(valid, vocab)), dim=1)
before = torch.load("checkpoint/valid_probs.pt")
print("same predictions:", (probs.argmax(1) == before.argmax(1)).sum().item(), "of", len(valid))
print("largest difference in a probability:", (probs - before).abs().max().item())
print("validation accuracy:", round((probs.argmax(1) == labels(valid)).float().mean().item(), 4))

The output, from python reload_router.py:

same predictions: 600 of 600
largest difference in a probability: 0.0
validation accuracy: 0.9833

load_state_dict() copies the saved tensors into a network that the code built; it checks that every name and shape matches. Identical, to the last digit, on the same computer. On another processor, tiny rounding differences in the last digits are possible; the predicted teams should not change.

Note model.eval() after the load. This router has no active dropout. But without eval(), a network with dropout gives answers that change between runs: the training-only behaviour of Module 3 and Module 4.

Why not save the whole object?

torch.save(model) also works, without state_dict(). It saves the whole Python object, which sounds simpler. But the file then holds a reference to the class BagOfWordsNet in ticketnet.py, not the class’s code. Priya tries to load it in a fresh process, first with PyTorch’s default, then in a folder without ticketnet.py:

default load:       _pickle.UnpicklingError: Weights only load failed. This file …
without ticketnet:  ModuleNotFoundError: No module named 'ticketnet'

PyTorch’s safe default (weights_only=True) refuses to rebuild Python objects, because loading an object can run code from the file. With the safety switched off, the object still needs the exact class code at the same place. If someone renames the file or changes the class, old checkpoints stop loading. So: save the state_dict, and keep the code that builds the network under version control, next to the checkpoint.

The vocabulary must match: one loud failure, one silent one

Tomás sends Amira only router.pt. She rebuilds the vocabulary herself, from the training and the validation tickets, and loads the weights:

size mismatch for layers.0.weight: copying a param with shape torch.Size([64, 2864]) from checkpoint, the shape in current model is torch.Size([64, 2987]).

Her vocabulary has 2,987 tokens, not 2,864, so PyTorch refuses. That is the lucky case: a loud error. If her vocabulary had the same size but another order, every word would land in another word’s column, and nothing would complain. Priya makes that happen on purpose: the same 2,864 tokens in reverse order. The network loads, runs and routes 24.0% of the validation tickets right. That is worse than always answering “delivery” (27.5%), with no error message at all.

What else to record

Weights, vocabulary and configuration rebuild the model. To repeat the experiment, someone also needs everything that made those weights, and an honest statement of what the result shows.

What a reproducible experiment record holds, filled in for Priya's router. The first two cards rebuild the model; all eight repeat the experiment.

Text description of the diagram

Eight cards, filled in for Priya’s router.

• Weights: router.pt, from model.state_dict(), 0.74 MB, with its SHA-256.
• Vocabulary or tokenizer: vocab.json, 2,864 tokens in order; for a pretrained model, the model ID and revision.
• Configuration: hidden 64, dropout 0, Adam, learning rate 0.001, batch 64, 12 epochs.
• Seeds: set_seed(0) before the model and before training.
• Data: train.csv and valid.csv, with their SHA-256 from SHA256SUMS.
• Code and libraries: ticketnet.py, Python 3.12.13, PyTorch 2.14.1.
• Hardware: Mac mini, CPU only, 2 threads.
• Evaluation and its limits: 98.3% on 600 validation tickets; the tickets are made up; the test set is still sealed.

Why each card matters:

• Seeds fix the starting weights and the order of the batches. Without set_seed(0), another run reaches another epoch-12 result.
• Library versions can change defaults and numbers. torch.load changed its default to weights_only=True in a recent version, which is why the whole-object file failed above.
• Hardware changes timings completely and floating-point results a little. A timing without its hardware means nothing.
• Data checksums prove that the data is the same file, byte for byte. A checksum is a short fingerprint computed from every byte; one changed character changes it. In the notebook, both files match the published SHA256SUMS; train.csv starts with 98487169346f8b0d.
• The evaluation and its limits say what the number means: which split, how many examples, what it cannot show. 98.3% on 600 made-up validation tickets does not promise 98.3% on real customers.

For a pretrained model, the model ID and revision replace the vocabulary file. SmolLM2’s tokenizer comes with the model, so pinning revision= fixes the weights and the tokenizer together. Add the dtype, because float32 and bfloat16 give different outputs (the previous lesson showed it).

An experiment record for SmolLM2

An experiment record is a short text file, often EXPERIMENT.md, next to the code and the checkpoint. Here is the record of the timing experiment behind the previous lesson:

EXPERIMENT.md

# Experiment: SmolLM2-135M generation time with and without the KV cache

## Question
How long does SmolLM2-135M take to write 40 tokens on our CPU, with and
without the KV cache? Are the tokens the same?

## Model and code
- Model: HuggingFaceTB/SmolLM2-135M, revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2
- Loaded as float32 (the files on Hugging Face are bfloat16); tokenizer of the same revision
- Code: m07_smollm.py; Python 3.12, PyTorch 2.14.1, transformers 5.19.0

## Settings
- Prompt: "Larkfield is an online shop for home and garden products.
  A customer wrote: my parcel is late. The support team" (25 tokens)
- Greedy choice, 40 new tokens, use_cache=True and use_cache=False
- torch.manual_seed(0); one warm-up run, then the median of 5 runs

## Hardware
- Mac mini (Apple silicon, arm64), macOS 26, CPU only, 4 threads,
  no other heavy programs running

## Data
- None: one fixed prompt

## Results
- With the KV cache: 0.46 s. Without: 1.01 s. The 40 tokens are identical.
- Greedy text: " is very helpful. I am looking for a new parcel. I am looking for ..."
- Checksum of the 40 token IDs (first 12 hex digits of SHA-256): 8def2cec688d

## Limits
- One prompt of 25 tokens: the gap grows with longer texts; not measured here.
- One computer: other CPUs, other thread counts or a GPU give other times.
- Medians of 5 reduce timing noise from other programs; they do not remove it.
- Speed only: this says nothing about the quality of the text (a base model).

The checksum of the token IDs lets someone confirm the same output without comparing 40 tokens by eye. The limits section is not an apology. It tells the reader which questions the result answers and which it does not.

When to save, and what it costs

Save a checkpoint at the end of training, and during long runs at the best validation epoch. That is early stopping: keep the best, not the last. Keep a record for every result that someone will act on or quote. Commit the code and the record to Git; keep large checkpoints next to them, named and checksummed.

What it costs: a few minutes per experiment, and disk space. The router’s checkpoint is 0.74 MB; SmolLM2’s files are 269 MB, and a checkpoint for further training also needs Adam’s state, twice the weights. You can skip a checkpoint for a quick look that nobody will repeat. Do not skip the record of a number that goes into a decision.

Guided practice: save, reload, compare

1. In the notebook, run section 1. You should see epoch 12 with 'valid_acc': 0.9833 and four tensors in the state_dict.
2. Run the cell that saves the folder checkpoint. You should see router.pt with 737,107 bytes.
3. Run the two cells of section 2. You should see same predictions: 600 of 600.
4. Run section 4. Write down which failure gives an error and which one gives none.
5. Run section 5. You should see matches SHA256SUMS: True twice.

Show the answer

The vocabulary built from the training and validation tickets gives a loud error: a size mismatch, 2,864 against 2,987. The reversed vocabulary gives no error at all and routes 24.0% of the tickets right. A shape check catches only the first. Only a comparison of predictions after the reload catches both.

Your turn: rebuild instead of load

Amira says: “I don’t need vocab.json. I’ll build Vocab from the same train.csv with min_count=2, as Priya did.” Predict: will her network give the same 600 predictions? Then explain when her shortcut breaks.

Show the answer

Yes, today. Vocab sorts the tokens, and the same file with the same tokenizer and the same min_count gives the same 2,864 tokens in the same order. But the shortcut depends on four things that nobody recorded. The file must be byte for byte the same (check the SHA-256), and min_count must still be 2. The tokenize() code must be unchanged, and Vocab must still sort the same way. A fixed typo in one ticket can change the vocabulary. Saving vocab.json removes all four risks for 28 KB.

Common mistake

It worked on my laptop

Symptom: Tomás trains a router, sees 98% on his laptop and sends Amira router.pt. On the help-desk server, loading fails with size mismatch for layers.0.weight. Tomás “fixes” it by building a vocabulary of the right size. Now it loads, but the server routes about one ticket in four correctly, and nobody gets an error.

Cause: the checkpoint held only the weights. The vocabulary that gives each column its meaning stayed on Tomás’s laptop, with his seeds, library versions and data version. Nobody wrote down how the 98% was measured, so nobody could check the server against it.

Fix: ship the checkpoint folder: router.pt, vocab.json, config.json, and the code version that builds the network. Add EXPERIMENT.md with the seeds, versions, hardware, data checksums and the evaluation. After every reload on a new machine, predict a fixed set of tickets in a fresh process. Compare with the saved predictions: 600 of 600, or stop.

What it means for a real AI service: “it works on my machine” is not a result. A model is ready to hand over only when a fresh process, with only the shipped files, reproduces the recorded predictions.

See it end to endClassify clothing imagesCase study

The clothing case study saves its network and an experiment record, and reloads it with identical predictions.

Knowledge check

Check your understanding

What does model.state_dict() contain, and why is it safer to save than the whole model object?

Show the answer

It contains the learned tensors, by name: for the router, layers.0.weight (64, 2864), layers.0.bias (64), layers.3.weight (5, 64) and layers.3.bias (5). It is plain data, so PyTorch loads it with the safe default and any code that builds the same network can use it. A whole saved object holds a reference to its class, so it needs the exact class code at the same place, and the safe default refuses to load it.

Check your understanding

Predict: you reload the router with a vocabulary of the same 2,864 tokens in a different order. What happens when you call load_state_dict(), and what happens to the accuracy?

Show the answer

load_state_dict() succeeds, because every name and shape matches. The accuracy collapses, because each word now lands in another word’s column. With the reversed order, the router got 24.0% of the validation tickets right, below the 27.5% of always saying “delivery”. Only a comparison of predictions catches it.

Module practice

Guided practice → independent task. You have seen Priya save, reload and record the router, and read the record of the timing experiment. Now run SmolLM2-135M and compare its outputs under controlled settings, and write the record yourself.

1. In section 6 of the notebook, load SmolLM2-135M with the pinned revision, as float32.
2. Generate 40 greedy tokens twice, and check that the texts are identical.
3. Generate 25 tokens at temperature 0.8 (top_p=0.95) with seed 1, twice, and check that they are identical.
4. For temperatures 0.3, 0.8 and 1.2, generate 25 tokens with seeds 1, 2 and 3. Run the check cell.
5. Fill in the four ... lines of EXPERIMENT.md: the question, the settings, one result line per temperature, and the limits. Run the check cell.

Show a model answer

On the course’s computer, both repeats were identical. The nine samples (new lines shown as spaces):

Temperature	Seed 1	Seed 2	Seed 3
0.3	“ is available to help. The post How to Find a Good Online Store appeared first on The Gardener’s Companion.”	“ is very helpful. I am looking for a new home. I have a 1200 sq ft house. I”	“ is very helpful and I have been able to get my parcel delivered to my house. I have a question about the”
0.8	“ is currently at my front door, and I am still waiting for their response. I am the only one with the”	“ is there to help you out. They’ve got all the right tips and tools to make it happen, and they know exactly”	“ is there to help you with the delivery. They are really friendly and supportive. It takes 2-3 hours to get”
1.2	“ is currently at my front lawn, looking at a patch on my driveway and have been asking me where to send my package”	“ is there but my parcel is not there. Can you fix my parcel to arrive to me early for delivery? The customer email”	“ took my item and returned it to me but they didn’t refund me since I’m not supposed to have it back. What”

The record’s four lines, for example:

## Question
How do greedy choice and sampling at three temperatures change the continuation
of one Larkfield prompt, and does each setting repeat?

## Settings
- Greedy: 40 new tokens, run twice
- Sampling: temperatures 0.3, 0.8 and 1.2, top_p 0.95, seeds 1, 2, 3, 25 new tokens each

## Results
- Temperature 0.3: two of three samples start like the greedy text ("is very helpful").
- Temperature 0.8: three different, fluent continuations.
- Temperature 1.2: three different continuations; one invents a refund story.

## Limits
- One prompt and three seeds: a description, not a measurement of quality.
- A base model: it continues the text; it does not answer the customer.
- Sampled text can differ on another processor even with the same seeds; greedy text usually matches.
- No timings: they depend on the computer and on what else runs on it.

The notebook fills in the model, revision, library versions and hardware from the running program, so they cannot be copied wrong. Seed 1 at temperature 0.3 shows a base model’s habit: it ends with a sentence that looks like the footer of a blog’s web feed. Web pages are part of its pretraining text.

Module check

Check your understanding

Retrieval: name the three stages in the life of a pretrained model, and say which of them change the weights. Then name what the KV cache stores.

Show the answer

Pretraining changes all the weights, on a huge amount of text. Adaptation changes them a little by fine-tuning, or not at all by prompting. Inference changes nothing: it runs forward passes only. The KV cache stores the keys and values of every token so far, in every block, so that each decode step computes only the new token.

Check your understanding

Explain or predict: Amira runs the module practice on a Colab computer with an Intel processor. Which of your results should she get exactly, which may differ, and why?

Show the answer

She should get the same parameter count, the same shapes and the same tokenization, because they depend only on the pinned model files. The greedy text will usually match. The sampled texts may differ even with the same seeds. Tiny rounding differences between processors can change one draw, and every later token follows from it. Her timings will differ completely, because they depend on the hardware. That is why the record states the hardware, and why it checks the greedy output with a checksum rather than the samples.

Check your understanding

Completion check: deliver a reproducible experiment. List what your EXPERIMENT.md and its folder must contain so that another person can repeat it, and state its hardware and data limitations.

Show the answer

The folder holds the code (or its Git commit) and the record. The record names the question, and the model ID, revision and dtype. (For the router: router.pt, vocab.json and config.json.) It lists the exact settings: prompt, greedy or sampling, temperature, top_p, seeds, number of tokens. It gives the library versions (Python, PyTorch, transformers) and the hardware (processor, CPU or GPU, threads). For data, it gives the files with their SHA-256 checksums, or “none: one fixed prompt”. It gives the results with a checksum of the greedy token IDs, and the evaluation with its size. Hardware limits: timings and sampled texts hold for this computer only; another processor can change both. Data limits: one prompt is a description, not a measurement. The router’s 98.3% holds for 600 made-up validation tickets, not for real customers, and not for the sealed test set.

Recap

This lesson: a checkpoint is model.state_dict() saved with torch.save(), with the vocabulary (or the model ID and revision) and the configuration next to it. A fresh process must reproduce the predictions: 600 of 600, difference 0.0. Do not save the whole object. Record the seeds, library versions, hardware, data checksums, the evaluation and its limits in EXPERIMENT.md.

This module: a pretrained model goes through pretraining, adaptation (fine-tuning or prompting) and inference. SmolLM2-135M is Module 6’s transformer at scale: 30 blocks, grouped-query attention, tied embeddings, 134.5 million parameters. Generation is a loop of forward passes; the KV cache makes each step cheap and costs memory. Memory is parameters × bytes, and quantization trades memory, speed and quality. Training needs about four times the weights’ memory, so large models are out of reach of ordinary hardware.

Reset procedure: if a notebook gives other numbers, restart the kernel and run all the cells from the top. The router’s numbers come from set_seed(0). If the model download fails, check the internet connection (in Kaggle, turn on Internet) and run the cell again. If a data checksum is wrong, delete the folder ticket-router and run the setup cell again. To start the practice again, delete the folder checkpoint and the file EXPERIMENT.md.

Next: the case studies apply the whole course to real open data, starting with clothing images.

Next: From counts to meaning (optional)