Final assignment

What you will be able to do

Use every module of the course on one new decision. You train a small network, make it honest about when it is unsure, and report its results in a way that another person can repeat. Then you show that you can read a transformer: its shapes, its attention and what it costs to run.

Before you startWhat to know and to have readyAbout 240 minutes

• You completed Modules 1 to 7. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• Your ticket-router project works: data/ and ticketnet.py from get_data.py, and PyTorch in your virtual environment or in Colab. If something is broken, run python get_data.py --reset (Set up PyTorch).
• Make a new folder in your project for this assignment: assignment/. Put every script, file and report in it.
• You can do Parts 1 to 5 in a notebook (local, Colab or Kaggle). Part 6 needs the SmolLM2-135M download (about 270 MB) and about 1 GB of free memory.
• Tested with Python 3.12, PyTorch 2.14.1 (CPU) and transformers 5.19.0 on macOS.

The decision: send unsure tickets to a person

Amira has added the router to the help-desk app. It sends every new ticket to a team at once. Grace likes the speed, but she has one worry: “When the router is wrong, the ticket waits in the wrong queue for hours. When it is unsure, I would rather a person reads it.”

So Grace sets a rule. The router may send at most 1 in 20 new tickets (5%) to a person for manual routing. Every other ticket goes straight to the team that the router chooses. Omar asks the question that decides whether it is worth it: “How many tickets will still go to the wrong team?”

Priya asks you to build the router with a neural network, as in this course, and to answer Omar with evidence. Tomás offers a shortcut: “Train it, then pick the cut-off on the test set, so we hit 5% exactly.” You will see in Part 4 why that is not allowed.

The tickets are synthetic: a language model wrote them for this course. No real customers.

Part 1: shapes first

Uses Tensors and shapes and The forward pass.

1. Build the vocabulary from the training tickets only, with Vocab (minimum count 2). Write down its size.
2. Make the bag-of-words tensors for train, validation and test. Print each shape.
3. Make a BagOfWordsNet with 32 hidden neurons. Pass the first 8 validation tickets through it, and print the shape after each layer.
4. In assignment/REPORT.md, under Shapes, explain each dimension of every printed shape in one line, and count the parameters by hand. Check your count against count_parameters.

Part 2: train three sizes, keep the best epoch

Uses The training loop, Batches, epochs and validation, Overfitting controls and state_dict() from Save and inspect results.

1. Write your own training loop in assignment/train.py, with the five steps in order. Do not call ticketnet.train; you may copy from it. Use Adam with learning rate 0.001, batches of 64, shuffled every epoch, and seed 0.
2. After every epoch, measure the training and validation loss and accuracy in evaluation mode, with no gradients. Log one line per epoch.
3. Early stopping: keep a copy of the weights from the epoch with the lowest validation loss. Stop when the validation loss has not improved for 3 epochs (a patience of 3), or after 30 epochs. At the end, load the best weights. Copy them with copy.deepcopy(model.state_dict()): a plain model.state_dict() keeps changing as the model trains.
4. Train networks with 32, 64 and 128 hidden neurons in the same way.
5. Choose with one rule, written before you look: take the smallest network whose best validation loss is within 0.005 of the lowest. Smaller is cheaper to run and to explain, so it wins a tie.
6. Plot the training and validation loss of the chosen network. Mark the best epoch.

One change at a time

Do not change the learning rate, the vocabulary or the seed between the three sizes. If you change two things at once, you cannot say which one made the difference.

Part 3: a cut-off for “unsure”

Uses Loss: how wrong is the network?, and The precision–recall trade-off in the machine learning course.

The network’s confidence for a ticket is its highest probability after softmax. A ticket with a confidence below a cut-off goes to a person.

1. Compute the confidence of every validation ticket.
2. Find the cut-off: the highest value (rounded to 3 decimals) at which at most 5% of validation tickets fall below it.
3. Report on validation: how many tickets go to a person, the accuracy on the tickets that are routed automatically, and how many of the network’s errors the person catches.
4. Answer Omar in two sentences: out of 1,000 new tickets, about how many go to a person, and about how many go to the wrong team?

Part 4: the test result, once

Uses Train, validation and test sets in the machine learning course.

1. Before you open the test set, write under Frozen choices in your report: the network size, the best epoch, the vocabulary size and the cut-off. Commit the file, or save a copy with the time.
2. Run the chosen network on the test set once. Report the same numbers as in Part 3 Step 3, and the overall accuracy.
3. Compare test with validation. If the share of tickets for a person is above 5%, say so plainly. Explain why a cut-off chosen on 600 tickets can give a different share on 600 new ones. Then answer Tomás: why must you not move the cut-off on the test set now?
4. Propose what Grace should do if the share stays above 5% on new tickets. For example: check the first weeks of real tickets, and choose a new cut-off on that data, never on the test set.

Part 5: make it repeatable

Uses Save and inspect results.

1. Save the chosen network’s state_dict, the vocabulary (as a list of tokens) and the cut-off in assignment/.
2. Write assignment/predict.py. It loads the three files in a fresh process, routes the validation tickets, and prints how many go to each team and how many to a person.
3. Check that the fresh process gives the same predictions as your notebook for all 600 validation tickets.
4. Train the chosen size again with seeds 1, 2 and 3. Report the best validation accuracy of each: how much does it move?
5. Write assignment/EXPERIMENT.md as in the lesson: question, code and versions, settings, hardware, data (with the SHA-256 of each data file), results, limits. hashlib.sha256(Path("data/train.csv").read_bytes()).hexdigest() computes one; compare it with C06/data/SHA256SUMS in the labs repository.

Part 6: an annotated transformer walkthrough

Uses Attention step by step, Transformer blocks, Position and masking, Training versus inference and Inference costs.

Write assignment/WALKTHROUGH.md. A reader who knows Python but not transformers must be able to follow it.

1. Attention by hand. Choose 3 tokens and give each a query, a key and a value of 2 numbers (your own numbers). Compute the scores, divide by √2, apply softmax, and compute the weighted sums. Show every number. Then compute the same with PyTorch, and show that the results agree to 4 decimals.
2. The causal mask. Apply a causal mask to your example. Show which scores are hidden and the new weights. Explain in two or three sentences why a model that learns to predict the next token must not see the future tokens during training.
3. SmolLM2’s shapes. Load SmolLM2-135M. Choose a prompt of your own about a Larkfield ticket. Print the token pieces and IDs, the shape of the embedding output, the shape of the hidden state after the blocks, and the shape of the logits. Explain every dimension. model(**inputs, output_hidden_states=True) also returns .hidden_states: 31 tensors of the same shape. The first is the embedding output; the last is the hidden state after the 30 blocks and the final norm.
4. One token at a time. Generate 10 tokens greedily with your own loop (not only generate). Show the top 3 next tokens with their probabilities at the first step.
5. What it costs. For your prompt plus the 10 tokens, compute by hand the KV cache size in fp32 (the formula from the lesson), and the memory of the weights in fp32 and bf16. Then say why training this model on your computer would need far more memory than running it.
6. What attention does not tell you. In three to five sentences: what can attention weights suggest about which words influence an output, and why are they not a complete explanation?

What to submit

The folder assignment/ with:

• train.py, predict.py, and the notebook or scripts for Part 6;
• the saved network, vocabulary and cut-off;
• REPORT.md with these sections:

# Larkfield ticket router: report
## Shapes                       (Part 1: every shape explained, the parameter count by hand)
## Training                     (Part 2: the three sizes, best epochs, the choice and the rule)
## Curves                       (Part 2: the loss plot of the chosen network)
## Cut-off for a person         (Part 3: the cut-off and the validation numbers)
## Frozen choices               (Part 4: written before the test)
## Test result                  (Part 4: once; comparison with validation; the answer to Tomás)
## Repeatability                (Part 5: fresh-process check, the seed spread)
## Limits                       (synthetic tickets, one language, what the router cannot know)

• EXPERIMENT.md and WALKTHROUGH.md.

Check your work

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a reason. To complete the course, all four required items must be demonstrated.

• Correct tensor shapes (required): every shape in Parts 1 and 6 is printed and explained, and the parameter count by hand matches.
• A reproducible training loop (required): your own loop with the five steps in order; predict.py in a fresh process gives identical predictions; the seed spread is reported.
• Training and validation results kept apart (required): the size and the cut-off are chosen on validation only, the choices are frozen before the test, and the test is used once.
• A clear attention and inference explanation (required): the hand calculation agrees with PyTorch, the causal mask is explained, SmolLM2’s shapes and costs are correct, and the limits of attention weights are stated.

Show the key numbers, to check your results

With seed 0 and PyTorch 2.14.1 on a CPU. Your numbers can differ a little on other computers and versions.

• Vocabulary 2,864. Train (4696, 2864), validation (600, 2864), test (600, 2864). The 32-neuron network has 91,845 parameters; 64 neurons 183,685; 128 neurons 367,365.
• Best validation loss: 32 neurons 0.0632 (epoch 15), 64 neurons 0.0630 (epoch 11), 128 neurons 0.0665 (epoch 12). The rule chooses 32 neurons (within 0.005 of the lowest).
• Cut-off 0.771. Validation: 29 tickets (4.8%) go to a person; accuracy on the automatic tickets 99.5% (3 errors); the person catches 6 of the network’s 9 errors.
• Test, once: 36 tickets (6.0%) go to a person, above Grace’s 5%. Accuracy on the automatic tickets 98.9% (6 errors); overall 96.7%.
• Seeds 1, 2 and 3: best validation accuracy 97.8%, 98.3% and 98.3%.
• Part 6: SmolLM2’s KV cache is 46,080 bytes per token in fp32. The weights are 538 MB in fp32 and 269 MB in bf16.

Recap

You turned a network into a decision that a team can rely on. You chose its size and cut-off on validation only, and you reported the test result once, even when it missed a target. You made the result repeatable in a fresh process. You also read a transformer from tiny tensors to a real pretrained model, with its costs and its limits. If you have not done one yet, choose a case study for the certificate: Classify clothing images, Deep network or gradient boosting?, Fine-tune a small transformer or Train a tiny language model.