Regressions and operational value

What you will be able to do

Decide whether the adapter ships. You look for what it does worse than the system in use, and why. Then you weigh time, memory and cost, and write the decision with its evidence.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The ladder of baselines and the holdout result, from Compare baselines.
• The coverage table of the training data (no French, no rules H2 to H5), from Clean, split and check for leakage.
• Blockers and acceptable regressions, from Trade-offs and regressions in Evaluating and Testing AI Systems.
• A threshold turns a score into a yes or no, from Thresholds and calibration in Machine Learning: From Problem to Reliable Model.

What you need on your computer

• ~/projects/ticket-adapter after Compare baselines, with .venv active: python -m pytest shows 30 passed, 2 deselected. If it does not, copy the snapshot end-of-m04, and do the copy step of that lesson again.

No model run is needed: every reply and probability is saved. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are written but not tested.

The fastest lifeguard

A swimming pool tests a new lifeguard for two weeks. He is the fastest swimmer they have ever seen. His uniform is perfect, and he is never late. During the tests, 23 swimmers act as if they are in trouble. He blows his whistle for none of them.

On most days, nobody is in trouble. So if you count “right decisions” over all the hours, he scores well. But nobody hires him. The rare moment is the reason the job exists.

The problem: an average hides the rare cases

On the holdout cases, the adapter gets 87 of 97 teams right and 65 of 98 decisions. Those are averages. In the last lesson, the adapter also sent 0 of 24 tickets to a person. An average over 98 tickets cannot show you which kinds of ticket it fails.

A regression is something that worked in the system that you have, and fails in the new one. To find them, look where failure costs most. Look at the slices (groups of cases of one kind), at safety, and at what the model could do before training.

Look at the slices

This is still the final reading of the holdout: you choose nothing with it.

1. Copy this lesson’s files from the snapshot end-of-m05:

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m05
   for f in adapt/regressions.py adapt/costs.py prices.toml docs/result-memo.md tests/test_operations.py; do cp "$S/$f" "$f"; done
   cp -R $S/data/general data/
   cp -R $S/runs/general $S/runs/probes runs/

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m05"
   foreach ($f in 'adapt\regressions.py', 'adapt\costs.py', 'prices.toml', 'docs\result-memo.md', 'tests\test_operations.py') { Copy-Item "$S\$f" $f }
   Copy-Item -Recurse "$S\data\general" data\
   Copy-Item -Recurse "$S\runs\general", "$S\runs\probes" runs\

2. Run the tests: python -m pytest. Expect 34 passed, 2 deselected. Your project now matches the snapshot end-of-m05.

3. Show the decisions per slice for three systems:

   python -m adapt compare --split holdout --by slice --systems chat-small words adapter-long

Note: the holdout cases are for the final decision only. Read them once, at the end.
98 cases (holdout)
system          answers     team  caught  false  decisions  s/answer
chat-small       98/98    97/97   24/24       4     94/98      2.813
    boundary      decisions 13/14
    difficult     decisions 29/29
    french        decisions 10/10
    needs_person  decisions 21/21
    normal        decisions 11/11
    unanswerable  decisions 10/13
words            98/98    86/97    8/24       0     74/98      0.000
    boundary      decisions 14/14
    difficult     decisions 28/29
    french        decisions 2/10
    needs_person  decisions 8/21
    normal        decisions 11/11
    unanswerable  decisions 11/13
adapter-long     98/98    87/97    0/24       0     65/98      0.237
    boundary      decisions 13/14
    difficult     decisions 28/29
    french        decisions 1/10
    needs_person  decisions 0/21
    normal        decisions 10/11
    unanswerable  decisions 13/13
caught = needs a person, sent to one; false = sent to a person without a rule; s/answer = median

Two rare slices of the holdout. The hosted assistant gets them all right; the adapter gets 0 of 21 tickets that need a person and 1 of 10 French tickets.

Text description of the chart

Needs a person (21 cases): hosted assistant 100%, word-count model 38% (8 of 21), adapter 0%. French (10 cases): hosted assistant 100%, word-count model 20% (2 of 10), adapter 10% (1 of 10).

On the ordinary slices, the adapter is close to the hosted assistant. It loses 29 decisions, almost all in two slices. It sends no ticket to a person. And it gets 1 of 10 French decisions, where on dev it got 7 of 10. Ten cases are too few to measure French. But no training record is in French, so do not trust it with French tickets.

Why the adapter never says true

When the adapter writes its answer, it reaches the word after "needs_human":. There it has two choices, true or false, and it gives each a probability. Greedy decoding takes the larger one. So the adapter says true only when P(true), the probability of true, is above 0.5.

The course’s authors saved P(true) for every dev ticket in runs/probes/lora-r8-long.json. Three examples, with every value visible:

Ticket	Rule	P(true)	P(false)	The adapter says
T-65270, a sharp edge on a compost bin near children	H1 safety	0.38	0.62	false
T-64709, the hammock that “isn’t safe to use”	H1 safety	0.12	0.88	false
T-81026, a complaint to the consumer protection office	H2 legal	0.0006	0.9994	false

• Rule H1, safety (in the training data)
• Rules H2 to H5 (never in the training data)
• Above the line at 0.5, the adapter says true

P(true) for the 23 dev tickets that need a person, lowest first. No point reaches the line at 0.5, so the adapter never says true.

Text description of the chart

The 13 tickets under rules H2 to H5 are all between 0.0002 and 0.024: on this scale they lie on the floor. The 10 safety tickets under rule H1 are between 0.013 and 0.38, and 9 of them are above 0.08. The highest point, 0.38, is still below the line at 0.5.

The chart shows what the adapter learned. It learned a little of rule H1: safety tickets get a higher P(true) than the others, but never enough. It learned nothing of rules H2 to H5. The cause is the data. Only 78 of the 900 training records need a person, all under rule H1. No record shows a legal threat, a request against the policy, someone else’s order, an unreadable ticket, or French. The coverage table in Module 2 showed these zeros before anyone trained.

You could lower the threshold: say true when P(true) is above 0.1, not 0.5. Count what that does:

1. Create try_threshold.py in the project folder:

   try_threshold.py

   import json
   
   rows = json.load(open("runs/probes/lora-r8-long.json", encoding="utf-8"))["rows"]
   dev = [r for r in rows if r["set"] == "dev"]
   needs = [r for r in dev if r["needs_human"]]
   for t in (0.5, 0.1, 0.05):
       h1 = sum(r["p_true"] >= t for r in needs if r["rule"] == "H1")
       other = sum(r["p_true"] >= t for r in needs if r["rule"] != "H1")
       alarms = sum(r["p_true"] >= t for r in dev if not r["needs_human"])
       print(f"threshold {t}: caught H1 {h1}/10, H2-H5 {other}/13, false alarms {alarms}")

2. Run it: python try_threshold.py. Then delete the file.

threshold 0.5: caught H1 0/10, H2-H5 0/13, false alarms 0
threshold 0.1: caught H1 8/10, H2-H5 0/13, false alarms 3
threshold 0.05: caught H1 9/10, H2-H5 0/13, false alarms 9

A lower threshold catches most safety tickets, at the price of false alarms. It never catches a ticket under H2 to H5. It is a patch, not a fix: the model cannot report a rule that it never saw.

What the adapter forgot

Training can also break what a model could do before: catastrophic forgetting, from Full and parameter-efficient tuning. The project asks the base model and both adapters 20 ordinary questions:

python -m adapt regressions

base     right 12/20   replied with ticket JSON  0/20
         g01: 'The capital of France is Paris.'
         g02: '7 + 5 = 12'
         g03: '"Merci" in French is "merci".'
lora-r8  right 10/20   replied with ticket JSON  0/20
         g01: ''
         g02: '7 + 5 = 12'
         g03: '"Bonjour."'
lora-r8-long right 11/20   replied with ticket JSON  0/20
         g01: ''
         g02: '7 + 5 = 12'
         g03: "'Au revoir.'"

No adapter answers ordinary questions with ticket JSON, and the scores are close: 12, 10 and 11. But both adapters give an empty reply to “What is the capital of France?”, and translate “thank you” wrongly. Twenty questions can only show a large loss, and there is none. They cannot prove that nothing was lost. LoRA changed 0.34% of the weights, and removing the adapter gives the base model back.

Time, memory and cost

python -m adapt cost

822 tickets a day (assumption: Larkfield's average day).

Hosted model now (chat-small, recorded run, dev):
  measured: US$0.35 per 1,000 tickets (median 1902 tokens in, 276 out)
  = US$0.28 a day, US$8.53 a month

Local adapter (this computer, 2 threads, dev):
  measured: median 0.24 s per ticket = 3.2 minutes of CPU a day
  assumption: it runs on a computer you already have; its price, power and upkeep are not counted
  measured: training took 16.4 minutes once (794,124 tokens seen)

Hosted fine-tune (list prices from prices.toml; prices change: check the current page; not run here):
  price: training gpt-4.1-nano on the same tokens: US$1.19
  price: hosting the fine-tuned model: US$1.70 an hour = US$1,241 a month (730 hours), before any ticket
  (token counts here are this course's tokenizer; the hosted model counts its own way: an assumption)

Each number says what kind it is. Measured comes from a saved run. Price comes from a published price list, which changes. Assumption is a number that someone chose. The adapter is fast (0.24 s against 2.98 s) and used about 1.26 GB of memory in the measured runs. It keeps tickets on Larkfield’s computers. Its training time, 16.4 minutes, counts the training steps only; with validation and saving, the run took about 19 minutes, as in Monitor and recover. It saves at most US$8.53 a month: US$0.000346 per ticket × 822 tickets a day (an assumption) = US$0.284 a day. Even 100 times the tickets would save only about US$853 a month. The chat-small model has no fine-tuning price, so a hosted fine-tune means another model, and hosting it costs more than the assistant.

The adaptation scorecard

The scorecard puts the targets, written before training, beside the holdout results and the regression checks.

The adaptation scorecard: the holdout results of three systems against targets.toml, then the regression checks.

Text description of the diagram

The adapter meets the format, false-alarm and speed targets. It misses the team, decisions and needs-a-person targets. The hosted assistant misses only the speed target, which was written for a local model.

Discuss: ship it for routing only?

The scorecard is on the screen, and Tomás has a new plan.

• Tomás: “The adapter routes well. Ship it for the team, and let the word list send safety tickets to a person.”
• Kwame, reading the cost report: “It saves US$8.53 a month, at our assumed volume.”
• Omar: “What do customers see?”
• Grace: “A ticket about an injury that waits in the wrong queue.”

Before you read on, answer these:

1. Is a lower threshold, or the word list, a fix for the safety decision?
2. Does the adapter meet the team target on its own?
3. What evidence would make you accept a routing-only adapter?

The course’s answer. Do not ship. A lower threshold catches some H1 tickets and none under H2 to H5. The word list caught 8 of 24 on holdout. Each is a patch on a model that never learned the rules. Routing alone also fails: 87 of 97 teams against a target of 95%, while the hosted assistant gets 97 of 97. So customers would get slower help, to save about US$8.53 a month. A lower validation loss, perfect JSON and speed do not outweigh a missed safety ticket.

What would change the answer: reviewed training records for every needs-a-person rule and for French. Then a new run, measured on dev and on new held-out cases. Or a design where the adapter only routes and the hosted model or a person decides needs_human, if the routing reaches 95%. Either must be measured before anyone believes it.

When to stop, and what it costs

Stop when the evidence answers the question, even when the answer is no. Write the decision down as a result memo: what you tried, the results on the same cases, why, and what would change the decision. It takes little time. It saves the next team from trying the same thing blind, and it keeps the evidence for a later attempt.

Common mistake

Choosing by the lowest loss

Symptom: the 6-epoch run has the lowest validation loss of the course (0.0193). The team ships it, and no ticket ever reaches a person.

Cause: the loss is an average over every answer token of the 100 validation records. Only 9 of them need a person, so their true token is a small part of that average.

Fix: judge the model by the targets, the slices and the blockers, on held-out cases. Use the loss to choose between runs of the same setup, not to decide whether to ship.

Module practice

Guided practice → independent task. Open docs/result-memo.md. It is Priya’s result memo, the worked example. For each number in its table, find the command that prints it: compare, compare --split holdout, regressions or cost. Then write your own.

Your task: write a result memo for the first adapter, adapter (2 epochs), in docs/my-result-memo.md. Use only saved results: python -m adapt targets adapter, python -m adapt failures adapter, the holdout table of the last lesson, and regressions. Accept or reject it, and say what would change your decision.

Show a model answer

adapter on dev against targets.toml:
  met      answers                     target >= 0.99  measured 1.00
  NOT MET  team                        target >= 0.95  measured 0.81
  NOT MET  needs a person caught       target >= 1.00  measured 0.00
  met      false alarms                target <= 12    measured 0
  NOT MET  decisions                   target >= 0.88  measured 0.67
  met      median seconds per answer   target <= 1.0   measured 0.21
3 of 6 targets met

# Result memo: the 2-epoch adapter (lora-r8)

**Decision:** do not ship. Keep the hosted assistant.

## What we tried
A LoRA adapter (rank 8, query and value layers, 460,800 trainable parameters) on
SmolLM2-135M-Instruct, 2 epochs on 900 cleaned records (configs/lora-r8.toml).
Validation loss 0.0584.

## Results (same cases, python -m adapt compare)
- Dev: valid 102/102, team 81/100, needs a person caught 0/23, false alarms 0,
  decisions 68/102, 0.22 s. Targets met: 3 of 6.
- Holdout (read once): team 74/97, caught 0/24, decisions 57/98.
- Hosted assistant, holdout: team 97/97, caught 24/24, decisions 94/98.
- Word-count model, holdout: decisions 74/98, caught 8/24.
- General questions: 10/20 right (base 12/20); empty reply to "capital of France".

## Why
19 dev tickets go to the wrong team and all 23 that need a person are missed.
The training data has 78 needs-a-person records, all rule H1, and no French.

## What would change the decision
Reviewed records for rules H2 to H5 and French, then a new run, measured on dev
and on new held-out cases. Or a design where the adapter only routes and reaches
95% of teams. Training longer is not enough: the 6-epoch run also caught 0/23.

The 2-epoch adapter routes worse than the 6-epoch one and has the same blocker. The decision does not depend on its speed or its loss.

See it end to endShould this team fine-tune?Case study

In the case study, more training data makes the intents better (143 of 150) and the stop rule worse: the adapter catches 40 of 50 out-of-scope messages, one fewer than before. The rare class got rarer, and the memo says no-go.

Module check

Check your understanding

Retrieval: what is a regression, and what is catastrophic forgetting? Give the evidence for each from this module.

Show the answer

A regression is something that works in the system you have and fails in the new one. The adapter gets 0 of 24 holdout tickets that need a person (hosted: 24 of 24). It gets 1 of 10 French tickets (hosted: 10 of 10). Catastrophic forgetting is when training on a new task makes a model lose old skills. The 20 general questions show no large loss: 12 right for the base model, 11 for the adapter. But the adapter gives an empty reply to “capital of France”, and 20 questions cannot prove that nothing was lost.

Check your understanding

Explain or predict: Tomás sets the threshold for needs_human to 0.02 on dev. What happens to the H1 tickets, the H2 to H5 tickets and the false alarms? Why can no threshold fix this adapter?

Show the answer

It catches 9 of the 10 H1 tickets and 1 of the 13 others (T-81034, at 0.024), with 15 false alarms. The H2 to H5 tickets have a P(true) between 0.0002 and 0.024, almost the same as ordinary tickets. A threshold low enough to catch them sends many ordinary tickets to a person too. The model cannot separate what it never learned. The fix is in the data: reviewed examples of every rule.

Check your understanding

Completion check: accept or reject the 6-epoch adapter. Use evidence other than the training loss, and say what would change your decision.

Show the answer

Reject it. On the holdout cases, read once, it sends 0 of 24 tickets that need a person to one. The target is every one, and the go/no-go memo made this a blocker. It also misses the team target (87 of 97, target 95%) and the decisions target (65 of 98, target 88%). It gets 1 of 10 French decisions. It meets the format, false-alarm and speed targets. It would save about US$8.53 a month at an assumed 822 tickets a day. None of that outweighs a missed safety ticket. Its lower validation loss (0.0193) did not change the safety result. Reviewed records for rules H2 to H5 and French, and a new run measured again, could change the decision.

Recap

This lesson: a regression is something that worked before and fails now. Look at the slices, the safety cases and the old skills. The adapter sends 0 of 24 holdout tickets to a person, because its P(true) never reaches 0.5. It learned a little of rule H1 and nothing of the rules that its data never showed. A lower threshold is a patch. Catastrophic forgetting is not large here (11 of 20 against 12), but 20 questions prove little. Label every cost number as measured, price or assumption.

This module: compare against a ladder of strong baselines on the same cases, and read the holdout once. Judge the adapter by the targets written before training, not by its loss. The result memo says: do not ship the adapter, and keep the hosted assistant. Models and prices change fast: these numbers are a snapshot of one model and one set of runs; the method lasts.

Reset: if your project differs, copy the snapshot end-of-m05. Your project: 34 passed, 2 deselected.

Next, you package the adapter anyway, so that a later attempt can load it, verify it and roll it back.

Next: Package and deploy an adapter