Retrain or revise

What you will be able to do

When the evidence says that the assistant is failing, choose the change that removes the cause. It can be a data fix, a prompt change, a new index or a new model. A symptom alone never chooses the change. Then send that change through the gate and a canary like any other release.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Drift, delayed labels and the RAG regressions, from Module 4: Data and concept drift, Delayed labels and LLM/RAG regressions.
• The gate and the canary, from Evaluation gates and Controlled rollouts.

What you need on your computer

• Your project after the last lesson: python -m pytest prints 213 passed.
• The saved telemetry in data/telemetry/. This lesson reads saved files only: no Docker, no services.

The car that pulls to one side

Your car pulls to the left. You could buy a new car, and it would stop pulling. It would also cost a fortune, and you would never learn why the old one pulled.

A mechanic asks questions first. Is one tyre soft? Is the wheel alignment off? Is the road itself sloped? Each question has a quick check, and each answer points to a different repair: air, an alignment, or nothing at all. A new car is the last option, for when nothing else works.

The assistant has the same choice. “Retrain the model” is the new car.

The problem: one symptom, three different causes

Look at three of the course’s saved runs. In each one, customers got worse answers:

• “Last week”: many tickets without an answer, and agreement with the agents fell to 83.7%.
• The policy-silent block of the drift run: agreement fell to 85.3%, below the 90% SLO.
• The stale-index run: answers quoted a delivery price that was no longer true.

Tomás has one answer for all three: “Retrain the model.” It would fix none of them. The model belongs to the provider, and it did not change in any of the three. The first cause was a new prompt. The second was a gap in the shop’s policy. The third was an old copy of a document. A new model would still read the same prompt, the same policy and the same old document.

The same symptom in three saved runs. A different check points to a different cause in each, and so to a different change.

Text description of the diagram

All three runs show the same symptom: worse answers for customers. In “last week”, the release history shows a release at minute 5, and the quality report shows 10.3% of v3’s tickets without text. The cause is the prompt, so the change is a rollback and a revised prompt. In the policy-silent tickets, there is no release and no language drift, but agreement fell to 85.3%. The cause is the policy, so the change is the policy text, then the prompt. In the stale-index run, there is no release. The index check says that the index is older than the documents, and 22 answers cite old sources. The cause is the index, so the change is a rebuild.

The idea: diagnose, then make the smallest change that fixes the cause

Retrain means changing the model: fine-tuning it on your own labelled tickets, or moving to another model. Revise means changing what the model is given: the data, the prompt, the policy or the documents it searches. This course names four kinds of change:

Change	Fixes	Example at Larkfield	Cost
Data fix	wrong inputs, labels or documents	a wrong label in the evaluation set; a wrong document	small; a new evaluation
Prompt change	instructions, policy text, output format, token limit	the policy says nothing about a new payment option	small; the gate and a canary
New index	answers that search an old copy of the documents	the delivery price changed, the index did not	very small; one command
Model adaptation	a task that the model cannot do with any prompt you tried	none so far	large: labelled data, training, a full evaluation, and a new dependency

Every change, small or large, is a new release. It gets a manifest, it goes through the gate, and it reaches customers through a canary.

A tiny example: read the evidence, not the alarm

Before any change, you judge a candidate with numbers. Here is the shadow run of prompt v3 against v2, checked against the project’s rollback rules. Every step is visible:

Run Python in your browser

Judge a candidate with the rollback rules

The box prints each rule with its calculation. It finds 42 answers without text, a cost change of +39.8% against +20%, and a p95 change of +66.4% against +50%. All 3 rules are met, so it says reject. Try the other candidate in the last comment.

Python code (you can change it)# The shadow run of prompt v3 against the active prompt v2 (ops canary --shadow). shadow = {"no_text": 42, "cost": 0.471, "p95": 8.22} active = {"no_text": 0, "cost": 0.337, "p95": 4.94} # The rollback rules of operations/rollout.toml. rules = {"no_text_max": 1, "cost_increase": 0.20, "p95_increase": 0.50} cost_change = shadow["cost"] / active["cost"] - 1 p95_change = shadow["p95"] / active["p95"] - 1 print(f"answers without text: {shadow['no_text']} (rule: {rules['no_text_max']} is enough)") print(f"cost: {shadow['cost']} / {active['cost']} - 1 = {cost_change:+.1%} (rule: +{rules['cost_increase']:.0%})") print(f"p95: {shadow['p95']} / {active['p95']} - 1 = {p95_change:+.1%} (rule: +{rules['p95_increase']:.0%})") failed = [ shadow["no_text"] >= rules["no_text_max"], cost_change > rules["cost_increase"], p95_change > rules["p95_increase"], ] print(f"{sum(failed)} of 3 rules met: {'reject' if any(failed) else 'go on with the canary'}") # Try: a candidate with no_text 0, cost 0.36 and p95 6.9.

answers without text: 42 (rule: 1 is enough)
cost: 0.471 / 0.337 - 1 = +39.8% (rule: +20%)
p95:  8.22 / 4.94 - 1 = +66.4% (rule: +50%)
3 of 3 rules met: reject

The numbers say what is wrong: no text, more cost, more time. They do not yet say why. That needs the questions of the decision tree.

The decision tree

The project’s docs/improve.md writes the questions as a table: symptom, what to check first, the likely cause, the change. The tree puts them in the order that saves the most time. The cheap, common causes come first, and model adaptation comes last.

From a symptom to the change that fixes its cause. Each question has a command that answers it. Model adaptation is the last box.

Text description of the diagram

Start from the symptom and ask five questions in order. Did it start right after a release? Then roll back first and revise the release. Do answers cite an old document? Then rebuild the index. Did the provider’s model version change? Then pin it, or evaluate the new one. Do tickets wait while the answers are right? Then it is an outage, not a quality problem. Do the agents’ reviews agree less? If not, only the inputs changed: change nothing, and review the new kind of ticket. If they do, read the tickets that the agents corrected. A gap in the policy leads to a policy and prompt change. Wrong labels or data lead to a data fix. Only when every prompt fails, and labelled data exists, does the tree reach model adaptation. Every change then becomes a new release, through the gate and a canary.

Walk the tree with the saved runs

“Last week”: the first question answers it

python -m ops quality data/telemetry/release-full.jsonl.gz --by release

release                         1.5.0-4fafd1       1.5.0-767b56
------------------------------  -----------------  -----------------
tickets                         602                1,798
  ok                            601 (99.8%)        1,602 (89.1%)
  no_answer                     1 (0.2%)           186 (10.3%)
  dead_letter                   0 (0.0%)           10 (0.6%)
sent to a person                162 (27.0% of ok)  475 (29.7% of ok)
tokens in (median)              1,902              1,477
tokens out (median)             260                627
triage call p50 / p95 (s)       2.89 / 5.17        5.56 / 8.34
time to a draft p95 (s)         5.2                7.9
cost per 1,000 tickets (US$)    0.338              0.474
agrees with the agent's review  288/307 (93.8%)    623/744 (83.7%)

The two columns are two releases of the same service, with the same model on the same kind of tickets. Only the prompt differs. The tokens out more than doubled, from 260 to 627. v3 makes the model reason longer, and some answers run out of tokens before any text. So the cause is the prompt. The change: roll back now, then revise the prompt, for example with the larger token limit of EXP-002 in Experiment records. Then send the revision through the gate.

The policy-silent block: the last question

No release happened in the drift run, the index was current and the model version did not change. The reviews do agree less:

python -m ops labels data/telemetry/drift.jsonl.gz --windows 1000

3,000 tickets, 1,445 reviewed (48%); review delay median 120 s, p95 121 s
window       tickets  reviewed  agrees (95% interval)  no answer  to a person  French
-----------  -------  --------  ---------------------  ---------  -----------  ------
     1-1000     1000       474  95.1% [92.8%, 96.7%]        0.1%        25.1%    5.4%
  1001-2000     1000       495  95.8% [93.6%, 97.2%]        0.1%        24.0%   24.9%
  2001-3000     1000       476  85.3% [81.8%, 88.2%]        0.3%        24.8%    3.4%

So the tree says: read the tickets that the agents corrected. Priya reads them in the help desk tool. The course’s authors did the same count with the saved run: they matched each reviewed ticket with its practice ticket. In the last block, 57 of the 70 corrected tickets came from one slice: questions that the shop’s policy does not answer.

Block	Reviews of policy-silent tickets	Corrected	Agreement on that slice
1–1,000	52	11	78.8%
1,001–2,000	31	7	77.4%
2,001–3,000	197	57	71.1%

The assistant was always weak on these tickets, below 80%. In the last block, they became 41% of the reviews instead of 11%. The model did not get worse; the policy has no answer for these questions, and they became common.

The change is a prompt change: Grace writes the missing policy text, for example which team handles questions about the new payment option. Then Priya adds the corrected tickets to the evaluation set, so that the next gate tests them. A new model would read the same silent policy and guess, as this one did.

The stale-index run: the second question

python -m ops rag data/telemetry/stale-index.jsonl.gz --windows 1.5

The rows from minute 3 to 6 show old sources 8 and 14, with index behind 14 and 26. The model and the prompt did not change; the documents did. The change is a new index: python -m scripts.reindex. And a prevention: run --check after every document change.

When is a new model the right change?

Sam asks, “So do we never retrain?”

Priya answers: “We do, when the evidence says the model itself is the limit.” Model adaptation is the right change when all of these are true:

• You tried the cheaper changes: the prompt, the policy, the data and the index, each with an evaluation.
• The evaluation shows the same failures with every prompt you tried, on tickets that people agree about.
• You have enough labelled tickets to train on, and separate frozen tickets to evaluate on.
• Someone owns the new model: its cost, its updates and its evaluation.

Moving to a bigger model is also model adaptation, and the gate judges it like any other candidate. You do that in the task below.

What it costs: a fine-tune needs hundreds or thousands of labelled tickets, a training run, a full evaluation and a new release. The cheaper changes need an afternoon. A drift alert never justifies any of them alone, as Data and concept drift showed.

Module practice: assess a candidate release

This module’s practice task: assess a candidate release against quality, latency and cost targets. First, a worked example with every step visible. Then you assess a different candidate yourself.

Worked example: prompt v3, offline and live

Assess prompt v3 against the targets of operations/gates.toml and operations/rollout.toml:

Target	Limit	v3 offline (holdout)	v3 live (shadow, 543 tickets)	Result
Critical tickets	0 failed	T-65663 failed	(not judged)	fail
Answers without text	0 offline; 1 live stops it	8 of 98	42 of 543 (7.7%)	fail
Decisions, lower end of v3 − v2	≥ −0.03	−0.184	no reviews in a shadow	fail
Cost per 1,000 tickets	≤ US$0.40; ≤ +20% live	US$0.480	0.471 / 0.337 − 1 = +39.8%	fail
Triage p95	≤ 8.0 s; ≤ +50% live	8.534 s	8.22 / 4.94 − 1 = +66.4%	fail

Recommendation: reject. The offline evidence and the live shadow agree on every target. The live numbers come from real ticket traffic, not from the evaluation set. They confirm that the holdout result was not bad luck.

Your turn: the bigger model

Kwame proposes another candidate: “Keep prompt v2, but use the bigger deployment, chat-strong. A better model must be safer.” The evaluation course ran prompt v2 on chat-strong once, on its 102 development tickets. Its harness printed, on the same tickets as the v2 baseline:

strong-v2-chat-strong-r1-160302 on dev (102 cases)
valid answers 102/102 | team 100/100 | needs a person: caught 23/23, false alarms 15 | reply criteria passed 100/102

decisions  baseline 90 | candidate 87 | difference -0.029 [-0.098, 0.039] (n=102) | better 5, worse 8

The evaluation course measured its cost at US$5.149 per 1,000 tickets, with a triage p95 of 5.283 s. No critical ticket failed. For v2 on the same dev tickets: 9 false alarms.

1. Write operations/evidence/assistant-v2-strong.json with the same fields as assistant-v3.json. Use the run ID strong-v2-chat-strong-r1-160302 and the split dev.
2. Run python -m ops gate assistant-v2-strong.
3. Write your recommendation in three to five sentences: the decision, the checks that decide it, and what the evidence cannot say.

Worked solution

{
  "prompt": "assistant-v2",
  "model": "chat-strong (gpt-6.1-sol-2026-09-29)",
  "source": "the evaluation course's recorded run strong-v2-chat-strong-r1-160302 (real calls)",
  "evaluation_set": "d4c96eaa10c0",
  "split": "dev",
  "run": "strong-v2-chat-strong-r1-160302",
  "cases": 102,
  "valid_answers": 102,
  "answers_without_text": 0,
  "needs_human_caught": [23, 23],
  "cost_per_1000_usd": 5.149,
  "latency_median_s": 2.872,
  "latency_p95_s": 5.283,
  "blockers": [],
  "decisions_difference_vs_baseline": [-0.029, -0.098, 0.039]
}

python -m ops gate assistant-v2-strong

Release gate for assistant-v2-strong (baseline assistant-v2), evidence: strong-v2-chat-strong-r1-160302 on the dev split of d4c96eaa10c0
check                                            value  limit     result  detail
----------------------------------------------  ------  --------  ------  -----------------------
blockers (critical cases failed)                     0  <= 0      pass    none
answers without text (holdout)                       0  <= 0      pass    of 102 tickets
decisions: lower end of (candidate - baseline)  -0.098  >= -0.03  FAIL    -0.029 [-0.098, +0.039]
needs a person: share caught                     1.000  >= 1.0    pass    23 of 23
cost per 1,000 tickets (US$)                     5.149  <= 0.4    FAIL    baseline 0.339
triage time p95 (s)                              5.283  <= 8.0    pass    baseline 4.896
Decision: DO NOT SHIP (2 failed: decisions: lower end of (candidate - baseline), cost per 1,000 tickets (US$))

A good recommendation: “Do not ship chat-strong. It costs US$5.149 per 1,000 tickets, about 15 times v2 and 13 times the budget. Its decisions are not shown to be as good: 87 against 90 correct, and the interval reaches 9.8 points worse. It has no blocker and no answer without text, and it catches every ticket that needs a person. But it sends 15 tickets to a person without need, against v2’s 9. The evidence is also on the 102 development tickets only; a release decision needs the frozen holdout.”

Three details matter. The gate’s first line says on the dev split: the table’s label still says “holdout”, so read the header. A safer-sounding model is not a safer release; the evidence decides. And the gate compares runs, not prompts: the prompt is v2 in both, but the runs differ, so the difference is real.

Delete the file when you have checked it. The snapshot end-of-m05 has only the evidence of v2 and v3.

Completion check: recommend a rollout or a rejection

Omar writes after the recorded canary: “Only 3 of 25 canary tickets went wrong, and none was a safety ticket. 88% is fine. Ship v3 to everyone for a week, and we will look again.”

Write the reply in five sentences or fewer. Recommend a rollout or a rejection, with the evidence from ops gate, ops canary and the shadow run. Say what new evidence would change your decision.

Worked solution

“I recommend that we reject v3 and keep v2. The gate fails all six checks. They include a safety ticket without an answer (T-65663) and a cost of US$0.480 per 1,000 against our budget of US$0.40. The canary agrees: 3 of 25 tickets had no text, against 2 of 2,375 for v2. The 543 shadow tickets agree too: 7.7% without text, +40% cost and +66% triage time. A week for everyone would repeat ‘last week’, where 10 tickets that needed a person reached nobody. A new candidate, such as EXP-002’s larger token limit, must first pass the gate.”

A good answer rejects, and it quotes evidence from all three sources with numbers. It explains why “only 3 of 25” is not small: 12% of customers got no answer, and a safety ticket can be the next one. It names what would change the decision: a new candidate with new evidence, not more time with the same one.

See it end to endMonitor a model on public data that changesCase study

The case study compares three retraining policies on a year of public data. The drift alarm fires every month, so “retrain on drift” becomes a monthly schedule, and a model trained on one odd month learns only that month.

Knowledge check

Check your understanding

Name the four kinds of change in this lesson, with one Larkfield cause that each one fixes.

Show the answer

A data fix: wrong labels, documents or inputs, such as a wrong label in the evaluation set. A prompt change: the instructions or the policy text, such as a policy that says nothing about a new payment option. A new index: answers that search an old copy of the documents, such as the old delivery price. Model adaptation: a task that the model cannot do with any prompt tried, with labelled data to train on. Larkfield has no such case yet.

Check your understanding

Predict: agreement with the agents falls from 95% to 86%, and no release, document change or model-version change happened. What do you check next, and which change do you expect?

Show the answer

Read the tickets that the agents corrected, and look for a common slice. In the drift run, 57 of 70 corrected tickets were questions that the policy does not answer. That slice was always weak, and it became common. The expected change is the policy text and then the prompt, with the tickets added to the evaluation set. It is not a new model: the model did not change, and a new one would read the same silent policy.

Check your understanding

Practical check: assess the chat-strong candidate with your project. Which command, output lines and recommendation prove it?

Show the answer

With the evidence file of the task, python -m ops gate assistant-v2-strong prints Decision: DO NOT SHIP (2 failed: …). The two failed checks are the cost (5.149 against 0.4) and the lower end of the decisions difference (−0.098 against −0.03). The header says on the dev split. The recommendation rejects the candidate, quotes both failed checks and says that the evidence is from dev tickets only.

Recap

This module put evidence in the way of every change. Now you can:

• Combine offline quality, blockers, latency and cost into one release gate that a script enforces, and read why prompt v3 fails all six checks.
• Release a candidate safely: a shadow run with no customer at risk, then a canary for a share of the tickets. Write rollback rules in advance, and give three pieces of evidence of recovery.
• Explain how the canary’s share trades speed of learning against customers touched. The recorded canary caught v3 at the 9th canary ticket; 25 customers got v3, against 1,798 “last week”.
• Walk from a symptom to its cause with the decision tree. Choose the smallest change: a data fix, a prompt change, a new index, or model adaptation last.
• Assess a candidate release against quality, latency and cost targets, and recommend a rollout or a rejection with evidence.

Next, Module 6 handles what goes wrong anyway: runbooks, a provider outage and a post-incident review.

Next: Runbooks