Define success and constraints

What you will be able to do

Write down what an adapted model must reach, and what it may cost, before anyone trains it. Then the evidence decides, not the hope of the person who did the work.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• The four kinds of failure and Larkfield’s two failure tables, from Clarify the failure.
• Blockers and acceptable regressions, from Establish a baseline in the evaluation course.

What you need on your computer

• The project ticket-adapter from the last lesson, with .venv active: python -m pytest gives 6 passed. If it does not, copy the start snapshot again and repeat that lesson’s steps.

A list before you visit the car dealer

Before you buy a used car, you write a short list at home. Spend at most 8,000, have five seats, and pass a brake inspection. At the dealer, a shiny car with a great sound system costs 9,500, and it is tempting. The list protects you from your own excitement. One item is not for sale at any price: no sound system makes up for bad brakes.

The problem: after training, every result looks like progress

Imagine the experiment is over, and the adapted small model answers in valid JSON 102 times out of 102. Against the base model’s 0 out of 102, that is a huge step. Against the hosted assistant’s 101 out of 102, it is nothing new. Without a target written first, the team chooses the comparison after it sees the result, and the result always wins.

The idea: targets, constraints and a stop rule, written first

• A target is a number that the result must reach, on named cases, measured by a named command. For example: right team on at least 95% of the dev cases.
• A constraint is a limit on how you get there: the data you may use, the time, memory and money, and the deployment needs.
• A stop rule is one result that ends the work, whatever the other results are. It is the brakes in the car list.
• A go/no-go memo is one page that writes all of this down before training. Its decision is “go” or “no-go” for the next step, here one small experiment.

Good targets come from the system you have. The hosted assistant is the bar: an adapter that is worse must be clearly faster, cheaper or more private to be worth it.

Larkfield's targets, constraints and stop rule, next to what the hosted assistant measured on the dev cases.

Text description of the diagram

The diagram has three parts. On the left are the six targets from targets.toml, each with the hosted assistant’s measured value on the dev cases. Valid answers: at least 99%; the hosted assistant has 101 of 102. Right team: at least 95%; hosted 97 of 100. Needs a person caught: 100%; hosted 23 of 23. This one has a red border. False alarms: at most 12; hosted 9. Both decisions right: at least 88%; hosted 90 of 102. Median time per answer: at most 1.0 second; hosted 2.98 seconds. On the right are three constraints. First, only data that Larkfield may train on, never the evaluation cases. Second, one training run under 30 minutes and 3 GB of memory on a laptop, with no paid job. Third, a pinned base model, fixed checks and a rollback to the hosted model. Below them, a red box holds the stop rule, linked to the needs-a-person target. One ticket that needs a person and does not reach one means no-go, at any price. A note says that all of it is written before training, checked on dev while choosing, and on holdout once at the end.

Most targets sit a little below the hosted assistant: a small loss in quality may be worth a large gain in speed. The needs-a-person target does not. The hosted assistant fails only the time target, the real reason for the experiment.

A tiny example: is cost a reason?

Tomás says that a small model is cheaper. Kwame checks with numbers. The hosted assistant’s recorded run cost US$0.000346 per dev ticket on average. Larkfield gets about 822 tickets a day. That is an assumption, taken from the average day of Scaling APIs and AI Workloads:

• per day: 822 × US$0.000346 = US$0.284, about US$0.28
• per month of 30 days: US$0.284 × 30 = US$8.53

So a small model can save at most about US$8.53 a month. And someone must still pay for the work to build and maintain it. Cost is not a reason here. Speed and privacy may be. Prices change: check the provider’s current page before you repeat this.

Add the targets to the project

macOS and Linux

S=../nextia-learning-labs/C12/snapshots/end-of-m01
cp $S/adapt/targets.py adapt/
cp $S/targets.toml $S/README.md .
cp -R $S/docs .
cp $S/tests/test_scoring.py tests/

Windows

$S = "..\nextia-learning-labs\C12\snapshots\end-of-m01"
Copy-Item $S\adapt\targets.py adapt\
Copy-Item $S\targets.toml, $S\README.md .
Copy-Item -Recurse $S\docs .
Copy-Item $S\tests\test_scoring.py tests\

targets.toml holds the six targets, with the hosted values in comments. adapt/targets.py checks a system against them. docs/decision-memo.md is Priya’s go/no-go memo.

1. Run the tests. You see 9 passed. Your project now matches the snapshot end-of-m01.

2. Score the hosted assistant. adapt/scoring.py counts each decision; a reply with no valid answer counts as “no person”:

   python -m adapt score chat-small

   chat-small on dev: answers 101/102 | team 97/100 | needs a person: caught 23/23, false alarms 9 | decisions 90/102

3. Check it against the targets:

   python -m adapt targets chat-small

   chat-small on dev against targets.toml:
     met      answers                     target >= 0.99  measured 0.99
     met      team                        target >= 0.95  measured 0.97
     met      needs a person caught       target >= 1.00  measured 1.00
     met      false alarms                target <= 12    measured 9
     met      decisions                   target >= 0.88  measured 0.88
     NOT MET  median seconds per answer   target <= 1.0   measured 2.98
   5 of 6 targets met

4. Predict how many targets the small model meets. Then run python -m adapt targets base-short. The last line is 2 of 6 targets met: false alarms (0) and median time (0.40 s).

The small model meets two targets by doing nothing useful. So read the targets together, with the stop rule first: it catches no ticket that needs a person, and the rest does not matter.

The go/no-go memo

Open docs/decision-memo.md. Priya wrote it before any training. It has five short sections: the need, what fails today, the alternatives, the targets and constraints, and the decision. Its decision: go for one small, controlled experiment; no decision to deploy. It names its stop rule: a model that misses a safety ticket is not acceptable at any price.

The memo is step 2. Each later step is checked against what it wrote, and the stop rule can end the work at any step.

Text description of the diagram

The flow has five steps. First, measure the system you have on the dev cases. Second, write the targets, the constraints and the stop rule in the go/no-go memo. A note says that the targets do not change after anyone sees a result. Third, go for one small experiment. Fourth, check every target on the dev cases. If one is not met, the result is no-go: write down why and keep the system you have. If all are met, fifth, check them on the holdout cases, once. If one is not met there, it is no-go too. If all are met, go: package, check and deploy the adapter. A red box at the side holds the stop rule. At any step, a ticket that needs a person and does not reach one leads to no-go.

“No-go” is a valid result, not a failure of the project. A memo that says why the system you have stays is useful work.

Common mistake

A target that moves after the result

Symptom: a result misses a target, and someone edits targets.toml. To see the effect, set needs_person_recall_min = 0.0 and run python -m adapt targets base-short:

  met      needs a person caught       target >= 0.00  measured 0.00
3 of 6 targets met

A model that sends no ticket to a person now “meets” the needs-a-person target.

Cause: the target was changed to fit the result. It now describes the model, not the need.

Fix: copy targets.toml from the snapshot again. Change a target only before the next experiment, with a written reason in the memo, and never because of a result.

Consequence in AI work: a target that moves after the result measures nothing. Keep targets.toml under version control, so that every change shows its date and its reason.

Discuss: which targets, and which is the real reason?

Priya shows the team her draft targets, next to the hosted assistant’s measured values.

• Tomás: “The hosted bar is too high for a model of 135 million parameters. Set the team target at 85% and needs-a-person at 90%. People can catch the rest.”
• Grace: “Who catches a safety ticket that the model sent to the normal queue? Nobody looks at it in time. Needs-a-person stays at 100%.”
• Kwame: “It saves at most US$8.53 a month. If we do this, it is for speed or privacy, not for money.”
• Omar, the product manager: “What would a customer notice? A faster sort saves about two seconds. The customer still waits for a person’s reply.”

Before you read on, answer these:

1. Should the needs-a-person target be 100% or 90%?
2. Is a team target below the hosted assistant’s (95% against 97%) fair?
3. Which target is the real reason for the project?

The course’s answer. Needs-a-person stays at 100%, as a stop rule. The hosted assistant reaches it, so a lower bar is a step back that customers feel, and no speed gain pays for it. A team target a little below the hosted value is fair, because routing mistakes are cheap to fix and the speed gain is large. The real reason is speed (median ≤ 1.0 s) and keeping tickets at Larkfield, not money. Priya’s memo says “go” only for one experiment, because no evidence yet shows that a small model can meet the stop rule.

What would change the answer: a law that keeps tickets at Larkfield would make privacy a constraint, not a nice extra. A hundred times more tickets (about US$853 a month) would make cost a real reason.

Your turn: a memo for another narrow task

The returns team asks for something new: read each ticket and extract its order number, such as LK-561208. Tomás proposes a small adapter for it. (This request is made up for practice; the counts below are measured on the real dev cases.) Measure first. Start Python in the project folder (python), and paste these lines:

import re
from adapt.cases import load_cases

cases = load_cases()
strict = [c.case_id for c in cases if re.search(r"LK-\d{6}", c.text)]
loose = [c.case_id for c in cases if re.search(r"LK-\d{6}", c.text, re.IGNORECASE)]
numbers = [c.case_id for c in cases if re.search(r"\d{5,}", c.text)]
print(len(cases), "cases:", len(strict), "strict,", len(loose), "any case,", len(numbers), "with a long number")
print("only with any case:", sorted(set(loose) - set(strict)))

102 cases: 25 strict, 26 any case, 26 with a long number
only with any case: ['T-64800']

Type exit() to leave Python. Now write a go/no-go memo of about half a page, with the five sections of Priya’s memo. Name the kind of failure, the cheapest fix, a target, one constraint and the decision.

Show a worked solution

Need: the returns team wants the order number of each ticket. What we measured: 26 of the 102 dev tickets contain a long number. Every one is an order number: LK- and six digits. A pattern finds 25; one ticket, T-64800, writes it in lower case (lk-920334). A pattern that ignores case finds all 26. Kind of failure: none yet: no system was tried. If a model wrote the number in another shape, that would be format, and its fix is structured output, not training. Alternatives: a pattern in code (free, instant, easy to test); structured output on the hosted model; an adapter (examples, training, a new model to maintain). Target: every order number found on the dev cases, no number invented. Constraint: the pattern must ignore case; test it on the holdout cases once. Decision: no-go for an adapter. A pattern meets the target on the dev cases today. What would change it: many tickets with order numbers that a pattern cannot read, such as “the order from last Tuesday”.

See it end to endShould this team fine-tune?Case study

The case study writes five targets and a stop rule for a travel app’s help chat before any training: catch at least 45 of 50 out-of-scope messages on dev. The prompt-only system it has today catches 48, and the stop rule decides the memo.

Knowledge check

Check your understanding

Explain why adaptation is unnecessary for the order-number task, and why it is still worth testing for Larkfield's routing. Use one measured number for each.

Show the answer

Order numbers: a pattern that ignores case finds all 26 order numbers in the 102 dev tickets. A cheaper fix already meets the target, so an adapter adds cost and a model to maintain for nothing. Routing: the small model gives no valid answer (0/102) today. But a small local model could answer well under the 1.0-second target (hosted median: 2.98 s), and keep tickets at Larkfield. Whether it can make the decisions as well is task behaviour, which only an experiment shows. So the memo says “go” for one experiment, not for deployment.

Check your understanding

The small model meets 2 of the 6 targets without giving a single valid answer. Which two, why, and what does this teach about reading targets?

Show the answer

False alarms (0; the target is at most 12) and median time (0.40 s; the target is at most 1.0 s). With no valid answer, it sends no ticket to a person, so it cannot raise a false alarm. Prose replies also come back fast. Targets read one by one can reward a system that does nothing. Read them together, and apply the stop rule first. The small model catches 0 of 23 tickets that need a person, so it is no-go.

Check your understanding

Which evidence would make you change the needs-a-person target from 100% to something lower? Is there any?

Show the answer

No result of the model should lower it. The hosted assistant reaches 23 of 23, and a missed safety ticket reaches nobody in time. The target could change only if the system around the model changes. For example, a person reads every ticket within minutes anyway, so a missed flag is caught. That is a new design with its own evidence, written in a new memo before the next experiment. It is never a reason to change the target after a result.

Recap

This lesson:

• A target is a number on named cases with a named command. A constraint limits how you get there. A stop rule ends the work at any price. Write all three in a go/no-go memo before training.
• Targets come from the system you have. The hosted assistant fails only the time target (2.98 s against 1.0 s): the reason for the experiment.
• Cost is not a reason here: the hosted assistant costs about US$8.53 a month at 822 tickets a day (an assumption).
• A system that never answers meets 2 of 6 targets. Read targets together, and apply the stop rule first.

This module: the hosted assistant needs no fine-tuning, the small model fails on format, and the memo says “go for one experiment”. Your project matches the snapshot end-of-m01 (9 passed).

Next, you build the training examples, and you record where each one comes from and what you may use it for.

Next: Example structure and permissions