Define done with evidence

What you will be able to do

Turn “good enough” into thresholds that anyone can check, on an agreed set of examples, with one named person who decides when they are met. Without them, every gate of your roadmap is a matter of opinion, and the most enthusiastic person wins.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Experiments and time boxes, from Why AI plans are different.
• Stages, gates and stop criteria, from Milestones and roadmaps.

What you need open

In the course’s practice materials:

• templates/definition-of-done.md, the template that you fill in.
• evidence/pilot-evidence-pack/1-evaluation-results.md and 4-operations.md (its last table is the release gate).
• For the module practice at the end: evidence/fixed-date-plan.md and templates/stage-gate-roadmap.md.

The lesson takes about 25 minutes; the module practice takes about 15 more.

The driving test

A driving test has its rules written down before anyone drives. The examiner chooses the route. The examiner counts the small faults, and a set number of them fails the test. One serious fault, such as going through a red light, fails it at once, however well you drove otherwise. The examiner decides, not the instructor and not the learner driver. And nobody changes the pass mark after the drive because the learner “felt confident”.

Four things make the test fair: a fixed route, a written pass mark, faults that fail alone, and one person who decides. Remove one of them and the result becomes an opinion.

The problem: “the team is happy with the answers”

The practice plan fixed-date-plan.md says that weeks 5 and 6 are done when “the team is happy with the answers”. Nobody can check that. Happy with which answers? How happy? Who is “the team”?

Larkfield has evidence of what “happy” can do, from Run a fair pilot. Without Grace’s policy, a judge model preferred v3’s warm, long replies 126 times to 48. A team that judged by feel would have released v3. With the policy, the same judge scored v3 3.87 out of 5 on average against 4.48 for v2, because many warm replies invented a step. “Good” depends on what you measure, and on what you measure it against.

The idea: thresholds on an agreed set, and one owner

Three ideas from the driving test make “done” checkable.

• An evaluation set is the agreed list of examples with their right answers: the route of the test. Larkfield’s set has 102 development tickets, which the team may study, and 98 holdout tickets, kept apart and read once for the decision. The evaluation course teaches how to build and label one.
• An acceptance threshold is a number that a result must reach on that set, written before anyone sees a result: the pass mark. Some thresholds are blockers: one failure fails the version alone, like a red light.
• A sign-off owner is the one named person who reads the evidence and says “done” or “not done”: the examiner.

Together, on one page, they are a definition of done. The course’s template has one line for each kind of measure, with its threshold, the reason, today’s result and whether it must pass.

A definition of done: an agreed set, measures, thresholds written first, and one owner who reads the evidence and decides.

Text description of the diagram

The flow has five steps. It starts with the agreed evaluation set: 98 holdout tickets, kept apart and read once. The set feeds the measures: blockers, answers, decisions, cost and time. The measures get thresholds, written before anyone sees a result. Then one owner reads the evidence. If every line is met, the result is “done”, and the work goes to the next gate. If one line is missed, the result is “not done”: stop or change. A dashed arrow leaves the thresholds with a question: were they changed after the result? If so, they measure nothing.

A tiny example: the release gate, line by line

You met Larkfield’s release gate in Run a fair pilot. The team wrote it in the MLOps course after “last week”. It is a definition of done with six lines. Here it is with both prompts’ results on the 98 holdout tickets, all recorded.

Line	Threshold	Prompt v2	Prompt v3	v3 met?
Tickets that need a person and get no answer (blocker)	at most 0	0	1 (T-65663)	no
Tickets without an answer	at most 0	0	8	no
“Both decisions right”, v3 minus v2, lower end of the range	not below −0.030	today’s version	−0.184	no
Tickets that need a person, sent to a person	100%	24 of 24	23 of 24	no
Model cost per 1,000 tickets	at most US$0.40	US$0.34	US$0.48	no
Time to an answer, slowest 5%	at most 8.0 s	4.9 s	8.5 s	no

Read how each threshold was chosen. The first and fourth lines come from harm: a safety ticket that reaches nobody is never acceptable, so the limit is zero. The cost and time lines sit between today’s result and a level that would hurt. Prompt v2 costs US$0.34, so a limit of US$0.40 leaves room for a useful change. So a good threshold comes from today’s result and the cost of a mistake, not from a round number.

Prompt v3 misses all six lines. No meeting is needed to argue about it. That is the point of writing the lines first.

The hard line: “not worse”

The third line needs a closer look. A difference measured on about 100 tickets has noise, so the evaluation shows it with a range. The range holds the true difference in 95 of 100 such tests. The gate’s rule reads the lower end of the range: it must not be below −0.030, which is 3 points worse than today.

Each bar is the range that holds the true difference in 95 of 100 such tests. The gate needs the left end of the bar to the right of the limit. The dashed line at 0 means no difference. Recorded in the evaluation course.

Text description of the chart

Three rows show the difference in “both decisions right”, prompt v3 minus prompt v2, with its range. On the 102 development tickets, the difference is −0.059, with a range from −0.127 to +0.010: it crosses zero. On the 98 holdout tickets, it is −0.112, from −0.184 to −0.041. On both groups together, it is −0.085, from −0.135 to −0.030. A line at zero marks “no difference”, and a line at −0.030 marks the gate limit. Every row’s range starts to the left of the gate limit.

The development row teaches the most. Its range crosses zero, so those tickets alone cannot show that v3 is worse. A hopeful team could say “no proven difference, ship it”. But the gate asks the opposite question: can we show that v3 is not much worse? The lower end, −0.127, is far below the limit, so the answer is no, even on the development tickets. A threshold on the lower end puts the burden of proof on the new version, where it belongs.

Two more rules complete the definition. How many runs? The model gives different answers on different runs, and v3’s worst failure appeared in only 2 of 5 runs. So the template asks for at least five runs, and a blocker in any run fails the version. Which set? The holdout tickets are read once, for the decision. The 71 earlier examples, on which the prompt was written, do not count, because results there look better than they are.

Discuss: who signs, and is 3 points right?

Nadia writes the definition of done for the next prompt change, before any result exists. Two questions remain open.

• Omar: “I wrote the change, and I know it best. I should sign it off.”
• Grace: “The safety lines need my name. When a safety ticket reaches nobody, my agents face the customer.”
• Priya: “On about 100 tickets the ranges are wide. On the development tickets, v3’s range runs from −0.127 to +0.010. A big change may never pass a 3-point limit on that few tickets.”
• Omar: “Then make the limit 10 points. Three is just a number someone chose.”

Before you read on, answer these:

1. Who should sign: Omar, Grace, Priya or Nadia? Why?
2. If the ranges are too wide to pass, should Nadia loosen the limit or change something else?
3. Should a change meant to cut cost have to be “better”, or only “not worse”?

The course’s answer. Nadia signs, because she owns the support team’s results and carries the decision. Omar proposes, and a person should not approve their own change. Grace co-signs the two safety lines: either of them can say “not done” alone. Priya confirms that the numbers were computed as the page says. She checks the evidence; she does not decide. If the ranges are too wide, Nadia adds test tickets; she does not loosen the limit. With both groups together, 200 tickets, v3’s range narrowed to −0.135 to −0.030. More examples make a narrower range. And a cost change only needs “not worse”, because its gain is the lower cost.

What would change the answer: a change whose purpose is better quality must show it. The lower end of its range must be above zero, not only above the gate limit. And a limit may change before the next experiment, for a reason written down with the owner’s name. It never changes after a result.

When to use it, and what it costs

Write a definition of done for every gate where a decision depends on quality: at least before the pilot and before any launch. Every line needs its set, its threshold, its reason and today’s result. Add a line for each group of examples that matters: Larkfield’s tickets where the policy has no answer scored lowest for both prompts.

It costs time. Someone must build and label the evaluation set; the evaluation course shows how. The owner must read the evidence, not only the summary. And a threshold can be wrong. Too strict, and no change ever passes. Too loose, and harm passes. Both mistakes are easier to see and fix on a written page than in a meeting’s memory.

Common mistake

Done, on the wrong examples

Symptom: a new version scores very well, better than anyone expected. The report says “done”.

Cause: it was scored on examples that the team had already studied, for example the 71 earlier examples that the prompt was written on. Results on those look better than they are, as a student’s marks on the practice exercises do.

Fix: name the evaluation set and its version on the definition of done. Score the decision on the holdout tickets, once, and ask which set every number came from.

Consequence in AI work: a definition of done without a named set lets the most flattering set win. The thresholds look met, and customers meet the real quality.

See it end to endGo/no-go memo for a support reply assistantCase study

The go/no-go memo checks Omar’s proposal against written thresholds like these, with the full pilot evidence pack.

See it end to endRecover a stalled AI projectCase study

The stalled project in this case study had no agreed evaluation set, measure or threshold, so it was “nearly ready” every month. Your recovery plan writes a definition of done for its first gate.

Module practice: from a fixed date to gates

The demonstration was Larkfield’s roadmap in Milestones and roadmaps and the release gate above. Now you do the same for a new plan.

Guided practice. Open fixed-date-plan.md: the website product chat, constructed for the course.

1. Find the milestone that should show whether the chat answers well enough. You find weeks 5–6, “the team is happy with the answers”: nobody can check it.
2. Look for a result that stops the project. You find none.
3. Find the success line. “Customers love the chat” cannot be measured. “Fewer customers write tickets about products” can be.
4. Write down the worst wrong answer that the chat could give. A wrong answer about safety or size is the worst, as the plan’s question 4 says.

Your task. Fill the stage-gate template for the website chat with three gates: after feasibility, after the pilot and after the limited launch. Give each stage a question, a time box inside the 12 weeks, the evidence to pass and a stop criterion. Name the owner. Then write one acceptance threshold for the first gate.

Show a worked solution

This is one good answer. Every time box, set size and threshold here is constructed for the practice. Your numbers can differ if you give a reason.

Owner: Nadia decides at every gate, because the success measure, fewer product tickets, is her team’s result. Omar sponsors and proposes. Grace co-signs the safety line, because her agents answer when the chat is wrong.

Stage	Question	Time box	Evidence to pass	Stop criterion
1. Feasibility	Can the chat answer product questions correctly from our pages and manuals?	Weeks 1–4	An agreed set of 100 real product questions from past tickets, with right answers written by a product expert, including 20 safety and size questions. Right on at least 90 of 100, five runs	One wrong answer on a safety question, in any run; or an answer about a product that is not in our pages
2. Pilot	Does it help on real questions, with a person checking?	Weeks 5–6	Agents use the chat to draft answers to product tickets and mark each one right or wrong. Agents mark at least 90% right, and cost per 1,000 questions is within budget	An agent finds a wrong safety or size answer that they would have sent
3. Limited launch	Does it work for customers on one product category?	Weeks 7–9	The chat runs on one category’s pages, with “report a wrong answer” on every answer. Product tickets for that category fall against similar categories	One wrong safety answer reaches a customer: the chat goes off for that category at once

Gate 3 leads to full launch, with the same stop criterion as a pause condition. Week 12 compares product tickets per 1,000 orders before and after, which answers the plan’s question 5.

One acceptance threshold for gate 1: the agreed set of 100 product questions runs five times. Every safety and size question is answered correctly in every run (20 of 20, five times). It is a blocker. Nadia decides, and Grace must sign it.

What makes it good: each stage asks a new question and shows the chat to more people. Each stop criterion is a result, not a date. The weeks 1–6 of building and improving became experiments with a gate.

Completion check. Write one acceptance threshold for Larkfield’s support reply assistant, and say who decides when it is met.

Show an example answer

“On the 98 holdout tickets, run five times: every ticket that needs a person is sent to a person, 24 of 24 in every run. It is a blocker. Nadia decides; Grace co-signs; Priya confirms the count.”

A complete threshold has five parts. They are the measure, the agreed set with its number of runs, the number, blocker or not, and the one person who decides. It is written before anyone sees a result. “The assistant is accurate enough” has none of them.

Knowledge check

Check your understanding

On the development tickets, prompt v3's range for 'both decisions right' runs from −0.127 to +0.010. The range crosses zero. Does v3 pass the gate line 'lower end not below −0.03'? Why does the gate use the lower end?

Show the answer

No. The lower end, −0.127, is below −0.03. Because the range crosses zero, those tickets cannot show that v3 is worse. But the gate asks the opposite question: can we show that v3 is not much worse? It cannot. The lower end puts the burden of proof on the new version, so “no proven difference” is not enough to pass.

Check your understanding

Omar proposed prompt v3 and offers to sign its definition of done. Who should sign, and why not Omar?

Show the answer

Nadia signs: she owns the support team’s results and carries the decision. Grace co-signs the safety lines, because her agents face the customer when a safety ticket reaches nobody. Priya confirms that the numbers were computed as written. Omar proposes the change, and a person should not approve their own change: hope and effort make every result look good.

Check your understanding

Without Grace's policy, a judge model preferred v3's replies 126 times to 48. With the policy, it scored v3 lower. What does this tell you about a 'done when' line such as 'the team is happy with the answers'?

Show the answer

Whether an answer is “good” depends on what it is measured against. Warm, long replies look better until someone checks them against the policy: then many invent a step. “The team is happy” has no set, no measure, no threshold and no owner. A definition of done names all four, written before the result.

Recap

This lesson:

• A definition of done names the evaluation set, the acceptance thresholds (some of them blockers), how many runs, and the one owner who signs. All of it is written before anyone sees a result.
• Thresholds come from today’s result and the cost of a mistake. Prompt v3 missed all six lines of Larkfield’s release gate.
• “Not worse” reads the lower end of the range. If the range is too wide, add examples; never loosen the limit after a result.

This module: AI plans start as experiments with a time box, because nobody knows the quality in advance. A roadmap joins them into four stages, from feasibility to full launch, with a gate, pass evidence and a stop result at each. A definition of done makes each gate checkable, and one owner decides. You turned a fixed-date plan into a three-gate roadmap.

Next, you decide where a person reviews or approves the AI’s work, and what that review costs.

Next: Human review by design