Review work you cannot write yourself

What you will be able to do

Review an AI experiment, a demo or a proposal without reading its code. Ask for ten pieces of evidence, read them, and decide: accept, ask for more, or reject.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• The four number labels (recorded, price, assumption, constructed), from Read the evidence pack.
• The release rule and its six checks, from Define done with evidence.
• Why the builder does not judge, from Team roles by stage.

What you need

• The template templates/experiment-review-checklist.md, copied into your document.
• The evidence pack’s 1-evaluation-results.md and 2-error-examples.md, open in your browser.

A building inspector’s report

You want to buy a house, and you are not a builder. An inspector gives you a report: “The house is in good condition.” You could not build the house, and you cannot check the wiring yourself. But you know which questions to ask about the report. Which rooms did the inspector go into? Did they go up on the roof, or look at it from the street? What did they compare the house with? Did a second inspector see the same things? What did the report not check? What will the repairs it lists cost in a year?

None of these questions needs you to build anything. Each one asks for evidence that someone else can produce. A good report answers them easily. A report that cannot answer them tells you something too.

The problem: a convincing slide

Omar brings Nadia a demo and one slide about prompt v3, the new version of the assistant’s written instructions. In the demo, v3’s replies are warmer and longer. The slide says:

Prompt v3: shorter and warmer. The request to the model is shorter: 1,477 tokens (pieces of words) against 1,902. A judge model compared the two replies to the same tickets, and preferred v3’s reply 126 times to 48.

Every number on the slide is true. All of them are recorded in the course Evaluating and Testing AI Systems. A leader who approves from this slide ships a dangerous version. In a practice replay in the course MLOps and LLMOps: Operating AI in Production, it left 186 of 1,798 tickets without an answer. Ten tickets that needed a person reached nobody. (The answers are recorded; the week is a situation constructed for that course.)

Nadia cannot read the prompt files or the evaluation scripts. She does not need to, any more than you need to rewire the house to read the inspector’s report.

The idea: review the evidence, not the code

An experiment review is a short meeting, or a written exchange, before a decision. In it, someone who did not do the work checks the evidence for a result. The reviewer does not redo the work. They ask for the pieces of evidence that every sound result has, read them, and decide.

The course’s checklist has ten lines in four groups. You need no technical knowledge to ask any of them.

The ten lines of the experiment-review checklist, in four groups. Each line asks for evidence that the author can produce; none needs you to read code.

Text description of the diagram

Why: line 1, the decision. What will we do differently if the result is good, or bad? Compared with what: line 2, the baseline, today’s way on the same examples. Line 3, the evaluation set: which examples, how many, who labelled them, and were they kept apart? Line 4, the measure that matters to users. How sure: line 5, results for the rare and risky groups. Line 6, the range, and how many examples and runs. Line 7, at least five real failures, read by a person. What it costs and hides: line 8, cost per unit and per month, with people’s time. Line 9, what was not tested. Line 10, whether someone can repeat it. The review ends with a decision: accept, ask for more, or reject. A missing line is a question, not a failure.

You already know the words on the list. The baseline is today’s way, from Problems before tools: without it, “86 of 100” means nothing. The evaluation set and its holdout come from Run a fair pilot. The range comes from Define done with evidence. The checklist puts them in the order in which a reviewer asks for them.

A tiny example

A vendor shows you five customer emails and five excellent drafts, and says “it works”. Run the checklist in your head. Line 2: there is no baseline. Line 3: five examples, chosen by the vendor, not kept apart. Line 6: one run on five examples cannot tell you much. Line 7: no failures shown, so you do not know what a failure looks like. Four questions, and no code. You do not reject the vendor. You ask for the same test on 100 of your own emails, with today’s replies as the baseline.

The checklist on Omar’s slide

Nadia asks Priya for the evidence behind the slide, and gets the evaluation pack. Here is her review. Every number is recorded in the evaluation course, except where a label says otherwise.

#	Line	What Nadia found
1	The decision	Replace prompt v2 with v3 for every ticket. Stated.
2	The baseline	Prompt v2, on the same tickets. On the 98 holdout tickets: both decisions right 94 (v2) against 83 (v3). The slide had no baseline line.
3	The evaluation set	102 development and 98 holdout tickets, made up and labelled by one person with Grace’s policy. The holdout was kept apart.
4	The measure that matters	Tickets that need a person, sent to one: 24 of 24 (v2) against 23 of 24 (v3). Tickets with an answer: 98 against 90 of 98. The slide’s judge saw no policy; with the policy, it scored v3 lower: 3.87 against 4.48 out of 5.
5	The groups	Both decisions right, v2 against v3: boundary tickets 93% against 82%, difficult 92% against 80%, “the policy has no answer” 70% against 56%.
6	The range and the runs	See the two charts below.
7	Error examples	T-64740: no answer on a safety ticket (a parasol that fell toward a child). T-65350: a warm reply that invents a “saved-card section”.
8	Cost	Model cost per 1,000 tickets: US$0.34 (v2) against US$0.48 (v3), so v3 is dearer, not cheaper. In agent time, v3 costs about US$475 more per 1,000 drafted tickets. Human review by design computed it from recorded scores and assumed minutes.
9	Not tested	Real customers, agents’ real review time, a second labeller.
10	Repeatable	Yes: the dataset version and every answer are saved.

Line 6 needs a picture. First, the range of the difference in “both decisions right”, v3 minus v2:

The difference between prompt v3 and v2, with its 95% range. On the development tickets alone, the range crosses zero; on the holdout tickets and on both groups together, the whole range is below zero (recorded).

Text description of the chart

Three rows, with a dashed line at zero, no difference. Development tickets (102): −0.059, range −0.127 to +0.010; the range crosses zero. Holdout tickets (98): −0.112, range −0.184 to −0.041; the whole range is below zero. Both groups (200): −0.085, range −0.135 to −0.030; the whole range is below zero.

Define done with evidence read these ranges for the release gate. For a reviewer, one point matters: the development tickets alone cannot show that v3 is worse, and the holdout tickets do. A small test can hide a real difference, so ask for the range, not only the average. The evaluation course explains how a range is computed, in Uncertainty and slices.

Second, the number of runs. The model does not give the same answer every time, as What AI does well and badly showed. So the team ran the 102 development tickets five times with each prompt:

Five runs of the same test. Prompt v2's runs all score higher than prompt v3's. Prompt v3 left a ticket that needed a person without an answer in only two of the five runs (recorded).

Text description of the chart

Prompt v2: 90, 96, 94, 92, 95 in runs 1 to 5. Prompt v3: 84, 81, 76, 80, 81. Prompt v3 gave no answer on 11, 16, 19, 14, 14 tickets in the five runs. A ticket that needed a person got no answer in run 1 (T-64740) and in run 3 (T-81036), and in no other run.

Every v2 run scores higher than every v3 run. But look at the most serious failure: a ticket that needs a person, with no answer. It appeared in only two of the five runs. A single run could have missed it. When a report says “no problems found”, ask how many times the test was run.

Models change fast, so these numbers are a snapshot of one model at one time. The checklist lasts.

Discuss: approve, ask for more, or reject?

Nadia has her review. The team meets.

• Omar: “The judge preferred v3 126 times to 48. Customers like warm replies. And the request is shorter, so it must cost less.”
• Tomás: “On the development tickets, the range includes zero. You cannot say that v3 is worse.”
• Priya: “The holdout tickets say it is worse, and they were kept apart for exactly this decision.”
• Grace: “One safety ticket with no answer is one too many. Who would even know? No reply means nobody is told.”

Before you read on, answer these:

1. Which lines of the checklist did Omar’s slide answer?
2. Is Tomás right? What does his point really show?
3. What do you decide, and what are your three questions to the author?

The course’s answer. The slide answered none of the ten lines well. It had no baseline and no groups, and it showed a judge that never saw the policy. Its cost claim was wrong: the shorter request gave a longer answer, so v3 costs more. Tomás is right about the development tickets, and his point shows why the holdout exists. Development tickets alone are too few to see a difference of this size. The holdout tickets, read once, show it. Reject the release of v3 as it is. The safety ticket without an answer (T-64740 and T-65663) is a blocker, and the holdout range is entirely below zero. Nadia’s three questions to Omar’s team: Why do some tickets get no answer, and does any change stop it? What do agents spend on rewriting v3’s drafts? What would a new version need to show on a fresh holdout set?

What would change the answer: a new version that answers every ticket and sends every ticket that needs a person to one. It must also score at least as well as v2 on a fresh holdout set, in at least five runs. That is a new experiment and a new review, not a second look at this one.

When to use the checklist, and what it costs

Use it for every experiment report, demo, vendor proposal and release request that asks you for a decision. Do not use it to grade people: a missing line is a question for the author, not a mark against them. For a small, cheap-to-undo change, lines 1, 2, 7 and 9 are often enough. What it costs: about 30 minutes to read a report with the list, and a day or two for the author to fill the gaps. The slide in this lesson shows the cost of skipping it.

Common mistake

Approving on the average

Symptom: a report says “decisions right: 84 of 102”, and the leader approves, because 82% sounds good.

Cause: one number with no baseline, no groups and no range. The average hides the rare and risky tickets, where the cost is.

Fix: ask for lines 2, 5 and 6 before you decide: today’s way on the same tickets, the results by group, and the range.

Common mistake

Asking to read the code

Symptom: a leader who cannot read code asks for the code “to be safe”, reads it for an hour, and approves because it “looks professional”.

Cause: the belief that only someone who can write the work can review it. Code that looks clean says nothing about how the product behaves on real tickets.

Fix: ask for the evidence instead. A technical reviewer can check the code; the leader checks that the result answers the decision.

Your turn

Change one condition. Omar takes the judge’s comparison off the slide and brings it alone. “A strong judge model compared v2 and v3 on 200 tickets. It preferred v3 126 times, v2 48 times, with one tie.” Use lines 2, 4, 6 and 7 of the checklist, and the pack’s 2-error-examples.md (Example 3). What do you find, and what do you ask?

Show a worked solution

• Line 2, the baseline: present. The judge compared v3 with v2, today’s version, on the same tickets.
• Line 4, the measure that matters: missing. The judge did not see Grace’s policy, so it rewarded the reply that sounded more helpful. With the policy, the same judge scored v3 lower: 3.87 against 4.48 out of 5.
• Line 6, the range: a warning. The judge’s verdict changed with the order of the two replies on 25 of 200 pairs.
• Line 7, error examples: Example 3, ticket T-65479. Without the policy, the judge preferred v3’s reply, “sign in and update your saved details”, in both orders. With the policy, it scored that reply 2, because the policy has no such self-service step.

Ask: What did the judge know about the policy? How do its scores compare with a person’s on the same replies? Show me ten replies where the judge preferred v3.

See it end to endGo/no-go memo for a support reply assistantCase study

In the case study, a judge model without the support policy preferred prompt v3 on 126 tickets and prompt v2 on 48. With the policy, it scored v3 lower. You read what the judge was told before you use its verdict.

Knowledge check

Check your understanding

A report says 'our new model gets the right team for 89 of 100 tickets'. Which checklist line do you ask about first, and why?

Show the answer

Line 2, the baseline: what does today’s way score on the same 100 tickets? Without it, 89 of 100 means nothing. If today’s way gets 70, it is a big improvement; if it gets 97, it is a step back. After that, ask line 4 (is “right team” the measure that matters to users?) and line 5 (the rare and risky groups).

Check your understanding

Predict: on the 102 development tickets, v3 minus v2 is −0.059, with a range from −0.127 to +0.010. What can you conclude, and what do you ask for?

Show the answer

You cannot conclude that v3 is worse, or that it is as good. The range includes zero, so these tickets are too few to show the difference. Ask for the holdout tickets, which were kept apart for the decision. On them, the range is −0.184 to −0.041, entirely below zero: v3 is worse. You can also ask for more runs or more tickets, but never choose the result that you like best.

Check your understanding

A team reports 'we ran the safety test and found no problem'. Which line of the checklist does this touch, and what do you ask?

Show the answer

Line 6, the range and the runs, and line 5, the groups. Ask how many times the test was run, and on how many safety tickets. In Larkfield’s five runs, a ticket that needed a person got no answer from prompt v3 in only two runs. One run could easily have shown “no problem”. Also ask for line 7, the error examples, so that a person reads what the failures look like.

Recap

• You do not need to read code to review AI work. An experiment review asks for evidence that every sound result has, and decides: accept, ask for more, or reject.
• The checklist has ten lines in four groups. Why: the decision. Compared with what: baseline, evaluation set, the measure that matters. How sure: groups, range and runs, error examples. What it costs and hides: cost, not tested, repeatable.
• Omar’s slide was true and misleading. It had no baseline and a judge without the policy. Its cost claim was wrong: US$0.48 against US$0.34 per 1,000 tickets.
• The development tickets alone could not show that v3 is worse (−0.127 to +0.010); the holdout could (−0.184 to −0.041). The most serious failure appeared in 2 of 5 runs.
• A missing line is a question for the author, not a failure.

Next, you make reviews a habit: experiment reviews, error-analysis sessions and decision records, and a policy for coding assistants.

Next: Team rituals and AI-assisted work