Map goals to evidence

What you will be able to do

Decide what “good” means for Larkfield’s support assistant before you measure anything. You turn what the support team, the customers and the product manager need into criteria that anyone can check in the same way.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What Larkfield’s assistant does: it reads a ticket and returns the team, needs_human and a reply draft, as validated data. This is the project of Building Reliable Applications with LLM APIs, described in What an LLM API is.
• Why one good answer proves little, from Compare prompts in the same course.

What you need

• A browser, and a text file or a sheet of paper for the practice. This lesson has no code to run. The next lesson sets up the project.
• No account, no API key and no payment, in this lesson or in any other lesson of the course.

A driving test checks named skills

A driving examiner does not write “drives well” on the form. The form lists skills: checks the mirrors before a turn, stops fully at a stop sign, keeps a safe distance, parks within the lines. Each item is something that the examiner can see happen, or not happen. Two examiners who watch the same drive mark almost the same boxes.

Some items fail the test at once, however well the rest of the drive went. A driver who goes through a red light fails, even with a perfect parallel park. The form names those items too. So “a good driver” becomes a list of checks, agreed before the test, and a few of them are never acceptable.

The problem: “the new version feels better”

Larkfield’s support assistant works. In Building Reliable Applications with LLM APIs, Amira built it with prompt v2. It answers every ticket with a team, needs_human and a reply draft. Now Omar, the product manager, wants a change. His team wrote prompt v3: warmer replies that answer fully in one message. The prompt is also shorter, so each ticket should cost less and come back faster.

Tomás, an AI engineer, tries it first. He pastes two replies into the team chat: “Look how much better v3 is. Ship it?” Here they are, next to the replies of the current prompt for the same tickets.

Ticket	Current assistant (prompt v2)	Omar’s candidate (prompt v3)
T-64852: the tracking of a garden bench has not changed for several days.	“I’m sorry your tracking hasn’t updated. Please share your order number so our delivery team can check the shipment status and advise on the expected delivery date.”	“Good afternoon, D. Whitfield—I understand it’s frustrating when tracking hasn’t changed for several days and you’re left without a delivery estimate. Please reply with your LK order number so our delivery team can check the shipment status; I can’t confirm its current location or delivery date from the details provided. Standard delivery in Canada takes 3–5 business days. If the order is more than 5 business days late, the shipping fee is refunded, and if it is more than 15 business days late, you can choose a replacement or a full refund.”
T-65129: a fire pit arrived crushed, with a bent leg and a cracked bowl.	“I’m sorry your fire pit arrived damaged. Please send us photos of the damage within 14 days of delivery. We will then send a replacement at no cost, or a full refund if none is in stock.”	“I’m sorry your fire pit arrived in such poor condition, especially with both the leg and bowl damaged. Please send us clear photos of the crushed box and the broken fire pit within 14 days of delivery. Once we receive them, our team can arrange a replacement at no cost; if a replacement isn’t in stock, they’ll issue a full refund. You don’t need to return the damaged item.”

All four replies are real. Both prompts ran on the same model, chat-small (gpt-6-luna on Azure), and the course saved every reply.

The v3 replies are warmer and longer, and many readers would prefer them. But read the last sentence for the fire pit: “You don’t need to return the damaged item.” Grace’s policy does not say that. It may be a promise that the team must now keep. And two replies cannot answer the question of Grace, the support lead: “Is the new version better, and did it break anything that matters?” Tomás chose two tickets out of hundreds, after he read them.

“Feels better” is a vague quality claim: a statement about quality that no one can check. Two people can read the same replies and disagree, and neither can prove the other wrong.

The idea: a criterion you can check

A criterion is a rule that says what a good output must do, written so that anyone can check it. The word comes from Greek, where it means “a means of judging”. Its plural is criteria. An evaluation is a planned check of a system against criteria, on a fixed set of cases.

A criterion is measurable when two people who check the same output get the same answer. “The reply is helpful” fails this test. “The reply asks the customer for a photo of the damage” passes it: you read the reply, and the request is there or not. This is the driving-test form again. The examiner does not judge “drives well”; she checks “stops fully at the stop sign”.

A measurable criterion has four parts. Here they are for one need of Grace’s: “The replies are helpful.”

The four parts of a measurable criterion, for one of Grace's needs. The last question tests the result.

Text description of the diagram

The flow starts with the vague claim “The replies are helpful.” Part 1 says what to check in one output: the reply asks the customer for a photo. Part 2 says on which cases: the 14 cases of the development set whose next step is a photo. Part 3 says how to check it: a rule in code, or a reader who follows written instructions. Part 4 sets the bar: 14 of 14 replies. Then comes the test. Two people check the same reply: do they get the same answer? If yes, the criterion is measurable. If no, it is still vague: make part 1 or part 3 more exact, and check again.

1. What to check in one output. Policy rule 3 says that a customer with a damaged or wrong item sends a photo within 14 days. So a helpful reply to such a ticket asks for that photo. This is one fact about one reply.

2. On which cases. Not every ticket needs a photo. A question about a password needs the “Forgot password” link instead. So the criterion names its cases: the tickets whose expected next step is a photo. The course keeps 102 development cases (dev) for everyday work, and 14 of them need a photo. Module 2 explains why other cases are kept apart.

3. How to check it. Code can look for the words “photo” or “picture”. A reader can check too, with a short written instruction. Both are fine, if the method is written down. Module 3 builds the code checks, and Module 4 builds the written instructions for readers.

4. The bar. How many of the 14 must pass? For a step that the policy requires, Grace says all of them: 14 of 14. For other criteria, a bar such as “95 of every 100” can be fine.

Now apply it to Tomás’s two replies. Both fire-pit replies ask for photos within 14 days, so both pass. The tracking ticket needs a different next step: the team checks the shipment. Both tracking replies say that the delivery team will check it, so both pass that criterion too. On these criteria, v3 is not better than v2. It is the same, with more words.

The sentence “You don’t need to return the damaged item” needs a second criterion: “The reply promises nothing that the policy does not give.” This one is harder to check, because a reader must compare the reply with the policy. But it is still checkable. It also makes Grace decide a question that “feels better” hid: does the policy allow this promise?

Where the evidence comes from

A criterion needs evidence: the facts that show whether an output passes. For Larkfield, most of the evidence is stored with each test case. An evaluation set is a fixed set of cases, each with the answers that a good system gives. This course’s evaluation set has 271 tickets, and you open it in the next lesson. Here are the labels of the fire-pit ticket:

T-65129 in data/cases.jsonl (two of its fields)

"criteria": {"must": ["photo"], "must_not": ["claims_action", "asks_secret"]}
"expected": {"needs_human": false, "next_step": "photo", "rule": "", "team": "delivery"}

next_step: "photo" is part 2 of the criterion above: it puts this ticket among the 14. must_not lists what the reply must never do. It must not claim that an action was done (claims_action), such as “we refunded you”. It must not ask for a secret such as a password (asks_secret). The course author wrote these labels by applying Grace’s written policy, and scripts checked them. It is one person’s reading. Module 2 shows how the labels were made, and how you check them.

Grace, the customers and Omar need different things. The diagram maps each need to a criterion and to its evidence. The five kinds are the ones that most AI products need: correctness, usefulness, safety, latency (how long an answer takes) and cost.

Five vague claims, each turned into a criterion and linked to the evidence that checks it. The labels come from each case; the time and the cost come from each saved output.

Text description of the diagram

Five rows. Each starts with a vague claim and the person who makes it, then the measurable criterion, then the evidence.

Correctness, from Grace: “It understands the tickets.” The criterion: the team is right on at least 95 of every 100 development cases that have a team. The evidence: expected.team in each case.

Safety, from Grace: “It is safe.” The criterion: every ticket that the rules H1 to H5 cover goes to a person, and none is missed. The evidence: expected.needs_human and expected.rule.

Usefulness, from the customer: “The replies are helpful.” The criterion: the reply contains the case’s next step, for example “send a photo within 14 days”. The evidence: expected.next_step and criteria.must.

Latency, from Omar: “It is fast.” The criterion: 95 of every 100 answers arrive in 10 seconds or less. The evidence: the time of each saved output.

Cost, from Omar: “It is cheap.” The criterion: at most US$1 for every 1,000 tickets. The evidence: the tokens and the cost of each saved output.

Three things in the map are worth a second look.

First, the bars are decisions, not mathematics. Why 95 of 100 for the team, and none missed for safety? A wrong team costs a few minutes: an agent moves the ticket. A missed safety ticket can hurt someone, because no person reads it in time. Grace and Omar set the bars from the cost of each mistake. The evaluation only measures against them.

Second, the latency criterion uses a high share, not the average. “95 of every 100 answers in 10 seconds or less” is the p95 time, the 95th percentile. An average can look fine while one customer in ten waits too long. Measure usage and cost explains the median and p95.

Third, a reply can come with the right team and still not help: it can forget the photo. So the map checks the decisions and the reply separately.

Some quality is hard to measure

“The reply sounds warm” is a real need: customers notice it. But “warm” is hard to check. You have three honest choices.

• Make it more exact. “The reply says sorry when the customer reports a problem” is checkable, and it covers part of “warm”.
• Use a rubric. A rubric is a short scoring guide, with an example for each score, so that two readers score the same reply in the same way. Module 4 writes one with Grace.
• Accept it as a judgement. Say so in the report, and never let taste overrule a safety criterion.

Do not drop a need only because it is hard to measure. Criteria that check only the easy things give a confident answer to the wrong question.

When to write criteria, and what it costs

Write the criteria before you look at the new version’s outputs. If you read the v3 replies first, you notice what v3 does well, such as warmth. Then you may write criteria that v3 passes, without meaning to. The driving test works because the form exists before the drive.

Writing criteria costs time, and it starts arguments: Grace and Omar may disagree about a bar. It is better to have that argument now than after a release. When the policy changes, the criteria and their evidence must change with it. For a quick experiment, a look at a few outputs is enough. You need written criteria when a decision depends on the result, such as a release.

Guided practice: make one claim measurable

Omar says: “The new assistant must handle French tickets well.” Make it measurable with the four parts.

1. Find what one output must do. Read policy rule 7: “Answer in the language of the customer: English or French.” Write part 1: the reply is in French.
2. Choose the cases. Write part 2: the tickets written in French. The development cases have 10 of them.
3. Choose how to check it. Write part 3: a reader who reads French, or code that detects the language.
4. Set the bar. A reply in the wrong language is useless to the customer. Write part 4: 10 of 10.
5. Test it. Ask: would two people who check the same reply get the same answer? If yes, the criterion is ready.

Show an example answer

“On the 10 French development cases, every reply is in French: 10 of 10. A reader who reads French checks each reply.” This is measurable: two readers agree on whether a reply is in French.

It does not cover all of “handle French tickets well”: the decisions must also be right on French tickets. Module 3 shows how to report every measure for the French tickets separately.

Your turn: success criteria before you see the candidate

This is the module practice. You have seen only two of the candidate’s replies, and Tomás chose them. Module 4 shows you all of them. Write your criteria now, before you know the results.

1. Open a text file, success-criteria.md, or take a sheet of paper. Keep it: you use it again in Establish a baseline.
2. Write one criterion for each of the five kinds: correctness, usefulness, safety, latency and cost. Give each one its four parts.
3. Add one criterion of your own that the map above does not have. Read Grace’s policy for ideas, or use the two replies above.
4. Mark the criteria that must never fail, whatever the other results are.
5. Test each criterion: would two people get the same answer for the same output?

Show an example answer

• Correctness: on the development cases that have a team, the team is right on at least 95 of every 100. Checked by code against expected.team.
• Safety (never fails): every ticket that a rule H1 to H5 covers has needs_human: true. All of them, checked by code against expected.needs_human.
• Usefulness: on every case with a next step, the reply contains it. The bar is all of them for steps that the policy requires, such as the photo. Checked by code first; a reader checks the cases that the code cannot decide.
• Safety (never fails): no reply says that a refund, a replacement or a change was made. The assistant only drafts; a person acts (policy rule 8). Checked by code, then by a reader.
• Safety (never fails): no reply shares anything about another customer’s order or account.
• Latency: p95 time of 10 seconds or less over all cases.
• Cost: at most US$1 for every 1,000 tickets, at the provider’s current prices.
• Own criterion: every ticket gets a valid answer. A missing answer means that no team, no needs_human and no reply reach the agent.

Your list can differ. What matters is that each item has the four parts, and that someone else could check it without asking you what you meant.

Common mistake

Criteria written after reading the outputs

Symptom: Tomás writes the criteria after he reads twenty v3 replies. His list says: “Replies are warm, specific and complete, 3 to 6 sentences.” v3 passes every criterion. Grace reads the list and asks: “Where is safety? Where is cost?” No criterion mentions them.

Cause: the criteria came from the outputs, not from the needs. Tomás noticed what v3 does well, and he wrote it down as the goal. A list written this way describes the new version, so the new version passes it. Things that v3 does badly, or does not do, never reach the list.

Fix: start from the needs, not from the outputs. Ask each person what they need: Grace for the support team, Omar for the product, and the policy for the customers. Write the criteria with their four parts, and agree on them before anyone runs the new version. If a new idea comes from reading outputs later, add it as a new criterion, and check both versions against it.

Knowledge check

Check your understanding

Which of these are measurable criteria, and which are vague quality claims? (a) The replies are more professional. (b) On the 23 development cases that need a person, needs_human is true on all 23. (c) The assistant understands customers better than before. (d) The p95 time is 10 seconds or less. (e) The reply never asks for a password or a full card number.

Show the answer

(b), (d) and (e) are measurable. Each one names what to check in one output, and two people who check it get the same answer. needs_human is true or not; a time is under 10 seconds or not; a reply asks for a password or not. (b) and (d) also name their cases and their bar.

(a) and (c) are vague. “More professional” and “understands better” mean different things to different readers, and no one can prove them right or wrong. To fix (a), name what “professional” means in one reply, for example “the reply has no slang and says what happens next”. Then add the cases and the bar.

Check your understanding

Grace writes: 'Replies must be correct.' Name the four parts that are missing, with one example for each.

Show the answer

1. What to check in one output: for example, the reply contains the next step that the policy gives for this case.
2. On which cases: for example, the development cases that have a next step.
3. How to check it: for example, a phrase rule in code, and a reader with written instructions where the code cannot decide.
4. The bar: for example, all cases, for steps that the policy requires.

Then test it: two people who check the same reply must get the same answer.

Check your understanding

Why should the criteria exist before anyone looks at the new version's outputs?

Show the answer

If you read the outputs first, the criteria tend to describe what the new version does well, so the new version passes them. The criteria should come from what people need: the support team, the customers and the product. Written first, they also check what the new version does badly or forgets. A driving test works in the same way: the form exists before the drive.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Calibrate a model judge against human ratingsMeasure how well a model judge agrees with several human raters on an open dataset, find its order bias, and decide where people must still review.
• Evaluate summaries against expert ratingsCompare automatic measures of summary quality with expert ratings on an open dataset, and find which measure agrees with the experts on which quality.
• Regression-test a classifier release by sliceCompare two versions of a classifier on an open labelled set overall and by slice, with intervals, and make a release decision with blockers.

Recap

• “It feels better” is a vague quality claim. Two good replies, chosen after reading, prove nothing about hundreds of tickets.
• A criterion says what a good output must do. It is measurable when two people who check the same output get the same answer.
• A measurable criterion has four parts: what to check in one output, on which cases, how to check it, and the bar.
• For Larkfield, the needs fall into five kinds: correctness, usefulness, safety, latency and cost. Most of the evidence is in the labels of each case of the evaluation set. The time and the cost come from each saved output.
• Bars are decisions about the cost of each mistake. Some criteria must never fail.
• Write the criteria before you look at the new version’s outputs.

Next, you choose at which level to check each criterion: a plain function, one part of the assistant, a whole ticket, or production. You also set up the project.

Next: Choose evaluation layers