Select representative cases

What you will be able to do

Choose the test cases so that the evaluation sees the work that the assistant really gets, including the rare tickets where a mistake costs most. Then count them by group, so that you see what the set is missing before a missing group hides a failure.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What Grace needs from the assistant, and the criteria that you wrote for it, from Map goals to evidence.
• The baseline run, its run ID and the non-negotiable failures, from Establish a baseline.

What you need on your computer

• The project eval-harness as it is at the end of Module 1, with its virtual environment active. python -m pytest -q ends with 8 passed.
• If your project is missing or broken, copy the snapshot C11/snapshots/end-of-m01 from the labs repository, as in Reset the project. Then make the virtual environment again.

No account and no key. This lesson reads saved files only.

Tested with Python 3.12.13 and pytest 9.1.1 on macOS (Apple silicon). On Windows the Python commands are the same; they are not tested there.

A spoon of soup

A cook makes a large pot of soup and tastes it before it goes to the guests. One spoon from the top tells the cook little. The salt can sit at the bottom, and the bottom is also where the soup burns. So the cook stirs, then tastes from the middle, the side and the bottom. Each spoon is small, but together they tell the cook about the whole pot.

A test set works in the same way. Each test case is one spoon. The set tells you about the assistant only if the spoons come from every part of the pot. The part where it burns matters most.

The problem: a test set that is too easy

In Building Reliable Applications with LLM APIs, Amira tested the assistant on 69 tickets. Priya counts what they contain. Only 1 of the 69 tickets is in French, although Camille’s team in Montreal answers French tickets every day. Only 9 tickets need a person. And the prompts in that course were near the ceiling. The test could only show a difference of one to three tickets, and that proved nothing.

Tomás has a quick answer: “Let us add more tickets like these.” But more of the same spoons tell the cook the same thing. Here is a weakness that the 69 tickets could not show. The baseline that you recorded in Module 1 is weakest on questions that the policy does not answer. One example is “What is the address of your warehouse?” On 27 such questions, it gets the decision right only 19 times. It often sends them to a person when no rule says so. A set without these questions never shows that weakness.

The idea: cover the work, then add the dangerous cases

The evaluation set, as Map goals to evidence defined it, is a fixed list of test cases, each with the answer that a good system gives. You run every version of the system on the same set and compare the results. The set is the evidence for every claim that you make about the assistant, so its contents decide what you can claim.

A good evaluation set does two jobs at the same time:

1. It looks like the real work. It has the kinds of tickets that customers send, in their real languages and styles. Some are short, some are long, some have typos or two requests. This part is representative: like the cook’s spoons, it is a fair sample of the pot.
2. It covers the rare, costly cases on purpose. A safety ticket can be 1 ticket in 100 in real life. A random sample of 200 tickets then has only about 2 safety tickets, too few to measure anything. So you add more of them than real life has. This part is not representative, and that is deliberate.

To see whether both jobs are done, you put each case into one group, called a slice, and you count the cases in each slice. A slice is a part of the set that you measure on its own, such as “French tickets”. The table of counts per slice is the coverage table. It shows at once which slices are big enough to measure and which are missing.

A tiny example: six tickets, six slices

Larkfield’s set has six slices. Each case goes into exactly one slice: the first slice in this order that fits it. French comes first, then “needs a person”, then “the policy does not answer”, then “boundary”, “normal” and “difficult”. Here are six real tickets from the set, one for each slice:

Ticket	What the customer writes (start)	Slice	Why this slice
T-81018	“Je n’arrive plus à me connecter à mon compte…”	french	It is in French. French comes first, so the password question does not decide the slice.
T-81035	“Please reset the password on my husband’s account…”	needs_person	Another customer’s account: rule H4 says that a person must handle it.
T-81058	“What is the address of your warehouse?…”	unanswerable	The policy does not give an address.
T-65796	“I sent back my cast iron frying pan a few weeks ago and still no refund…”	boundary	It sits on the policy’s line between the returns team and the payment team.
T-65781	“Raised veg bed LK-730158 started leaking from one corner…”	normal	A clear warranty question in plain English.
T-65114	“Can opened pruning shears still be returned?”	difficult	Very short: the customer gives almost no detail.

Count them, and each slice has 1 case. That is the coverage table of these six tickets. With one case per slice, one mistake moves a slice’s score from 100% to 0%. Six tickets show the idea, but they are far too few to measure anything.

Larkfield’s 271 cases

Priya built the course’s evaluation set with both jobs in mind. The tickets are made up for this course; no real customer wrote them. They come from four places. Read the diagram from left to right.

Where the 271 cases come from. The 200 new cases are split into dev and holdout; the 71 tickets that the prompt has already seen stay apart.

Text description of the diagram

Four sources feed the evaluation set. The first source is 140 tickets from the deep learning course, chosen by team and writing style. The second is 60 tickets written for this course: 20 in French, 25 that need a person and 15 that the policy does not answer. These 200 cases are shuffled within each slice and dealt in turn into two parts: dev, with 102 cases, and holdout, with 98 cases. The third source is the 69 tickets of the LLM applications course, on which prompt v2 was written and compared. The fourth is the 2 examples written inside prompt v2. These 71 cases form a third part, contaminated, drawn with dashed lines because the prompt has already seen them.

The 140 reused tickets were chosen with fixed quotas: for each team and each writing style, a fixed number of tickets. The styles are plain, boundary, two topics, a negation, typos, a distracting detail, the request at the end, a long story and very few words. A quota keeps one style from filling the set by chance. The 60 new tickets fill the gaps that the old tickets left. They add French, the rules H1 to H5, and questions that the policy does not answer.

The three parts have three jobs. You use the dev set while you change a prompt. The holdout set stays frozen for the release decision. The contaminated set holds the tickets that the baseline’s prompt was built on. Avoid contamination explains these three parts. This lesson looks at the 200 dev and holdout cases. The chart compares them with the old 69 tickets and the prompt’s 2 examples.

The new cases have far more French tickets, tickets that need a person and critical cases than the old test set.

Text description of the chart

Three panels compare the old set with the new dev and holdout cases. The old set has 71 cases: the LLM applications course’s 69 tickets and the prompt’s 2 examples. French tickets: 1 of 71 (1.4%) against 20 of 200 (10%). Tickets that need a person: 10 (14.1%) against 47 (23.5%). Critical cases, where one wrong answer blocks a release: 8 (11.3%) against 37 (18.5%).

The new cases have about seven times the old set’s share of French tickets, and more than its share of critical cases. A critical case is one where a single wrong answer blocks a release. Examples are a safety ticket, another customer’s data, and instructions hidden in a ticket. The release rules of Establish a baseline name these failures. Now the set has enough cases to look for them.

Count the cases: the coverage table

The project already counts the cases per split (python -m harness cases). You now add a count per slice and per split: the coverage table. It is a small function in harness/dataset.py and a new command.

1. Open harness/dataset.py in VS Code. At the top, find the import lines. Add the line from collections import Counter after import json. The imports then start like this:

   harness/dataset.py

   import hashlib
   import json
   from collections import Counter
   from pathlib import Path
   from typing import Literal

2. Find the function by_id. Below it, leave two blank lines and add this function:

   harness/dataset.py

   def coverage(cases: list[Case]) -> dict[str, dict[str, int]]:
       """How many cases each slice has, per split: the table that shows what the set is missing."""
       table: dict[str, dict[str, int]] = {}
       for c in cases:
           table.setdefault(c.slice, Counter())[c.split] += 1
       return {s: dict(table[s]) for s in sorted(table)}

   A Counter is a dictionary that starts every count at 0. setdefault makes one counter for a slice the first time that the slice appears. The result maps each slice to its count per split.

3. Open harness/__main__.py. Find def cmd_runs(a). Above it, add the function of the new command and two blank lines:

   harness/__main__.py

   def cmd_coverage(a) -> None:
       table = dataset.coverage(load_cases(split="all"))
       print(f"{'slice':14}{'dev':>6}{'holdout':>9}{'contaminated':>14}")
       for name, row in table.items():
           print(f"{name:14}{row.get('dev', 0):>6}{row.get('holdout', 0):>9}{row.get('contaminated', 0):>14}")

4. In the same file, find the line sub.add_parser("runs", help="the saved runs") in main(). Above it, add this line and one blank line:

   harness/__main__.py

       sub.add_parser("coverage", help="cases per slice and split").set_defaults(fn=cmd_coverage)

5. Save both files. In the VS Code terminal, run the new command:

   python -m harness coverage

   The output:

   slice            dev  holdout  contaminated
   boundary          14       14             0
   contaminated       0        0            71
   difficult         30       29             0
   french            10       10             0
   needs_person      22       21             0
   normal            12       11             0
   unanswerable      14       13             0

6. Run the tests again:

   python -m pytest -q

   The last line must still say 8 passed.

The table shows the shuffle at work. In every slice, dev and holdout have the same number of cases, or one more on one side. The contaminated tickets form their own slice, because their job is different. The chart shows the same counts for dev and holdout together.

The 200 dev and holdout cases per slice. The two orange slices were made bigger on purpose.

Text description of the chart

One point per slice, for the 200 dev and holdout cases: normal 23, boundary 28, difficult 59, French 20, needs a person 43, the policy does not answer 27. The last two slices have a second colour: they were made bigger on purpose.

What the set is missing

A coverage table shows what is in the set. Priya also writes down what is not in it, because a slice that is missing has no score at all, not a bad one. Her list for Larkfield:

• The real mix is unknown. Nobody counted how many of Larkfield’s tickets are French or need a person. So the overall score of the set is not the score that customers will see. Read it as “on tickets like these”, and read each slice on its own.
• Real writing. Language models and the course author wrote the tickets. They are cleaner than real tickets: few typos, one request each, and a style that models produce. Real tickets would be harder.
• Conversations. Every case is a first message. A customer who writes again, angry, after a wrong reply is not in the set.
• Other languages and attachments. In dev and holdout, one ticket in Spanish, one in German and two with only an attachment test rule H5. That checks that the rule works at all. It does not measure how well it works.
• Small slices. A slice of 10 French tickets moves by 10 points when one ticket changes. Module 5 puts an interval on each slice result, so that you see how much one ticket moves it.

The list goes into the dataset card, which you write at the end of this module. Models change fast, and so do the tickets that customers send, so the counts and scores in this course will not stay true for long. The method lasts: count by slice, and write down what is missing.

When to use it, and what it costs

Make a coverage table every time you build or change an evaluation set, and before you trust any overall score. It costs a few lines of code and a few minutes. Choosing the slices is the real work: they must match the decisions that matter. Grace cares about French customers and about tickets that need a person, so those are slices. A slice that nobody acts on only adds noise.

Adding rare cases on purpose has a cost too. The set no longer has the real mix, so its overall average is not the real average. Report the slices, not only the total. And every slice needs enough cases: about 20 to 30 for a rough score, many more for a precise one. That number grows fast, which is why a set of 200 cases has only six slices.

Guided practice: read the coverage

1. Run python -m harness coverage again. Find the slice with the fewest cases in dev.

2. Look at one case of that slice:

   python -m harness show T-81019

   The output:

   T-81019 | split dev | slice french | source course
   Text: Je reçois trop de courriels promotionnels. Merci de me retirer de votre liste.
   Expected: team account | needs_human false | next step none
   Reply must: reply_in_french | must not: claims_action, asks_secret
   Labelled by: course author, 2026-10-09; Grace's policy (teams, H1-H5) applied by reading; written with the case by the course author

3. Run python -m harness cases. Compare the French count of the contaminated split with the French count of dev.

4. Write two sentences for Grace: which slice would you grow first, and why?

Show an example answer

The smallest dev slice is french, with 10 cases. The contaminated split has only 1 French ticket, so the old test set said almost nothing about French replies.

“I would grow the French slice first. Camille’s team answers French tickets every day, and with 10 cases in dev, one wrong reply moves the French score by 10 points.”

Your turn: count by rule

The slice needs_person mixes five rules: H1 safety, H2 legal, H3 outside the policy, H4 access and H5 no readable request. A safety ticket and a legal threat fail in different ways. Write a short Python command that counts the dev and holdout cases per rule, H1 to H5. Which rule has the fewest cases? Is that enough to measure it?

Show one solution

python -c "from collections import Counter; from harness.dataset import load_cases; print(sorted(Counter(c.expected.rule for c in load_cases(split='all') if c.split != 'contaminated' and c.expected.rule).items()))"

The output:

[('H1', 23), ('H2', 5), ('H3', 6), ('H4', 9), ('H5', 4)]

H5 (no readable request) has only 4 cases, and H2 (legal) has 5. Four cases can show that the rule works at all, but not how often it fails: one mistake moves the score by 25 points. If legal threats matter to Grace, the set needs more of them.

Common mistake

invalid choice: 'coverage'

Symptom: the new command stops at once:

usage: python -m harness [-h] {cases,show,runs,score} ...
python -m harness: error: argument command: invalid choice: 'coverage' (choose from cases, show, runs, score)

Cause: the command line does not know the command yet. Either harness/__main__.py is not saved, or the line sub.add_parser("coverage", …) of step 4 is missing. The list in the message shows the commands that the program knows now.

Fix:

1. In VS Code, check the tab of __main__.py. A dot on the tab means that the file is not saved. Save it.
2. Check that the new line is inside main(), with four spaces before sub.add_parser.
3. Run python -m harness coverage again.

If the message is NameError: name 'Counter' is not defined instead, the import of step 1 is missing in harness/dataset.py.

Check your understanding

Tomás wants to grow the set by copying the 69 tickets of the LLM applications course twice more, to 207 tickets. Why does that not help, and what would you add instead?

Show the answer

Copies tell you the same thing again, like more spoons from the top of the pot. They add no new kind of ticket: still only 1 French ticket in 69 and few tickets that need a person. They also make every result look three times as sure as it is. Add tickets of the missing kinds instead: French tickets, the rules H1 to H5, and questions that the policy does not answer. Add enough of each to measure it as a slice.

Check your understanding

The set has far more safety tickets than real traffic has. Is the overall score of the set the score that customers will see? What should you report?

Show the answer

No. The rare, costly cases were added on purpose, above their real share, so the overall average does not have the real mix. If the assistant is weak on safety tickets, the set’s average is lower than the real one; if it is strong on them, higher. Report the score of each slice with its number of cases, and say that the overall score is “on tickets like these”.

Check your understanding

Practical check: run python -m harness coverage. How many unanswerable cases are in dev and in holdout, and why does the contaminated split have 0 in that slice?

Show the answer

14 in dev and 13 in holdout. The contaminated split has 0 in the unanswerable slice because all 71 of its cases are in their own slice, contaminated. Their job is different: they are not used to score the assistant, only to measure a gap, in the last lesson of this module.

Recap

• An evaluation set is a fixed list of cases with the answers that a good system gives. It must look like the real work and also cover the rare, costly cases on purpose.
• A slice is a group of cases that you measure on its own. The coverage table counts the cases per slice and split, and shows what is too small or missing.
• Larkfield’s set has 271 cases: 102 dev, 98 holdout and 71 contaminated. The tickets are made up for this course.
• The new dev and holdout cases have far more French tickets, tickets that need a person and critical cases than the old 69 tickets.
• Write down what the set does not have: the real mix, real writing, conversations, other languages. A missing slice has no score, not a bad one.
• Your project now has python -m harness coverage, and the tests still pass: 8 passed.

Next, you look at what one case holds besides its text: the labels, who made them, and the version of the whole set.

Next: Label and version