Before Larkfield compares models, choose the tickets that the comparison uses. Only a test set that looks like the real work, with its hard and strange cases, shows the differences that matter.
Before you startWhat to know and to have readyAbout 35 minutes
What you need to know
What Larkfield’s assistant must return (language, category, needs_human and a reply), and the kinds of errors that models make, from Uncertainty and errors.
A sample and a population, from Samples and populations in Essential Mathematics and Statistics for AI. A test set is a sample of the tickets that will come.
What you need
A browser. You install nothing and need no API key: every reply in this module is a recorded output.
Hosted models change often: your own results will differ.
Imagine a driving test on an empty car park, on a sunny day. Every learner drives straight ahead, stops and parks. Nearly everyone passes. Then the new drivers meet rain, a roundabout and a cyclist who does not signal, and some of them are not ready.
The test was not wrong. Each thing it checked was real. But it checked only the easy part of driving, so it could not tell a safe driver from a lucky one. A good driving test includes the situations where drivers make mistakes: busy junctions, bad weather, unclear signs.
A test for a language model has the same problem. The test cases decide what you can see.
Tomás wants to settle the choice quickly. He takes three English tickets: a late parcel (S01), the return of an unused garden hose (S02) and a double charge (S04). He runs them through the five hosted models and reads the replies. All fifteen replies give the right policy answer.
“They are all good,” says Tomás. “Take the one at the top of the leaderboard.”
Grace is not convinced. Her customers in Quebec write in French. Some messages mix French and English. Some ask for things the policy does not cover, such as a discount on a 9,000-dollar business order. The assistant must not invent an answer to those. None of that is in Tomás’s three tickets. His test cannot show a difference, because he chose tickets where every model succeeds.
A test set is a fixed list of cases that you run through every candidate, so that you compare them on the same work. It is representative when it contains the kinds of cases that the real work contains. That includes the rare cases that cost the most when they go wrong.
For Larkfield’s assistant, five kinds of tickets matter:
Routine: the everyday questions that the policy answers directly. They are most of the volume, so a model must be reliable on them.
Difficult: tickets that need two rules at once, or a careful reading of a number. “My order is 6 business days late, and the watering can was dented” needs the delivery rule and the damage rule.
Multilingual: the same work in every language that customers use, and messages that mix languages. Larkfield needs English, French and a mix of the two.
Ambiguous: tickets where the customer’s meaning is unclear. “Je veux annuler.” (I want to cancel.) Cancel what? The right action is to pass the ticket to a person who can ask.
Unanswerable: tickets that the assistant must not answer alone: an injury, a price-match request, a large business order. Rule 8 of the policy sends them to a person.
The last two kinds test abstention: the model’s ability to see “I should not answer this” and pass the ticket on. Abstention is one of the four criteria of the rubric in the next lesson.
The course team wrote 24 tickets for this course. They are synthetic: written to look like real tickets, with no real customer data. The figure shows the mix.
ENEnglish
FRFrench
mixMixed English and French
●Must go to a person
Routine, English 6
S01EN
S02EN
S03EN
S04EN
S05EN
S06EN
Routine, French 6
S07FR
S08FR
S09FR
S10FR
S11FR
S12FR
Difficult 3
D01EN
D02EN●
D03FR
Mixed-language 3
M01mix
M02mix
M03mix
Ambiguous 3
A01EN●
A02FR●
A03EN●
Unanswerable 3
U01EN●
U02FR●
U03EN●
Larkfield's test set: 24 synthetic tickets in six groups. The colour of each chip is the customer's language; an orange dot marks the tickets that must go to a person. Hover over a chip to read the ticket.Text description of the figure
Six groups of tickets. Routine English: S01 to S06 (late parcel, return, broken pot, double charge, password, warranty). Routine French: S07 to S12, the same six situations in French. Difficult: D01 (late and damaged, English), D02 (late refund and a new card, English), D03 (17 days late, French). D02 must go to a person after a correction of its key (see “The answer key comes first”). Mixed-language: M01 and M02 (French and English mixed; the customer is French-speaking), M03 (starts in French, asks for an answer in English). Ambiguous: A01 (“It doesn’t work”), A02 (“Je veux annuler”), A03 (a plant that “looks sad”); all three must go to a person. Unanswerable: U01 (price match), U02 (an injury, French), U03 (a 9,000-dollar business order); all three must go to a person.
Notice three choices in this mix.
First, the routine tickets come in language twins. S01 (English, 7 days late) and S07 (French, 8 days late) are the same situation. So are S04 and S10 (a double charge), and the other four pairs. If a model answers the English twin well and the French twin badly, the problem is the language, not the situation.
Second, routine tickets are half of the set, not 90%. In real traffic, routine tickets are probably most of the volume. A test set does not need the real proportions, because its job is not to measure the average. Its job is to find where each model fails, so it gives the rare, costly cases more room than real traffic does.
Third, six of the 24 tickets were written to go to a person. A model that never abstains can still answer the other tickets well. Without these six, you would never see that weakness.
With every ticket, the assistant receives Larkfield’s support policy and the task instructions. Open the box to read the policy: you need it for the practice of this module.
Larkfield’s support policy (8 rules), as the assistant receives it
Delivery. Standard delivery takes 3 to 5 business days in Canada. If an order is more than 5 business days late, the customer gets free shipping on the order (the shipping fee is refunded). An order more than 15 business days late is treated as lost: we send a replacement or a full refund, as the customer prefers.
Returns. Unused items can be returned within 30 days of delivery for a full refund. The customer pays return shipping, except when the item is damaged or wrong. Plants and seeds cannot be returned, but a dead plant on arrival is replaced (photo needed within 7 days).
Damaged or wrong items. Send a photo within 14 days of delivery. We send a replacement at no cost, or a full refund if no replacement is in stock.
Warranty. Garden tools and furniture have a 2-year warranty against defects. Normal wear, misuse and rust from leaving tools outside are not covered.
Payments. Refunds go back to the original payment method within 5 to 10 business days. We cannot change a payment method after an order is placed. A double charge is refunded within 3 business days after we confirm it.
Accounts. For security, support never asks for or accepts passwords or full card numbers. Password resets are done only through the “Forgot password” link.
Language. Answer in the language of the customer: English or French.
Anything else (legal threats, injuries, price matching, business orders over 5,000 dollars, questions this policy does not answer): do not answer; send the ticket to a person.
A test case has two parts: the input, and what a good output must contain. Here is the case for S07:
S07 (routine, French)
Ticket: Bonjour, ma commande L-20877 devait arriver il y a 8 jours ouvrables
et je n'ai toujours rien reçu. Que pouvez-vous faire?
Key: language fr · category delivery · needs_human false
Must say: free shipping: the shipping fee is refunded (rule 1: more than 5 days late)
Must not: apply the 15-day "lost order" rule as if it applies now
The answer key records the right labels and the facts that a good reply must contain or avoid. Grace wrote the keys before anyone ran a model. This order matters. If you write the key after you read the outputs, you tend to accept what the models said. Then the key stops being a fair judge.
Keys can be wrong too. The key of D02 says that the assistant can answer alone: a lamp was returned 3 weeks ago, the refund has not arrived, and the customer’s card has expired. But the policy says nothing about a late refund or an expired card, and rule 8 sends “questions this policy does not answer” to a person. Every hosted model sent D02 to a person. The policy agrees with the models, so Grace corrected the key and wrote down why. When many careful replies disagree with a key, check the key against the policy before you blame the models.
The key is a guide for the person who scores, not a list of exact words. “Les frais de livraison vous seront remboursés” gives the free-shipping fact without the words “livraison gratuite”. The next lesson shows why a person must read the replies instead of searching them for the key’s words.
The course team ran all 24 tickets through eight model runs. Then a person read and scored every reply. Each cell below is one reply. The number is its score out of 8 on the rubric of the next lesson. A blue cell is a reply that a person can send after a light check.
Ready to send after a light check
Not ready
No answer
Every recorded reply, scored by reading it: 24 tickets (grouped by kind) by 8 model runs. The number is the rubric total out of 8. Blue: ready to send after a light check. Orange: not ready. Grey: no answer at all. Text description of the chart
A grid of 8 model runs by 24 tickets, grouped by kind. Claude Haiku 5.5, Claude Opus 5.5, Gemini 3.5 Flash and Gemini 3.1 Pro (preview): every cell is blue, 24 of 24 ready. Claude Sonnet 5.5: blue except S09 and S12, both routine French (22 ready). Gemma 3 4B: orange on every routine and every mixed-language ticket, and on D01, D03 and A03; blue only on D02, A01, A02, U01, U02 and U03 (6 ready). Mistral 7B: 12 ready; orange on S02 and S06 (English), S07, S08, S10 and S12 (French), D01, D02, A03, U01, U03 and M03. Qwen3 4B with a 4,000-token limit: 14 ready; grey (no answer) on D01, D02, A03 and M02, where it reached its token limit while reasoning.
Read the grid by groups of columns, and compare it with Tomás’s test.
On S01, S02 and S04, Tomás’s tickets, the five hosted models are all blue. His test was correct. It just could not separate them.
Claude Sonnet 5.5’s only two misses are French tickets. In S12, its reply says a crack “peut donc être couverte, car l’usure normale […] ne sont pas couvertes” (can be covered, because normal wear is not covered). The reasoning does not make sense, and an English-only test would never show it.
Mistral 7B invents rules on unanswerable tickets. For U03 it wrote: “as per our policy, we do not offer discounts for business orders over 5,000 dollars”. The policy says no such thing. It says: send the ticket to a person.
Qwen3 4B fails where the work is hard. On four hard tickets (two difficult, one ambiguous, one mixed-language), it reasoned until it reached its 4,000-token limit. It returned no answer.
Each kind of ticket found a different weakness. That is the purpose of a representative test set.
Twenty-four tickets is a small sample. One reply is about 4 percentage points of the score. A model with 22 of 24 and a model with 24 of 24 may be equally good. The difference may come from which tickets the team happened to write.
This is the idea of Samples and populations in the mathematics course: the test set is one sample, and another sample would give other numbers. Lesson 3 puts a range around each score, so you see how much 24 tickets can tell you. For now, keep two rules:
24 tickets can show that a model is not ready. Gemma 3 4B was ready on 6 of 24. Bad luck in the choice of tickets cannot explain that.
24 tickets cannot show that a model is the best. Four hosted models were ready on all 24. That means “no failure found in 24 tickets”, not “never fails”.
Grow the test set over time. Every real ticket that goes wrong after launch is a candidate for the next version of the set.
Models also change fast. The numbers in this module are a snapshot. The method lasts longer than the numbers: when a new model comes out, run the same 24 tickets and score them the same way.
A test set can also miss parts of the policy. Check which rules the 24 tickets test.
Open the policy box above. It has eight rules: 1 delivery, 2 returns, 3 damaged or wrong items, 4 warranty, 5 payments, 6 accounts, 7 language and 8 send to a person.
Make a table with one row per rule. In a second column, write the IDs of the tickets that need the rule. Use the text description of the ticket figure above.
For each rule, ask: does a ticket test the “no” side of the rule as well as the “yes” side? For example, rule 2 says that seeds cannot be returned. Is there a ticket where the right answer is “no”?
Write down two parts of the policy that no ticket tests.
Expected result: every rule has at least one ticket, but some “no” sides are missing.
Example answer
Rule
Tickets
1 Delivery
S01, S07, D01, D03, M01
2 Returns
S02, S08, M03 (seeds: the answer is “no”), A03 (a plant)
3 Damaged or wrong
S03, S09, D01, M02
4 Warranty
S06, S12
5 Payments
S04, S10, D02
6 Accounts
S05, S11
7 Language
all 24; M01, M02 and M03 test it hardest
8 Send to a person
A01, A02, A03, U01, U02, U03
Parts that no ticket tests:
The warranty exclusions. No ticket describes rust on a tool left outside, where the right answer is “not covered”.
A legal threat. Rule 8 names legal threats, but no ticket contains one.
A late return. No ticket asks to return an item after 30 days.
A dead plant with a photo. A03 is vague on purpose; no ticket tests the clear 7-day case.
Larkfield starts to sell gift cards. Grace adds rule 9: “Gift cards cannot be refunded or exchanged for cash. A lost gift card can be replaced if the customer has the order number.”
Write three new test tickets for rule 9. Give each one an answer key: language, category, needs_human, what the reply must say and what it must not say. Make one routine English ticket, one French ticket, and one ticket that is harder than routine.
Example answer
G01 (routine, English): “Can I get my money back for a gift card I don’t need?” Key: en, payment, needs_human false. Must say: gift cards cannot be refunded or exchanged for cash. Must not: offer a refund.
G02 (routine, French): “J’ai perdu la carte-cadeau que j’ai achetée. J’ai le numéro de commande.” Key: fr, payment, needs_human false. Must say: the card can be replaced with the order number. Must not: ask for a card number or a password (rule 6).
G03 (difficult, English): “I lost a gift card my aunt gave me. I don’t have the order number. Can you cancel it and refund her?” Key: en, other, needs_human true: the policy allows a replacement only with the order number, and no refund. Must not: promise a replacement or a refund.
A good set also has a ticket where the right answer is “no”, and one where the customer gives too little information. G03 does both.
Common mistake
The test set has English tickets only
Symptom. A model passes every English test ticket. After launch, French customers wait longer, because many of their simple tickets go to a person.
Cause. Some models treat the same situation differently in another language. In the recorded runs, Gemma 3 4B answered S01 (English, 7 days late) itself but sent S07 (French, 8 days late) to a person. In five more samples at temperature 0.8, it did the same five times out of five. Gemini 3.1 Pro answered the English warranty ticket S06 itself and sent its French twin S12 to a person.
Fix. Write the routine tickets as language twins: the same situation in each language. Compare each pair. A model that treats the twins differently has a language problem, even when each reply looks fine on its own.
For a real AI service. Test every language and channel that your customers use, with the same scenarios, before you compare overall scores.
Omar suggests that the test set should be 24 random tickets from last month. About 90% of them would be routine English tickets. What would this test set be good for, and what would it miss?
Show the answer
It would estimate the average quality on today’s traffic quite well. But it would contain almost no French, mixed-language, ambiguous or unanswerable tickets. So it could not find the weaknesses that cost the most. Those are French errors, invented answers to tickets that must go to a person, and language bias. A test set for comparing models includes more of the rare, costly kinds on purpose.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why did Grace write the answer keys before anyone ran a model?
Show the answer
A key written after reading the outputs tends to copy what the models said, including their mistakes, so it stops being a fair judge. Writing the key first fixes what “correct” means, in the same way for every model.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a model gives a perfect reply to S04 (English, double charge) and sends S10 (the same situation in French) to a person. What does this suggest, and what do you check next?
Show the answer
It suggests a language problem, not a problem with the payment rule, because S04 and S10 are language twins. Check the other twins (S01 and S07, S05 and S11, S06 and S12) for the same pattern. Run the pair a few more times to see whether the difference is stable or random.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
A test set is a fixed list of cases that every candidate model runs. It is representative when it contains the kinds of work that matter: routine, difficult, multilingual, ambiguous and unanswerable cases.
Write the answer key before you run any model.
Language twins show a language problem directly.
In the recorded runs, each kind of ticket found a different weakness: French errors, invented rules, no answer on hard tickets.
24 tickets can show that a model is not ready. They cannot prove that a model is the best.
Next, you score the replies with a rubric, so that every reply is judged in the same way.
