Score outputs consistently

What you will be able to do

Turn “this reply looks good” into a score that another person would give too. A rubric with clear score levels lets Larkfield compare 192 replies from eight model runs fairly, and see exactly where each model loses points.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Larkfield’s 24 test tickets, their answer keys and the policy, from Build a representative test set.
• Hallucination (an invented fact) and inconsistent replies, from Uncertainty and errors.

What you need

• A browser. The explorer and the practice use recorded replies.

Hosted models change often: your own results will differ.

Two teachers, one essay

Give the same essay to two teachers and ask for a mark out of 10. One gives 6, the other 8. Neither is careless. One cares most about spelling, the other about ideas, and they have different pictures of “a 6”.

Schools fix this with a marking scheme. It lists what to check: ideas, structure, language. It also describes each mark with an example: “2 points: the argument has a clear reason and one example”. Teachers who use the same scheme give much closer marks. Many exams also hide the student’s name, so that the marker cannot favour a student they know.

Larkfield needs both ideas: a scheme that says what a good reply is, and scoring without names.

The problem: “this one looks better”

Tomás reads two replies to the French warranty ticket S12 and prefers the longer one, from Claude Sonnet 5.5, because it “explains more”. Camille, the bilingual agent in Montreal, prefers the shorter one, from Claude Opus 5.5, because the longer one does not make sense in French. Both are reading carefully. They are measuring different things.

Amira suggests a shortcut: let a program search each reply for the words in the answer key, such as “30 days” for the return ticket. It is fast and it never gets tired. The figure shows three real replies with both checks.

S02 Return an unused garden hose bought 10 days ago.

Gemini 3.1 Pro (preview)

Hello! You can return your unopened garden hose for a full refund since it is within our 30-day return window for unused items. Please note that you will be responsible for the return shipping costs. Let us know if you would like to proceed and we will help you with the next steps.

Word search for the key

✗ fail The key wants "30 days"; the reply says "30-day".

A person reads it

✓ pass Correct and complete: 30 days, full refund, the customer pays return shipping.

M03 Bonjour! Quick question in English please: can I return seeds I don’t need?

Gemini 3.1 Pro (preview)

Hello! Unfortunately, we cannot accept returns for seeds. Our policy states that plants and seeds are non-returnable. Please let us know if you have any other questions.

Word search for the key

✗ fail The key wants "cannot be returned"; the reply says "non-returnable".

A person reads it

✓ pass Correct, in English as the customer asked.

U03 A landscaping company wants 60 benches, about 9,000 dollars, with a discount.

Mistral 7B

Thank you for your inquiry. We would be happy to help with your order. However, as per our policy, we do not offer discounts for business orders over 5,000 dollars. A member of our team will contact you shortly to discuss your order.

Word search for the key

✓ pass The labels match the key (other, needs_human true); the key lists no required words.

A person reads it

✗ fail It invents a rule. The policy has no discount rule; it says: send to a person.

Three recorded replies, each checked in two ways: a program searches the reply for the answer key's words, and a person reads it with the policy. The word search fails a correct reply twice and passes an invented rule.

Text description of the figure

Card 1, ticket S02, Gemini 3.1 Pro: the reply says the hose can be returned “within our 30-day return window” for a full refund, and the customer pays return shipping. The word search fails it, because the key’s words are “30 days”. A person passes it. Card 2, ticket M03, Gemini 3.1 Pro: “we cannot accept returns for seeds … plants and seeds are non-returnable”. The word search fails it (“cannot be returned” is missing); a person passes it. Card 3, ticket U03, Mistral 7B: “as per our policy, we do not offer discounts for business orders over 5,000 dollars”. The word search passes it, because the labels are right and the key lists no words. A person fails it: the rule is invented.

What this tells us. A word search is wrong in both directions. It fails correct replies that use other words, and it passes replies with an invented rule, because it cannot know what is not allowed. On the 24 tickets, the word search found a missing fact in 6 of Gemini 3.1 Pro’s replies. A person found that all 6 were correct. The search for forbidden words is worse. The key of S11 forbids “envoyer votre mot de passe” (send your password). Five models wrote “nous ne pouvons pas vous envoyer votre mot de passe”: we cannot send you your password. That is the right refusal, and the word search failed all five. A program finds words; a person reads meaning. The automatic check is useful as a first filter, but a person must read the replies.

The idea: a rubric with anchors

A rubric is a scoring guide: a short list of criteria (the things you check), and for each criterion, a description of every score level. Each description is an anchor: it ties a score to a clear, recognisable kind of reply, with an example. Anchors are what make two readers give the same score. “Correctness: 1” means nothing alone. “Correctness 1: one small error that does not change what the customer gets, for example an invented ‘24 hours’” is something two people can agree on.

A rubric is the marking scheme of the two teachers. It turns a feeling into a reason that someone else can check.

Larkfield’s four criteria

Grace and Camille chose four criteria. Each is scored 0, 1 or 2, so a reply gets 0 to 8 points.

Correctness: is everything it says true? The facts, promises and labels must agree with the policy and the ticket. The worst errors change what the customer gets. Gemma 3 4B told the customer of S01, whose parcel was 7 days late, about the 15-day rule for lost orders. It never mentioned the free shipping that the customer had a right to. That is a 0.

Completeness: does it give everything the customer needs? A reply can be true and still leave the customer without an answer. Mistral 7B answered S07 (8 days late, French) with only “we will check and contact you”. Nothing in it is false, but the customer does not learn about the free shipping. That is a 0 for completeness and a 2 for correctness: the criteria measure different things.

Format compliance: can the application use it as it is? The assistant must return one valid JSON object with the four fields. The reply must be in the customer’s language, in 2 to 5 sentences. For D01, Mistral 7B wrote a good reply inside broken JSON ("category": "delivery", "returns"), so the application received nothing. That is a 0. A French reply that ends with “Thank you for your patience.” in English is a 1.

Appropriate abstention: does it send the right tickets to a person? Abstention means that the model passes the ticket to a person: needs_human: true, and a reply that says a person will contact the customer. It is right for the 6 tickets of rule 8 and for gaps in the policy. Answering one of them alone is the dangerous error, so it scores 0. Sending a routine ticket to a person costs staff time but harms no customer, so it scores 1.

The anchors in one table:

Score	Correctness	Completeness	Format compliance	Appropriate abstention
2	Everything is true	All the points this ticket needs	Valid JSON, customer’s language, 2–5 sentences, no visible language error	Abstains exactly when the policy says so
1	One small error that does not change the outcome (an invented “24 hours”, a wrong routing label)	One needed point missing	Usable after a small fix (an English sentence or a grammar error in a French reply, 6+ sentences, text around the JSON)	Too cautious: sends an answerable ticket to a person
0	A wrong or invented rule that changes what the customer gets	The main answer missing	The application cannot use it (broken JSON, wrong language)	Answers alone a ticket that must go to a person, or gives no answer

Grace adds one decision rule. A reply is ready to send after a light check when correctness is 2 and no criterion is 0. A person can fix a small mistake in seconds, but nobody can trust a reply with a wrong rule.

Worked example: Camille scores S12

The ticket, in French: “My wooden garden bench has a crack after one year. I use it normally. Is it covered?” Rule 4 says that garden furniture has a 2-year warranty against defects, and normal wear is not covered. The key: fr, warranty, needs_human false; the reply must mention the 2-year warranty.

Claude Sonnet 5.5 replied:

Bonjour, les meubles de jardin comme votre banc sont couverts par une garantie de 2 ans
contre les défauts. Comme votre banc a un an et que vous l'utilisez normalement, une
fissure peut donc être couverte, car l'usure normale, une mauvaise utilisation et la
rouille due à des outils laissés dehors ne sont pas couvertes. Nous n'avons toutefois pas
de procédure détaillée pour la suite dans notre politique, mais n'hésitez pas à répondre
à ce message si vous avez d'autres questions.

Camille scores one criterion at a time, and writes a reason for each.

Criterion	Score	Reason
Correctness	1	The second sentence says the crack “can be covered, because normal wear is not covered”. The logic does not work, and rust on tools does not apply to a bench. The reply also tells the customer that the policy has no next step. No fact changes what the customer gets, so 1, not 0.
Completeness	2	It gives the 2-year warranty against defects and the exclusions.
Format	2	Valid JSON, French, 3 sentences.
Abstention	2	The policy answers the question; the reply does not send it to a person.

Total: 7 of 8. Not ready, because correctness is below 2. Tomás’s “it explains more” was the problem: the extra sentence is the confused one. A reader who does not read French well would not see it. This is why Camille scores the French replies.

Score blind

Tomás expects the most expensive model to win. If he knows which model wrote a reply, that belief can push his score up or down without him noticing. So the team scores blind: the replies carry the letters A to D, and the model names come only after the scores.

Try it. Each ticket has four real replies from different models. Put them in order, best first. Then reveal the models and the rubric scores.

Explore

Rank four replies before you see the models

S07 · French Bonjour, ma commande L-20877 devait arriver il y a 8 jours ouvrables et je n'ai toujours rien reçu. Que pouvez-vous faire?

Your order, best first: A, B, C, D. Move the replies, then reveal the models and the scores.

1. 1Reply A

   Bonjour, nous sommes désolés que votre commande L-20877 n'ait pas encore été livrée après 8 jours ouvrables. Nous allons vérifier l'état de votre commande et vous contacter pour vous informer de nos prochaines étapes.

   needs_human: yes · category: delivery
2. 2Reply B

   Bonjour, nous sommes désolés du retard de votre commande L-20877. Comme elle a plus de 5 jours ouvrables de retard, la livraison vous est offerte : les frais de livraison vous seront remboursés sur votre moyen de paiement d'origine, dans un délai de 5 à 10 jours ouvrables. Si la commande dépasse 15 jours ouvrables de retard, elle sera considérée comme perdue et nous vous enverrons, à votre choix, un remplacement ou un remboursement complet.

   needs_human: no · category: delivery
3. 3Reply C

   Bonjour, je suis désolé que votre commande soit en retard. Selon notre politique, si une commande est retardée de plus de 15 jours ouvrables, nous vous enverrons un remplacement ou un remboursement complet, selon votre préférence. Je vais transmettre votre dossier à un agent qui pourra vous aider plus rapidement. Veuillez patienter, un membre de notre équipe vous contactera dans les plus brefs délais.

   needs_human: yes · category: delivery
4. 4Reply D

   Je suis désolé d'apprendre que votre commande L-20877 est en retard. Comme votre livraison a plus de 5 jours ouvrables de retard, nous allons vous rembourser vos frais de port. Si le retard dépasse 15 jours ouvrables, nous considérerons le colis comme perdu et nous vous proposerons un remplacement ou un remboursement complet selon votre choix.

   needs_human: no · category: delivery

Real replies to three tickets, in a fixed order with the model names hidden. Move each reply up or down, best first, then reveal the models and the rubric scores. The status line counts the pairs that you put in the same order as the rubric.

Text description of the explorer

Three tickets, four replies each. The letters are fixed; the models are revealed only after ranking.

S07 (French, order 8 business days late). A, Mistral 7B: only “we will check your order and contact you” (correctness 2, completeness 0, format 2, abstention 1; total 5; not ready). B, Claude Opus 5.5: free delivery, the shipping fee refunded to the original payment method in 5 to 10 days, and the 15-day rule as the next step (8; ready). C, Gemma 3 4B: applies the 15-day lost-order rule to an 8-day delay, misses the free shipping and sends the ticket to a person (0, 0, 1, 1; total 2; not ready). D, Gemini 3.5 Flash: refunds the shipping fee (“frais de port”) and gives the 15-day rule as the next step (8; ready).

S12 (French, a crack in a garden bench). A, Claude Sonnet 5.5: confused logic and “our policy has no next step” (1, 2, 2, 2; total 7; not ready). B, Mistral 7B: asks for a photo that the policy does not require, does not say if the bench is covered (1, 1, 2, 1; total 5; not ready). C, Claude Opus 5.5: 2-year warranty, covered if it is a defect, normal wear excluded (8; ready). D, Gemini 3.1 Pro: correct facts, but sends the ticket to a person to judge it (2, 2, 2, 1; total 7; ready).

U03 (a 9,000-dollar business order asks for a discount). A, Gemini 3.5 Flash: sends it to a person, citing the 5,000-dollar rule (8; ready). B, Mistral 7B: invents “we do not offer discounts for business orders over 5,000 dollars” (0, 2, 2, 2; total 6; not ready). C, Claude Haiku 5.5: sends it to a person, no discount promised (8; ready). D, Gemma 3 4B: sends it to a person; the JSON is inside a code fence (2, 2, 1, 2; total 7; ready).

Did you put Gemma’s S07 reply (C) in the middle? It is polite, in French and mentions a refund. It scores lowest, because the refund rule it quotes does not apply yet. In U03, Mistral’s reply (B) sounds helpful and has the right labels. But its one invented sentence makes it the only reply that is not ready.

All 192 replies, one rubric

The course team scored every reply of the eight model runs with this rubric: 24 tickets × 8 runs = 192 replies. The scores, with a one-line reason for each, are in reference/c07/scores/scores.csv, and Lesson 1 showed them as a grid. Where did the models lose points?

Model run	Ready of 24	Score (0–10)	Correctness 0	Completeness 0	Format 0	Abstention 0
Claude Haiku 5.5	24	9.84	0	0	0	0
Claude Sonnet 5.5	22	9.90	0	0	0	0
Claude Opus 5.5	24	10.00	0	0	0	0
Gemini 3.5 Flash	24	9.95	0	0	0	0
Gemini 3.1 Pro (preview)	24	9.90	0	0	0	0
Gemma 3 4B (local)	6	5.78	6	6	2	0
Mistral 7B (local)	12	7.45	4	2	1	0
Qwen3 4B, 4,000-token limit (local)	14	6.51	6	6	4	4

The score out of 10 is the mean total of the 24 replies ÷ 8 × 10, so 10 means full marks on every criterion of every reply; Module 5 uses it. The other columns count the replies with a 0 on each criterion.

The hosted models lost only single points: a confused French sentence, a cautious hand-over to a person, an English link name in a French reply. The local models lost whole criteria: wrong rules (Gemma 3 4B on 6 tickets), wrong or invented rules (Mistral 7B on 4), and no answer at all (Qwen3 4B on 4). No model answered alone a ticket that had to go to a person; the 4 abstention zeros of Qwen3 are the tickets with no answer.

Two readers, one rubric

A rubric is only useful if another reader gives the same scores. To check, the team gave a second reader 28 of the replies: 4 from each of the seven main runs (every run except Qwen3 4B), chosen at random. The reader had the rubric, the policy and the tickets, and no model names. The agreement rate is the share of replies where the two readers gave the same score.

The first comparison found 5 replies with a different score on at least one criterion. Two of the differences had one cause. Gemma 3 4B wraps its JSON in a Markdown code fence: a line of three backticks (```) before it and after it. The first reader had not counted that as “extra text around the JSON”. The second reader had. The anchor was clear; the first reader had missed it. So the team rescored all 25 replies with a code fence (Gemma’s 24 and one from Gemini 3.1 Pro), not only the two in the sample.

After that fix:

• On all four criteria together, the readers agreed on 24 of 28 replies.
• Criterion by criterion: correctness 26 of 28, completeness 27, format 26, abstention 28.
• On Grace’s decision, ready or not ready, they agreed on 28 of 28.

The remaining differences are honest judgement calls. Is “n’est pas endommagée ni erronée” (Claude Haiku 5.5, S08) a calque, an English phrase copied word for word, that a Quebec reader notices? The second reader said yes; the first said no. Camille would say it is unusual but clear. The rubric cannot remove every judgement, but it moves the disagreements to small, visible places.

In a real team, two people do this. Grace scores the English replies and Camille the French ones. Both score a shared sample to check their agreement. When they disagree, they do not average. They talk, sharpen the anchor, and rescore every reply that the change affects.

Later: a model as the judge

The second reader here was an AI model (Claude, used through the course’s authoring tools), given only the rubric. When a language model scores outputs, people call it LLM as a judge. The course Evaluating and Testing AI Systems teaches it. It can score thousands of replies in minutes, so it helps when the test set grows. But it has known risks. A judge model can prefer long replies, replies in a familiar style, or replies from its own model family. This judge belongs to the same family as three of the candidates. It can also miss the same French errors that a weak model makes. So use it only after you measure its agreement with people on a sample, as here. Keep a person on the French replies and on every reply that decides the choice.

Guided practice: score three replies

Ticket M01, in Quebec French mixed with English: “Allô, mon order est late de 6 jours, c’est pas correct. Je veux mon shipping back.” The key: fr, delivery, needs_human false; the reply must give the free shipping (the shipping fee refunded).

1. Read the three replies. All three are real.

   X  Merci pour votre message et désolé pour le retard de votre commande. Selon notre
      politique, si une commande arrive plus de 5 jours ouvrables après le délai standard
      de livraison, les frais de livraison vous sont remboursés. Nous pouvons appliquer ce
      remboursement lorsque le retard est confirmé pour votre commande. Si vous nous
      donnez le numéro de votre commande, nous pourrons vérifier la date de livraison prévue.
      [needs_human: false]
   
   Y  Bonjour, nous sommes désolés que votre commande soit en retard de 6 jours. Nous
      allons vous réfunder le frais de livraison. Un de nos agents vous contactera bientôt
      pour vous fournir plus d'informations.  [needs_human: true]
   
   Z  Je suis désolé que votre commande soit en retard. Selon notre politique, si une
      commande est retardée de plus de 15 jours, nous vous enverrons un remplacement ou un
      remboursement complet, comme vous préférez. Un membre de notre équipe contactera
      bientôt pour traiter votre demande. Nous vous remercions de votre patience.
      [needs_human: true]

2. Score each reply on the four criteria, 0 to 2, with the anchor table. Write a reason of one line for each score.

3. Add the totals and decide: ready or not ready?

4. Open the example answer and count how many of your 12 scores match.

Expected result: X is ready with 8. Y is ready only after a person fixes two words. Z is not ready.

Example answer (the reference scores)

Reply	Correct	Complete	Format	Abstention	Total	Ready
X (Claude Haiku 5.5)	2	2	2	2	8	yes
Y (Mistral 7B)	2	2	1	1	6	yes, after a fix
Z (Gemma 3 4B)	0	0	1	1	2	no

• X: the right rule, careful wording, asks for the order number. Good French.
• Y: the right fact, but “réfunder” is not French and “le frais” should be “les frais” (format 1). It sends an answerable ticket to a person (abstention 1). A person fixes two words and sends it, so it is ready by Grace’s rule, but it is not good.
• Z: applies the 15-day rule to a 6-day delay and never mentions the free shipping (correctness 0, completeness 0). “Contactera” has no object, and the JSON is in a code fence (format 1). It sends the ticket to a person (abstention 1).

If you gave Y a 2 for format, read it aloud: a French reader notices “réfunder” at once. This is the kind of difference that a shared sample and a talk between readers resolve.

Your turn: change one rule

Amira changes the application: it now removes a Markdown code fence before it reads the JSON. Grace updates the format anchor: a code fence is no longer a breach.

Which of Gemma 3 4B’s scores change, and does its number of ready replies change? Use the anchor table and the fact that all 24 of Gemma’s replies have a code fence.

Example answer

Gemma’s format score goes from 1 to 2 on the 19 replies where the code fence was the only format problem. Five keep a lower score, because they also have English in a French reply or a grammar error (S09, M01, M02 at 1; S12 and D03 at 0). Its total rises by 19 points over the set.

The number of ready replies does not change: 6 of 24. A format score of 1 never blocked “ready”; correctness did. A change in the application can change the scores of a criterion without changing the decision.

Common mistake

The rubric changes halfway through

Symptom. The scores of the first models scored are lower, or higher, than the later ones for the same kind of reply. A model seems to improve when you score it again.

Cause. The readers changed an anchor while they scored, for example after a discussion, but did not rescore the replies they had already scored. In this module, the code-fence rule changed after the second reading. Without a full rescore, the two sampled Gemma 3 4B replies would have had format 1. The other Gemma replies with the same code fence would have kept format 2.

Fix. Write every anchor change down with its date. Then rescore every reply that the change affects, for every model, before you compare.

For a real AI service. Keep the rubric with a version number next to the scores. When you add a model next year, score it with the same version, or rescore all models with the new one.

Check your understanding

Check your understanding

A reply says nothing false, but it does not tell the customer the one fact they need. Which criterion catches this, and why is it a separate criterion from correctness?

Show the answer

Completeness catches it. Correctness asks whether everything said is true; completeness asks whether everything needed is said. Mistral 7B’s S07 reply (“we will check and contact you”) is correct (2) but not complete (0). If the two were one criterion, a vague, safe reply would score as well as a full answer.

Check your understanding

Why does answering an unanswerable ticket alone score 0 for abstention, while sending a routine ticket to a person scores 1?

Show the answer

The two errors have different costs. Sending a routine ticket to a person costs some staff time, but the customer still gets a right answer. An answer to an injury, a legal threat or a large business order can harm the customer or the company. Mistral 7B’s invented discount rule shows how. The rubric scores the dangerous error lower.

Check your understanding

Predict: Tomás scores the replies with the model names visible, and he believes that the largest model is the best. Which way will his scores be wrong, and how does blind scoring help?

Show the answer

He will tend to score the largest model’s replies higher and the small models’ replies lower than the rubric says. The effect is largest on borderline replies, such as a cautious hand-over or a small language error. Blind scoring removes the name, so the only thing he can react to is the reply itself.

Recap

• A rubric lists criteria and anchors: a description and an example for every score level.
• Larkfield’s four criteria: correctness, completeness, format compliance and appropriate abstention, each 0 to 2. A reply is ready when correctness is 2 and no criterion is 0.
• A word search for the key’s words is wrong in both directions. A person must read the replies.
• Score blind, then reveal the models.
• Check agreement between two readers on a shared sample; when they differ, sharpen the anchor and rescore everything it affects.
• An LLM as a judge can help later (see the evaluation course), after you measure its agreement with people.

Next, you add time and money to quality: how long each model takes, how many tokens it uses and what a ticket costs.

Next: Measure practical performance