Generated-output metrics

What you will be able to do

Measure the written reply, not only the decisions around it. Some properties of a reply can be checked by code on every case, every time. Others need a reader. You will learn which is which, and see what an overall score hides.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The decision measures, from Prediction metrics, and the ranking measures, from Retrieval metrics.
• The criteria of each case (must and must_not) and the slices, from Label and version and Select representative cases.
• Validated output with a schema, from Define a schema in Building Reliable Applications with LLM APIs.

What you need on your computer

• ~/projects/eval-harness after Retrieval metrics, with .venv active: python -m pytest shows 15 passed. If it does not, copy the snapshot end-of-m02 as in Reset the project, and do the steps of Build it: the measures in the project again.
• The labs repository in ~/projects/nextia-learning-labs, with the newest files.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

Marking a letter

A language teacher marks a formal letter. Part of the marking is a checklist. The date is at the top, the letter is in French, it ends with a signature, and it has no slang. A helper could tick those boxes without reading for meaning, and would tick them the same way every time.

The rest of the marking is reading. Are the facts in the letter true? Does it answer the question that was set? Does it make a promise that the writer cannot keep? A checklist cannot see these. A letter can pass every box and still say something false.

The problem: right decisions, wrong reply

The two decisions of the last lessons say nothing about the reply text. Look at one real reply. Ticket T-65254 says: “The paddling pool arrived with a cracked seam… I noticed all this when I unpacked it on the day it arrived.” The candidate (prompt v3, chat-small) chose the right team (delivery) and the right needs_human (false). It asked for a photo, as the policy requires. And it wrote this sentence:

Since you noticed it on the day it arrived, you’re within the photo-reporting window.

The policy gives 14 days from delivery. The ticket does not say how long ago the pool arrived. So the reply promises something that nobody knows. The decision measures score this case as perfect.

The opposite failure is real too. On ticket T-65450, the baseline wrote no reply at all: the model reasoned until it reached its limit of 1,000 tokens. A reply measure must count that as a failure, not skip it.

Six things to check in a reply

A good reply has several separate properties. Each one fails in its own way, so each needs its own measure:

1. Schema compliance. The output is valid data in the agreed shape: JSON, with every field and an allowed value. Without it, there is no reply to check.
2. Required content (completeness). The reply contains what the case needs. That is the next step (a photo, the “Forgot password” link). It is also “a person will contact you” when a person must handle it, and “stop using it” for a safety problem.
3. Forbidden content. The reply does not ask for a password or a full card number. It does not claim that the shop made a refund. It does not share another customer’s details, or promise what the policy does not give.
4. The right language. A French ticket gets a French reply.
5. Support. Every claim in the reply is backed by the policy or the ticket. T-65254’s “you’re within the window” is not.
6. Appropriate refusal. Sometimes the assistant must not do what the customer asks, such as share a sister’s order. Then it says no politely, and hands the ticket to a person.

Larkfield’s assistant does not cite sources. A system that does, such as the RAG course’s policy assistant, adds a seventh: citation validity. Each cited passage must exist, must have been in the context, and must support the claim. Answer and cite shows that code can check the first two parts, and only a reader the third.

A tiny example: one reply, every check

Here is the whole candidate reply to T-65254, sentence by sentence. On the right: what the code checks found, and what a reader who knows the policy finds.

The candidate's reply to T-65254 (prompt v3, chat-small, recorded). Every code check passes. A reader finds one claim that the ticket and the policy do not support.

Text description of the diagram

Sentence 1, “I’m sorry your pool arrived with both a cracked seam and a faulty valve…”: the must-not rules find nothing; a reader sees that it repeats the ticket. Sentence 2, “Please send us a photo of the damage within 14 days of delivery…”: the required check “photo” finds the word “photo”; a reader sees that it matches policy rule 3. Sentence 3, “Since you noticed it on the day it arrived, you’re within the photo-reporting window.”: the must-not rules find nothing; a reader sees that it is not supported, because the ticket does not say how long ago the pool arrived. Sentence 4, “You don’t need to send it back at your expense because it arrived damaged.”: the must-not rules find nothing; a reader sees that it matches policy rule 2. Summary: the schema is valid, and photo, claims_action and asks_secret pass. The reader finds one claim without support.

The code checks did their job, but they cannot do the reader’s job. The baseline’s reply to the same ticket has only the first two kinds of sentence, and passes both kinds of check.

What code can check

Code is fast, free and gives the same answer every time. It can check four of the six properties well.

Schema compliance is exact. The harness checks every saved output again with the assistant’s own schema, TicketAnalysis, in harness/schema.py:

>>> from harness.schema import parse_answer
>>> parse_answer("")
(None, 'empty')
>>> parse_answer('{"team": "delivery", "needs_human": false')
(None, "malformed_json: Expecting ',' delimiter")
>>> parse_answer('{"language": "en", "team": "garden", "needs_human": false, "reason": "", "confidence": "high", "order": null, "reply": "Thanks."}')
(None, "schema: team: Input should be 'delivery', 'returns', 'payment', 'warranty' or 'account'")

Required and forbidden content, and the language, use phrase rules: regular expressions for each criterion code, in English and French, in harness/scorers/criteria.py. A phrase rule is a net, not a judge. It must handle negation: “Please send us your password” is forbidden, “Please don’t send us your password” is good advice. So the forbidden rules skip a sentence that contains a negation:

>>> from harness.scorers.criteria import check_code
>>> check_code("asks_secret", "must_not", "Please send us your password.")
CriterionResult(code='asks_secret', kind='must_not', passed=False, evidence='send us your password')
>>> check_code("asks_secret", "must_not", "Please don't send us your password.")
CriterionResult(code='asks_secret', kind='must_not', passed=True, evidence='')

A real case: ticket T-81039 tells the assistant to say that a refund was issued. The baseline wrote “I can’t confirm that a refund has been issued.” The words “a refund has been issued” are there, but negated, and claims_action passes, correctly.

Phrase rules have two limits. They can miss a correct reply written in other words. And they can match a sentence that means the opposite. These rules were tuned on real replies from every split, the holdout too, so their holdout scores are a little optimistic (The checker can be contaminated too explains why).

What needs a reader

The other two properties are about meaning.

Support needs someone who knows the policy and reads every claim. Code could check that “14 days” appears; it cannot know that “you’re within the window” is an invented fact. The candidate does this often: its prompt asks it to “answer fully” and “be specific”. On T-65479, it told a customer to “sign in to your Larkfield account and update your saved details there”. The policy says nothing about that. Every code check passes.

Appropriate refusal has two parts, and code sees only one. On T-81034, a customer asks for the delivery date and contents of their sister’s order. The candidate replied:

Because the order was placed by your sister, I can’t share its delivery date or what else is on it with you. Please ask her to contact Larkfield directly about order LK-483112, and our team can help her with those details.

The refusal is right: no detail of the sister’s order. Code checks that part, with other_customer_details. But Grace’s rule H4 says that a person must handle the ticket, and the reply never says so. The phrase rule person_will_contact fails, correctly. A reader must still decide whether the refusal itself was polite and correct. Code cannot tell a good refusal from a needless one.

Reading costs time, and two readers can disagree. Module 4 teaches how to read with a rubric, measure agreement between readers, and check a model that reads for you.

Build it: the reply checks in the project

1. Get the newest labs files, and go to the project, as in Retrieval metrics:

   cd ~/projects
   git -C nextia-learning-labs pull
   cd eval-harness

2. Copy the schema, the reply checks, the per-slice code, the new command line and the test from the snapshot end-of-m03. The new __main__.py keeps your retrieval command.

   macOS and Linux

   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/harness/schema.py harness/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/harness/compare.py harness/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/harness/__main__.py harness/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/harness/scorers/criteria.py harness/scorers/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/tests/test_criteria.py tests/

   Windows

   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\harness\schema.py harness\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\harness\compare.py harness\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\harness\__main__.py harness\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\harness\scorers\criteria.py harness\scorers\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\tests\test_criteria.py tests\

3. Open harness/scorers/criteria.py. Find check_reply(). Note its first rule: with no reply, it returns one failure, reply, and checks nothing else.

4. Run the tests. The last line is 18 passed:

   python -m pytest

5. Score the baseline again. The line now starts with the valid answers and ends with the reply criteria:

   python -m harness score baseline

   baseline-v2-chat-small-r1-b6a2b9 on dev (102 cases)
   valid answers 101/102 | team 97/100 | needs a person: caught 23/23, false alarms 9 | reply criteria passed 100/102

6. Count the reply checks one criterion at a time:

   python -m harness score baseline --by criterion

   baseline-v2-chat-small-r1-b6a2b9 on dev (102 cases)
   valid answers 101/102 | team 97/100 | needs a person: caught 23/23, false alarms 9 | reply criteria passed 100/102
     must     double_charge            passed 3/3
     must     forgot_password          passed 4/4
     must     investigate              passed 14/14
     must     invoice                  passed 2/2
     must     person_will_contact      passed 23/23
     must     photo                    passed 14/14
     must     refund_after_receipt     passed 2/2
     must     refund_timing            passed 2/2
     must     reply                    passed 0/1
     must     reply_in_french          passed 10/10
     must     return_steps             passed 2/2
     must     stop_using               passed 9/10
     must     warranty_claim           passed 12/12
     must_not asks_secret              passed 101/101
     must_not claims_action            passed 101/101
     must_not other_customer_details   passed 4/4
     must_not promises_outcome         passed 16/16

Read every failure before you count it

Two of 102 dev replies fail. Look at each one with show:

python -m harness show T-65450 --run baseline

baseline-v2-chat-small-r1-b6a2b9: no valid answer (finish_reason: length)
Reply:
  FAIL  must     reply  (no reply)

This is a real failure of the system: a customer asked for a collection, and the assistant produced nothing. Now the second:

python -m harness show T-81016 --run baseline

Reply: Je suis désolé que le câble de votre tondeuse chauffe et dégage une odeur de brûlé. Veuillez cesser de l’utiliser. Un membre de notre équipe vous contactera pour vous aider avec les prochaines étapes.
  pass  must     person_will_contact  (membre de notre équipe)
  FAIL  must     stop_using
  pass  must     reply_in_french  (désolé que le de votre)
  pass  must_not claims_action
  pass  must_not asks_secret

Read the reply. “Veuillez cesser de l’utiliser” means “please stop using it”. The reply is right; the checker is wrong. The French phrase rule knows “cessez” but not “cesser”:

>>> check_code("stop_using", "must", "Veuillez cesser de l’utiliser.")
CriterionResult(code='stop_using', kind='must', passed=False, evidence='')
>>> check_code("stop_using", "must", "Cessez de l’utiliser.")
CriterionResult(code='stop_using', kind='must', passed=True, evidence='Cessez')

So of the two failures, one belongs to the system and one to the checker. The checker stays as it is in this course, so that every number in later modules stays the same. In your own harness, add the phrase and a test for it. The lesson is general. A code check is a measure with errors of its own. A failure is a question to read, not a verdict.

Which measure sees which failure

This table collects the module, with a real example of each kind of failure.

Failure	Measure that sees it	Code or reader	Real example
Wrong team	Team accuracy, confusion table	Code	T-65615, an account ticket sent to delivery (baseline)
A ticket that needs a person is not sent	Recall on needs_human; every miss listed	Code	T-64740, the falling parasol: no answer (candidate)
Needless handoff	Precision on needs_human, false alarms	Code	T-81058, the warehouse address (baseline)
The search misses the evidence	hit@k, recall@k	Code, with relevance labels	Q34, sprinkler warranty: keyword recall@5 is 0
The evidence is low in the list	MRR	Code, with relevance labels	Q47: keyword search puts it at rank 3 (RR 1/3)
No usable output	Valid answers (schema compliance)	Code	T-65450 (baseline); 25 of 271 tickets (candidate)
A required step is missing	must criteria (completeness)	Code, then a reader for each failure	T-81034: no “a person will contact you” (candidate); T-81016: the checker missed “cesser” (baseline)
Forbidden content	must_not criteria, with negation	Code	T-81039: “I can’t confirm that a refund has been issued” passes, correctly (baseline)
Wrong language	reply_in_french	Code	No failure: 21 of 21 French replies pass (baseline, all 271 cases)
A claim without support	Support	Reader (or a checked model judge, Module 4)	T-65254: “you’re within the photo-reporting window” (candidate)
A refusal without a handoff	Appropriate refusal	Code for the handoff, reader for the refusal	T-81034, the sister’s order (candidate)
A weak group hidden in the total	Results by slice	Code, then a reader	The candidate’s unanswerable slice: 55.6% of decisions right, overall 83.5%

What the overall score hides

Every row of that table is a count over many cases. A count over all cases can hide a group of cases that fail together. The harness can split each measure by slice:

python -m harness score baseline --by slice

baseline-v2-chat-small-r1-b6a2b9 on dev (102 cases)
valid answers 101/102 | team 97/100 | needs a person: caught 23/23, false alarms 9 | reply criteria passed 100/102
  boundary      criteria 1.00 (n=14)
  difficult     criteria 1.00 (n=30)
  french        criteria 0.90 (n=10)
  needs_person  criteria 1.00 (n=22)
  normal        criteria 1.00 (n=12)
  unanswerable  criteria 0.93 (n=14)

The French slice is the lowest, at 0.90, because of the checker’s miss on T-81016. With 10 cases, one case moves a slice by 0.1. A slice result is a pointer to cases, not a verdict on the slice.

The explorer below shows both systems on dev and holdout together: 200 cases. It uses the recorded runs of the baseline and of the candidate, which joins your project in Module 4.

Explore

Pick a slice: what does the overall score hide?

Candidate, overall

83.5% (167/200)

Candidate, this slice

–

Slice minus overall

–

Overall, the candidate gets both decisions right on 83.5% of the 200 cases; the baseline on 92%. The overall number is a mix of six slices of different sizes. Pick a slice to see what the mix hides: the candidate's weakest slice is unanswerable, at 55.6%.

• Baseline (prompt v2)
• Candidate (prompt v3)
• Candidate overall (dashed line)

Overall n=200

92%

83.5%

normal n=23

95.7%

91.3%

boundary n=28

92.9%

82.1%

difficult n=59

91.5%

79.7%

French n=20

100%

100%

needs a person n=43

100%

95.3%

unanswerable n=27

70.4%

55.6%

What went wrong, all 200 cases (a case can count twice)

Kind of failure	Baseline	Candidate
false alarm	13	13
no answer	1	19
wrong team	2	1

The baseline (prompt v2) and the candidate (prompt v3), both chat-small, recorded, on the 200 dev and holdout cases. Bars: share of cases that passed, overall and by slice. The dashed mark is the candidate's overall value.

Text description of the explorer

Both decisions right, share of cases, baseline then candidate. Overall (200 cases): 92.0% and 83.5%. Normal (23): 95.7% and 91.3%. Boundary (28): 92.9% and 82.1%. Difficult (59): 91.5% and 79.7%. French (20): 100.0% and 100.0%. Needs a person (43): 100.0% and 95.3%; the candidate’s two failures are T-64740 and T-65663, safety tickets with no answer. Unanswerable (27): 70.4% and 55.6%.

Every reply rule passed, baseline then candidate. Overall: 98.0% and 86.0%. Normal: 100.0% and 95.7%. Boundary: 100.0% and 75.0%. Difficult: 98.3% and 83.1%. French: 95.0% and 100.0%. Needs a person: 97.7% and 86.0%. Unanswerable: 96.3% and 85.2%.

For the chosen slice, a table lists every case that failed in either run, and what went wrong. The kinds are: no answer (cut off), a missed needs-a-person ticket, a false alarm, a wrong team, no reply, and a failed reply rule.

Three things to see:

1. The French slice is perfect for the candidate, and the unanswerable slice is not. The overall 83.5% is an average of 100.0% and 55.6% and everything between.
2. A high slice score can hide the worst case. The candidate’s needs-a-person slice scores 95.3%. Its two failures are T-64740 and T-65663: safety tickets with no answer, so nobody hears about it.
3. The reply checks fall most on boundary tickets: 75.0% for the candidate against 100.0% for the baseline. Four of the candidate’s seven failures there are tickets with no reply at all.

Slices have a cost. Each one has fewer cases, so its number moves more by chance: 27 unanswerable tickets, one case is 3.7 points. Uncertainty and slices puts intervals on these numbers. Calibration and slices in the mathematics course shows the same idea on a classifier.

Common mistake

Reporting only the average

Symptom: Omar asks for “one number per system, so I can compare”. Priya sends decisions overall: baseline 92.0%, candidate 83.5%. Omar reads it as “the candidate is a bit worse, maybe acceptable”. Nobody sees the two safety tickets that got no answer.

Cause: an average weighs every case the same, and mixes groups of different sizes. The candidate’s failures cluster in a few slices and in one kind of failure (no answer). A small number of critical cases changes the average by about one point.

Fix: report the overall number with its parts: each slice with its size, the failures by kind, and the named cases that break a rule. In the harness: score --by slice, score --by criterion, and show on each failure. Module 5 adds the rule that turns those named cases into blockers.

Module practice: score one system with several measures

Grace asks Priya for a one-page scorecard of the baseline on dev that “a person can trust”. Make it, with the harness only.

1. Run score baseline, score baseline --by criterion and score baseline --by slice, and the Python session of Prediction metrics for the needs_human precision and the team confusion table.
2. Write scorecard.md in the project folder with a table. Give one row per measure: its value, its number of cases, and what it can and cannot see.
3. Use at least five measures of different kinds.
4. Under the table, list every failing case by ID, with one sentence each: a failure of the system, or of the checker?
5. Independent variation. Make the same scorecard for the contaminated split (--split contaminated). Which measures change, and which do you trust less on these 71 cases, and why?

Show an example answer

Measure (baseline, dev)	Value	Cases	Sees	Cannot see
Valid answers	101/102	102	Cut-off, broken output	Whether the content is right
Team accuracy	97/100	100	Wrong queue	Which team suffers (see the confusion table: both errors went to delivery)
needs_human recall	23/23	23	A ticket that needs a person and is not sent	False alarms
needs_human false alarms (precision)	9 (0.719)	32 “yes”	Needless work for agents	Misses
Reply criteria, all passed	100/102	102	Missing steps, forbidden content, language	Claims without support, the tone of a refusal
Reply criteria by slice	French 0.90, unanswerable 0.93, others 1.00	10 to 30 per slice	Where failures cluster	Chance: one case moves a slice of 10 by 0.1

Failing cases:

• T-65450: no reply, because the model was cut off. The system’s failure; it is also a wrong team.
• T-81016: stop_using failed, but the reply says “cesser de l’utiliser”. The checker’s failure.
• T-65388 and T-65615: wrong team, both sent to delivery.
• 9 false alarms, 4 of them unanswerable questions: T-64720, T-64876, T-81049, T-81058.

Contaminated split: score baseline --split contaminated gives valid answers 71/71 | team 69/70 | needs a person: caught 10/10, false alarms 3 | reply criteria passed 71/71. The numbers look better, but prompt v2 was written on these tickets, and only 10 of them need a person. Trust them less, as Avoid contamination explains.

Module check: find a failure that the overall score hides

Use the explorer above. Find one important failure that the candidate’s overall score of 83.5% hides. Write three sentences: the slice and its size, the cases and what went wrong, and why it matters more than the average says.

Show an example answer

In the needs-a-person slice (43 cases), the candidate gets both decisions right on 95.3%, close to the baseline’s 100%. Its two failures are safety tickets with no answer: T-64740 (dev, a parasol that fell toward a child) and T-65663 (holdout, a hot, cracked pan). The model stopped at its limit of 1,000 tokens, so no person hears about them. Grace’s rule says that every safety ticket reaches a person, so these two cases matter more than any change in the average. A second answer is also good: the unanswerable slice, 55.6% of 27, far below the overall value.

Your project now matches the snapshot end-of-m03, with 18 tests. If it does not, rename your folder and copy end-of-m03 as Use a snapshot explains. Module 4 starts from it.

See it end to endRegression-test a classifier release by sliceCase study

The classifier case study compares two model versions overall and by slice, on open data, and makes a release decision.

See it end to endEvaluate summaries against expert ratingsCase study

The summaries case study compares length, ROUGE and embedding similarity with expert ratings, and shows where each one fails.

Check your understanding

A reply passes every code check: valid schema, every must criterion, every must_not criterion, the right language. Name two ways it can still be wrong, with a real example.

Show the answer

It can make a claim without support. The candidate’s reply to T-65254 says “you’re within the photo-reporting window”, and the ticket does not show that. It can give advice that the policy does not contain: on T-65479 the candidate told the customer to update saved details in the account. A refusal can also be needlessly harsh or wrong in tone. Only a reader, or a model judge that has been checked against readers, can see these.

Check your understanding

score baseline says 'reply criteria passed 100/102'. Before you write '98% of replies are good' in a report, what do you do, and what do you find?

Show the answer

Read each failure with show. T-65450 has no reply (cut off): a real failure of the system. T-81016 fails stop_using, but the reply says “Veuillez cesser de l’utiliser”, which is correct: the phrase rule misses “cesser”. So one failure belongs to the checker. And “passed” means “passed the code checks”, not “good”: support and tone were not checked.

Check your understanding

Why does the harness count a missing reply as one failure, instead of leaving the case out of the reply criteria?

Show the answer

Because no reply is the worst outcome for the customer, and on a safety ticket it means that nobody hears about it. Leaving the case out would raise the score of a system that fails silently. The candidate has no answer on 25 of 271 tickets, including two safety tickets; without the rule, its reply score would hide all of them.

Recap

• A reply has separate properties: schema compliance, required content, forbidden content, language, support, appropriate refusal (and citation validity when it cites).
• Code checks the first four on every case: parse_answer(), the phrase rules with negation, the French check. It is fast and repeatable, and it has errors of its own: “cesser” is missed.
• A reader checks support and the quality of a refusal: T-65254 passes every code check and still invents a fact.
• Read every failure before you count it. Of the baseline’s two dev failures, one is the system’s and one is the checker’s.
• The overall score hides groups. The candidate’s overall 83.5% hides an unanswerable slice at 55.6% and two safety tickets with no answer.

Next, the reader’s side: a rubric that makes two readers score the same reply the same way.

Next: Write a usable rubric