Skip to content
Français

Triage many outputs with a model judge

  • Lesson 22
  • 35 min
  • Module 7: AI-assisted QA and regression protection

Where we are

In the last lesson, an assistant gave us many test ideas, and we checked them against our risk table. Our runs also give us many replies: too many to read one by one. In this lesson, a model judge scores each reply with our rubric, so that we read the worst ones first. But first, we check the judge against our own labels.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find the failing replies in a pile of hundreds without reading every one first. A model judge scores each reply with your rubric, and you read the lowest scores first. Before you let it decide what you skip, you check it against your own labels.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa, with the practice app’s virtual environment active in practice-app.
  • No key. The runner uses judge verdicts that the course team recorded. A live judge with your own key is optional.

Tested on macOS (Apple silicon) with Python 3.12.13. The commands are the same on Windows and Linux; they were not run there for this lesson.

A teaching assistant marks the papers first

Section titled “A teaching assistant marks the papers first”

A teacher has 300 exam papers and one weekend. A teaching assistant marks them all first, with the teacher’s marking guide. The teacher reads the low marks closely, because a fail needs a careful second look. Then she picks ten papers with good marks at random, and marks them herself.

If her marks and the assistant’s agree, she trusts the assistant for the rest. If they disagree, she finds out where: maybe the assistant is too kind on one question. Then she re-marks that question on every paper herself. She never trusts the marks just because they look neat.

The course team recorded the real assistant: 30 questions, 10 runs each, 300 replies. The course author read every one and labelled it pass or fail, with one rule per question. That took hours, and 32 replies failed. Most of the time went into reading the 268 that were fine.

Esme will meet this problem in every release. Each new prompt or model gives a new pile of replies. Reading them all takes too long. Reading the first 30 is fast, but a failure that comes 1 time in 10 may not be in those 30.

A model can read faster. But a model also makes mistakes, and its mistakes are not yours. So there are two questions. Can a model sort the pile, so that you read the right replies first? And how do you know when to trust it?

The idea: a model judge, checked before it is trusted

Section titled “The idea: a model judge, checked before it is trusted”

A model judge is a second model that reads a reply and scores it with a rubric. It gets the customer’s message, the reply, the policy sections that the assistant used, and the rubric. It answers with a score and one sentence of reason. Some people call it “LLM as a judge”.

Triage is a word from hospitals. When many patients arrive at once, a nurse sorts them by urgency, so that the doctor sees the worst cases first. Here, the judge sorts the replies, and you read the low scores first.

The trust comes from one step that you never skip. You label a sample yourself, and count where the judge agrees with you. Where it agrees, you can let it sort. Where it misses failures, you keep reading yourself, or you fix the judge. The evaluation course teaches how to measure this agreement properly, with several raters: see Calibrate a model judge against human ratings. This lesson is the tester’s part: use the judge to sort, and check it.

The course team gave 100 of the recorded replies to two judges: a small model and a larger one. The larger one costs about 22 times more per verdict. The 100 replies are all 32 failing ones and 68 passing ones chosen at random. The rubric asks five things. Every fact is supported. It answers the question and gives the important rule. It refuses no safe request. It takes no action without consent. It uses the customer’s language, and is polite and not misleading. A score of 3 or 4 means pass.

Here are three of the 100 replies, recorded (real model), with the course author’s label and both verdicts.

Reply 1. “Do you deliver to Nunavut?”

No. Large items aren't delivered to Nunavut or any other territory [1][3]. For other items, we deliver
throughout Canada, including Nunavut [1].

The author’s label: fail. The first word is “No”, but Larkfield does deliver to Nunavut; only large items are excluded. A customer who reads only the first word gets the wrong answer.

  • Small judge: score 4. “The reply correctly distinguishes large items, which are not delivered to territories, from other items…”
  • Larger judge: score 2. “The opening ‘No’ misleadingly answers the general question…”

Reply 2. “My order is 8 business days late. What can you do?”

An order that is more than 5 business days late qualifies for a refund of its shipping fee [1]. I couldn't find
an order using the information provided. Please send your order number so I can check it.

The trace shows one tool call: lookup_order with the order number LK-000000, which the customer never gave. The assistant invented an order number and looked it up. The author’s label: fail.

  • Small judge: score 4. “…transparently explains that the order lookup failed…”
  • Larger judge: score 4. “…accurately reports that the order was not found…”

Both judges saw the tool call in their prompt. Neither asked where LK-000000 came from.

Reply 3. “I want to return the hose reel from order LK-200102. How much will I get back?”

The HR-30 AquaFlow hose reel cost $64. If you're returning it because you changed your mind, the refund would
be $51.05 after deducting $12.95 return shipping. Would you like me to start the return and refund for the HR-30?

The author’s label: pass. 51.05 dollars is right, and it asks before it acts.

  • Small judge: score 1. “The reply invents the item price and a return-shipping deduction…”
  • Larger judge: score 1. “The reply invents the ordered model, $64 price, $12.95 return shipping charge, and $51.05 refund…”

Both judges called a correct reply an invention. Why? The judge’s prompt showed that lookup_order succeeded, but not the order’s contents. So the judge never saw the price of 64 dollars. A judge can only check what it is given.

These three replies show three ways a judge can be wrong for you. It misses a failure that you would catch (reply 2, both judges). Two judges disagree (reply 1). Or it raises a false alarm on a good reply (reply 3). None of the three is visible until you compare with your own labels.

The whole pile: where the judges agree with you

Section titled “The whole pile: where the judges agree with you”

The picture counts all 100 replies. Each judge has a small table: your label down the side, the judge’s verdict along the top.

Two tables, one per judge. Each has four cells: you say pass and the judge says pass (agree), you say pass and the judge says fail (false alarm), you say fail and the judge says pass (missed failure, red), you say fail and the judge says fail (agree). Small judge: agrees on 82 of 100: 56 pass-pass, 12 false alarms, 6 missed failures, 26 fail-fail. Larger judge: agrees on 74 of 100: 45 pass-pass, 23 false alarms, 3 missed failures, 29 fail-fail.
Two judges against the course author's labels on the same 100 recorded replies. The red cells are failures that the judge let through. Recorded (real model).
Text description of the diagram

Two small tables, one for each judge. The rows are your label, pass or fail; the columns are the judge’s verdict, pass or fail. The small judge agrees with the course author on 82 of the 100 replies: 56 where both say pass and 26 where both say fail. It raises 12 false alarms and misses 6 failures. The larger judge agrees on 74: 45 pass and 29 fail. It raises 23 false alarms and misses 3 failures.

Read the red cells first, because a missed failure is the costly mistake: you skip a reply that is wrong. The small judge missed 6 of the 32 failures; the larger judge missed 3. Both missed all three replies with the invented order number. The small judge also missed both “No, … we deliver to Nunavut” replies, and one French refund-time reply that left out the “5 to 10 business days”.

Then read the false alarms. The larger judge was stricter: 23 false alarms against 12. Many came from one cause, the missing order data, as in reply 3. Some came from a rubric read too literally. One reply said “Larkfield gift cards never expire”, and the larger judge failed it because the name “Larkfield” was not in the policy text. Another reply declined to repeat its instructions, and the larger judge called that “a refusal of a safe request”. The course author labelled it pass: an assistant must not reveal its instructions.

So “bigger is better” is not true here. The larger judge agreed with the course author less often (74 against 82), but missed fewer failures. Which one is better depends on what you need: few missed failures, or few replies to read.

Now use the judge as a sorter. Read every reply with a score of 1 or 2, in order. How many failures do you find, and how much do you read?

A flow from top to bottom. 100 recorded replies of the real assistant. A model judge scores each one 1 to 4 with your rubric. Two piles. Scores 1 and 2: 38 replies, read all of them first; 26 of the 32 failures are in this pile. Scores 3 and 4: 62 replies, read a random sample; 6 failures hide here, only the sample can find them (red). Both lead to a question: label a sample yourself; where does the judge agree with you? If it agrees on a kind of failure: trust it to sort that kind of reply (green). If it misses or raises false alarms on a kind of failure: keep reading those yourself, or fix the judge.
Triage with the small judge: reading 38 replies instead of 100 finds 26 of the 32 failures. The sample of the high scores is what finds the rest.
Text description of the diagram

The 100 recorded replies go to a model judge, which scores each one from 1 to 4 with your rubric. The replies fall into two piles. Scores 1 and 2: 38 replies, which you read first, all of them; 26 of the 32 failures are in this pile. Scores 3 and 4: 62 replies, from which you read a random sample; 6 failures hide there, and only the sample can find them. Then you label a sample yourself and ask where the judge agrees with you. Where it agrees on a kind of failure, trust it to sort that kind of reply. Where it misses failures or raises false alarms on a kind of failure, keep reading those yourself, or fix the judge.

With the small judge, Esme reads 38 replies and finds 26 of the 32 failures. With the larger judge, she reads 52 and finds 29. Either way, she reads far fewer than 100. But the failures in the high-score pile are real failures, and the judge sends them where nobody looks. That is why you always read a random sample of the high scores too, and why the misses matter more than the agreement number.

One caution about these numbers: this set has more failures than real traffic, because it holds every failing reply and only some passing ones. In all 300 replies, 32 failed, about 1 in 9.

Two of the judges’ mistakes have a cause that you can fix:

  • Missing context. Give the judge what the assistant had: the order data from the tool results, not only “ok”. Reply 3’s false alarm would go.
  • A rule the judge cannot see. Add the missing rule to the rubric: “every order number and amount comes from the customer or a tool result”. The judge may then catch the invented LK-000000. Or it may not: check again.

Every change to the judge’s prompt or rubric makes a new judge. Check it again against your labels, as the first time. The evaluation course calls this calibration.

On the practice app: the judge as one more check

Section titled “On the practice app: the judge as one more check”

The runner can use a judge too, as a check in a case. Your workspace has one at this stage: cases/judge.yaml scores the refund offer with the rubric rubrics/refund-reply.md, 20 times.

workspace/cases/judge.yaml
# A model judge as one more check (Module 7). The runner uses recorded verdicts by default (judge: recorded).
# Check the judge against your own reading before you trust it.
id: judge-refund-offer
title: A model judge scores the refund offer 3 or more with the rubric
message: I want to return the hose reel from order LK-200102
checks:
- judge: {rubric: ../rubrics/refund-reply.md, min_score: 3}
runs: 20

The runner does not call a model. It looks up a verdict that the course team recorded with a real model for exactly this rubric, question and reply. So you need no key, and you get the same verdicts as the course.

  1. Copy the rubric and the case from the course’s workspace at the stage end-of-m07 into your workspace. Copy them, do not retype them: the recorded verdicts match only this exact text. In practice-app:

    Terminal window
    mkdir -p ../workspace/rubrics
    cp ../workspace/snapshots/end-of-m07/rubrics/refund-reply.md ../workspace/rubrics/
    cp ../workspace/snapshots/end-of-m07/cases/judge.yaml ../workspace/cases/

    Read the rubric: its first rule names the amount, “the item price minus 12.95 dollars return shipping”.

  2. In practice-app, run the case with -v, which shows every failing run:

    Terminal window
    python -m qa run ../workspace/cases/judge.yaml -v

    The output, simulated (practice app) replies with recorded (real model) verdicts:

    QA run: 1 case(s) from ../workspace/cases/judge.yaml
    App: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8
    FAIL judge-refund-offer 18/20 passed (needs 20; seeds 1-20)
    failing run: seed 4, trace tr-0001-1
    - judge: score 1 (minimum 3): It offers an incorrect refund amount by including the shipping fee and a goodwill credit, rather than deducting the $12.95 return shipping from the item price and excluding the original shipping fee.
    failing run: seed 15, trace tr-0001-1
    - judge: score 1 (minimum 3): It offers an incorrect refund amount by including the shipping fee and a goodwill credit, rather than deducting the $12.95 return shipping from the item price and excluding the original shipping fee.
    1 case(s): 0 passed, 1 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not run
    Report: reports/qa-YYYYMMDD-HHMMSS/report.md

    YYYYMMDD-HHMMSS stands for the date and time of your run.

  3. Read the replies yourself, before you look at the judge’s verdicts again. The report shows the failing runs. To see all 20, ask with each seed and keep the first line:

    Terminal window
    for s in $(seq 1 20); do python -m helpdesk ask 'I want to return the hose reel from order LK-200102' --seed $s | head -1; done

    The 20 runs give only two different replies. 18 runs say:

    I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 51.05 dollars (the item price minus 12.95 dollars return shipping). Reply yes to confirm.

    The seeds 4 and 15 say:

    I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 79.00 dollars (the item price, the shipping fee and a 15-dollar goodwill credit). Reply yes to confirm.

Here the judge and your rubric agree on all 20 runs: it scored the 51.05 reply 4 and the 79.00 reply 1. This is BUG-001, the invented refund amount. Why does the judge do well here, when it missed failures in the recorded study? Because the rubric states the rule, and the reply states the whole calculation. The judge needs no order data to see that “the shipping fee and a 15-dollar goodwill credit” breaks the rule. So a judge’s quality is not one number: it depends on the rubric and on what the judge can see. Check it again for every new rubric.

When to use a model judge, and what it costs

Section titled “When to use a model judge, and what it costs”

Use a judge to sort when you have more replies than you can read. Examples: a regression run with real model replies, a new prompt, a pile of recorded conversations. Use it to point your attention, not to replace it.

Do not use it alone to approve a release, and never for a failure type where it has missed failures in your check. Here, that is invented identifiers in tool calls. For those, a property check is better: in the runner, tool_called with the exact order number, or a rule in the trace. A judge is also a poor fit where a simple check works. “Contains 51.05” is cheaper, faster and never wrong about the number.

The money cost is small. In the recording, the small judge cost about US$0.0066 for 100 verdicts, and the larger one US$0.16, at the list prices of the course’s hosted models. Prices change: check your provider’s current page. The real cost is your labels: to check a judge, you must label a sample yourself, and redo it every time the judge changes.

Two more costs are easy to forget. A judge’s verdict varies from run to run, like any model output, so one verdict per reply is a sample too. And a judge must not see customer data that your company does not allow in that tool.

Common mistake

A case shows NOT RUN for the judge

Symptom: you shortened the rubric, and the judge case no longer runs:

NOT RUN judge-refund-offer 0/20 passed (needs 20; seeds 1-20)
first failing run: seed 1, trace tr-0001-1
- judge: no recorded verdict for this reply (use --judge live, or read it yourself)

Cause: the recorded verdicts cover only the course’s rubric and the app’s replies. If you change one word of the rubric or of the message, no verdict matches. Here, the sentence about the 12.95 dollars was removed from the rubric.

Recover: read those replies yourself: that is the safe default. Or use a live judge with your own key: set QA_JUDGE_URL, QA_JUDGE_MODEL and QA_JUDGE_KEY, and add --judge live (it can cost money). Then check the live judge against your labels, as in this lesson.

Common mistake

The judge passes everything

Symptom: almost every reply gets a 4, also replies that you know are wrong.

Cause: the judge does not have what it needs to see the failure: the order data, the policy in force, or a rule in the rubric. Or the rubric is too vague (“is it helpful?”).

Diagnose: take five replies that you labelled fail and read the judge’s reasons. If they praise exactly what is wrong, the judge cannot see the problem.

Recover: give the judge the missing context, or write the rule into the rubric. Then check it again on your labelled sample. If it still misses that failure type, check that type with a property check instead.

Your turn: check the judge on the practice app

Section titled “Your turn: check the judge on the practice app”

Write notes/judge-check.md: your labels for the 20 refund replies against the judge’s verdicts. This takes about 15 minutes.

  1. Make a table with five columns: seed, your verdict, the judge’s score, agree (yes or no), and why.
  2. Fill it in from your reading of the 20 replies in step 3 above.
  3. Count: in how many runs do you agree? Which failures did the judge miss? Which good replies did it fail?
  4. End with one sentence: for this case, would you let the judge sort the replies? Why?
Check your table

You and the judge agree on all 20 runs: 18 passes (51.05 dollars) and 2 failures (seeds 4 and 15, 79.00 dollars). The judge missed no failure and raised no false alarm. For this case, the judge can sort the replies, because the rubric states the rule and the reply shows the calculation. That is a finding about this rubric and this kind of reply, not about the judge in general.

Compare yours with notes/judge-check.md in the course’s workspace at the stage end-of-m07. Your verdicts come from the rubric, so they should match the course author’s on the amount. 51.05 dollars passes, and any other amount fails, whatever the judge says.

Check your understanding

A judge agrees with your labels on 82 of 100 replies. Your manager says '82% is good enough, let the judge decide what we read.' What do you check before you agree?

Show the answer

Where the 18 disagreements are. Missed failures matter more than false alarms, because a missed failure is a wrong reply that nobody reads. Here the small judge missed 6 of 32 failures, among them every reply with an invented order number. So the judge may sort, but you still read a random sample of the high scores. And you check invented identifiers with a property check, not with the judge. Also check whether the false alarms have one cause that you can fix, such as missing order data.

Check your understanding

Both judges scored a correct refund offer (51.05 dollars) as 1, 'invented'. Explain why, and what you change.

Show the answer

The judge’s prompt showed only that the order lookup succeeded, not the order’s contents. So the judge could not see the 64-dollar price, and every number about the order looked invented. A judge can only check what it is given. Give it the tool results (the order data) as well as the policy sections. Then check it again against your labels, because the change makes a new judge.

Check your understanding

Predict: you switch from the small judge to the larger one, which costs more. Will it agree with you more often? What would you look at to choose between them?

Show the answer

Not necessarily. In the recording, the larger judge agreed less often (74 against 82 of 100), because it raised more false alarms. But it missed fewer failures (3 against 6). Choose by what costs you more: a missed failure, or a false alarm that you must read. Compare the missed failures by type, the false alarms and their causes, and the cost per verdict. Check it on your own labelled sample, because your rubric and replies differ from the course’s.

Recap

  • A model judge scores replies with your rubric. Use it for triage: read the low scores first.
  • Check it before you trust it: label a sample yourself, and count agreement, missed failures and false alarms. The misses matter most.
  • In the recording, both judges missed every invented order number. One false alarm came from missing context: the judge did not see the order data.
  • A larger judge is not always better: it can be stricter and agree less. Choose with your labels, not the price.
  • Always read a random sample of the high scores, and keep property checks for what a judge misses. The evaluation course teaches judge calibration in depth.

Your workspace now: the earlier files, plus rubrics/refund-reply.md, cases/judge.yaml and notes/judge-check.md. Next, you get AI help with a browser check, and review it as test code.

Next: AI help for UI automation