Model-based judging

What you will be able to do

Let a model score replies at scale, but only after you have measured where it agrees with people and where it does not. Then write down the cases that it must never decide alone.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• Larkfield’s rubric, from Write a usable rubric.
• Percent agreement and Cohen’s kappa, and your own ratings in my_ratings.csv, from Calibrate reviewers.

What you need

• Your project at the end of the last lesson: python -m pytest gives 21 passed.
• An internet connection for one pip install.

You need no account and no key. Every judgment in this lesson was recorded once from a real model, and the project replays it. A live judge, on your own computer, is optional in the next lesson.

A new teaching assistant

A professor has 300 essays to mark, and she hires a teaching assistant. Before she trusts his marks, she marks 20 essays herself and compares. Most marks match. But she finds three habits. He gives more to long essays. When he compares two essays, he tends to prefer the one he read second. And he is kind to essays written in his own style.

She does not fire him. She writes down where his marks are good enough, and where she reads every essay herself: every failing essay, and every essay that may involve cheating. Then she checks his marks again each term.

The problem: too many replies for people

A person needs about a minute to score one reply with the rubric. The development and holdout cases have 200 tickets. Two systems give 400 replies, so one comparison takes a reviewer most of a day. Omar’s team wants to try a new prompt every week.

Tomás has an answer: “Let a model read them.” A model judge is a language model that reads a reply, with instructions such as a rubric, and returns a score or a choice. The approach is often called LLM-as-a-judge. It reads 400 replies in minutes, for about a dollar.

But a judge is a measuring instrument, like a new reviewer. Before you trust it, you check it the same way: on a sample that people have scored.

The idea: calibrate the judge like a reviewer

There are two common kinds of model judge.

• A reference-based judge scores one reply from 1 to 5, with the rubric and the case’s reference notes. It answers: “How good is this reply, for this case?”
• A pairwise judge reads two replies to the same ticket and chooses the better one, or a tie. It gets no reference notes. It answers: “Which reply should the agent start from?”

Both kinds are calibrated with the same loop as human reviewers. The judge rates the sample that people rated. You compare with agreement and kappa, and you decide where the judge is good enough.

The human review and model-judge calibration loop. People set the standard; the judge earns its place one question at a time, and is checked again after every change.

Text description of the diagram

1. People rate a fixed sample with the rubric, each on their own.
2. They compare and discuss. The rubric gets clearer.
3. The model judge rates the same sample.
4. Compare the judge with the people: agreement, kappa, and the items where they part.
5. Is the judge good enough for this question? If not, change the judge’s prompt or model, or keep people for this question, and try again from step 3.
6. If yes, use the judge for that question. A person reviews what the judge cannot judge.
7. When something changes (the model, the prompt, the rubric or the cases), check again from step 3.

Note “for this question”. A judge can be good at “is this reply bad?” and poor at “is it a 4 or a 5?”. You measure each question separately.

Add the judges to the project

The judges need a few new files: the judge code, the prompts, the adapter that talks to a model and the saved judgments. They also need the candidate’s saved replies, which arrive now, and one new package, openai, for the optional live judge.

1. In the project folder, with the environment active, set the variable S again if you opened a new terminal, as in lesson 1. Then copy the new requirements and install them:

   macOS

   cp $S/requirements.txt .
   python -m pip install -r requirements.txt

   Windows

   Copy-Item $S\requirements.txt .
   python -m pip install -r requirements.txt

   Linux

   cp $S/requirements.txt .
   python -m pip install -r requirements.txt

   The last line:

   Successfully installed anyio-4.15.1 h11-0.16.0 httpcore2-2.13.1 httpx2-2.13.1 idna-3.20 jiter-0.17.0 openai-3.27.0 sniffio-1.3.1 truststore-0.10.4

2. Copy the saved judgments, the new runs, the judge code, the prompts, the settings example and the tests:

   macOS

   cp -R $S/recordings .
   cp $S/outputs/*.jsonl $S/outputs/runs.json outputs/
   cp $S/harness/judge.py $S/harness/providers.py $S/harness/config.py $S/harness/costs.py harness/
   cp $S/harness/prompts/* harness/prompts/
   cp $S/.env.example .
   cp $S/tests/test_judge.py $S/tests/test_providers.py $S/tests/test_criteria.py tests/

   Windows

   Copy-Item -Recurse $S\recordings .
   Copy-Item $S\outputs\*.jsonl, $S\outputs\runs.json outputs\
   Copy-Item $S\harness\judge.py, $S\harness\providers.py, $S\harness\config.py, $S\harness\costs.py harness\
   Copy-Item $S\harness\prompts\* harness\prompts\
   Copy-Item $S\.env.example .
   Copy-Item $S\tests\test_judge.py, $S\tests\test_providers.py, $S\tests\test_criteria.py tests\

   Linux

   cp -R $S/recordings .
   cp $S/outputs/*.jsonl $S/outputs/runs.json outputs/
   cp $S/harness/judge.py $S/harness/providers.py $S/harness/config.py $S/harness/costs.py harness/
   cp $S/harness/prompts/* harness/prompts/
   cp $S/.env.example .
   cp $S/tests/test_judge.py $S/tests/test_providers.py $S/tests/test_criteria.py tests/

3. Run the tests:

   python -m pytest

   The last line:

   ============================== 34 passed in 0.33s ==============================

4. List the runs:

   python -m harness runs

   The output:

   baseline-v2-chat-small-r1-b6a2b9            271 cases  contaminated, dev, holdout gpt-6-luna-2026-09-22  started 2026-10-09T14:56:12-04:00
   candidate-v3-chat-small-r1-31b45f           271 cases  contaminated, dev, holdout gpt-6-luna-2026-09-22  started 2026-10-09T15:01:11-04:00
   strong-v2-chat-strong-r1-160302             102 cases  dev                        gpt-6.1-sol-2026-09-29  started 2026-10-09T15:34:30-04:00

   Three runs now: the baseline, the candidate (prompt v3, same model) and prompt v2 on the stronger model, used below.

By default, the judges use a mock provider. It finds the saved judgment for each request and returns it, so it needs no network and no key. The course recorded the judgments with two models on Azure: chat-small (gpt-6-luna) and chat-strong (gpt-6.1-sol). Models change fast, and a new version would score some replies differently. The numbers here are a snapshot of 9 October 2026; the method of checking them lasts.

The reference-based judge

Score every development reply of the baseline, then of the candidate:

python -m harness judge baseline
python -m harness judge candidate

The output:

T-81040: no verdict: the provider's content filter refused the judge request (jailbreak detected: True)
chat-strong judge (judge_reference_v1) on baseline-v2-chat-small-r1-b6a2b9, dev: 101 replies scored
scores: 1: 0  2: 2  3: 12  4: 25  5: 62 | mean 4.46 | critical: 0
agreement with the code checks (bad = critical or score 1-2 / a criterion failed): 0.98, kappa 0.49
T-81040: no verdict: the provider's content filter refused the judge request (jailbreak detected: True)
chat-strong judge (judge_reference_v1) on candidate-v3-chat-small-r1-31b45f, dev: 101 replies scored
scores: 1: 2  2: 25  3: 6  4: 32  5: 36 | mean 3.74 | critical: 2
agreement with the code checks (bad = critical or score 1-2 / a criterion failed): 0.86, kappa 0.60

The judge scores the candidate lower: a mean of 3.74 against 4.46, with 25 replies at score 2. Eleven of those 25 are not replies at all. Prompt v3 made the model think until its token limit on 11 development tickets, so it wrote nothing. The other 14 are real replies that invent a procedure or a timeline. Prompt v3 asks the model to “be specific” and to “answer fully in this one message”. When the policy has no specific answer, the model makes one up.

T-65479 from lesson 1 is one of them. The judge’s recorded reason for the candidate’s reply: “The reply invents a self-service step by telling the customer to update saved address and card details in their account, which the policy and reference notes do not establish.” That is the same reading as the ladder in lesson 1.

The last line compares the judge with the code checks of Module 3. Both say “bad” or “not bad” for each reply. They agree on most replies, because most replies are fine; kappa shows that the agreement above chance is moderate.

The first line is new. For T-81040 the judge gave no verdict. The next lesson explains why, in What the judge cannot judge.

Compare the judge with people

Is the judge right? Compare it on the 30 calibration replies, as you did for yourself. The harness has no command that writes the judge’s scores as a sheet, so write a small script. Save it as judge_sheet.py in the project folder:

judge_sheet.py

"""Write a model judge's scores for the 30 calibration replies as a sheet that `harness agree` reads.

    python judge_sheet.py chat-strong

The judgments are the recorded ones (the mock provider): no account, no cost.
"""
import csv
import json
import sys

from harness.dataset import DATA, by_id, load_cases
from harness.judge import parse_verdict, reference_request
from harness.providers import MockProvider, ProviderError
from harness.run import find_run, load_outputs

judge = sys.argv[1] if len(sys.argv) > 1 else "chat-strong"
key = json.loads((DATA / "calibration" / "key.json").read_text(encoding="utf-8"))
cases = by_id(load_cases(split="dev"))
provider = MockProvider()
rows = []
for item, where in sorted(key.items()):
    case = cases[where["case_id"]]
    out = load_outputs(find_run(where["run_id"]))[case.case_id]
    needs = out.answer.get("needs_human") if out.answer else None
    try:
        verdict = parse_verdict(provider.complete(reference_request(case, out.reply, needs, judge)))
        rows.append({"item": item, "score": verdict.score or "", "note": verdict.reason})
    except ProviderError:
        rows.append({"item": item, "score": "", "note": "no verdict: the provider refused the request"})
with open(f"judge_{judge}.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["item", "score", "note"])
    writer.writeheader()
    writer.writerows(rows)
print(f"Wrote judge_{judge}.csv: {sum(1 for r in rows if r['score'])} of {len(rows)} replies scored")

The script uses key.json to find each item’s case and run, and builds the same request as harness judge. Run it for the stronger judge, then compare its sheet with the reference ratings:

python judge_sheet.py chat-strong
python -m harness agree data/calibration/reference_ratings.csv judge_chat-strong.csv

The output:

Wrote judge_chat-strong.csv: 28 of 30 replies scored
28 replies scored by both
exact agreement 0.46 | within one point 0.82 | Cohen's kappa 0.22
bad (1-2) or not: agreement 0.96, kappa 0.78
rows: first sheet, columns: second sheet
  2: 2->2
  3: 3->1  4->1  5->3
  4: 3->2  4->4  5->5
  5: 2->1  3->1  4->2  5->6
discuss first (2 or more points apart): R13, R15, R17, R19, R23

Read it with care. This is agreement with the course author’s ratings: one person’s reading, not agreement with people in general. On the exact score, the judge is loose: kappa 0.22, with 5 items two points apart, R13 among them. On “bad or not”, it is close: kappa 0.78. It found both bad replies, and called only one other reply bad.

Now the weaker judge:

python judge_sheet.py chat-small
python -m harness agree data/calibration/reference_ratings.csv judge_chat-small.csv

The first three lines of the output:

Wrote judge_chat-small.csv: 28 of 30 replies scored
28 replies scored by both
exact agreement 0.32 | within one point 0.54 | Cohen's kappa 0.17
bad (1-2) or not: agreement 0.61, kappa 0.16

chat-small is unreliable on both questions. On all 200 baseline replies of the development and holdout cases, it marks 37 as critical, against 0 for chat-strong. Most of its “critical” reasons are the misreading of lesson 1: it calls “a member of our team will contact you” a promised service. A cheap judge that cries “harmful” on safe replies would block every release.

The course author’s ratings are one person’s reading. Yours are the second set of human ratings. Compare them with the judge:

python -m harness agree my_ratings.csv judge_chat-strong.csv

This is the module practice of the next lesson.

Pairwise judging and order bias

The pairwise judge compares the baseline (A) and the candidate (B) on each development ticket. The harness asks it twice: once with the baseline’s reply first, and once with the candidate’s reply first. A careful reader gives the same verdict both times.

python -m harness pairwise baseline candidate

The output:

chat-strong judge, baseline-v2-chat-small-r1-b6a2b9 (A) against candidate-v3-chat-small-r1-31b45f (B), dev: 102 pairs, both orders
A better: 28 | B better: 60 | tie: 1 | the verdict changed with the order: 13
order changed the verdict on: T-64707, T-64873, T-65054, T-65137, T-65353, T-65460, T-65567, T-65820, T-65855, T-81036, T-81039, T-81044, T-81050

On 13 of 102 tickets, swapping the places of the two replies changed the verdict. This is order bias, or position bias: the judge’s choice depends on where a reply is shown, not only on what it says. The harness counts those tickets as “inconsistent”, and neither system gets the win. Explore them.

Explore

Does the order change the judge's verdict?

Tickets, both orders

102

Candidate better

60

Baseline better

28

Tie

1

Order changed the verdict

13 (12.7%)

• The order changed the verdict
• Candidate better in both orders
• Baseline better in both orders
• Tie in both orders

The order changed the verdict: the reply shown second won both times, so each system "won" once.

Ticket T-64707 · slice difficult

Ordered the bathroom cabinet last Tuesday, order LK-561208. It came Friday, all fine, put it up Saturday. Sunday I was going through the statements because my partner asked me to check the card. Two charges for the cabinet, same day, same amount, both from you. Bank shows both as posted. I only meant to pay once. I did not place a second order and the confirmation emails show only one. The cabinet is in the bathroom and I don't need a second one. I have screenshots of both charges if you need them. Please refund the extra one and tell me when it's done.

Baseline reply (215 characters)

I'm sorry you were charged twice. Our payment team will review the charges, and a confirmed double charge is refunded within 3 business days after confirmation. You can share the screenshots to help with the review.

Candidate reply (518 characters)

I’m sorry you’ve had to spend time sorting out a duplicate charge, especially when you only placed one order. Our payment team will check the two posted charges against order LK-561208 and, if they confirm it was a duplicate, refund the extra charge within 3 business days of confirmation. The refund goes to your original payment method; please allow 5 to 10 business days after it is issued for it to appear. You can reply with the screenshots if you’d like to share them, and there’s no need to place another order.

Order 1: baseline shown as A, candidate as B

The judge chose: B, the candidate

“Draft B more clearly makes the refund conditional on confirming the duplicate charge and explains the original payment method and expected timing. Both drafts follow the payment policy without claiming a refund has already been issued, but B is more complete and specific to the order.”

Order 2: candidate shown as A, baseline as B

The judge chose: B, the baseline

“B accurately gives the specific double-charge refund timeline: within 3 business days after confirmation. A adds a further 5–10-business-day wait after issuance, which the policy does not specify for confirmed double charges.”

Real pairwise judgments of the baseline's and the candidate's replies on the 102 development tickets, each asked in both orders (judge_pairwise_v1; Azure, recorded 9 October 2026).

Text description of the explorer

The explorer shows one square per development ticket, grouped by outcome, and the details of one ticket at a time. For the chat-strong judge, the candidate is better in both orders on 60 tickets and the baseline on 28, with 1 tie. The order changed the verdict on 13 (12.7%). On 11 of those 13, the reply shown second won both times; on 2, the reply shown first won both times. For the chat-small judge: candidate 67, baseline 19, tie 1, order changed 15 (14.7%). For each ticket, the explorer shows the ticket, both replies with their length, and for each order the judge’s choice and its recorded reason. Example: on T-65460 (chat-strong), the judge chose the second reply in both orders, so each system “won” once.

Two patterns appear. First, when the order changes the verdict, the reply shown second usually wins: 11 of the 13 times for chat-strong. On all 200 development and holdout tickets, the second place won 218 times and the first 180. Second, the candidate wins most consistent verdicts, 60 to 28. Its replies are also more than twice as long: a median of 386 characters against 178 on the development tickets.

Is the candidate better, then? Look at one ticket where the judge is consistent. On T-81034, a customer asks about their sister’s order. The policy says that a person must handle such a ticket (rule H4).

The same two replies to T-81034, read by the same model as two judges (chat-strong, recorded). The pairwise judge has no reference notes; the reference-based judge has them.

Text description of the diagram

The baseline replies: “Since the order was placed by your sister, a member of our team will contact you about what information we can share.” The candidate replies: “I can’t share its delivery date or what else is on it with you. Please ask her to contact Larkfield directly… and our team can help her.”

The pairwise judge, without the reference, chose the candidate in both orders: “B explicitly protects the sister’s order details and directs the order owner to contact Larkfield.” The reference-based judge, whose notes say that a person must handle the ticket, scored the baseline 5 and the candidate 2: “it does not say that a person will handle this ticket, as required for an access request.”

The same model gives opposite answers, because it was asked different questions. Without a reference, a judge rewards what looks good on the page: a clear, warm, complete answer. It cannot know that Larkfield’s rule wants a person on this ticket, or that a “specific” step is invented. With the reference, it checks the reply against the case. A pairwise judge measures preference, not correctness.

See it end to endCalibrate a model judge against human ratingsCase study

The judge calibration case study asks two model judges in both orders and writes a policy for what the judge may decide alone.

See it end to endEvaluate summaries against expert ratingsCase study

The summaries case study measures two model judges with a rubric per quality against three experts, and meets articles the content filter refused.

Knowledge check

Check your understanding

What is the difference between a reference-based judge and a pairwise judge, and why did they disagree about the candidate?

Show the answer

A reference-based judge scores one reply with the rubric and the case’s reference notes. The notes give the team, whether a person must handle it, the next step and what the reply must not do. A pairwise judge compares two replies and has no reference. Without the reference, the judge rewards what reads well: the candidate’s longer, warmer, more specific replies won 60 to 28 on the development tickets. With the reference, the same model sees that many of those specific steps are invented, and that some replies leave out the person who must handle the ticket. So it scores the candidate lower: 3.74 against 4.46.

Check your understanding

Predict: you swap the order in which a pairwise judge sees two replies. A careful judge gives the same verdict. What did the chat-strong judge do on the 102 development tickets, and how does the harness handle it?

Show the answer

On 13 of 102 tickets, the verdict changed with the order. On 11 of them, the reply shown second won both times. This is order bias. The harness asks every pair in both orders, turns the second answer back into the first order’s names, and keeps a winner only when both orders agree. Otherwise it records “inconsistent”, and neither reply wins.

Recap

• A model judge scores a reply with the rubric and reference notes, or picks the better of two replies without them. Calibrate it against people, one question at a time.
• Against the course author’s reference ratings, chat-strong is close on “bad or not” (kappa 0.78) and loose on the exact score (0.22). chat-small is unreliable (0.17). Your own ratings are a second human check.
• Without a reference, the judge prefers the longer, warmer candidate, 60 to 28 on the development tickets. With the reference, it scores the candidate lower, because the candidate invents steps.
• Order bias is measured, not assumed: the order changed 13 of 102 verdicts. The harness keeps a winner only when both orders agree.
• Your project has the judges and the saved judgments: python -m pytest gives 34 passed.

Next, you check the judge for two more habits, self-preference and drift. You also find the cases that it cannot judge, and you decide where a person must review.

Next: Model-based judging, continued