Calibrate a model judge against human ratings

What you will be able to do

Find out whether a model judge can replace people for one job: choosing the better of two chatbot answers. You measure how often real people agree with each other, how often two model judges agree with them, and where the judges fail. Then you write a judge policy and test it once.

Case studyPortfolio piece

Dataset

MT-Bench human judgments (LMSYS)

Licence

CC BY 4.0 (votes); Apache-2.0 (prompts, references)

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Human agreement as the ceiling
• Majority labels
• Pairwise judges in both orders
• Cohen's kappa with intervals
• Order and length bias
• Slices by category
• A written judge policy
• One test on holdout

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. It downloads two files (about 2 MB) and needs only Python and numpy, so it runs in Colab or Kaggle with no setup. It runs in a few seconds. You need no account and no key: every model judgment was recorded once, and the notebook replays it.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study goes further than Module 4. These lessons matter most:

• Percent agreement and Cohen’s kappa, from Calibrate reviewers.
• Pairwise judges, both orders and order bias, from Model-based judging.
• Bootstrap intervals, paired differences and slices, from Uncertainty and slices.
• Dev and holdout sets, from Avoid contamination.

What you need on your computer (only for the local route; Colab and Kaggle need none of it)

• Python 3.12 or later with numpy, and the notebook, downloaded from GitHub, in VS Code or JupyterLab. The course project is not needed.
• The notebook’s first cell downloads the data into a new folder judge-calibration. To start again, delete that folder.

Tested on macOS with Python 3.12 and numpy 2.5.3, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.

The situation

Ines leads evaluation at a small company that builds a general chat assistant. Every week, the team compares two versions of the assistant on a fixed set of questions. The questions cover writing, role-play, reasoning, maths, coding, extraction, science and humanities. For each question, a reviewer reads the two answers and chooses the better one, or a tie. Two reviewers spend most of a day on it, every week.

Daniel, an engineer on the team, has a proposal. “Let a model choose. The MT-Bench paper says that GPT-4 agrees with people over 80% of the time, as often as people agree with each other. Our Azure models are newer than GPT-4. We can stop the reviews.”

Ines does not say no. She asks for evidence first. “Show me where the judge agrees with people, on questions like ours. Then show me where it does not. I want a written rule: what the judge decides alone, and what a person still reads.”

A wrong verdict can ship the worse version; reading everything costs two reviewers a day each week. Ines wants the judge where it is as good as a person, and people where it is not.

What you will build

You will build a calibration study of two model judges, and a judge policy from it. The study measures four things on real human votes. How often do people agree with each other? How often does each judge agree with them? Does the verdict depend on the order or the length of the answers? Where does the judge fail, by category? The policy turns those numbers into rules. At the end, you test the policy once, on questions that it has never seen.

The skills come from the whole course. Dev and holdout sets come from Avoid contamination, and agreement and kappa from Calibrate reviewers. Judges in both orders come from Model-based judging, and intervals and slices from Uncertainty and slices.

The data

In 2023, the authors of the MT-Bench paper asked six chatbots of that time the same 80 questions. The chatbots were GPT-4, GPT-3.5, Claude v1, Vicuna-13B, Alpaca-13B and LLaMA-13B. Then they asked people to compare the answers in pairs. The result is the MT-Bench human judgments dataset: about 3,300 votes, each saying which of two answers is better, or a tie.

Who are the people? The paper says: “58 expert-level human labelers … mostly graduate students”, from more than ten universities. They were paid US$20 to judge 20 questions. Some of the paper’s authors voted too. In the data, each person has a pseudonym, such as expert_17 or author_4. These are real human ratings, made by those people. The model verdicts in this case study are model outputs, and the page keeps the two apart.

Source and licence

MT-Bench human judgments, LMSYS (Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685), huggingface.co/datasets/lmsys/mt_bench_human_judgments, licensed under CC BY 4.0. Categories, reference answers and the judge prompts come from lm-sys/FastChat (Apache-2.0). Changes: the course selected 320 first-turn items with votes from 2 or more human judges (40 per category), kept some fields, wrote every label in a fixed model order, and added the category, a dev/holdout split by question and, for questions 101–130, the reference answer.

The course also considered HelpSteer3 (CC BY 4.0). It publishes “only the three annotations that agree with one another the most”, so it makes people look more consistent than they are. MT-Bench keeps every vote and every pseudonym.

One item

One item is one question, the first answers of two chatbots, and the votes of the people who compared them. A vote is A (the answer of model_1 is better), B (the answer of model_2 is better) or tie. model_1 is always the model whose name comes first in alphabetical order, so a label means the same thing on every line.

Field	What it means
item_id, question_id, category	The item; the MT-Bench question (81 to 160); one of 8 categories of 10 questions
split	dev or holdout (see below)
question, answer_1, answer_2	The question and the two answers of 2023, from model_1 and model_2
reference	For reasoning, maths and coding: a reference answer, written by GPT-4 for MT-Bench
human_ratings	The votes: the person’s pseudonym and A, B or tie
gpt4_2023	The paper’s own GPT-4 judge, asked in both orders in 2023 (inconsistent = the order changed its verdict)

First checks

The notebook loads the sample and counts:

items = [json.loads(line) for line in open("data/mtb_judge_sample.jsonl", encoding="utf-8")]
dev = [it for it in items if it["split"] == "dev"]
holdout = [it for it in items if it["split"] == "holdout"]
votes = [r for it in items for r in it["human_ratings"]]
print(len(items), "items |", len(dev), "dev |", len(holdout), "holdout")
print(len(votes), "human votes from", len({r["rater_id"] for r in votes}), "people")
print("votes per item:", dict(sorted(Counter(len(it["human_ratings"]) for it in items).items())))

The output:

320 items | 155 dev | 165 holdout
834 human votes from 65 people
votes per item: {2: 187, 3: 89, 4: 31, 5: 10, 6: 2, 7: 1}

What this tells us. The sample has 320 items with 834 votes from 65 people. Most items have only two votes, so “the people” on one item are often just two people. The sample takes 40 of the 484 eligible items from each category, five times the sample in Calibrate reviewers.

Frame the problem

Before any judge runs, Ines and Daniel agree on four things.

• The unit: one item, a question with two answers.
• The judge’s job: choose the better answer, or a tie. It is the same job as the reviewers’.
• The split, by question. Each MT-Bench question appears in several items, with different pairs of models. If a question sat on both sides, the policy could learn about it on dev and meet it again on holdout. So in each category, 5 questions go to dev: 155 items to calibrate the judges and write the policy. The other 5 go to holdout: 165 items to test the policy once, at the end.
• What counts as “right”. “Which answer is better?” has no answer key. There are only people’s votes. So the judge is compared with people, and the agreement between people is the ceiling.

That last point is the new idea of this case study.

The idea: people set the ceiling

Think of two referees who watch the same match. They disagree on some calls, because some calls are hard. A third referee who agrees with each of them as often as they agree with each other is as good as a referee. You cannot ask for more, because there is no true call to compare with.

A judge is the same. Compare its agreement with one person with the agreement of one person with another person: the same kind of pair. That is the ceiling. Agreement with the majority of the people on an item is a second, useful view: “would the judge give the team’s usual answer?”. But a majority is easier to agree with than one person, so never compare those two numbers with each other. This case study makes that mistake once, and catches it.

The plan of the study. Everything is chosen on dev; holdout is used once, at the end.

Text description of the diagram

320 items with 834 human votes are split by question into dev (155 items) and holdout (165 items). On dev: 1, the ceiling (person against person); 2, the two judges, each asked in both orders; 3, the comparison with the majority and with each person; 4, where the judge fails (order, length, category, reference); 5, the policy. Then 6: on holdout, the policy is applied once.

The ceiling: how often do people agree?

The notebook takes every pair of two people who voted on the same dev item. It measures their agreement and kappa. The interval is a bootstrap over items, with 2,000 resamples, because the votes on one item are not independent.

per_item = [human_pairs(it) for it in dev]
print("pairs of people:", sum(len(p) for p in per_item))
print("agreement:", interval(per_item, 0))
print("kappa:    ", interval(per_item, 1))
print("agreement without ties:", interval([[p for p in ps if "tie" not in p] for ps in per_item], 0))

The output:

pairs of people: 376
agreement: (0.638, 0.563, 0.711)
kappa:     (0.451, 0.337, 0.554)
agreement without ties: (0.834, 0.759, 0.898)

What this tells us. Two people agree on 64% of the votes, with kappa 0.45. Most disagreements are one “tie” against one winner. Without ties, they agree on 83%. That is the “over 80%” of the paper, which also leaves out the ties. Daniel quoted a number without its condition. With ties, the ceiling is about two votes in three, and the interval runs from 56% to 71%: 155 items do not pin it down closely.

The people also split evenly on 33 of the 155 dev items, for example one A and one B. Those items have no majority, and they are left out of every comparison with the majority. They are not hard only for a judge; they are hard.

Baselines

Two baselines frame the judges.

• Pick the longer answer. No model and no reading: choose the answer with more characters. A judge that cannot beat this adds nothing.
• GPT-4 in 2023. The dataset includes the verdicts of the paper’s own judge, asked in both orders. It is the published result that Daniel quoted.

The judges, asked twice

The course asked two model judges on Azure: chat-small (gpt-6-luna) and chat-strong (gpt-6.1-sol). Both got the MT-Bench paper’s own pairwise prompt, word for word. So the only change from the paper is the model. It says, in part: “Please act as an impartial judge … Avoid any position biases … Do not allow the length of the responses to influence your evaluation.”

Each item was asked twice, as in Model-based judging. The first time, answer 1 was “Assistant A”; the second time, answer 2 was. For reasoning, maths and coding, the course also asked the paper’s second prompt, which adds the reference answer. That makes 1,760 recorded calls, 480 of them with the reference, for about US$2.70 in all. Models change fast, and a new version would vote differently on some items. These verdicts are a snapshot of 9 October 2026; the method lasts.

Here is one real dev item, worked by hand.

One dev item, and the recorded chat-strong verdicts in both orders. The judge found the same bugs both times, and still chose the answer that it read second.

Text description of the diagram

Coding question 129 asks for a function that finds the kth smallest element of two sorted lists in linear time. Answer 1 (claude-v1, 1,007 characters) gives code with bugs and an explanation. Answer 2 (llama-13b, 92 characters) is one sentence with no code. Both people chose answer 1.

The chat-strong judge saw answer 1 first and chose answer 2: “B is preferable to a misleading, broken implementation.” It saw answer 2 first and chose answer 1: “B is the more useful response, despite needing correctness fixes.” The second place won both times, so the verdict is inconsistent. It counts as a tie, and the policy sends it to a person.

Both reasons name the same bugs. The judge’s reading did not change; its weighing did, and the place decided it. Asked once, it would give one confident verdict.

The notebook turns each recorded letter into the fixed direction, then combines the two orders. The same verdict twice is the verdict. Two different verdicts are inconsistent, and count as a tie, as in the paper.

for judge in ["chat-small", "chat-strong", "gpt-4 (2023)"]:
    print(f"{judge:13}", dict(Counter(verdicts[(judge, "pair-v2")].get(it["item_id"], "missing") for it in dev)))

The output:

chat-small    {'B': 67, 'A': 65, 'inconsistent': 12, 'tie': 11}
chat-strong   {'B': 68, 'A': 62, 'inconsistent': 8, 'tie': 17}
gpt-4 (2023)  {'B': 59, 'A': 49, 'inconsistent': 27, 'tie': 20}

Order bias: the share of the 155 dev items where swapping the two answers changed the judge's verdict. Today's judges are much steadier than GPT-4 was in 2023, but not free of it.

Text description of the chart

A column chart. chat-small: 12 of 155 items, 7.7%. chat-strong: 8 of 155 items, 5.2%. gpt-4 (2023): 27 of 155 items, 17.4%.

What this tells us. The order changed chat-strong’s verdict on 8 of 155 dev items, and chat-small’s on 12. GPT-4 in 2023 changed on 27. So order bias is much smaller than in the paper, but it has not gone. When chat-strong’s verdict changed, the answer shown second won both times on 5 items, and the first on 1.

A fair comparison

Now the main table: every judge on the same dev items, with the same two measures, with intervals. “With the majority” uses the 122 dev items that have one. “With each person” puts every human vote against the judge’s verdict on the same item: the same kind of pair as the ceiling.

for judge in ["longer answer", "chat-small", "chat-strong", "gpt-4 (2023)"]:
    v = verdicts[(judge, "pair-v2")]
    print(f"{judge:15}{str(interval(vs_majority(v, dev), 0)):32}{str(interval(vs_majority(v, dev), 1)):24}"
          f"{str(interval(vs_people(v, dev), 0)):30}{interval(vs_people(v, dev), 1)}")

The output (the notebook prints the heading and the ceiling first):

               with the majority: agreement    kappa                   with each person: agreement   kappa
person (ceiling)                                                        (0.638, 0.563, 0.711)         (0.451, 0.337, 0.554)
longer answer  (0.582, 0.5, 0.672)             (0.313, 0.196, 0.434)   (0.537, 0.473, 0.605)         (0.252, 0.165, 0.343)
chat-small     (0.73, 0.642, 0.803)            (0.573, 0.438, 0.684)   (0.639, 0.572, 0.699)         (0.439, 0.338, 0.525)
chat-strong    (0.738, 0.653, 0.811)           (0.588, 0.454, 0.701)   (0.646, 0.579, 0.704)         (0.449, 0.349, 0.537)
gpt-4 (2023)   (0.754, 0.675, 0.828)           (0.626, 0.507, 0.734)   (0.651, 0.59, 0.708)          (0.472, 0.382, 0.555)

Agreement with one person's vote, on the dev items, with 95% bootstrap intervals over items. The grey row is the ceiling: one person against another. The orange row is the baseline that picks the longer answer.

Text description of the chart

One row each, with its interval. person vs person: 63.8% (56.3 to 71.1). longer answer: 53.7% (47.3 to 60.5). chat-small: 63.9% (57.2 to 69.9). chat-strong: 64.6% (57.9 to 70.4). gpt-4 (2023): 65.1% (59 to 70.8).

What this tells us. Read it like with like. Against one person, both Azure judges agree as often as another person does: 65% and 64%, against a ceiling of 64%. GPT-4 of 2023 is at the same level, so the paper’s main result holds with today’s models too. The longer-answer rule is clearly lower, 54%: the judges read more than length. Against the majority, every number is higher, about 74%. That is what makes the majority column tempting, and wrong, to compare with the ceiling.

One more surprise. chat-small agrees with people as often as chat-strong: the paired difference with the majority is +0.008, with an interval from -0.033 to 0.057. On Larkfield’s tickets, in Model-based judging, chat-small was the unreliable judge. A judge’s quality belongs to a task, not to a model.

Where the judge fails

Length

The prompt says: “Do not allow the length of the responses to influence your evaluation.” The notebook checks it. For every decisive verdict (A or B), is the chosen answer the longer one? It does the same for the human majority, as a baseline. People may prefer longer answers too, and a longer answer is often more complete.

people (majority)  picks the longer answer in 75% of 95 decisive verdicts
chat-small         picks the longer answer in 63% of 132 decisive verdicts
chat-strong        picks the longer answer in 64% of 130 decisive verdicts
gpt-4 (2023)       picks the longer answer in 76% of 108 decisive verdicts

What this tells us. Longer answers usually win, with people too. Today’s judges pick the longer answer less often than the people do. So there is no sign of a length bias beyond the people’s own. And where chat-strong overrules a clear majority, it chose the longer answer only 4 times in 14. On Larkfield’s tickets, in Model-based judging, the judge did prefer the longer replies: check it for every task.

Category

An overall number blends the categories. So split it, as in Uncertainty and slices, with the counts and the intervals next to every number. Each category has only 14 to 24 dev items.

By category, on dev: one person against another (grey) and chat-strong against one person (blue; orange where the policy sends the category to a person), with 95% bootstrap intervals. Each category has 14 to 24 items, so the intervals are wide.

Text description of the chart

For each category, two rows. writing (14 items): person vs person 58.3% (30.4 to 82.9), chat-strong vs person 54.1% (33.3 to 75). roleplay (22 items): person vs person 61.5% (44.2 to 79.1), chat-strong vs person 56.5% (38.2 to 75). reasoning (19 items): person vs person 60.9% (42.6 to 82), chat-strong vs person 72% (55.6 to 87.8). math (18 items): person vs person 56.2% (38.1 to 73.9), chat-strong vs person 62% (45.8 to 76.6). coding (22 items): person vs person 81.6% (67.7 to 97), chat-strong vs person 66% (46.8 to 85.4). extraction (24 items): person vs person 72.4% (52 to 93.2), chat-strong vs person 75.8% (60.6 to 89.8). stem (16 items): person vs person 61.8% (37.2 to 85), chat-strong vs person 67.5% (50 to 83.7). humanities (20 items): person vs person 58.8% (39.4 to 81.6), chat-strong vs person 60.4% (43.9 to 77.4).

What this tells us. Coding stands out. People agree with each other there more than anywhere else, 81.6%. Code is either right or wrong, and the people saw it. The judge reaches only 66%. Question 129 above is one of those items. In writing and role-play, the judge is a few points below people, and people themselves agree least. In reasoning, maths, science and extraction, the judge is at or above the people. Every interval is wide, so treat these as signals, not proof.

With and without a reference

For reasoning, maths and coding, a judge that reads two answers may make the same mistake as the answers. The paper’s fix is a reference answer in the prompt. The notebook compares the same judge on the same dev items, with and without it, with a paired bootstrap of the difference.

chat-small   49 items | without 0.735 | with the reference 0.735 | difference +0.000 [-0.082, +0.082]
chat-strong  50 items | without 0.780 | with the reference 0.760 | difference -0.020 [-0.060, +0.000]

What this tells us. The reference does not help these judges: the difference is 0 or slightly negative, and both intervals include 0. Today’s judges see the mistakes of the 2023 answers without help, and a reference costs a person’s work to write and check. Here it is not worth it.

Refusals and unreadable verdicts

In Model-based judging, continued, Azure’s content filter refused every judgment of Larkfield’s injection ticket. Here, the filter refused none of the calls: these questions contain no instructions aimed at the judge. On dev, one chat-small judgment with the reference explained its choice but never wrote the [[A]] format; one more did the same on holdout. The policy treats a missing verdict like a refusal: a person decides. Count these outcomes on their own, every run.

Common mistake

Asking only once

Symptom: a team asks its judge once per pair, with the answers in a fixed order. The judge looks decisive, and its agreement with people looks fine.

Cause: one order hides every item where the place, not the content, decided. Asked once, chat-strong gives no inconsistent verdict at all, and its agreement with the majority is 73%, against 74% with both orders. The average hardly moves. What changes is what you know: the 8 items where the place decided look exactly as certain as the others.

Fix: ask in both orders, always. Count the items where the verdict changed, treat them as ties, and send them to a person. The second call doubles the cost of the judge, which is still small next to a person’s time.

The judge policy

The policy turns the dev numbers into rules. Its first version was written down before any judgment was analysed; the run log has the time. Its rule 3 compared the judge’s agreement with the majority with the agreement between two single people. That is the mistake that the ceiling section warns about. With that rule, the judge would decide every category alone, coding included. The dev numbers showed the flaw. The rule was fixed on dev, before any holdout number was computed. Version 2 compares like with like:

1. The judge is the Azure judge that agrees with the human majority more often on dev: chat-strong. The lead over chat-small is within the noise, so a team that cares more about cost could choose chat-small and calibrate it the same way.
2. The prompt: for reasoning, maths and coding, use the reference prompt if, on dev, it agrees with the majority at least as often. That is the case for math and reasoning, where it was equal.
3. A category goes to a person when the judge agrees with one person more than 0.05 less often than one person agrees with another, in that category. On dev, that is coding.
4. Every item whose verdict changed with the order goes to a person. So does every refused or unreadable judgment.
5. A person reads a random sample of the judge’s decisions every week. The notebook cannot test this rule, but it is how the team will notice a change.

The notebook computes rules 1 to 3 from dev:

writing     pair-v2       judge vs person 0.541 | person vs person 0.583 -> judge alone
roleplay    pair-v2       judge vs person 0.565 | person vs person 0.615 -> judge alone
reasoning   pair-math-v1  judge vs person 0.740 | person vs person 0.609 -> judge alone
math        pair-math-v1  judge vs person 0.640 | person vs person 0.562 -> judge alone
coding      pair-v2       judge vs person 0.660 | person vs person 0.816 -> a person
extraction  pair-v2       judge vs person 0.758 | person vs person 0.724 -> judge alone
stem        pair-v2       judge vs person 0.675 | person vs person 0.618 -> judge alone
humanities  pair-v2       judge vs person 0.604 | person vs person 0.588 -> judge alone

What this tells us. Only coding goes to a person by category. Writing and role-play pass, but only just: role-play is exactly 0.05 below. With a margin of 0, they would go to a person too. The notebook’s last task asks you to make that change and measure what it costs.

The test, once

The holdout items are 40 other questions. Nothing in the policy was chosen on them. The notebook applies the policy once and measures two things: how much work the judge takes over, and how well it does there.

decisions = {it["item_id"]: route(it, policy) for it in holdout}
print("decisions:", dict(Counter(d for d, _ in decisions.values())))
alone = [it for it in holdout if decisions[it["item_id"]][0] == "judge"]
decided = {i: v for i, (d, v) in decisions.items()}
print(f"the judge decides alone on {len(alone)} of {len(holdout)} items ({len(alone) / len(holdout):.0%})")
print("there, agreement with the majority:", interval(vs_majority(decided, alone)), "| kappa", interval(vs_majority(decided, alone), 1))
print("there, person vs person:           ", interval([human_pairs(it) for it in alone]))
print("judge on every holdout item, no policy:", interval(vs_majority(verdicts[(policy["judge"], "pair-v2")], holdout)))

The output:

decisions: {'judge': 139, 'person: the order changed the verdict': 9, 'person: category': 17}
the judge decides alone on 139 of 165 items (84%)
there, agreement with the majority: (0.766, 0.688, 0.845) | kappa (0.62, 0.499, 0.741)
there, person vs person:            (0.657, 0.568, 0.745)
judge on every holdout item, no policy: (0.758, 0.68, 0.832)

What this tells us. On new questions, the judge takes over 84% of the work. People still read 26 items: 17 coding items and 9 items where the order changed the verdict. Where the judge decides, it agrees with one person 67% (61 to 74), and people agree with each other 66% (57 to 75) on the same items. The policy held: the judge is as good as a second person where it decides alone.

One warning from holdout. By category, extraction did not repeat its dev result: there the judge agrees with one person 43%, and people with each other 54%. With about 20 items per category, a category’s rank can change from one sample to the next. Coding stayed below the people (53% against 58%). That is why rule 5, the weekly sample, is part of the policy.

A new comparison, end to end

Next week, a new pair of answers arrives. The policy decides who judges it. Here, one holdout item plays the new comparison:

new = next(it for it in holdout if it["item_id"] == "mtb-q116-gpt-4-vs-llama-13b")
decision, verdict = route(new, policy)

The output:

math | x+y = 4z, x*y = 4z^2, express x-y in z
judge in order 12: A | in order 21: A
decision: judge | verdict: A | the people said: ['A', 'A', 'A', 'A']

What this tells us. The judge gave the same verdict in both orders: answer 1, from GPT-4, is better. Maths is a category that the judge may decide, so the decision is judge, with no person. All four people who voted on it agree. A coding comparison, or any verdict that changes with the order, would go to the reviewers’ queue instead.

Recommendations

Ines writes to the team in their words, not in model words:

• Let the judge decide most weekly comparisons. On new questions, it decided 84% of them alone, and there it agreed with each reviewer as often as the reviewers agree with each other.
• Keep people for coding. On code, reviewers agree with each other more than anywhere else, and the judge does not reach them.
• Ask the judge twice, in both orders, every time. Send every verdict that changes with the order to a person. It was about one comparison in twenty.
• Read a sample every week. A person reads about 10 of the judge’s decisions each week, chosen at random. One category already moved between dev and holdout.
• Cost. One chat-strong judgment cost about US$0.003, so a week of 160 comparisons in both orders costs about US$1. chat-small agreed with people as well, for less than a tenth of the cost; it is worth its own calibration.

And what not to use it for:

• Not for a release decision on its own. A judge’s preference between two answers is one input. Blockers and critical cases are decided by release rules, as in Trade-offs and regressions.
• Not for other kinds of questions without a new calibration: other languages, other topics, longer conversations, or the team’s own users.
• Not for scoring people. It compares two machine answers. It is not a tool to grade students, staff or applicants.

Limits

• 2023 answers. The six chatbots are old, and their mistakes are easy to see. Pairs of strong, modern answers are harder to tell apart, for people and for judges.
• One group of raters. The voters were mostly graduate students, paid per question. They are not the team’s users. Their taste is one taste.
• The votes may not be fully independent of GPT-4. When a vote differed from GPT-4, the collection tool showed GPT-4’s judgment and asked whether it was reasonable. The paper says that people changed their choice in 34% of those cases. The data does not say which votes changed.
• Small categories. About 20 items per category on dev, and the same on holdout. Category decisions are rough, as extraction showed.
• One snapshot. Two Azure models on 9 October 2026, one prompt, no temperature setting. A new model version needs a new calibration.
• The first turn only. The second turn of MT-Bench asks the judge to read a whole conversation. That is not tested here.

Responsible use

• Say who rated what. The human votes are real votes by the MT-Bench study’s paid judges, mostly graduate students, and some of the paper’s authors. The model verdicts are model outputs. Never put them in one column called “ratings”, and never call a model’s verdict a human label, even when it agrees with people.
• A majority overrules a minority. On many items, the people split. A judge that agrees with the majority still disagrees with someone. Keep the split items visible, and keep a person in the loop where people disagree most.
• Recheck after every change. A new model version, a new prompt or new kinds of questions move the numbers. Give the judge prompt a version, as in Model-based judging, continued.

Make it a portfolio piece

A calibration study shows an employer that you do not trust a number until you have measured it. Make it yours with one change. For example, use another judge model, such as a free local one with Ollama. Or use another dataset with several human ratings per item, or your own team’s questions with two colleagues as raters. Then make it public, and easy to read in five minutes.

What to put in the repository:

• README.md, the page that people read first (template below).
• The notebook, run from top to bottom, with its outputs.
• JUDGE_POLICY.md: the judge, the prompt and its version, the rules, the numbers behind each rule, and when to check again.
• The data source and licence, with the attribution text from the box above. CC BY 4.0 lets you share the sample with credit; say what you changed.
• If you record your own judgments: the recording script, the exact requests, and a run log with the model version, the date and the cost.

A README template:

# Can a model judge replace people for pairwise reviews?

## Question
Choose the better of two chatbot answers: when may a model judge decide alone?

## Data and licence
MT-Bench human judgments (Zheng et al., 2023), CC BY 4.0: 320 items, 834 votes by
people (mostly graduate students), 8 categories. Split by question: dev 155, holdout 165.

## Method
Ceiling = person vs person. Judges: chat-small and chat-strong (Azure), the paper's
prompt, both orders; GPT-4 (2023) and "pick the longer answer" as baselines.
Agreement and kappa with the majority and with each person; bootstrap intervals over items.

## Results (dev)
Person vs person 0.64 (kappa 0.45). chat-strong vs person 0.65, chat-small 0.64,
longer answer 0.54. Order changed the verdict on 5% (chat-strong) vs 17% (GPT-4, 2023).
No gain from a reference answer. Coding: judge 0.66 vs people 0.82.

## Policy, tested once on holdout
The judge decides alone except coding and order-dependent verdicts: 139 of 165
items (84%); there, judge vs person 0.67, person vs person 0.66.

## Limits
2023 answers; one group of raters; about 20 items per category; one snapshot.

How to describe it in an interview, in two sentences: “I measured two model judges against 834 real human votes. They agree with a person as often as two people do, except on code. My judge policy, tested once on new questions, let the judge take 84% of the reviews at human-level agreement.”

Go further:

• Ask each judge twice in the same order. How often does it change its verdict without any swap?
• Compare the judges on the items where people split evenly. Predict first: what should a good judge say there?
• Replace the 0.05 margin with the lower end of the interval of the difference between the judge and people. How many categories change?
• Run the same study on the second turn of MT-Bench, where the judge must read a whole conversation.

Knowledge check

Check your understanding

Daniel says: the paper shows that GPT-4 agrees with people over 80% of the time. On this sample, two people agree on only about two votes in three. Who is right, and what does each number measure?

Show the answer

Both numbers are real, and they measure different things. The “over 80%” counts only the votes where neither side said “tie”. With ties, people agree on about two votes in three, because many disagreements are one “tie” against one winner. A number about agreement is only useful with its conditions: with or without ties, against one person or against the majority, and with its interval. The fair ceiling for a judge is one person against another person, measured the same way as the judge.

Check your understanding

The first version of the policy compared the judge's agreement with the majority with the agreement between two people. Why is that unfair, and in whose favour?

Show the answer

A majority is easier to agree with than one person. It smooths out each person’s own taste, and the items without a majority are left out. So the judge’s number is measured on easier ground than the people’s number, and the rule favours the judge. In version 1, the judge would decide every category alone, coding included. A fair rule compares the same kind of pair: the judge against one person, and one person against another person.

Check your understanding

On a coding item, the judge chose the answer that it read second, in both orders, and gave a confident reason each time. What does the policy do with such an item, and why not keep the first verdict?

Show the answer

It sends the item to a person. A verdict that changes with the order depends on the place, not on the answers, so neither verdict says which answer is better. Keeping the first verdict would turn a coin toss into a confident label. For agreement numbers, such an item counts as a tie, as in the MT-Bench paper.

Recap

• Two people agree on about two votes in three (kappa 0.45): the ceiling. The paper’s “over 80%” leaves out the ties.
• Compare like with like: the judge against one person, with person against person. Agreement with the majority is higher, and it is not the ceiling.
• Both Azure judges reached the ceiling overall, as GPT-4 did in 2023. The cheaper judge was as good as the stronger one on this task.
• Ask a pairwise judge in both orders. Here the order still changed about one verdict in twenty; asked once, those verdicts look certain.
• The judge policy came from dev: people keep coding and every order-dependent verdict. Tested once on holdout, the judge decided 84% of items at human-level agreement.

Next, a different kind of rating: experts score summaries on four qualities, and you find which automatic measure agrees with them, and on which quality.

Next: Evaluate summaries against expert ratings