Calibrate reviewers

What you will be able to do

Find out whether two reviewers really read the rubric the same way. You rate a fixed sample yourself, compare your scores with the course author’s, and measure the agreement with a number that discounts luck.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Larkfield’s rubric and its ladder, from Write a usable rubric.
• Shares and proportions, from Essential Mathematics and Statistics for AI.

What you need

• Your project at the end of the last lesson, with harness/prompts/rubric.md: python -m pytest gives 18 passed.
• About 25 minutes of quiet time to rate 30 replies. You can stop and start again.

You need no account and no key.

Two teachers before the marking starts

In the last lesson, two teachers got a mark scheme. Before they mark 200 exams, they each mark the same ten, alone, and then compare. Where their marks differ, they talk: one missed a wrong date, or the scheme can be read two ways. This meeting is called moderation. It is the cheapest moment to find a problem, before 200 marks depend on it.

But how do they know that their agreement is good? Suppose both teachers give “pass” to nine scripts of ten. They agree on 9 of 10. That sounds good, until you notice that nine of the ten scripts were easy passes. Two teachers who said “pass” without reading would also agree on 9 of 10.

The problem: agreement that luck can give

Larkfield has the same problem. Most replies are fine, so most scores are 4 or 5. Two reviewers who both give “4” to almost everything agree on many replies, by luck. A simple share of agreement cannot tell careful readers from careless ones.

Here is the extreme case, a made-up example from the project’s tests. Ten replies, and one of them is bad. Reviewer 1 finds it. Reviewer 2 says “not bad” to all ten without reading. They agree on 9 of 10, or 90%. Yet reviewer 2 added nothing: the 90% came from the mix of replies, not from reading.

So you need two things. You need a fixed sample that every reviewer rates alone, before they talk. And you need a measure that removes the agreement that luck alone would give.

Who rated the reference ratings

The course gives you a first reviewer’s scores for the sample, the reference ratings. You need to know who made them.

The course author wrote the labels and the reference ratings by applying Grace’s written policy and rubric, and scripts checked them. In this module, you are the second rater.

So the reference ratings are not the truth. They are one person’s careful reading. The course author rated with the rubric before any model judge ran, and saw the replies without the system names. The course author had read some of these replies before, during the label review of Module 2. When your score differs from the reference, either reading can be the better one. That is what the discussion at the end of the lesson is for.

Rate the sample yourself

The calibration sample is 30 replies. It has 15 development tickets, two from each slice and three more critical cases, each answered by the baseline and by the candidate. The replies are blind: shuffled, numbered R01 to R30, with no system name. Rate them before you look at anything else.

1. In the project folder, with the environment active, set the variable again if you opened a new terminal. Then copy the sample, the public data of the end of this lesson, the agreement code, the new command line and its test:

   macOS

   S=../nextia-learning-labs/C11/snapshots/end-of-m04
   cp -R $S/data/calibration $S/data/public data/
   cp $S/harness/agreement.py $S/harness/__main__.py harness/
   cp $S/tests/test_agreement.py tests/

   Windows

   $S = "..\nextia-learning-labs\C11\snapshots\end-of-m04"
   Copy-Item -Recurse $S\data\calibration, $S\data\public data\
   Copy-Item $S\harness\agreement.py, $S\harness\__main__.py harness\
   Copy-Item $S\tests\test_agreement.py tests\

   Linux

   S=../nextia-learning-labs/C11/snapshots/end-of-m04
   cp -R $S/data/calibration $S/data/public data/
   cp $S/harness/agreement.py $S/harness/__main__.py harness/
   cp $S/tests/test_agreement.py tests/

   The folder data/calibration/ has a README.md and four data files. You need only items.jsonl now. Do not open reference_ratings.csv or key.json yet: one look, and your ratings are no longer your own.

2. Create a file rate.py in the project folder with this content. It shows one item at a time and saves your score after each one.

   rate.py

   """Rate the 30 calibration replies with the rubric, one at a time: python rate.py
   
   Your scores go to my_ratings.csv after each reply, so you can stop and start again later.
   """
   import csv
   import json
   from pathlib import Path
   
   ITEMS = Path("data/calibration/items.jsonl")
   SHEET = Path("my_ratings.csv")
   
   items = [json.loads(line) for line in ITEMS.read_text(encoding="utf-8").splitlines()]
   done = {}
   if SHEET.exists():
       with SHEET.open(encoding="utf-8", newline="") as f:
           done = {r["item"]: r for r in csv.DictReader(f) if r["score"].strip()}
   
   for item in items:
       if item["item"] in done:
           continue
       print(f"\n===== {item['item']} ({len(done) + 1} of {len(items)})")
       print(f"Ticket: {item['ticket']}")
       if item["attachments"]:
           print(f"Attachments: {item['attachments']}")
       print(f"Reference notes:\n{item['notes']}")
       print(f"Reply: {item['reply'] or '(no reply)'}")
       score = ""
       while score not in {"1", "2", "3", "4", "5"}:
           score = input("Score 1-5 (or q to stop): ").strip()
           if score == "q":
               raise SystemExit(f"Stopped. {len(done)} replies rated in {SHEET}.")
       note = input("Why, in a few words: ").strip()
       done[item["item"]] = {"item": item["item"], "score": score, "note": note}
       with SHEET.open("w", encoding="utf-8", newline="") as f:
           writer = csv.DictWriter(f, fieldnames=["item", "score", "note"])
           writer.writeheader()
           writer.writerows(done[i["item"]] for i in items if i["item"] in done)
   
   print(f"\nDone: all {len(items)} replies rated in {SHEET}.")

3. Open harness/prompts/rubric.md next to the terminal. Then run:

   python rate.py

   The first item:

   ===== R01 (1 of 30)
   Ticket: Thanks so much for the quick help with my bathroom cabinet delivery last week, everything arrived in one piece and it looks great. I really love the fittings. The only thing is I keep getting marketing emails about sales and new ranges every day and it's getting to be a bit much. Could you please take me off your marketing emails?
   Reference notes:
   - Team: account
   - A person must handle the ticket: no
   - The next step a good reply contains: no specific step from the policy; the reply must not invent one
   - The reply must not: say that a refund, replacement or change was made; ask for a password or a card number
   - Reply language: English
   Reply: I’m glad your bathroom cabinet arrived safely and that you love the fittings; I’m sorry the daily marketing emails have become too much. Our account team handles requests to opt out of marketing emails and will take care of yours. The policy doesn’t specify a processing time, so I can’t give you a reliable timeframe. You don’t need to send any further details for this request.
   Score 1-5 (or q to stop):

4. For each reply, climb the ladder, type the score, and write a few words on what decided it. Your notes matter later. Type q to stop; python rate.py continues where you stopped.

5. At the end, the script prints Done: all 30 replies rated in my_ratings.csv.

You can also rate in a spreadsheet: copy data/calibration/sheet_blank.csv to my_ratings.csv and fill in the score column. Keep the columns item, score and note.

Stop here until you have rated all 30. The rest of the lesson shows the reference ratings.

The idea: agreement above chance

A careless reviewer can match a careful one by luck. So compare the agreement you observed with the agreement that luck alone would give, and count only the part above luck. That measure is Cohen’s kappa, written with the Greek letter κ. The psychologist Jacob Cohen proposed it in 1960, for two people who sort the same items into the same categories.

Here is a tiny example, with every value visible: two raters, ten replies, and one question, “bad (score 1–2) or ok?”.

Cohen's kappa, worked by hand. The raters agree on 70% of the replies, but luck alone would give 62%, so kappa is only 0.21.

Text description of the diagram

The table counts the ten replies. Both raters said ok on 6 and bad on 1. On 3 they differ: rater 1 said ok and rater 2 bad on 2, and the other way round on 1. So rater 1 said ok 8 times, and rater 2 said ok 7 times.

1. Observed agreement: (6 + 1) / 10 = 0.70.
2. Chance agreement: suppose that each rater kept their own mix but chose at random. Both would say ok 0.8 × 0.7 = 0.56 of the time, and both bad 0.2 × 0.3 = 0.06. Together: 0.62.
3. Kappa: (0.70 − 0.62) / (1 − 0.62) = 0.08 / 0.38 = 0.21.

Read the three steps again in words. The percent agreement, 0.70, is the share of items on which the two raters gave the same label. The chance agreement, 0.62, is what two raters with these habits would reach by luck. Rater 1 says “ok” 80% of the time, so luck alone makes them match often. Kappa asks: of the room above luck, 1 − 0.62 = 0.38, how much did the raters fill? They filled 0.08, about a fifth.

So kappa is 1 when the raters always agree, and 0 when they agree only as often as luck gives. Below 0, they agree less often than luck. The formula, last:

κ = (observed agreement − chance agreement) / (1 − chance agreement)

The project computes it in harness/agreement.py, and its test is this worked example. Run it:

python -m pytest tests/test_agreement.py -v

The last lines:

tests/test_agreement.py::test_worked_example PASSED                      [ 33%]
tests/test_agreement.py::test_agreement_by_chance_alone_gives_kappa_zero PASSED [ 66%]
tests/test_agreement.py::test_within_one_point PASSED                    [100%]

============================== 3 passed in 0.00s ===============================

The second test is the careless reviewer of the problem section: 90% agreement, kappa 0.

Compare two score sheets

The new command agree compares two sheets of scores from 1 to 5. Try it first on the worked example, written as scores, so that you know the output before you see your own. Save these two files in a folder notes/:

notes/example_a.csv

item,score
E01,5
E02,4
E03,5
E04,4
E05,5
E06,3
E07,2
E08,4
E09,3
E10,2

notes/example_b.csv

item,score
E01,5
E02,4
E03,4
E04,5
E05,5
E06,4
E07,2
E08,2
E09,1
E10,4

Run:

python -m harness agree notes/example_a.csv notes/example_b.csv

The output:

10 replies scored by both
exact agreement 0.40 | within one point 0.70 | Cohen's kappa 0.20
bad (1-2) or not: agreement 0.70, kappa 0.21
rows: first sheet, columns: second sheet
  2: 2->1  4->1
  3: 1->1  4->1
  4: 2->1  4->1  5->1
  5: 4->1  5->2
discuss first (2 or more points apart): E08, E09, E10

Read it line by line.

• exact agreement is the share of items with the same score, here 4 of 10. within one point forgives a difference of one, 7 of 10. Cohen’s kappa is kappa on the five scores.
• bad (1-2) or not turns each score into bad or ok, and gives the worked example: 0.70 and 0.21. For a release, this is the question that matters most.
• rows and columns show where the sheets part. 4: 2->1 means: once, the first sheet said 4 and the second said 2.
• discuss first lists the items 2 or more points apart.

Now compare your own sheet with the reference ratings:

python -m harness agree data/calibration/reference_ratings.csv my_ratings.csv

Your numbers are your own, so no output is shown here. Write down your exact agreement, your kappa on five scores and your kappa on “bad or not”.

Read your result

What is a good kappa? There is no single line. A common rough guide calls 0.4 to 0.6 moderate and above 0.8 almost perfect. Treat it as a guide, not a law. Kappa depends on the number of items and on how the scores are spread.

The reference ratings are spread like this:

• Bad (1–2)
• Not bad (3–5)

The 30 reference ratings by score. Most are 4 or 5; only 2 replies are bad.

Text description of the chart

A column chart of the reference ratings: score 1, 0 replies; score 2, 2; score 3, 5; score 4, 12; score 5, 11. The two bad replies (score 2) are shown in orange.

Two things follow from this shape. First, 23 of the 30 scores are 4 or 5, so chance agreement on the exact score is high. Many matches would happen by luck, and kappa discounts them. Second, only 2 replies are bad. Your kappa on “bad or not” rests on those two and on the replies you called bad. If you agree on all 30, it is 1.00. Miss one of the two bad replies, and it falls to 0.65. With 30 items, read kappa as a rough signal, and read the items behind it.

How do people do on such tasks? The MT-Bench human judgments are a public dataset in which people compared two chatbot answers to the same question. They chose A, B or a tie. The project has a fixed sample of 60 questions, each judged by 2 or more people, 152 votes in all. Run:

python -m harness public

The output:

MT-Bench human judgments (CC BY 4.0), fixed sample: 60 items
human vs human         138 pairs | agreement 0.623 | kappa 0.431
human vs GPT-4 judge   152 pairs | agreement 0.697 | kappa 0.544

Two human judges gave the same vote on 62% of the pairs, with kappa 0.43. On 22 of the 60 questions, the people did not all vote the same. A GPT-4 judge agreed with the people more often than they agreed with each other. That does not prove that the model reads better. The two numbers are not measured the same way. The model’s changed verdicts count as ties. The pairs of people change from question to question, so each kappa mixes many rater pairs.

Where kappa values sit. Two people on a real judging task reached 0.43. Place your own result on the line.

Text description of the diagram

A number line for kappa. At 0, the raters agree no more often than luck gives; below 0, less often. At 1, they always agree. The worked example sits at 0.21. In the MT-Bench sample, two people reached 0.43, and people with a GPT-4 judge 0.54. Place your own kappa with the reference ratings on the line.

Data: MT-Bench human judgments, LMSYS (Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685), huggingface.co/datasets/lmsys/mt_bench_human_judgments (revision f7d2896d), licensed under CC BY 4.0. Changes: the course selected 60 first-turn items with votes from 2 or more human judges, kept some fields, and wrote every label in a fixed model order.

Discuss the disagreements

A number tells you how much you disagree; the items tell you why. Start with the discuss first list. For each item, read the reply, your note and the reference note in reference_ratings.csv. Then decide which of three causes it is.

1. One reader missed something. A sentence in the reply, or a line in the notes. The scores move; the rubric stays.
2. The rubric can be read two ways. Both readings are defensible. Add a sentence and an example to the review guide, with a new version.
3. A real judgement call. The rubric is clear, and you still weigh things differently. Write down both views. These items show the limit of agreement that you can expect.

Here is a real case, R13. A customer in Seattle asks whether Larkfield delivers to the United States. The notes say: no specific step from the policy, and the reply must not invent one. The reply is “Our delivery information covers standard delivery in Canada but does not specify delivery to the United States. I can’t confirm whether we deliver to Seattle.”

The reference rating is 3, with the note “correct but no next step”. A reader who follows the notes can give 5: the policy has no step, and the reply invents none. Who is right? Anchor 3 says “without the main next step”. But this ticket has no next step in the policy, so anchor 3 can be read two ways. That is cause 2. The fix is a line in the review guide: “When the notes give no step, a clear ‘we cannot confirm’ with no invented step is a 4 or a 5.”

The last file, key.json, says which case and which system each item comes from. Open it only after the discussion. Then you can see whether you scored one system higher, and on which tickets.

When to calibrate, and what it costs

Calibrate before the first review round, when a new reviewer joins, and after any change to the rubric. Also calibrate when the replies change a lot, as with a new prompt. Rate the sample alone, compare, discuss, then fix the review guide. Do not calibrate on the cases that you will report: a sample that the reviewers discussed is no longer a blind test.

The cost is people’s time. Thirty replies take each reviewer about 25 minutes, and the discussion about as long again. That is cheap next to a review of 200 replies with a rubric that two people read differently. The limit is size: 30 items give a rough kappa. Use them to find the problems, not to prove that the agreement is high.

Guided practice: kappa by hand

Two reviewers rated 20 replies as bad or ok. Both said ok on 14 and bad on 2. Reviewer 1 said ok and reviewer 2 bad on 3. On 1 reply, reviewer 1 said bad and reviewer 2 ok.

1. Draw the 2 × 2 table, with the totals for each reviewer.
2. Compute the observed agreement.
3. Compute the chance agreement.
4. Compute kappa.

Show the answers

1. Reviewer 1 said ok 14 + 3 = 17 times and bad 3 times. Reviewer 2 said ok 14 + 1 = 15 times and bad 5 times.
2. Observed: (14 + 2) / 20 = 0.80.
3. Chance: both ok 0.85 × 0.75 = 0.6375; both bad 0.15 × 0.25 = 0.0375; together 0.675.
4. Kappa: (0.80 − 0.675) / (1 − 0.675) = 0.125 / 0.325 = 0.38.

80% agreement, but kappa 0.38: most of the agreement is on “ok”, where luck already matches the two reviewers often.

Your turn: your three biggest disagreements

Take the first three items of your discuss first list. If it has fewer than three, take the items where you differ by one point on “bad or not”.

1. For each item, write the item, both scores, both notes and the cause: missed, rubric, or judgement call.
2. For each “rubric” cause, write the sentence that you would add to the review guide.
3. Save it as notes/calibration.md. The last lesson of this module uses it.

Show an example answer

“R13: reference 3, me 5. Reference: ‘correct but no next step’. Me: ‘nothing invented, says it cannot confirm’. Cause: rubric. The notes give no step, so ‘without the main next step’ can be read two ways. Guide: ‘When the notes give no step, a clear we cannot confirm with no invented step is a 4 or a 5.’

R08: reference 2, me 4. Reference: ‘does not say that a person will handle it’. Me: ‘protects the sister’s data’. Cause: I missed the notes. They say that a person must handle the ticket, and the reply never says so. I move to 2.”

Common mistake

High agreement, so the reviewers agree

Symptom: a team reports “our two reviewers agree on 92% of replies” and uses one reviewer from then on. Later, a safety reply that one of them would have flagged goes through. Here is a made-up example. Two reviewers rated 50 replies as bad or ok. Reviewer 1 found 4 bad replies. Reviewer 2 found none: they said ok to all 50.

Cause: they agree on 46 of 50, or 92%. But reviewer 2 never says “bad”, so chance agreement is (46 / 50) × 1 = 0.92. Kappa is (0.92 − 0.92) / (1 − 0.92) = 0. All the agreement came from the many easy replies. On the only replies that matter, the 4 bad ones, they never agreed.

Fix: report kappa next to percent agreement, and look at the confusion rows: how many replies each reviewer called bad. When bad replies are rare, put more of them in the calibration sample on purpose, as Larkfield’s sample does with its critical cases.

See it end to endCalibrate a model judge against human ratingsCase study

The judge calibration case study measures how often two people agree on real MT-Bench votes, and uses it as the ceiling for a model judge.

See it end to endEvaluate summaries against expert ratingsCase study

The summaries case study measures how well three experts agree with each other, and how little a crowd’s ratings agree with them.

Knowledge check

Check your understanding

Two reviewers agree on 85% of the replies. Why is that not enough to say that they read the rubric the same way, and what does Cohen's kappa add?

Show the answer

Some agreement comes from luck. When most replies are fine, two reviewers who both say “fine” most of the time match often without reading closely. Percent agreement counts these matches too. Cohen’s kappa first computes the agreement that luck alone would give, from each reviewer’s own mix of scores. Then it reports how much of the room above luck the reviewers filled: (observed − chance) / (1 − chance). Kappa 0 means no better than luck; 1 means they always agree.

Check your understanding

Predict: a reviewer gives every reply a 4. Their sheet is compared with the reference ratings. What is their kappa on five scores, roughly, and why?

Show the answer

About 0. They match the reference on the 12 items that the reference also rated 4, so exact agreement is 0.40. But that is exactly what luck gives: a rater who always says 4 matches whenever the other says 4. Chance agreement is 1 × 12/30 = 0.40, so kappa is (0.40 − 0.40) / (1 − 0.40) = 0.

Check your understanding

Who made the reference ratings, and what do you do when your score differs from one?

Show the answer

The course author made them, by applying Grace’s written policy and rubric, before any model judge ran; scripts checked them. So they are one person’s careful reading, not the truth. When your score differs, read the reply, both notes and the rubric again. Decide whether one reader missed something, whether the rubric can be read two ways, or whether it is a real judgement call. Fix the review guide for the second kind, and record both views for the third.

Recap

• Calibration means that every reviewer rates the same fixed sample alone, before they talk. Then they compare and discuss the differences.
• The course author wrote the reference ratings with Grace’s rubric. They are one person’s reading, and you are the second rater in this module.
• Percent agreement counts matches, including the ones that luck gives. Cohen’s kappa removes the agreement that luck alone gives: (observed − chance) / (1 − chance). In the worked example, 0.70 agreement is only 0.21 kappa.
• python -m harness agree gives exact agreement, within one point, kappa, the “bad or not” view and the items to discuss first.
• People disagree too: in the MT-Bench sample, two human judges reached kappa 0.43.
• Each disagreement has a cause: a missed detail, a rubric that can be read two ways, or a real judgement call. Only the second changes the review guide.

Next: Model-based judging