Measure agreement
- Lesson 6
- 35 min
- Module 2: Labels you can trust
Where we are
We have written guidelines. Do two people who follow them give the same label? In this lesson, Chloé and Grace label the same sample, and we measure how often they agree, with a simple share and with Cohen's kappa, which removes the agreement that comes by chance. Two teachers who mark the same exam show why.
Lesson 6 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMeasure agreement
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Measure how often two people give the same label to the same messages, and remove the agreement that comes by chance. Then find the routes where the guidelines are still unclear.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson: the 10-message example by hand and with scikit-learn, Chloé and Grace on 800 messages, the disagreement per route, your task on billing messages, and the failure case. Its setup cells install pandas and scikit-learn and download the practice data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s labelling guidelines and their hard cases, from Labelling guidelines.
- The project, its tests and
route compare, from Meet the inbox.
What you need on your computer (for the project steps; the notebook needs none of it)
- Your project folder from Meet the inbox, with its virtual environment active.
python -m pytest -qprints10 passed. - The course’s labs repository next to your project folder, as in The project on your computer. It has the project at each stage in
C36/snapshots/. This lesson adds the three files of the stageend-of-m02.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are given, but they were not run.
Two teachers mark the same exam
Section titled “Two teachers mark the same exam”A school wants to know if its exam marks are fair. So two teachers mark the same 100 papers, each without seeing the other’s marks. Then the school compares. If the two teachers give the same mark to most papers, the marking rules are clear. If they often differ, the rules are unclear, and a student’s mark depends on which teacher she got.
There is a trap in this check. Suppose the exam is easy, and both teachers give “pass” to 95 papers out of 100. Then they agree on many papers even if they mark without reading carefully: two people who write “pass” on almost everything agree almost always. A high share of agreement can come from luck.
The problem: agreement can come by chance
Section titled “The problem: agreement can come by chance”Chloé and Grace follow the same guidelines. Do they give the same route to the same message? Their agreement matters for two reasons. First, it tells Chloé where the guidelines are unclear. Second, it sets a limit on what the labels can teach a model, as you will see at the end of this lesson.
The first measure is simple. Agreement is the share of messages where the two people gave the same label. But, as with the teachers, part of that agreement can come by chance. If most messages go to one route, two people who guess that route agree often. We need a second number that removes this chance part.
The idea: Cohen’s kappa
Section titled “The idea: Cohen’s kappa”Cohen’s kappa compares the real agreement with the agreement that two people would reach by chance alone. It is named after Jacob Cohen, the psychologist who described it in 1960. Kappa is written with the Greek letter κ, so people often say “kappa”.
The chance agreement is the agreement two people would reach if each one picked labels at random, with their own usual shares. Suppose each person says A 80% of the time. Then they both say A by chance 0.8 × 0.8 = 64% of the time. Kappa then asks: of the agreement that chance leaves to win, how much did the two people really win?
kappa = (agreement − chance agreement) ÷ (1 − chance agreement)
Kappa is 1 when the two people always agree. It is 0 when they agree only as often as chance. It is below 0 when they agree less often than chance.
A tiny example: 10 messages
Section titled “A tiny example: 10 messages”Two people label 10 messages. Each label is A (billing) or B (returns_refunds). This is a practice problem we made on purpose.
Text description of the diagram
The top of the picture shows ten messages, numbered 1 to 10. Person 1 labels them A, A, A, A, A, A, A, A, B, B. Person 2 labels them A, A, A, A, A, A, A, B, A, B. Messages 8 and 9 are orange because the two labels differ. A stands for billing and B for returns_refunds.
Below is a two-by-two table, with person 1 in the rows and person 2 in the columns. Both said A on 7 messages. Person 1 said A and person 2 said B on 1 message. Person 1 said B and person 2 said A on 1 message. Both said B on 1 message. Each person said A 8 times and B 2 times.
The arithmetic: agreement is 7 plus 1, so 8 of 10, which is 0.80. Chance agreement is 0.8 times 0.8 plus 0.2 times 0.2, which is 0.68. Kappa is 0.80 minus 0.68, divided by 1 minus 0.68, which is 0.375.
Follow the numbers. The two people agree on messages 1 to 7 (both A) and on message 10 (both B): 8 of 10, so agreement is 0.80. Each person says A 8 times in 10 and B 2 times in 10. By chance, both say A 0.8 × 0.8 = 0.64 of the time, and both say B 0.2 × 0.2 = 0.04 of the time. So the chance agreement is 0.68.
Chance alone leaves 1 − 0.68 = 0.32 of agreement to win. The two people won 0.80 − 0.68 = 0.12 of it. So kappa = 0.12 ÷ 0.32 = 0.375. Most of their 80% came by chance. In the notebook, scikit-learn’s cohen_kappa_score gives the same 0.375.
How do you read a kappa? People often use these words: below 0.4 weak, 0.4 to 0.6 moderate, 0.6 to 0.8 good, above 0.8 very good. The words are a habit, not a law. So 0.375 is weak, even with 80% agreement.
Chloé and Grace on 800 messages
Section titled “Chloé and Grace on 800 messages”Chloé and Grace each labelled the same 800 training messages, without looking at each other’s labels. The 800 messages are a random sample of the training set. The labels are in data/double_labelled.csv. Here is every count:
Text description of the diagram
The table has one row for each of Chloé’s routes and one column for each of Grace’s routes. The order is orders, delivery, returns_refunds, billing, product, account and needs_person. Each cell counts the messages with that pair of labels.
Row orders: 109, 6, 1, 2, 1, 2, 2. Row delivery: 0, 159, 4, 3, 3, 1, 0. Row returns_refunds: 1, 7, 151, 12, 1, 1, 1. Row billing: 4, 1, 12, 105, 0, 4, 1. Row product: 4, 1, 3, 1, 96, 0, 1. Row account: 0, 3, 3, 1, 1, 61, 0. Row needs_person: 0, 2, 1, 0, 6, 0, 22.
The diagonal, where both chose the same route, adds up to 703 of 800 messages: 87.9%. The largest counts outside the diagonal are the two 12s between returns_refunds and billing. Then come 7 messages that Chloé labelled returns_refunds and Grace labelled delivery, 6 orders and delivery, and 6 needs_person and product.
Read the diagonal first: 703 of 800 messages, so the agreement is 87.9%. Then read outside the diagonal. The two largest counts are the 12s between returns_refunds and billing: the hard case of the last lesson, even with a rule for it. Next come returns_refunds and delivery (7), orders and delivery (6), and needs_person and product (6).
Now the chance part. Chloé and Grace each spread their labels over seven routes, with similar shares. For example, Chloé labelled 21.7% of the messages returns_refunds and Grace 21.9%. Add up “Chloé’s share × Grace’s share” for the seven routes, and you get a chance agreement of 0.169. So:
kappa = (0.879 − 0.169) ÷ (1 − 0.169) = 0.854
A kappa of 0.854 is very good. With seven routes, chance agreement is low, so kappa stays close to the plain agreement. The tiny example was different: with two labels and one of them common, chance did most of the work.
Where they disagree
Section titled “Where they disagree”The table shows pairs. Chloé also wants to know which routes are hardest. For each route, take the messages that either person gave that route, and count how often the other person chose something else:
Text description of the chart
A column chart with one column per route, highest first. needs_person: 38.9% of 36 messages. billing: 28.1% of 146. returns_refunds: 23.7% of 198. account: 20.8% of 77. product: 18.6% of 118. orders: 17.4% of 132. delivery: 16.3% of 190.
needs_person comes first: 38.9% of the 36 messages that either person labelled needs_person. This route has few messages, and the line between “an angry complaint” and “a complaint that needs a manager” is a judgement. billing comes second, at 28.1% of 146 messages, because of its hard case with returns_refunds.
There is a second way to count billing. Take the 125 messages whose reviewed label is billing. On those, Chloé and Grace disagree 16.0% of the time. The number is lower because it starts from the right answer, not from either person’s label. You compute it yourself in the notebook. Both numbers say the same thing: Chloé should add examples to the billing hard case.
Guided practice: add route agreement to the project
Section titled “Guided practice: add route agreement to the project”The project gets a command that prints these numbers from the data. It is the first change to the project in this course: the stage end-of-m02. Three files change: the new command, its test, and the README, which lists the command.
-
In your project folder, with the virtual environment active, check that the command does not exist yet:
Terminal window route agreementusage: route [-h] {info,compare,reset} ...route: error: argument command: invalid choice: 'agreement' (choose from info, compare, reset) -
Copy the three files of this lesson from the snapshot
end-of-m02. ChangeSto the place of your copy of the labs repository:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m02for f in larkroute/commands/agreement.py tests/test_agreement.py README.md; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m02"foreach ($f in 'larkroute\commands\agreement.py', 'tests\test_agreement.py', 'README.md') { Copy-Item "$S\$f" $f }The project finds each command by its file in
larkroute/commands/, socli.pydoes not change. -
Run the tests. Expect
12 passed:Terminal window python -m pytest -q -
Run the new command:
Terminal window route agreement
The output:
800 messages labelled twice: agreement 87.9%, Cohen's kappa 0.854Disagreement on the messages that either person gave this route: route messages disagree needs_person 36 0.389 billing 146 0.281returns_refunds 198 0.237 account 77 0.208 product 118 0.186 orders 132 0.174 delivery 190 0.163Chloé (rows) against Grace (columns):label_grace account billing delivery needs_person orders product returns_refundslabel_chloeaccount 61 1 3 0 0 1 3billing 4 105 1 1 4 0 12delivery 1 3 159 0 0 3 4needs_person 0 0 2 22 0 6 1orders 2 2 6 2 109 1 1product 0 1 1 1 4 96 3returns_refunds 1 12 7 1 1 1 151These are the same numbers as the diagram above. The command prints the routes in alphabetical order, so the rows and columns are in another order. Find the two 12s: billing against returns_refunds, both ways.
Open larkroute/commands/agreement.py. The function that does the work is three lines:
def agreement(df: pd.DataFrame, a="label_chloe", b="label_grace") -> dict: same = df[a] == df[b] return {"messages": len(df), "agreement": round(same.mean(), 3), "kappa": round(cohen_kappa_score(df[a], df[b]), 3)}Then open tests/test_agreement.py. Its first test is a tiny example: four messages, two labels, half of them the same. Agreement is 0.5 and kappa is 0, because two coins would agree half of the time too.
Your turn: agreement on billing messages
Section titled “Your turn: agreement on billing messages”In the notebook, section 4: take the messages whose reviewed label (route in train) is billing. Compute the share where Chloé and Grace disagree, rounded to 3 decimals, in billing_disagree.
Show the answer
merged = pairs.merge(train[["message_id", "route"]], on="message_id")billing = merged[merged["route"] == "billing"]billing_disagree = round((billing["label_chloe"] != billing["label_grace"]).mean(), 3)print(len(billing), "billing messages, disagreement", billing_disagree)125 billing messages, disagreement 0.16Chloé and Grace disagree on 16% of the billing messages: about 1 in 6.
A model cannot be more consistent than its labels
Section titled “A model cannot be more consistent than its labels”This inbox is simulated, so every message also has a reviewed label: the right answer we built into the data. In real work, you get such a label only from an expert review of each message. Against the reviewed labels, Chloé is right on 96.1% of the 800 messages, and Grace on 91.7%.
Now think about what a model learns. Suppose Grace had labelled all the training messages alone. Then 8.3% of the labels would teach the model a wrong route. The model learns her habits with the rules. A model trained on her labels that agrees with her on 99% of new messages has not become better than her. It has copied her mistakes very well.
The same limit applies to the test. If the test labels come from one person, a model is judged against that person’s habits. When two careful people agree on 87.9% of messages, a test score far above that number is a warning: the model may match one person’s habits, not the rule. In this course, the test set has the reviewed labels, so the scores are honest. Real projects do not have that luxury: there, the people’s agreement is the ceiling you can trust.
When to measure agreement, and what it costs
Section titled “When to measure agreement, and what it costs”Measure agreement when you write the first guidelines, and again after each important change to them. Let two people label the same random sample, separately. Here, the sample has 800 messages, 8% of the history. A smaller sample costs less, but it has few messages of the rare routes: even 800 messages give only 36 that either person labelled needs_person.
The cost is the second person’s time. At the course’s review cost of US$1.50 per message (an assumption), 800 extra labels cost about US$1,200, once. The next lesson shows that wrong labels cost far more than that over time.
Agreement does not prove that the labels are right. Two people can agree on a wrong rule, as the keyword guideline of the last lesson showed. It proves only that the rule is clear.
Common mistake
Agreement measured on easy messages
Symptom: A team wants a quick agreement check. One person picks 50 messages “to save time”, and two people label them. They agree on 92.0%. The team reports that the guidelines are clear.
Cause: The person picked short, clear messages. In the notebook, section 5 measures agreement on the 50 shortest of the 800 messages: 92.0%, against 87.9% on all 800. Short messages are often the easy ones. The hard cases, such as a refund and a charge in one message, are rarely short.
Fix: Take the messages for double labelling at random, or at random inside each route, and write down how you chose them. Report agreement with its sample size.
Consequence in AI work: An agreement check on an easy sample hides the hard routes. The team keeps unclear guidelines, the labels stay inconsistent on the hard cases, and the model fails exactly there.
Knowledge check
Section titled “Knowledge check”Check your understanding
Two labellers agree on 90% of messages. In one team, 90% of messages have the same label; in another team, the messages spread over seven labels. In which team is kappa higher, and why?
Show the answer
In the team with seven labels spread out. Kappa removes the agreement that comes by chance. When 90% of messages have one label, two people who give that label agree by chance most of the time. So chance agreement is high and kappa is low. With seven labels spread out, chance agreement is low: 0.169 for Chloé and Grace. So kappa stays close to the plain agreement: 0.854 for 87.9%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Compute Cohen's kappa: two people label 10 messages; they agree on 6; each person gives label A to 5 messages and label B to 5 messages.
Show the answer
Agreement = 0.6. Chance agreement = 0.5 × 0.5 + 0.5 × 0.5 = 0.5. kappa = (0.6 − 0.5) ÷ (1 − 0.5) = 0.2. This is weak agreement: only a little better than two coins.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical: you ran route agreement. Which hard case should Chloé improve first, and which number in the output shows it?
Show the answer
The hard case between billing and returns_refunds. The table shows the largest counts outside the diagonal there: 12 messages labelled billing by Chloé and returns_refunds by Grace, and 12 the other way. billing also has the second-highest disagreement per route (0.281). needs_person has the highest (0.389), so its rule needs examples too, but it has only 36 messages.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Agreement is the share of messages where two people give the same label. Part of it can come by chance.
- Cohen’s kappa removes the chance part: (agreement − chance agreement) ÷ (1 − chance agreement). In the 10-message example, 80% agreement gives a kappa of only 0.375.
- Chloé and Grace agree on 87.9% of 800 messages, kappa 0.854. They disagree most between
billingandreturns_refunds(12 + 12), and on 16% of the billing messages. route agreementprints these numbers; the project now has 12 tests.- A model trained on one person’s labels learns that person’s habits. When people agree on 87.9%, a much higher score against one person’s labels is a warning.
- Measure agreement on a random sample, never on messages someone picked.
You can now measure how consistent careful labels are. But Larkfield’s 6,000 training labels did not come from careful people: they came from the old ticket system. Next, we find the labels that are probably wrong.
Next: Find noisy labels