Skip to content
Français

Measure agreement

  • Lesson 6
  • 35 min
  • Module 2: Labels you can trust

Where we are

We have written guidelines. Do two people who follow them give the same label? In this lesson, Chloé and Grace label the same sample, and we measure how often they agree, with a simple share and with Cohen's kappa, which removes the agreement that comes by chance. Two teachers who mark the same exam show why.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 6 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMeasure agreement

What you will be able to do

Measure how often two people give the same label to the same messages, and remove the agreement that comes by chance. Then find the routes where the guidelines are still unclear.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson: the 10-message example by hand and with scikit-learn, Chloé and Grace on 800 messages, the disagreement per route, your task on billing messages, and the failure case. Its setup cells install pandas and scikit-learn and download the practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • Your project folder from Meet the inbox, with its virtual environment active. python -m pytest -q prints 10 passed.
  • The course’s labs repository next to your project folder, as in The project on your computer. It has the project at each stage in C36/snapshots/. This lesson adds the three files of the stage end-of-m02.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are given, but they were not run.

A school wants to know if its exam marks are fair. So two teachers mark the same 100 papers, each without seeing the other’s marks. Then the school compares. If the two teachers give the same mark to most papers, the marking rules are clear. If they often differ, the rules are unclear, and a student’s mark depends on which teacher she got.

There is a trap in this check. Suppose the exam is easy, and both teachers give “pass” to 95 papers out of 100. Then they agree on many papers even if they mark without reading carefully: two people who write “pass” on almost everything agree almost always. A high share of agreement can come from luck.

Chloé and Grace follow the same guidelines. Do they give the same route to the same message? Their agreement matters for two reasons. First, it tells Chloé where the guidelines are unclear. Second, it sets a limit on what the labels can teach a model, as you will see at the end of this lesson.

The first measure is simple. Agreement is the share of messages where the two people gave the same label. But, as with the teachers, part of that agreement can come by chance. If most messages go to one route, two people who guess that route agree often. We need a second number that removes this chance part.

Cohen’s kappa compares the real agreement with the agreement that two people would reach by chance alone. It is named after Jacob Cohen, the psychologist who described it in 1960. Kappa is written with the Greek letter κ, so people often say “kappa”.

The chance agreement is the agreement two people would reach if each one picked labels at random, with their own usual shares. Suppose each person says A 80% of the time. Then they both say A by chance 0.8 × 0.8 = 64% of the time. Kappa then asks: of the agreement that chance leaves to win, how much did the two people really win?

kappa = (agreement − chance agreement) ÷ (1 − chance agreement)

Kappa is 1 when the two people always agree. It is 0 when they agree only as often as chance. It is below 0 when they agree less often than chance.

Two people label 10 messages. Each label is A (billing) or B (returns_refunds). This is a practice problem we made on purpose.

Ten messages in a row. Person 1 labels them A A A A A A A A B B. Person 2 labels them A A A A A A A B A B. Messages 8 and 9 are orange: the two labels differ. Below, a two-by-two table: both A 7, person 1 A and person 2 B 1, person 1 B and person 2 A 1, both B 1. Then the arithmetic: agreement 7 plus 1 equals 8 of 10, 0.80. Chance agreement 0.8 times 0.8 plus 0.2 times 0.2 equals 0.68. Kappa equals 0.80 minus 0.68, divided by 1 minus 0.68, equals 0.375.
A constructed example. The two people agree on 8 of 10 messages, but both say A most of the time, so chance alone gives 0.68. Kappa is 0.375.
Text description of the diagram

The top of the picture shows ten messages, numbered 1 to 10. Person 1 labels them A, A, A, A, A, A, A, A, B, B. Person 2 labels them A, A, A, A, A, A, A, B, A, B. Messages 8 and 9 are orange because the two labels differ. A stands for billing and B for returns_refunds.

Below is a two-by-two table, with person 1 in the rows and person 2 in the columns. Both said A on 7 messages. Person 1 said A and person 2 said B on 1 message. Person 1 said B and person 2 said A on 1 message. Both said B on 1 message. Each person said A 8 times and B 2 times.

The arithmetic: agreement is 7 plus 1, so 8 of 10, which is 0.80. Chance agreement is 0.8 times 0.8 plus 0.2 times 0.2, which is 0.68. Kappa is 0.80 minus 0.68, divided by 1 minus 0.68, which is 0.375.

Follow the numbers. The two people agree on messages 1 to 7 (both A) and on message 10 (both B): 8 of 10, so agreement is 0.80. Each person says A 8 times in 10 and B 2 times in 10. By chance, both say A 0.8 × 0.8 = 0.64 of the time, and both say B 0.2 × 0.2 = 0.04 of the time. So the chance agreement is 0.68.

Chance alone leaves 1 − 0.68 = 0.32 of agreement to win. The two people won 0.80 − 0.68 = 0.12 of it. So kappa = 0.12 ÷ 0.32 = 0.375. Most of their 80% came by chance. In the notebook, scikit-learn’s cohen_kappa_score gives the same 0.375.

How do you read a kappa? People often use these words: below 0.4 weak, 0.4 to 0.6 moderate, 0.6 to 0.8 good, above 0.8 very good. The words are a habit, not a law. So 0.375 is weak, even with 80% agreement.

Chloé and Grace each labelled the same 800 training messages, without looking at each other’s labels. The 800 messages are a random sample of the training set. The labels are in data/double_labelled.csv. Here is every count:

A table of seven rows and seven columns. Rows are Chloé's label, columns are Grace's label, in the order orders, delivery, returns_refunds, billing, product, account, needs_person. The diagonal, in blue, has 109, 159, 151, 105, 96, 61 and 22. The largest counts outside the diagonal are 12 where Chloé said returns_refunds and Grace said billing, and 12 where Chloé said billing and Grace said returns_refunds. Below the table: same label on 703 of 800 messages, 87.9%.
Chloé (rows) against Grace (columns) on 800 messages. Blue: the same route. Orange: the pairs they confuse most. The two 12s are billing and returns_refunds.
Text description of the diagram

The table has one row for each of Chloé’s routes and one column for each of Grace’s routes. The order is orders, delivery, returns_refunds, billing, product, account and needs_person. Each cell counts the messages with that pair of labels.

Row orders: 109, 6, 1, 2, 1, 2, 2. Row delivery: 0, 159, 4, 3, 3, 1, 0. Row returns_refunds: 1, 7, 151, 12, 1, 1, 1. Row billing: 4, 1, 12, 105, 0, 4, 1. Row product: 4, 1, 3, 1, 96, 0, 1. Row account: 0, 3, 3, 1, 1, 61, 0. Row needs_person: 0, 2, 1, 0, 6, 0, 22.

The diagonal, where both chose the same route, adds up to 703 of 800 messages: 87.9%. The largest counts outside the diagonal are the two 12s between returns_refunds and billing. Then come 7 messages that Chloé labelled returns_refunds and Grace labelled delivery, 6 orders and delivery, and 6 needs_person and product.

Read the diagonal first: 703 of 800 messages, so the agreement is 87.9%. Then read outside the diagonal. The two largest counts are the 12s between returns_refunds and billing: the hard case of the last lesson, even with a rule for it. Next come returns_refunds and delivery (7), orders and delivery (6), and needs_person and product (6).

Now the chance part. Chloé and Grace each spread their labels over seven routes, with similar shares. For example, Chloé labelled 21.7% of the messages returns_refunds and Grace 21.9%. Add up “Chloé’s share × Grace’s share” for the seven routes, and you get a chance agreement of 0.169. So:

kappa = (0.879 − 0.169) ÷ (1 − 0.169) = 0.854

A kappa of 0.854 is very good. With seven routes, chance agreement is low, so kappa stays close to the plain agreement. The tiny example was different: with two labels and one of them common, chance did most of the work.

The table shows pairs. Chloé also wants to know which routes are hardest. For each route, take the messages that either person gave that route, and count how often the other person chose something else:

For each route (pers. = needs_person, bill. = billing, ret. = returns_refunds, acct. = account, prod. = product, ord. = orders, deliv. = delivery): of the messages that either person gave this route, the share where the other person chose another route.
Text description of the chart

A column chart with one column per route, highest first. needs_person: 38.9% of 36 messages. billing: 28.1% of 146. returns_refunds: 23.7% of 198. account: 20.8% of 77. product: 18.6% of 118. orders: 17.4% of 132. delivery: 16.3% of 190.

needs_person comes first: 38.9% of the 36 messages that either person labelled needs_person. This route has few messages, and the line between “an angry complaint” and “a complaint that needs a manager” is a judgement. billing comes second, at 28.1% of 146 messages, because of its hard case with returns_refunds.

There is a second way to count billing. Take the 125 messages whose reviewed label is billing. On those, Chloé and Grace disagree 16.0% of the time. The number is lower because it starts from the right answer, not from either person’s label. You compute it yourself in the notebook. Both numbers say the same thing: Chloé should add examples to the billing hard case.

Guided practice: add route agreement to the project

Section titled “Guided practice: add route agreement to the project”

The project gets a command that prints these numbers from the data. It is the first change to the project in this course: the stage end-of-m02. Three files change: the new command, its test, and the README, which lists the command.

  1. In your project folder, with the virtual environment active, check that the command does not exist yet:

    Terminal window
    route agreement
    usage: route [-h] {info,compare,reset} ...
    route: error: argument command: invalid choice: 'agreement' (choose from info, compare, reset)
  2. Copy the three files of this lesson from the snapshot end-of-m02. Change S to the place of your copy of the labs repository:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m02
    for f in larkroute/commands/agreement.py tests/test_agreement.py README.md; do
    cp "$S/$f" "$f"
    done

    The project finds each command by its file in larkroute/commands/, so cli.py does not change.

  3. Run the tests. Expect 12 passed:

    Terminal window
    python -m pytest -q
  4. Run the new command:

    Terminal window
    route agreement

The output:

800 messages labelled twice: agreement 87.9%, Cohen's kappa 0.854
Disagreement on the messages that either person gave this route:
route messages disagree
needs_person 36 0.389
billing 146 0.281
returns_refunds 198 0.237
account 77 0.208
product 118 0.186
orders 132 0.174
delivery 190 0.163
Chloé (rows) against Grace (columns):
label_grace account billing delivery needs_person orders product returns_refunds
label_chloe
account 61 1 3 0 0 1 3
billing 4 105 1 1 4 0 12
delivery 1 3 159 0 0 3 4
needs_person 0 0 2 22 0 6 1
orders 2 2 6 2 109 1 1
product 0 1 1 1 4 96 3
returns_refunds 1 12 7 1 1 1 151

These are the same numbers as the diagram above. The command prints the routes in alphabetical order, so the rows and columns are in another order. Find the two 12s: billing against returns_refunds, both ways.

Open larkroute/commands/agreement.py. The function that does the work is three lines:

def agreement(df: pd.DataFrame, a="label_chloe", b="label_grace") -> dict:
same = df[a] == df[b]
return {"messages": len(df), "agreement": round(same.mean(), 3), "kappa": round(cohen_kappa_score(df[a], df[b]), 3)}

Then open tests/test_agreement.py. Its first test is a tiny example: four messages, two labels, half of them the same. Agreement is 0.5 and kappa is 0, because two coins would agree half of the time too.

In the notebook, section 4: take the messages whose reviewed label (route in train) is billing. Compute the share where Chloé and Grace disagree, rounded to 3 decimals, in billing_disagree.

Show the answer
merged = pairs.merge(train[["message_id", "route"]], on="message_id")
billing = merged[merged["route"] == "billing"]
billing_disagree = round((billing["label_chloe"] != billing["label_grace"]).mean(), 3)
print(len(billing), "billing messages, disagreement", billing_disagree)
125 billing messages, disagreement 0.16

Chloé and Grace disagree on 16% of the billing messages: about 1 in 6.

A model cannot be more consistent than its labels

Section titled “A model cannot be more consistent than its labels”

This inbox is simulated, so every message also has a reviewed label: the right answer we built into the data. In real work, you get such a label only from an expert review of each message. Against the reviewed labels, Chloé is right on 96.1% of the 800 messages, and Grace on 91.7%.

Now think about what a model learns. Suppose Grace had labelled all the training messages alone. Then 8.3% of the labels would teach the model a wrong route. The model learns her habits with the rules. A model trained on her labels that agrees with her on 99% of new messages has not become better than her. It has copied her mistakes very well.

The same limit applies to the test. If the test labels come from one person, a model is judged against that person’s habits. When two careful people agree on 87.9% of messages, a test score far above that number is a warning: the model may match one person’s habits, not the rule. In this course, the test set has the reviewed labels, so the scores are honest. Real projects do not have that luxury: there, the people’s agreement is the ceiling you can trust.

When to measure agreement, and what it costs

Section titled “When to measure agreement, and what it costs”

Measure agreement when you write the first guidelines, and again after each important change to them. Let two people label the same random sample, separately. Here, the sample has 800 messages, 8% of the history. A smaller sample costs less, but it has few messages of the rare routes: even 800 messages give only 36 that either person labelled needs_person.

The cost is the second person’s time. At the course’s review cost of US$1.50 per message (an assumption), 800 extra labels cost about US$1,200, once. The next lesson shows that wrong labels cost far more than that over time.

Agreement does not prove that the labels are right. Two people can agree on a wrong rule, as the keyword guideline of the last lesson showed. It proves only that the rule is clear.

Common mistake

Agreement measured on easy messages

Symptom: A team wants a quick agreement check. One person picks 50 messages “to save time”, and two people label them. They agree on 92.0%. The team reports that the guidelines are clear.

Cause: The person picked short, clear messages. In the notebook, section 5 measures agreement on the 50 shortest of the 800 messages: 92.0%, against 87.9% on all 800. Short messages are often the easy ones. The hard cases, such as a refund and a charge in one message, are rarely short.

Fix: Take the messages for double labelling at random, or at random inside each route, and write down how you chose them. Report agreement with its sample size.

Consequence in AI work: An agreement check on an easy sample hides the hard routes. The team keeps unclear guidelines, the labels stay inconsistent on the hard cases, and the model fails exactly there.

Check your understanding

Two labellers agree on 90% of messages. In one team, 90% of messages have the same label; in another team, the messages spread over seven labels. In which team is kappa higher, and why?

Show the answer

In the team with seven labels spread out. Kappa removes the agreement that comes by chance. When 90% of messages have one label, two people who give that label agree by chance most of the time. So chance agreement is high and kappa is low. With seven labels spread out, chance agreement is low: 0.169 for Chloé and Grace. So kappa stays close to the plain agreement: 0.854 for 87.9%.

Check your understanding

Compute Cohen's kappa: two people label 10 messages; they agree on 6; each person gives label A to 5 messages and label B to 5 messages.

Show the answer

Agreement = 0.6. Chance agreement = 0.5 × 0.5 + 0.5 × 0.5 = 0.5. kappa = (0.6 − 0.5) ÷ (1 − 0.5) = 0.2. This is weak agreement: only a little better than two coins.

Check your understanding

Practical: you ran route agreement. Which hard case should Chloé improve first, and which number in the output shows it?

Show the answer

The hard case between billing and returns_refunds. The table shows the largest counts outside the diagonal there: 12 messages labelled billing by Chloé and returns_refunds by Grace, and 12 the other way. billing also has the second-highest disagreement per route (0.281). needs_person has the highest (0.389), so its rule needs examples too, but it has only 36 messages.

Recap

  • Agreement is the share of messages where two people give the same label. Part of it can come by chance.
  • Cohen’s kappa removes the chance part: (agreement − chance agreement) ÷ (1 − chance agreement). In the 10-message example, 80% agreement gives a kappa of only 0.375.
  • Chloé and Grace agree on 87.9% of 800 messages, kappa 0.854. They disagree most between billing and returns_refunds (12 + 12), and on 16% of the billing messages.
  • route agreement prints these numbers; the project now has 12 tests.
  • A model trained on one person’s labels learns that person’s habits. When people agree on 87.9%, a much higher score against one person’s labels is a warning.
  • Measure agreement on a random sample, never on messages someone picked.

You can now measure how consistent careful labels are. But Larkfield’s 6,000 training labels did not come from careful people: they came from the old ticket system. Next, we find the labels that are probably wrong.

Next: Find noisy labels