Skip to content
Français

Find noisy labels

  • Lesson 7
  • 35 min
  • Module 2: Labels you can trust

Where we are

We know how much two careful people agree. The old ticket system's labels are less careful: some messages ended up in the wrong queue and stayed there. In this lesson, we find the labels that are probably wrong with confident learning, and we review them instead of deleting them. A librarian who looks for books on the wrong shelf does the same.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 7 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFind noisy labels

What you will be able to do

Find the training labels that are probably wrong, without an answer key, so a person reviews a short list instead of 6,000 messages. Then measure what the fix is worth.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson: the old labels and what they cost, the cross-validated probabilities, cleanlab’s ranking, your fix of the top 30, and the failure case. Its setup cells install pandas, scikit-learn and cleanlab and download the practice data. About 25 minutes; the model cells take about a minute in all.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Agreement and Cohen’s kappa, from Measure agreement.
  • The comparison table and its cost of mistakes per 1,000 messages, from The cost of each mistake.
  • Cross-validation: train on some parts of the data and score the part the model did not see. It is in Train, validation and test sets in the machine learning course, and this lesson explains the part it needs.

What you need on your computer

  • Your project at the stage of the last lesson (python -m pytest -q prints 12 passed), for one route compare command.
  • The rest of the lesson runs in the notebook: cleanlab is not part of the project.

Think of a library with 6,000 books. Over the years, some books were put back on the wrong shelf. The librarian cannot open all 6,000 books to check. So she walks along the shelves and looks for books that seem out of place: a cookbook between two history books, a children’s book among the law books.

She has a good trick. She does not ask “where is this book now?” She asks “if I had never seen this shelf, where would I put this book?” When her answer is very different from the shelf, she takes the book out and checks it. Most of the books she takes out are on the wrong shelf. A few are on the right shelf after all: a history of cooking, for example. So she checks each one; she does not throw them away.

Larkfield’s 6,000 training messages have two labels. The reviewed label is the right route, which we built into this simulated inbox. The old label is the queue where the old ticket system put each message. In real work, the old labels are what a company has: years of queues, written by a system and by busy agents.

The old system made mistakes, and nobody moved the messages back. This inbox is simulated, so we can count them exactly: 674 of the 6,000 old labels are wrong (11.2%). In real work you do not have this answer key. You do not know which labels are wrong, or even how many.

What do the wrong labels cost? The project can train the same model on either set of labels. Run it in your project folder:

Terminal window
route compare --quick --labels old
Scoring 3 methods on the test messages (trained on the old labels)...
everything to a person: done
rules or a person: done
TF-IDF + logistic regression: done
method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msg
TF-IDF + logistic regression 0.913 0.909 0.750 0.768 0.920 0.896 1206.0 0.0 1206.0 0.14
everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00
rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.04

Your ms/msg will differ. Compare the first row with the same model on the reviewed labels, from route compare --quick: accuracy 0.913 against 0.962, and US$1,206 against US$296 per 1,000 messages. The accuracy drops by 5 points, but the cost of mistakes is four times higher. The old queue sent needs_person messages to delivery 42 times, so the model learned that a safety problem with a parcel is a delivery question. Its recall on needs_person fell from 1.000 to 0.768, and each of those mistakes costs US$40 in the cost table.

Here, the old labels make the model only a little cheaper than “everything to a person” (US$1,500). That is the problem this lesson solves.

Do what the librarian does. For each message, ask a model that never saw that message’s label which route it would choose. When the model gives the message’s own label a very low probability, the label is suspicious. Flag it, and let a person check it.

This idea is called confident learning. The name comes from its method. It uses the model’s confidence in each label to estimate how many labels of each kind are wrong. Then it flags the least likely ones. The open-source package cleanlab implements it.

Why must the model not have seen the label? Because a model believes what it learned. A model trained on all 6,000 old labels has learned the wrong ones too, so it would agree with them. The fix is cross-validation: cut the 6,000 messages into 5 parts, train on 4 parts, and score the fifth. Repeat 5 times, so each message is scored once by a model that did not learn from it.

A flow from top to bottom. 6,000 messages with the old labels. Cut them into 5 parts: train a model on 4 parts, score the 5th part, repeat 5 times. Each message gets probabilities from a model that did not see it. A question: is the probability of its own label very low? Yes leads to flag it: 622 flagged, lowest probability first, and then a person reviews the flagged messages: fix, keep or ask. No leads to keep the label.
Confident learning: probabilities from models that did not see the message, a flag where the message's own label is very unlikely, and a person who reviews the flags.
Text description of the diagram

The flow starts with the 6,000 training messages and their old labels. The next box says: cut them into 5 parts, train a model on 4 parts, and score the 5th part; repeat 5 times. So each message gets probabilities from a model that did not see it. Then a question: is the probability of the message’s own label very low? If no, the label stays. If yes, the message is flagged. cleanlab flags 622 messages here, with the lowest probability first. The last box says that a person reviews the flagged messages, and fixes the label, keeps it, or asks someone.

Here are five real training messages with their old label. The probabilities come from the cross-validated model, so the model did not see these labels.

Message (shortened) Old label Model’s probability for the old label Model’s choice Flag?
“Could you help me with order LK-152123? The colour isn’t what I expected, and I’d like to sort out a return.” account 0.000 returns_refunds (0.999) Yes, rank 1
“Ma commande LK-687451 devait arriver le 24/10 et je ne l’ai toujours pas reçue.” (My order was due on 24/10 and I still have not received it.) account 0.000 delivery (0.996) Yes, rank 2
“My card was charged $204.52, more than the order total. Please fix this. I’d also like to return the sun lounger.” billing 0.012 returns_refunds (0.978) Yes, rank 148
“Is the grill cover waterproof, and can it stay outside in winter?” product 1.000 product No
“Mon paiement est refusé chaque fois, mais ma carte fonctionne ailleurs.” (My payment is refused every time, but my card works elsewhere.) billing 0.997 billing No

The first two are clear: a return labelled account, a late parcel labelled account. The model gives the old label almost no chance. Their reviewed labels are returns_refunds and delivery, as the model said.

The third flag is the interesting one. The old label billing is right: the message asks for two things, and the guidelines say to label the first one, the wrong charge. The model is pulled by “return the sun lounger”. This is a wrong flag: a right label on a hard message. It is the first wrong flag in cleanlab’s list, at rank 148.

The last two messages get a high probability for their label, so cleanlab leaves them alone.

In the notebook, section 3 computes the cross-validated probabilities, and section 4 calls cleanlab:

from cleanlab.filter import find_label_issues
ranked = find_label_issues(labels=given, pred_probs=probs, return_indices_ranked_by="self_confidence", n_jobs=1)
flagged = tr.iloc[ranked]
print(len(ranked), "labels flagged")
# Only possible because this inbox is simulated: check the flags against the reviewed labels.
print(f"wrong among the flagged: {(flagged['old_queue'] != flagged['route']).mean():.3f}")
for k in (30, 100, 300):
print(f"top {k}: {(flagged.head(k)['old_queue'] != flagged.head(k)['route']).sum()} wrong")
622 labels flagged
wrong among the flagged: 0.797
top 30: 30 wrong
top 100: 100 wrong
top 300: 287 wrong

labels is the old label of each message, as a number. pred_probs is the table of probabilities: one row per message, one column per route. return_indices_ranked_by="self_confidence" sorts the flags by the probability of the message’s own label, the lowest first. n_jobs=1 keeps cleanlab in one process.

cleanlab flags 622 messages. The check against the reviewed labels is possible only because this inbox is simulated. It shows that 0.797 of the flags are real errors: 496 wrong labels and 126 right labels. The flags find 496 of the 674 wrong labels (a recall of 0.736). The other 178 wrong labels look normal to the model, so no method that starts from the model can find them.

The order matters more than the total. A person reviews from the top, so the top of the list must be almost all errors:

  • cleanlab ranking
  • Every flag a real error
  • Messages picked at random (11.2%)
Wrong labels found as a person reviews cleanlab's list from the top. The first 147 flags are all real errors; after 300, more flags are right labels on hard messages.
Text description of the chart

The horizontal axis shows the number of flagged messages reviewed, from 0 to 622. The vertical axis shows the number of wrong labels found. A dashed line shows a perfect list, where every flag is a real error: it goes straight up to 622. A second dashed line shows a person who checks messages at random: it finds about 11.2% wrong labels, so about 70 after 622 messages. The cleanlab line follows the perfect line exactly until 147: the first wrong flag is at rank 148. It finds 287 wrong labels in the first 300 flags, and then it rises more slowly, to 496 at 622. Per block of 100 flags, the real errors are 100, 95, 92, 73, 57 and 64, then 15 of the last 22.

The first 147 flags are all real errors. Per block of 100 flags, the real errors are 100, 95, 92, 73, 57 and 64. A person who checks messages at random finds a wrong label only once in 9 messages (11.2%). So the ranking saves most of the review work.

What are the wrong flags, the right labels that cleanlab flags? The commonest are pairs from the guidelines’ hard cases. The old label returns_refunds with the model saying billing comes 25 times, and the reverse 17 times. needs_person with the model saying returns_refunds comes 11 times. Many of them are messages with two requests, like the sun lounger above. A person who knows the guidelines sees in a few seconds that the label is right.

Chloé has time to review 30 messages today. She reviews the top 30 flags and writes the right route for each. In the notebook, the reviewed label plays her review.

  1. Open the notebook from the box Do this lesson in a notebook, and run the setup cells.
  2. In section 1, predict the share of wrong old labels. Then run the cell. You see 674 of 6000 old labels are wrong (11.2%).
  3. In section 2, run the two models. You see 0.913 and US$1,206.0 for the old labels, 0.962 and US$296.0 for the reviewed labels.
  4. In section 3, run the cross-validation cell. It takes about 30 seconds and prints (6000, 7).
  5. Read the five messages with the lowest probability for their old label. Decide for each one if the old label is wrong.
  6. In section 4, run cleanlab. You see 622 labels flagged and wrong among the flagged: 0.797.
  7. In section 5, do the Your turn task: fix the top 30, retrain and score.
  8. Run the check cell. It prints Check passed.
Show the answer to step 7
fixed = tr.copy()
top30 = fixed.index[ranked[:30]]
fixed.loc[top30, "old_queue"] = fixed.loc[top30, "route"]
row = rt.score("old labels, top 30 fixed", test, tfidf_model().fit(fixed["text"], fixed["old_queue"]).predict(test["text"]))
fixed_cost = row["total_per_1000"]
print(row["accuracy"], fixed_cost)
0.914 1138.0

Thirty fixed labels, 0.5% of the training set, move the accuracy only from 0.913 to 0.914. But the cost falls from US$1,206 to US$1,138 per 1,000 messages. The fixed labels were expensive mistakes.

Thirty fixes help a little. What if a person reviews all 622 flags? The chart compares the choices on the same test messages and the same cost table:

TF-IDF with logistic regression, trained on five versions of the labels (old; top 30 fixed; 622 flagged deleted; all 622 flags reviewed and fixed; all reviewed), scored on the same 2,000 test messages.
Text description of the chart

A column chart of the cost of mistakes per 1,000 test messages. Old labels: US$1,206, accuracy 0.913. Fix the top 30: US$1,138, accuracy 0.914. Delete the 622 flagged messages: US$693, accuracy 0.942. Review and fix all 622 flagged: US$503, accuracy 0.950. All reviewed labels: US$296, accuracy 0.962.

Reviewing and fixing all 622 flags gives 0.950 and US$503 per 1,000 messages. That is most of the way from the old labels (US$1,206) to the reviewed labels (US$296). The remaining gap comes mostly from the 178 wrong labels that cleanlab did not flag.

Is the review worth its cost? At US$1.50 per review (the course’s assumption), 622 reviews cost about US$933, once. The fix saves US$1,206 − US$503 = US$703 per 1,000 messages. At about 65 messages a day, that is about US$46 a day. So the review pays for itself in about 20 days, and it keeps saving after that.

Grace adds one practical point: the flagged list is also a list of the guidelines’ weak spots. When the same pair comes back again and again in the wrong flags, the hard case needs a clearer rule.

When to use confident learning, and what it costs

Section titled “When to use confident learning, and what it costs”

Use it when you inherit labels that nobody checked: an old queue, labels from a vendor, labels written by an LLM. It needs a model that is good enough to have an opinion. Here, TF-IDF reaches 0.913 even on the noisy labels, so its doubts mean something. With a weak model, the flags are mostly the model’s own mistakes.

It costs a few minutes of computing (5 trainings instead of 1) and a person’s review of the flags. It does not find the wrong labels that look normal: here, 178 of 674. And it flags right labels on hard messages, so it never replaces the review.

On this inbox, the old system’s errors are simple and the messages are regular, so the flags are unusually clean. On a real inbox, expect more wrong flags. The case study Route banking questions uses real customer questions, where the classes are much closer to each other.

Common mistake

Delete the flagged messages

Symptom: Tomás says: “Why pay for 622 reviews? cleanlab says these labels are wrong. Delete the messages and train on the rest.” The accuracy rises from 0.913 to 0.942, and the cost falls from US$1,206 to US$693 per 1,000 messages. It looks like a free win.

Cause: Deleting throws away the 126 right labels with the 496 wrong ones. The right labels are mostly hard messages: two requests, a refund and a charge. Those are exactly the messages the model needs to learn from. Deleting also removes 6 of the 44 training examples of account takeover, the rarest route, whose missed message costs US$200. The section 6 cell of the notebook shows both numbers.

Fix: Review the flags, from the top of the list. A person fixes each wrong label, keeps each right one, and asks a colleague when unsure. The review gives 0.950 and US$503: better than deleting, at a known cost.

Consequence in AI work: Automatic cleaning makes the training data easier, not better. The model looks good on easy messages and gets worse on the hard ones and on rare classes. Those failures are the expensive ones, and they hide inside a higher average.

Check your understanding

Why must the probabilities for confident learning come from cross-validation, and not from one model trained on all the labels?

Show the answer

A model trained on all the labels has learned the wrong labels too. So it tends to agree with them and gives them a high probability. With cross-validation, each message is scored by a model trained on the other parts only. That model never saw the message’s label, so its probability is an honest opinion: when it is very low, the label is suspicious.

Check your understanding

cleanlab flags 622 of Larkfield's 6,000 old labels, and 0.797 of the flags are real errors. A colleague says: “So 20% of cleanlab's flags are false: it is unreliable.” What do you answer?

Show the answer

The flags are a list for a person to review, not a decision. The top of the list is almost all real errors: the first 147 flags are all wrong labels, and 287 of the top 300. Random checking finds a wrong label only once in 9 messages (11.2%). The wrong flags are mostly right labels on hard messages, which a person who knows the guidelines confirms quickly. That is why the method is: review the flags, do not delete them.

Check your understanding

Practical: you fixed the top 30 flagged labels and retrained. Report the change in the costed measure, and say what a full review of the 622 flags gives and costs.

Show the answer

Fixing the top 30 moves the accuracy from 0.913 to 0.914. The cost of mistakes falls from US$1,206 to US$1,138 per 1,000 test messages. Reviewing all 622 flags gives 0.950 and US$503. The review costs about 622 × US$1.50 = US$933, once. It saves about US$703 per 1,000 messages, so it pays back in about 20 days at 65 messages a day.

Recap

  • The old queue gives 674 wrong labels of 6,000 (11.2%). We know this only because the inbox is simulated. Trained on them, TF-IDF falls from 0.962 to 0.913. The cost of mistakes rises from US$296 to US$1,206 per 1,000 messages.
  • Confident learning asks a model that never saw the label which route it would choose, with cross-validation. A very low probability for the message’s own label means a suspicious label.
  • cleanlab flags 622 labels, the least likely first: the top 147 are all wrong, and 0.797 of all flags are wrong. The wrong flags are right labels on hard messages.
  • Fixing the top 30 lowers the cost to US$1,138; reviewing all 622 to US$503, for about US$933 of review.
  • Deleting the flags gives US$693, removes 126 right labels and 6 of the 44 account takeover examples. Review; do not delete.

You can now find and fix wrong labels in a large set. But what if you have very few labels to start with? Next, we choose which messages a person labels first, and we check labels that an LLM wrote.

Next: When you have few labels