Skip to content
Français

Labels: how many, and how good

  • Lesson 12
  • 40 min
  • Module 3: Embeddings, CLIP and few labels

Where we are

Our learning curves told us how many labels we need. They did not ask whether the labels are right. In this lesson, we write a labelling guide and measure agreement between two labellers. Then we find the label errors that we planted, with embeddings and confident learning. First, two workers who sort the same box.

What you will be able to do

Make Larkfield’s labels good enough to trust. Write down the rules, measure whether two people follow them the same way, and find the wrong labels in the catalogue. A model learns its labels, mistakes included.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install NumPy, pandas, scikit-learn and cleanlab, and download the catalogue photos with their stored embeddings as zip files. It downloads no model. It needs no account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project route)

  • Your larkvision folder with vision probe from the last lesson. python -m pytest -q prints 11 passed.
  • In each new terminal: go to larkvision, activate .venv, and set LARKVISION_DATA again, as in Meet the Larkfield images.

Tested with Python 3.12.13, scikit-learn 1.9.1 and cleanlab 2.9.0 on macOS, in Jupyter and in a terminal. The Windows commands, Colab and Kaggle were not tested.

Hamid gives the same box of 12 returned items to two workers, one after the other. Each writes down a shelf for each item. They agree on 10 items and disagree on 2: a ceramic pot with cut flowers, and a tall jug with a small plant in soil.

Neither worker was careless. Hamid never said whether the shelf depends on the container or on what is inside it. Each worker made a sensible rule, and the two rules were different. The disagreement is a gap in the instructions.

The problem: a label is a decision, not a fact

Section titled “The problem: a label is a decision, not a fact”

Every method of this module learns from labels. The probe of the last lesson draws its line exactly where the labels say. If some plant pots are labelled “vase”, the probe learns that some plant pots are vases. If the test photos have wrong labels, even the score is wrong.

Larkfield’s labels have two kinds of problems. First, the catalogue has planted errors. The course changed 4 of the 70 plant-pot training labels to “vase”: about 6% of the class. Second, it has real gaps that nobody planted. The catalogue photos come from Open Images, labelled with Open Images’ own rules. Its “Flowerpot” class is not exactly Larkfield’s “plant pot”. Here are eight training photos from the top of this lesson’s suspect lists:

Eight photos in two rows. Top row, framed in red, all labelled vase. A plant in a red-brown pot, a plant in a black plastic pot, a pot of green seedlings, a bonsai tree in a flat pot. Bottom row. A white vase-shaped pot with a fern, labelled vase. A white mug with dry twigs in soil, labelled mug. A carved terracotta urn, labelled plant pot. A hanging glass globe with an air plant, labelled vase.
Top, framed: the 4 planted label errors, plant pots labelled vase. Bottom: 4 hard cases from the top of the suspect lists, with the labels that Open Images gave them. Photos, top then bottom: Joseph Novak, Chris Owen, Takahiro Yamagiwa, JCardinal18, bellergy2, Lauren Manning, mochichick, ProFlowers, CC BY 2.0, from Open Images.
Text description of the diagram

The top row has four photos framed in red. All are labelled vase, and all show a plant growing in soil in a pot. They are a plant in a red-brown pot, a plant in a black plastic pot, a pot of seedlings, and a bonsai tree. These are the planted errors. The bottom row has four hard cases. A white, vase-shaped pot with a fern growing in it is labelled vase. A white mug with dry twigs in soil is labelled mug. A carved terracotta urn with nothing in it is labelled plant pot. A hanging glass globe with an air plant is labelled vase.

The top row is easy. The bottom row is hard. Is a mug with a plant in it a mug or a plant pot? A good answer depends on what Larkfield sells. No model can decide that for Larkfield.

The fix is a labelling guide: a short document with the rules for each class and photos of the hard cases. It is written before the labelling starts, and updated with each new hard case. For each pair of classes that people confuse, it has:

  1. The deciding question. For Larkfield: “Which product would the customer buy?” The catalogue sells containers, so the container decides, not the plant.
  2. The rule. A plant pot has a drainage hole or a wide opening, and holds soil. A vase holds water and cut flowers, and is usually taller than wide.
  3. The hard cases, with photos, the decision and the reason. A mug with a plant in it is a mug: Larkfield sells it as a mug. A vase-shaped pot with soil and a fern is a plant pot.
  4. A way out. “If you cannot decide in 10 seconds, choose unsure and add a note.” Someone can check an unsure label later. Nobody checks a guess.

Priya’s first version covers the pairs that the embedding map showed: plant pot–vase, bowl–mug and chair–cushion.

How do you know that the guide works? Give the same photos to two people, and measure how often they agree. Here is a tiny example: two labellers, 12 photos, two classes. The labels are made up for this example.

B says plant pot B says vase Total
A says plant pot 5 1 6
A says vase 1 5 6
Total 6 6 12

The percent agreement is the share of photos where they agree: (5 + 5) / 12 = 0.833.

But two people who guess at random also agree sometimes. Here, each labeller says plant pot for half the photos. So two random guessers would agree on half the photos: 0.5 × 0.5 for “both plant pot” plus 0.5 × 0.5 for “both vase” = 0.5. Cohen’s kappa measures the agreement above chance:

kappa = (agreement − chance agreement) / (1 − chance agreement) = (0.833 − 0.5) / (1 − 0.5) = 0.667

A kappa of 1 means perfect agreement, and 0 means no better than chance. As a common rule of thumb, aim for 0.8 or more before you trust labels for training. At 0.667, the guide has gaps: find the two photos where A and B disagree, decide them, and add them to the guide.

In practice, two people label the same 50 to 100 photos from every class before the large job starts. They measure kappa, decide each disagreement, update the guide, and repeat. Later, a few shared photos keep checking the agreement.

For more than a few hundred photos, people use an open-source labelling tool such as Label Studio or CVAT. A person clicks a class for each photo, or draws boxes and masks for Modules 4 and 5. The catalogue’s manifest.csv is such an export, so you can skip the tools. To try one, label 20 catalogue photos with your guide and compare them with the manifest.

A guide prevents new mistakes. But the catalogue already has 453 labelled training photos, and people miss errors when they look at hundreds of similar photos. Can the computer point at the photos most likely to be wrong?

The idea uses the probe of the last lesson. Train a probe, and ask it about each training photo: “What probability do you give to this photo’s own label?” A photo labelled vase that looks exactly like the plant pots gets a low probability for “vase”. That photo is a suspect.

There is one trap. A probe that was trained on a photo has seen its label, so it tends to agree with it, even when the label is wrong. The fix is the cross-validation of the machine learning course. Cut the training photos into 5 folds. Train on 4 folds and score the 5th, 5 times. Every photo then gets an out-of-fold probability, from a probe that never saw it.

Five training photos, with the out-of-fold probability that the probe gives to each class. The numbers are made up for this example.

Photo Its label P(plant pot) P(vase) P(its own label) Suspect?
1 plant pot 0.95 0.05 0.95 no
2 vase 0.10 0.90 0.90 no
3 vase 0.97 0.03 0.03 yes: look first
4 plant pot 0.40 0.60 0.40 yes: look second
5 vase 0.45 0.55 0.55 no

Photo 3 is the clearest suspect: the probe is very sure that it is a plant pot, and it is labelled vase. Photo 4 comes second. Photo 5 is a close call, but the probe still prefers its label.

Confident learning makes this rule more careful. It first measures how confident the probe usually is for each class, because some classes are harder than others. Then it flags a photo when the probe gives another class more than that class’s usual confidence. The flagged photos are ranked by the probability of their own label, lowest first. The Python library cleanlab does all of this with one function, find_label_issues.

Five boxes from left to right. 5 folds: split the 453 training photos into 5 parts. Out-of-fold scores: train a probe on 4 parts, score the 5th part, repeat 5 times. Own-label probability: for each photo, the probability the model gives to its own label. A note says that no model scores a photo whose label it saw. Rank: lowest first, the photos whose label the model doubts most. A person looks: fix a wrong label, keep a right one, update the guide.
Finding suspect labels with confident learning. The computer ranks; a person decides.
Text description of the diagram

The diagram has five boxes in a row (one under the other on a phone). First, the 453 training photos are split into 5 parts. Second, a probe is trained on 4 parts and scores the 5th, five times, so every photo gets out-of-fold scores. Third, for each photo, we take the probability that the model gives to its own label. A note says that no model scores a photo whose label it saw. Fourth, the photos are ranked with the lowest probability first: the labels that the model doubts most. Last, a person looks at each one, fixes a wrong label, keeps a right one, and updates the guide.

Priya runs cleanlab on the out-of-fold probabilities of a CLIP probe. It flags 11 of the 453 training photos. These are the first ten (section 2 of the notebook):

Rank Photo Labelled The probe says
1 P-2f04b07d vase plant pot
2 P-2760f082 vase plant pot
3 P-20464783 bowl mug
4 P-1c7efe8e vase plant pot
5 P-157311db plant pot vase
6 P-2b646d6f vase plant pot
7 P-1ba1b277 chair cushion
8 P-67737a23 vase plant pot
9 P-427de7bf plant pot vase
10 P-19aa91d9 vase plant pot

The sealed key scores the search: all 4 planted errors are in the top 10, at ranks 1, 2, 4 and 6. With DINOv2 embeddings, cleanlab flags 12 photos, and the 4 planted errors are again in its top 10. Ten photos to look at, instead of 453.

The map of the last lesson shows why it works. The 4 planted errors, circled, sit deep inside the plant pot group, far from the vases:

The UMAP map of the 733 catalogue photos, coloured by label, as in the image embeddings lesson. Four red circles mark the 4 planted label errors. All four are black vase dots inside the orange plant pot group at the bottom, far from the vase group.
The UMAP map of the CLIP embeddings, with the 4 planted label errors circled in red. Their label says vase; their embeddings sit among the plant pots.
Text description of the diagram

The map is the one of the image embeddings lesson: one dot per catalogue photo, coloured by label. Four red circles mark the planted label errors. Each circle holds a black dot, the colour of vase, inside the orange group of plant pots at the bottom of the map. The real vases are further up. A photo whose colour does not match its neighbours is what the probe finds.

The other six photos in the top 10 are not planted errors. They are a large cup labelled bowl, a green funnel-shaped stand labelled plant pot, a chair with a jacket, and the glass globe. The other two are ceramic jars with green lizards, labelled plant pot, and a silver garden urn, labelled vase. Some are hard cases for the guide. The chair with a jacket is a chair: its label is right, and the probe is wrong. A suspect is a question for a person, not an answer.

Priya fixes the 4 planted errors, trains the probe again and runs the search again. Before you read on, predict: how much does the test accuracy rise?

CLIP probe on the training photos Test accuracy Suspects found by cleanlab
Labels as given 0.894 11
4 planted errors fixed 0.894 5

The test accuracy does not move. The DINOv2 probe moves by one photo, from 0.887 to 0.894. Four wrong labels among 453 photos are too few to change what the probe learns about plant pots: 66 right plant pots still outvote them. The suspect list, though, shrinks from 11 to 5.

So why fix them? Because labels are used for more than one training run:

  • Errors grow with their share. Real catalogues can have far more than 6%. In a class of 15 photos, one wrong label is 7%.
  • Errors in the validation and test sets change the score. A wrong test label counts a right answer as a mistake.
  • People and later models trust the labels too.

The same embeddings find another problem. A photo uploaded twice, a little cropped or brighter, has an embedding almost the same as the first one. Section 5 of the notebook lists the 10 most similar pairs of training photos. With CLIP, 5 of the top 10 pairs are second uploads that we planted (6 in all). With DINOv2, all 6 are in its top 10.

A duplicate matters most across splits: the model is then tested on a photo that it learned from. Find the same product compares this with perceptual hashing.

The project gets the search as a command, vision labels. It ranks the training photos where the probe prefers another label, by the out-of-fold probability of their own label.

  1. In your larkvision folder, with .venv active and LARKVISION_DATA set, copy the last files of this module from the snapshot:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m03
    for f in larkvision/commands/labels.py tests/test_probe.py README.md; do cp "$S/$f" "$f"; done
  2. Run the tests. You see 14 passed. The three new tests train a probe and search for a planted error on a tiny fake data folder:

    Terminal window
    python -m pytest -q
  3. Run the search with the DINOv2 embeddings, and score it with the sealed key:

    Terminal window
    vision labels --embeddings dinov2 --score

The output of step 3:

34 training photos where the model prefers another label. The first 10:
P-2f04b07d labelled vase the model says plant_pot (probability of the label 0.03)
P-225e8002 labelled plant_pot the model says vase (probability of the label 0.06)
P-2b646d6f labelled vase the model says plant_pot (probability of the label 0.07)
P-427de7bf labelled plant_pot the model says vase (probability of the label 0.07)
P-2760f082 labelled vase the model says plant_pot (probability of the label 0.07)
P-4c9f361a labelled mug the model says vase (probability of the label 0.08)
P-2b84b0ec labelled plant_pot the model says chair (probability of the label 0.12)
P-1c8c6fb7 labelled candle the model says lamp (probability of the label 0.14)
P-1c7efe8e labelled vase the model says plant_pot (probability of the label 0.14)
P-157311db labelled plant_pot the model says vase (probability of the label 0.19)
Planted label errors in your top 10: 4 of 4

The simple rule flags 34 photos, against 12 for cleanlab with the same embeddings, because it flags every close call too. Its top 10 still holds the 4 planted errors. Now your project matches the snapshot end-of-m03: vision probe and vision labels, with 14 tests.

Omar asks again: “Next month we add garden lanterns. How many photos must we label?” Priya’s plan uses the whole module:

  1. The guide first, with the lantern’s confusions with lamp and candle. Two labellers check it on the same 50 photos, until kappa is about 0.8.
  2. About 20 varied photos per class. The learning curve reached 0.929 with 20 per class. Five per class gave 0.869, and one draw gave only 0.819.
  3. Balanced classes, or balanced class weights: unbalanced classes gave 0.894 and found only 12 of 20 candles.
  4. The suspect search before each training run, with a person looking at each suspect.
  5. The lantern recall on validation photos. Label more only if it stays low.

When to use these checks, and what they cost

Section titled “When to use these checks, and what they cost”

Use a guide and an agreement check for every labelling job: they cost a few hours. Use the suspect search whenever labels come from someone else or from a public dataset. It only finds labels that disagree with the embedding, so it cannot find errors between classes that the embedding mixes. And a person must look at each suspect: 11 here, many more in a large catalogue.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The first installs the packages, with cleanlab.
  3. Run section 1. You see agreement 0.833 and kappa 0.667.
  4. Run section 2. You see 11 suspects and the table of the first ten.
  5. Predict how many of the ten are real errors. Then run the next cell, and look at the ten photos.

In section 3 of the notebook, look at the photos labelled vase among the first ten suspects. Write the IDs of those that clearly show a plant growing in soil in a pot into fixed_ids, sorted. There are four. Two other vase photos are hard cases: a glass globe with an air plant and a garden urn. Keep their labels, and write them down for the guide. Then run section 4 to measure the change.

Show the answer
fixed_ids = ["P-1c7efe8e", "P-2760f082", "P-2b646d6f", "P-2f04b07d"]

The check cell prints Check passed: fixed_ids is right., and the sealed key confirms 4 of 4. Section 4 then prints:

labels as given - test accuracy 0.894 - suspects 11
labels fixed - test accuracy 0.894 - suspects 5

The test score does not move, and the suspect list shrinks from 11 to 5. The five that are left are hard cases and right labels that the probe doubts: a decision for the guide, not for the code.

Common mistake

Let the model fix the labels

Symptom: Tomás sees that the four planted errors were in the top 10. “The model is right,” he says. “Let’s change every suspect to the model’s answer.” He writes one line of code that changes all 11 suspects. The probe trained on his new labels scores 0.894 on the test, as before, and the next search finds no suspect at all.

Cause: only 4 of the 11 suspects were real errors. His line changed 7 other labels without anyone looking at the photos. The chair with a jacket on it became a cushion. A small bowl of tea leaves became a mug. The test score did not show it, because 7 changes among 453 photos are too few to move the probe.

Fix: treat each suspect as a question for a person. Look at the photo, decide with the guide, and record the reason. Add the hard cases to the guide.

Consequence in AI work: when a model corrects its own training labels, its blind spots become the “truth”. Every later model learns them, and no later search shows them, because the labels and the model now agree.

Check your understanding

Two labellers agree on 10 of 12 photos, and each says plant pot for half of them. Compute Cohen's kappa, and say what you would do before the large labelling job.

Show the answer

The agreement is 10 / 12 = 0.833. Two random guessers who each say plant pot half the time agree 0.5 × 0.5 + 0.5 × 0.5 = 0.5 of the time. Kappa = (0.833 − 0.5) / (1 − 0.5) = 0.667. That is below the common target of about 0.8. So look at the two photos where they disagree, and decide them. Write the deciding rule and the photos into the guide. Then measure again on new shared photos before the large job starts.

Check your understanding

Why must the suspect search use out-of-fold probabilities, and not the probabilities of a probe trained on all the photos?

Show the answer

A probe trained on a photo has seen its label, and tends to agree with it, even when the label is wrong. The wrong label then gets a high probability and does not look suspect. With 5-fold cross-validation, each photo is scored by a probe that never saw its label. The probability then shows how well the label fits the other photos.

Check your understanding

Larkfield adds garden lanterns next month. Recommend how many photos to label first, and how to keep the labels good, with evidence from this module.

Show the answer

Guide first: the lantern rules and its confusions with lamp and candle. Two labellers check it until kappa is about 0.8. Then about 20 varied photos per class. The CLIP probe reached 0.929 with 20 per class. Five per class gave 0.869, with draws from 0.819 to 0.925. Keep the classes balanced or use balanced weights: unbalanced classes gave 0.894 and candle recall 0.60. Run the suspect search before each training run, with a person looking at each suspect. Then measure the lantern recall on validation photos, and label more only if it stays low.

Recap

  • A label is a decision. A labelling guide gives the deciding question, the rules, the hard cases and an “unsure” way out.
  • Cohen’s kappa measures agreement above chance: 0.667 in the example. Aim for about 0.8.
  • Confident learning ranks training photos by the out-of-fold probability of their own label. On the catalogue, cleanlab flags 11 photos (CLIP) and 12 (DINOv2), with all 4 planted errors in the top 10.
  • A suspect is a question for a person: 7 of the 11 were not errors, and some were right labels.
  • Fixing the 4 errors did not move the test accuracy (0.894) but shrank the suspect list from 11 to 5. Fix labels anyway: errors grow with their share, and they distort every score.
  • Similar embeddings also find photos uploaded twice: 5 to 6 of the 6 planted pairs.
  • For a new category: guide, about 20 varied photos per class, balanced classes, a suspect search, then measure.

You can now classify catalogue photos with few labels, and trust those labels. But each photo still gets one answer for the whole photo. Next, you find where each object is, and count the boxes on a pallet.

Next: Boxes and IoU