Skip to content
Français

When you have few labels

  • Lesson 8
  • 35 min
  • Module 2: Labels you can trust

Where we are

We can now find and fix wrong labels. Sometimes the problem is the opposite: we have too few labels. In this lesson, we label first the messages that the model is least sure about, and we let an LLM suggest labels that a person checks before we train on them. A new sorter who asks only about the letters he cannot place shows the idea.

Lesson 8 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceWhen you have few labels

What you will be able to do

Get a good model from few labels: let the model choose which messages a person labels next, and use an LLM’s labels only after a person has checked them.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson: the first 70 labels, what each strategy picks, the active-learning loop, the LLM’s recorded labels, and the “more or better labels” comparison. Its setup cells install pandas and scikit-learn and download the practice data. No key is needed: the LLM’s answers were recorded once. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Wrong labels, what they cost and how to find them, from Find noisy labels.
  • A model’s confidence: the highest of its probabilities for the routes. Thresholds and calibration in the machine learning course explains predicted probabilities; this lesson uses only the idea.

What you need on your computer

  • Nothing new: this lesson runs in the notebook. The project does not change.

Think of a new worker at a post office. On his first morning, the supervisor shows him ten letters for each town. Then he starts to sort. He is allowed to ask the supervisor about some letters, but she is busy.

Which letters should he ask about? Not the clear ones: “Main Street, Springfield” teaches him nothing new. He should ask about the letters he cannot place: an old address, a smudged postcode, a street name that exists in two towns. Each answer to one of those questions teaches him a rule he did not know. After a week, he sorts well, and he asked far fewer questions than a worker who asked about letters at random.

Larkfield’s 6,000 training messages have labels because this course built them. A company that starts a new router has messages, but few labels. Each label is a person’s time. At the course’s review cost of US$1.50 per message (an assumption), labelling all 6,000 costs about US$9,000.

Suppose Chloé labels only 10 messages of each route: 70 labels. The TF-IDF model trained on them reaches 0.524 on the test set. It is far from the 0.962 of the full labels. She must label more. But which messages? The usual answer is “a random sample”. This lesson asks whether the model can choose better.

Let the model choose, as the new sorter did. Train it on the labels you have. Score every message that has no label yet. Then give the person the messages the model is least sure about. Add their labels, train again, and repeat.

This is active learning: the model takes an active part in choosing its own training data. The rule “least sure first” is called uncertainty sampling. Here, the model’s uncertainty is its confidence: the highest of its seven probabilities. A low confidence means the model cannot place the message.

A loop. Start: 70 labelled messages, 10 per route. Train the model. Score every unlabelled message. Pick the 100 messages the model is least sure about. Chloé labels them. An arrow labelled add them, and repeat goes back to train the model.
Active learning with uncertainty sampling. Each round trains the model, scores the pool of unlabelled messages, and sends the 100 least confident to Chloé.
Text description of the diagram

The loop starts with 70 labelled messages, 10 for each route. The model is trained on them. It scores every message that has no label yet. The 100 messages that it is least sure about go to Chloé, who labels them. An arrow labelled “add them, and repeat” goes back to training the model. Each turn of the loop adds 100 labels.

The model trained on 70 labels scores four unlabelled messages. This is a practice problem we made on purpose.

Message Highest probability (route) Confidence Label it now?
“Where is my parcel? Tracking has not moved for a week.” 0.92 (delivery) 0.92 No: the model already knows
“Please cancel order LK-1234 before it ships.” 0.85 (orders) 0.85 No
“Can the courier leave it with my neighbour? Also please update my phone number.” 0.31 (delivery), 0.28 (account) 0.31 Yes
“See my last message about the other thing, please.” 0.19 (account) 0.19 Yes, first

With room for two labels, Chloé labels the last two. The first two would only confirm what the model knows. The last two show it something new: a message with two requests, and a vague message.

The notebook runs this loop on the real data. It starts from the same 70 labels. One copy of the loop adds 100 random messages per round; the other adds the 100 least confident. Before the loop, look at what each one picks first:

Route 100 least confident 100 random
orders 5 18
delivery 22 25
returns_refunds 28 16
billing 9 11
product 11 15
account 15 10
needs_person 10 5

The least confident 100 have confidence 0.203 or less; the median message in the pool has 0.379. They come more often from returns_refunds, account and needs_person: the routes with hard cases and few examples. They come rarely from orders, which the model already places well. The three least confident are a parcel message with an address change, a vague “see my last message”, and a password problem.

Ten rounds take each copy from 70 to 1,070 labels:

  • Least confident first
  • Random
TF-IDF with logistic regression, test accuracy after each round. Least confident first passes 90% at 670 labels; random needs 970.
Text description of the chart

A line chart of test accuracy against the number of labelled messages, from 70 to 1,070. Both lines start at 52.4% with 70 labels. Up to 270 labels, they are close: 69.7% and 70.5% at 170, 80.5% and 80.8% at 270. From 370 labels, the least confident line is higher: 85.5% against 84.8% at 370, 90.8% against 88.7% at 670, and 93.8% against 90.7% at 1,070. The random line first passes 90% at 970 labels (90.2%).

In the first rounds, the two lines are close: with 170 labels, random is even a little higher. The model is still too weak to know what it does not know. From 370 labels, least confident first pulls ahead: 0.855 against 0.848 at 370, 0.908 against 0.887 at 670, and 0.938 against 0.907 at 1,070.

Read the chart sideways too. Least confident first passes 0.90 at 670 labels. Random needs 970. For the same accuracy here, Chloé writes 300 fewer labels, about US$450 at US$1.50 each.

Remember that this inbox is easier than a real one. On a real inbox, the curves are lower and the gap can be larger or smaller. The case study Sort requests with few labels tries few labels on real requests in English and French.

Use it when labels are expensive and unlabelled messages are many: a new router, a new route, a new language. It needs a model that trains in seconds, because the loop trains it after each round. TF-IDF fits that well.

It has three costs. First, the picked messages are not a random sample: they are the hard ones. So never measure accuracy on them; keep a separate random test set, as this loop does. Second, the picks depend on the model that chose them. A different model later may have wanted other messages. Third, the hard messages are harder for the person too: Chloé needs more time per label, and she meets the guidelines’ gaps more often.

There is a faster way to get labels: ask an LLM, a large language model. A small hosted LLM labelled 5,000 training messages, 20 messages per call, in 250 calls. The whole job cost US$0.081. Each answer has a route and a confidence that the model wrote in its answer. The answers were recorded once, so the notebook replays them with no key.

How good are they? 91.4% of the LLM’s labels agree with the reviewed labels. A person who checks a random sample of 200 finds 90.5%, close to the full number. So 429 of the 5,000 labels are wrong. The commonest errors are product questions labelled returns_refunds (64) and orders labelled product (55). That is the keyword trap of Labelling guidelines: the LLM labels the words “return” or “item”, not the request.

That is a little below Grace’s 91.7% on the double-labelled sample, and far below Chloé’s 96.1%. Cheap labels with 429 errors are noisy labels, like the old queue’s. They need a check before you train on them.

The LLM wrote a confidence with each label. Does a low confidence mean a wrong label?

The LLM's 5,000 labels, grouped by the confidence it wrote. Low confidence means more errors, but even the top group has 5% errors.
Text description of the chart

A column chart on an axis from 0% to 100%. Labels with a written confidence below 0.80: 336 labels, 66.4% agree with people. From 0.80 to 0.90: 333 labels, 75.7%. From 0.90 to 0.95: 336 labels, 89.0%. 0.95 or more: 3,995 labels, 95.0%.

It helps. Below 0.80, only 66.4% of the labels are right; at 0.95 or more, 95.0%. So a person should review the LLM’s least confident labels first, as in active learning:

LLM labels reviewed, least confident first Errors left of 429 Review cost
0 429 US$0
250 333 US$375
500 267 US$750
1,000 198 US$1,500
2,000 98 US$3,000

But look at the top group again. It has 3,995 labels, and 5.0% of them are wrong: 198 errors written with a confidence of 0.95 or more. A review by confidence never reaches them before it has reviewed 1,000 other labels. This is also why a written confidence is not a probability until you have checked it on labelled data.

So the plan for LLM labels has two parts. Review the least confident labels, because there the errors are dense. And check a random sample, because some errors hide behind a high confidence. The random sample tells you the true error rate of the labels you keep.

Guided practice: run the loop and check the LLM

Section titled “Guided practice: run the loop and check the LLM”
  1. Open the notebook from the box Do this lesson in a notebook, and run the setup cells.
  2. In section 1, run the cell. You see 70 labels: test accuracy 0.5235.
  3. In section 2, predict which routes the least confident 100 come from. Then run the cell. You see 28 returns_refunds and 5 orders in the first column.
  4. In section 3, run the loop. It trains 22 models in about a minute. At 1,070 labels, you see 0.907 for random and 0.938 for least confident.
  5. Run the chart cell. The two lines separate after about 370 labels.
  6. In section 4, do the Your turn task: the smallest number of labels where random reaches 0.90.
  7. In section 5, run the LLM cells. You see agree with people: 91.4% and the four confidence groups.
  8. In section 6, count the wrong LLM labels with a confidence of 0.95 or more. The check prints Check passed.
Show the answers to steps 6 and 8
random_needs = int(table.index[table["random"] >= 0.90].min())
print(random_needs)
970
sure_but_wrong = int((~llm["agree"] & (llm["confidence"] >= 0.95)).sum())
print(sure_but_wrong)
198

This module has one big question: when you can spend more on labels, should you get more labels or better labels? The data can answer it. The same model learns from a growing number of old, noisy labels, and from the same numbers of reviewed labels:

  • Reviewed labels
  • Old queue labels (11.2% wrong)
TF-IDF with logistic regression on the same test messages. 2,000 reviewed labels beat all 6,000 old labels.
Text description of the chart

A line chart of test accuracy against the number of training labels, from 500 to 6,000. Reviewed labels: 84.5% with 500, 91.0% with 1,000, 93.4% with 2,000, 95.6% with 4,000 and 96.2% with 6,000. Old queue labels: 79.2% with 500, 85.2% with 1,000, 88.8% with 2,000, 90.1% with 4,000 and 91.3% with 6,000. The reviewed line is above the old line at every size. The reviewed point at 2,000 is above the old point at 6,000.

Here, better labels win. With 6,000 old labels, the model reaches 0.913. With only 2,000 reviewed labels, a third as many, it reaches 0.934. The cost of mistakes says the same: US$1,206 per 1,000 messages for the 6,000 old labels, US$655 for the 2,000 reviewed ones. And the old-label curve flattens: from 4,000 to 6,000 old labels, the accuracy rises only from 0.901 to 0.913. More noisy labels teach the model more of the noise.

This answer belongs to this inbox. The old queue here has 11.2% wrong labels. With fewer wrong labels, more labels win sooner; with more, better labels win by more. Run the same two curves on your own data before you decide.

Common mistake

Train on the LLM's labels because it said it was sure

Symptom: Tomás wants to skip the review. “The LLM labelled 5,000 messages for 8 cents. Let’s keep only the labels with a confidence of 0.95 or more. That is 3,995 labels, and the LLM is sure about them.”

Cause: “Sure” is a word the LLM wrote, not a measured probability. The 3,995 labels at 0.95 or more include 198 wrong ones (5.0%). The errors are not random: product questions about returns go to returns_refunds, orders go to product. A model trained on them learns those pairs as rules.

Fix: Check before you train. Review the least confident labels first, where errors are dense, and check a random sample of the rest to measure the true error rate. Then treat the remaining labels like any noisy labels: run confident learning on them, and review the flags.

Consequence in AI work: Cheap labels move the cost from labelling to mistakes. The model copies the LLM’s blind spots, and a test set labelled by the same LLM would not show them. Only labels checked by people show the real error rate.

Check your understanding

Why does active learning pick the messages the model is least sure about, and why should you never measure accuracy on those messages?

Show the answer

A message the model is already sure about teaches it little. A message it cannot place shows it something new, such as a hard case or a rare route. That is why least confident first reached 0.90 at 670 labels and random at 970. But the picked messages are the hard ones, not a random sample. Accuracy on them is too low, and it does not describe the inbox. Measure on a separate random test set.

Check your understanding

An LLM labels 5,000 messages and writes a confidence with each label. A colleague proposes to review only the labels below 0.95. What does the review catch, what does it miss, and what do you add?

Show the answer

The review covers the 1,005 labels below 0.95, where errors are dense (66.4% right below 0.80). It fixes the errors there: 429 errors fall to 198. It misses the 198 wrong labels that the LLM wrote with a confidence of 0.95 or more. Add a check of a random sample of the labels you keep, to know their real error rate. Then run confident learning on them to find likely errors.

Check your understanding

Completion check: does Larkfield need more labels or better labels? Give the evidence.

Show the answer

Better labels. TF-IDF with logistic regression learns from all 6,000 old queue labels (11.2% wrong). It reaches 0.913 on the test set, US$1,206 per 1,000 messages. Trained on only 2,000 reviewed labels, it reaches 0.934, US$655. The reviewed curve is above the old curve at every size. The old curve flattens: 0.901 at 4,000, 0.913 at 6,000. So the money is better spent on reviewing labels, for example the 622 flags of confident learning, than on more unchecked labels. This holds for this inbox; repeat the comparison on other data.

Recap

  • Labels cost a person’s time. With 70 labels, TF-IDF reaches only 0.524.
  • Active learning lets the model choose: label first the messages it is least sure about (uncertainty sampling). Here, it passes 0.90 at 670 labels; random needs 970. Keep a random test set: the picked messages are the hard ones.
  • An LLM labelled 5,000 messages for US$0.081, and 91.4% agree with people. Its written confidence helps to order a review, but 198 wrong labels have a confidence of 0.95 or more. Review the least confident, and check a random sample.
  • More or better labels? Here, better: 2,000 reviewed labels (0.934, US$655) beat 6,000 old labels (0.913, US$1,206).

You can now trust your labels: guidelines, agreement, noisy labels and few labels. But a model is only one way to use them. Next, we start the baselines with the simplest router of all: keyword rules that people can read and change.

Next: Keyword rules and regular expressions