Skip to content
Français

Active learning

  • Lesson 19
  • 35 min
  • Module 6: Learned matching: classifiers, embeddings and language models

Where we are

Our classifier learned from labels, and labels cost a reviewer's time. Some labels teach the model more than others. In this lesson, we label the pairs that the model is least sure about, and count how many labels it takes to beat Splink.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Choose which pairs a reviewer labels: the pairs that the model is least sure about. Simulate a reviewer, and compare these labels with the same number of random labels. Count how many labels it takes to beat Splink, and to beat the classifier on all 3,000 labels.

NotebookSimulate active learning in a notebookColab · Kaggle · GitHub

The notebook makes a pool of training pairs with six features each. You write the function that picks the pairs closest to 0.5. Then it runs eight rounds of 40 labels, for uncertain pairs and for random pairs, and draws the two curves. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of the last lesson. Or use the notebook only.

Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.

A student prepares for an exam. Her teacher has time for only ten questions. Which questions should she ask?

Not the ones she can already answer. “What is 2 + 2?” teaches her nothing. Not random ones either: most random questions are easy. She asks about the problems where she is unsure. Each answer there changes what she knows.

A model can do the same. It can tell us which pairs it is unsure about, and a reviewer labels those first.

The problem: labels cost time, and most teach little

Section titled “The problem: labels cost time, and most teach little”

A label costs a reviewer’s time. Ivana needs about 1.2 minutes for a pair and reviews about 300 pairs a day (both our assumptions, from data/costs.json). So:

  • 3,000 labels take 3,000 × 1.2 = 3,600 minutes: 60 hours, or 10 of her working days.
  • At 40 dollars an hour (our assumption), that is 2,400 dollars.

And most random pairs teach the model little. About 1 candidate pair in 40 is the same customer. A random pair is usually two strangers with nothing in common. The model already gets these right. Labelling them confirms what the model knows.

The question of this lesson: can a few well-chosen labels do the work of many random ones?

Active learning is a loop. The model chooses which examples a person labels next. The most common way to choose is uncertainty sampling: label the examples that the model is least sure about.

For a pair classifier, “least sure” means a score close to 0.5. A score of 0.02 says “surely different”. A score of 0.98 says “surely the same”. A score of 0.48 says “I do not know”.

The model scores four pairs that nobody has labelled yet. We may ask for two labels.

Pair Score Distance from 0.5 Label it?
A 0.10 0.40 no
B 0.48 0.02 yes (first)
C 0.90 0.40 no
D 0.55 0.05 yes (second)

We sort the pairs by their distance from 0.5 and take the two closest: B and D. The reviewer labels them. The model trains again with the new labels, scores the pool again, and the loop goes on.

A loop with four steps. 1: train logistic regression on the labels so far (20 at the start). 2: score every unlabelled pair in the pool. 3: pick the 40 pairs closest to 0.5 (example: scores 0.10, 0.48, 0.90 and 0.55; the closest two are 0.48 and 0.55). 4: the reviewer labels them (in the simulation, the true answer). Then back to step 1. After each round, the test F1 is measured with a threshold chosen on the validation pairs.
The active-learning loop that the command match active simulates: 12 rounds of 40 labels, after 20 starting labels.
Text description of the diagram

Four boxes in a circle, joined by arrows. Train on the labels so far. Score the pool of unlabelled training pairs. Pick the 40 pairs with a score closest to 0.5. A reviewer labels them. The worked example sits next to the third box. Of the scores 0.10, 0.48, 0.90 and 0.55, the two closest to 0.5 are 0.48 and 0.55. A side note says that the reviewer in the simulation is the true answer, so the reviewer never makes a mistake.

We cannot ask Ivana to label pairs for a lesson. So match active simulates her. Our records are synthetic, and the true customer of each record is known. When the loop “asks” for a label, it reads the true answer. This reviewer never makes a mistake; a real one does (Module 8 measures it).

  • The pool: every candidate pair of the training customers.
  • The start: 20 labels. Half are pairs that share an email or a phone (probably true pairs), half are random. So both answers are in the first labels.
  • Each round: train logistic regression on the labels so far, then add 40 labels.
  • The comparison: the same loop, but with 40 random pairs each round.

After each round, it measures F1 on the test pairs. The threshold comes from the validation pairs, as in the last lesson.

  1. Copy the two new files from the snapshot end-of-m06:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m06
    cp $S/larkfield_match/active.py larkfield_match/
    cp $S/larkfield_match/commands/active.py larkfield_match/commands/

    No new package is needed.

  2. Run the simulation. It trains 26 models, and each one scores about 1.1 million pairs, so it takes a few minutes:

    Terminal window
    match active
    Test F1 (threshold: best on validation / 0.5)
    labels uncertain pairs random pairs
    20 0.865 / 0.857 0.865 / 0.857
    60 0.809 / 0.799 0.844 / 0.840
    100 0.905 / 0.900 0.840 / 0.831
    140 0.914 / 0.915 0.843 / 0.843
    180 0.913 / 0.909 0.843 / 0.841
    220 0.921 / 0.919 0.842 / 0.838
    260 0.921 / 0.922 0.853 / 0.842
    300 0.924 / 0.715 0.874 / 0.845
    340 0.926 / 0.692 0.872 / 0.840
    380 0.923 / 0.921 0.873 / 0.840
    420 0.924 / 0.922 0.873 / 0.846
    460 0.925 / 0.922 0.873 / 0.845
    500 0.924 / 0.922 0.874 / 0.845

    On our busy laptop, it took about 5 minutes and 1.3 GB of memory.

Each line is one round. The first number in each column uses the threshold from the validation pairs. The second uses the fixed threshold 0.5.

  • uncertain pairs first
  • random pairs
  • Splink, no labels (0.797)
Test F1 of logistic regression after each round, on our synthetic records, threshold from the validation pairs. Uncertain pairs reach 0.905 at 100 labels. Random pairs stay below 0.88 after 500.
Text description of the chart

Two lines over 20 to 500 labels, and a dashed line at Splink’s F1 of 0.797. Both lines start at 0.865 with the same 20 labels. The uncertain line dips to 0.809 at 60 labels. Then it rises: 0.905 at 100 labels, 0.914 at 140, and about 0.92 from 220 on. The random line stays between 0.840 and 0.874. Both lines are above Splink at every point.

Three results:

  1. 140 uncertain labels give F1 0.914. That beats logistic regression on all 3,000 labelled pairs (0.911). It also beats LightGBM on them (0.902). 140 labels take Ivana about 2.8 hours; 3,000 take 60.
  2. 140 random labels give 0.843. Random pairs are mostly easy strangers, so each label teaches little.
  3. Even 20 labels beat Splink (0.865 against 0.797), with either way of choosing. Labels are worth a lot, even a few.

Why do the 3,000 labelled pairs do worse than 140 chosen ones? They were sampled by surname and region, not where the model is unsure. Many of them are easy, and their mix does not look like the hard pairs.

At 60 labels, the uncertain curve drops to 0.809. We found no single cause. At that point, 40 of the 60 labels are hard pairs from one round. A model trained on so few labels can change a lot from one round to the next. More rounds fix it. Do not stop after one round, and do not judge after one round.

The last lesson said: never use the threshold 0.5. Active learning shows why again.

  • threshold from validation
  • fixed threshold 0.5
The same uncertain labels, measured with two thresholds. With the fixed threshold 0.5, F1 falls to 0.715 at 300 labels and 0.692 at 340, then recovers.
Text description of the chart

Two lines over 20 to 500 labels. Both are close until 260 labels (about 0.92). At 300 labels, the fixed-threshold line falls to 0.715, and at 340 to 0.692. The chosen-threshold line stays at 0.924 and 0.926. From 380 labels on, both are close again (about 0.92).

Every round changes the labels, and the model’s scores move as a whole. We counted the test candidate pairs with a score of 0.5 or more. At 260 labels, there are 5,784. At 300 labels, there are 9,126. Only 5,531 test candidate pairs are true pairs. So a fixed threshold of 0.5 suddenly calls thousands of extra pairs the same. The best threshold on the validation pairs moved from 0.59 to 0.72 in that round. A threshold chosen after every round follows the model.

Be honest about what this costs in real work:

  • A validation set needs labels too. Here, the threshold uses the true answers of all validation pairs. In real work, Ivana must also label a random sample of candidate pairs to choose the threshold and to measure. That is a fixed cost, not counted in the 140.
  • Uncertain pairs are hard pairs. A reviewer may need more than 1.2 minutes for them, and may make more mistakes. We did not measure this.
  • The chosen labels are not a fair sample. Never measure the model on them: they are the hardest pairs only. Module 8 shows what happens when review labels are used without care.
Use it when Do not use it when What it costs
labels are expensive, and you can label in rounds you need a fair sample to measure: label random pairs for that one training and one scoring per round: about 12 s on our laptop for 1.1 million pairs (logistic regression)
the model already gives useful scores, from a few starting labels the reviewer cannot wait for rounds (one big batch only) a reviewer’s time on hard pairs: more minutes per pair, more mistakes
most pairs are easy, and a few are hard (the usual case in matching) the model is a poor judge of its own doubt a separate labelled set to choose the threshold

Common mistake

Measure the model on the labels you chose

Symptom: after 300 uncertain labels, TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. measures the model on those 300 pairs, with cross-validation. F1 is about 0.7. He concludes that the model is weak and buys more labels.

Cause: the uncertain pairs are the hardest pairs of all, by design. A score on them says nothing about the 1.1 million candidate pairs, where most pairs are easy. (The 0.7 here is a constructed example, not a run.) The same trap works the other way: labels sampled where a model is sure make it look too good.

Fix: keep a separate labelled sample, drawn at random from the candidate pairs of other customers, for the threshold and the measure. In the project, the validation and test splits play this role.

Consequence in AI work: any data that a model chose for itself is biased. This is true for active learning, for review queues and for user feedback.

Check your understanding

Why does a label on a pair with score 0.48 teach the model more than a label on a pair with score 0.02?

Show the answer

At 0.02, the model is already sure that the pair is different, and it is almost always right. The label confirms what it knows and changes little. At 0.48, the model cannot decide. The label tells it which side of the line the pair belongs on, and similar pairs then move too. In matching, most candidate pairs are easy strangers, so random labels are mostly of the first kind.

Check your understanding

140 uncertain labels give F1 0.914; the 3,000 labelled pairs give 0.911 with logistic regression. Does this mean that labels 141 to 3,000 are worthless? What would you tell Omar?

Show the answer

No. It means that which pairs we label matters more than how many. The 3,000 were sampled by surname and region, not where the model is unsure, so many were easy. The curve still rises a little after 140 (about 0.92 from 220 labels). Tell Omar three things. Start with a few hundred labels that the model chooses, in rounds. Keep a separate random sample to measure. Stop when the curve stops rising on validation. That costs a few hours of review, not ten days.

Check your understanding

With the fixed threshold 0.5, F1 falls to 0.692 at 340 labels; with the threshold from validation it is 0.926. What changed between the rounds, and what is the rule?

Show the answer

Each round adds 40 hard pairs, and the model is trained again. This can move all its scores up or down, so the point where “same” begins moves too. At 260 labels, 5,784 test pairs score 0.5 or more; at 300 labels, 9,126. A fixed 0.5 is then sometimes far from the best threshold. The rule: choose the threshold again on the validation pairs every time the model is trained again.

Recap

  • Active learning: the model chooses which pairs a person labels next. Uncertainty sampling picks the pairs with a score closest to 0.5.
  • In our simulation, the reviewer is always right. There, 140 uncertain labels give F1 0.914: better than all 3,000 labelled pairs (0.911 and 0.902). 140 random labels give 0.843.
  • Even 20 labels beat Splink without labels (0.865 against 0.797).
  • The curve is not smooth. Choose the threshold on validation after every round: with 0.5, F1 fell to 0.692.
  • Keep a random, labelled sample to choose the threshold and to measure. Never measure on the pairs the model chose.

Next, we try a different kind of feature: embeddings from a small language model, for names, addresses and product titles.

Next: Embeddings for names, addresses and products