Active learning
- Lesson 19
- 35 min
- Module 6: Learned matching: classifiers, embeddings and language models
Where we are
Our classifier learned from labels, and labels cost a reviewer's time. Some labels teach the model more than others. In this lesson, we label the pairs that the model is least sure about, and count how many labels it takes to beat Splink.
Lesson 19 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Active learning
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Choose which pairs a reviewer labels: the pairs that the model is least sure about. Simulate a reviewer, and compare these labels with the same number of random labels. Count how many labels it takes to beat Splink, and to beat the classifier on all 3,000 labels.
NotebookSimulate active learning in a notebook
The notebook makes a pool of training pairs with six features each. You write the function that picks the pairs closest to 0.5. Then it runs eight rounds of 40 labels, for uncertain pairs and for random pairs, and draws the two curves. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Logistic regression, LightGBM and the threshold from the validation pairs, from A supervised pair classifier.
- F1 and the test split, from Measure a matcher.
What you need on your computer
- The project
larkfield-matchwith the files of the last lesson. Or use the notebook only.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.
A student asks the right questions
Section titled “A student asks the right questions”A student prepares for an exam. Her teacher has time for only ten questions. Which questions should she ask?
Not the ones she can already answer. “What is 2 + 2?” teaches her nothing. Not random ones either: most random questions are easy. She asks about the problems where she is unsure. Each answer there changes what she knows.
A model can do the same. It can tell us which pairs it is unsure about, and a reviewer labels those first.
The problem: labels cost time, and most teach little
Section titled “The problem: labels cost time, and most teach little”A label costs a reviewer’s time. Ivana needs about 1.2 minutes for a pair and reviews about 300 pairs a day (both our assumptions, from data/costs.json). So:
- 3,000 labels take 3,000 × 1.2 = 3,600 minutes: 60 hours, or 10 of her working days.
- At 40 dollars an hour (our assumption), that is 2,400 dollars.
And most random pairs teach the model little. About 1 candidate pair in 40 is the same customer. A random pair is usually two strangers with nothing in common. The model already gets these right. Labelling them confirms what the model knows.
The question of this lesson: can a few well-chosen labels do the work of many random ones?
The idea: label where the model is unsure
Section titled “The idea: label where the model is unsure”Active learning is a loop. The model chooses which examples a person labels next. The most common way to choose is uncertainty sampling: label the examples that the model is least sure about.
For a pair classifier, “least sure” means a score close to 0.5. A score of 0.02 says “surely different”. A score of 0.98 says “surely the same”. A score of 0.48 says “I do not know”.
A tiny worked example
Section titled “A tiny worked example”The model scores four pairs that nobody has labelled yet. We may ask for two labels.
| Pair | Score | Distance from 0.5 | Label it? |
|---|---|---|---|
| A | 0.10 | 0.40 | no |
| B | 0.48 | 0.02 | yes (first) |
| C | 0.90 | 0.40 | no |
| D | 0.55 | 0.05 | yes (second) |
We sort the pairs by their distance from 0.5 and take the two closest: B and D. The reviewer labels them. The model trains again with the new labels, scores the pool again, and the loop goes on.
Text description of the diagram
Four boxes in a circle, joined by arrows. Train on the labels so far. Score the pool of unlabelled training pairs. Pick the 40 pairs with a score closest to 0.5. A reviewer labels them. The worked example sits next to the third box. Of the scores 0.10, 0.48, 0.90 and 0.55, the two closest to 0.5 are 0.48 and 0.55. A side note says that the reviewer in the simulation is the true answer, so the reviewer never makes a mistake.
How the simulation works
Section titled “How the simulation works”We cannot ask Ivana to label pairs for a lesson. So match active simulates her. Our records are synthetic, and the true customer of each record is known. When the loop “asks” for a label, it reads the true answer. This reviewer never makes a mistake; a real one does (Module 8 measures it).
- The pool: every candidate pair of the training customers.
- The start: 20 labels. Half are pairs that share an email or a phone (probably true pairs), half are random. So both answers are in the first labels.
- Each round: train logistic regression on the labels so far, then add 40 labels.
- The comparison: the same loop, but with 40 random pairs each round.
After each round, it measures F1 on the test pairs. The threshold comes from the validation pairs, as in the last lesson.
Run it in the project
Section titled “Run it in the project”-
Copy the two new files from the snapshot
end-of-m06:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m06cp $S/larkfield_match/active.py larkfield_match/cp $S/larkfield_match/commands/active.py larkfield_match/commands/Terminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m06"Copy-Item $S\larkfield_match\active.py larkfield_match\Copy-Item $S\larkfield_match\commands\active.py larkfield_match\commands\No new package is needed.
-
Run the simulation. It trains 26 models, and each one scores about 1.1 million pairs, so it takes a few minutes:
Terminal window match activeTest F1 (threshold: best on validation / 0.5)labels uncertain pairs random pairs20 0.865 / 0.857 0.865 / 0.85760 0.809 / 0.799 0.844 / 0.840100 0.905 / 0.900 0.840 / 0.831140 0.914 / 0.915 0.843 / 0.843180 0.913 / 0.909 0.843 / 0.841220 0.921 / 0.919 0.842 / 0.838260 0.921 / 0.922 0.853 / 0.842300 0.924 / 0.715 0.874 / 0.845340 0.926 / 0.692 0.872 / 0.840380 0.923 / 0.921 0.873 / 0.840420 0.924 / 0.922 0.873 / 0.846460 0.925 / 0.922 0.873 / 0.845500 0.924 / 0.922 0.874 / 0.845On our busy laptop, it took about 5 minutes and 1.3 GB of memory.
Each line is one round. The first number in each column uses the threshold from the validation pairs. The second uses the fixed threshold 0.5.
What the labels bought
Section titled “What the labels bought”- uncertain pairs first
- random pairs
- Splink, no labels (0.797)
Text description of the chart
Two lines over 20 to 500 labels, and a dashed line at Splink’s F1 of 0.797. Both lines start at 0.865 with the same 20 labels. The uncertain line dips to 0.809 at 60 labels. Then it rises: 0.905 at 100 labels, 0.914 at 140, and about 0.92 from 220 on. The random line stays between 0.840 and 0.874. Both lines are above Splink at every point.
Three results:
- 140 uncertain labels give F1 0.914. That beats logistic regression on all 3,000 labelled pairs (0.911). It also beats LightGBM on them (0.902). 140 labels take Ivana about 2.8 hours; 3,000 take 60.
- 140 random labels give 0.843. Random pairs are mostly easy strangers, so each label teaches little.
- Even 20 labels beat Splink (0.865 against 0.797), with either way of choosing. Labels are worth a lot, even a few.
Why do the 3,000 labelled pairs do worse than 140 chosen ones? They were sampled by surname and region, not where the model is unsure. Many of them are easy, and their mix does not look like the hard pairs.
The curve is not smooth
Section titled “The curve is not smooth”At 60 labels, the uncertain curve drops to 0.809. We found no single cause. At that point, 40 of the 60 labels are hard pairs from one round. A model trained on so few labels can change a lot from one round to the next. More rounds fix it. Do not stop after one round, and do not judge after one round.
Choose the threshold every round
Section titled “Choose the threshold every round”The last lesson said: never use the threshold 0.5. Active learning shows why again.
- threshold from validation
- fixed threshold 0.5
Text description of the chart
Two lines over 20 to 500 labels. Both are close until 260 labels (about 0.92). At 300 labels, the fixed-threshold line falls to 0.715, and at 340 to 0.692. The chosen-threshold line stays at 0.924 and 0.926. From 380 labels on, both are close again (about 0.92).
Every round changes the labels, and the model’s scores move as a whole. We counted the test candidate pairs with a score of 0.5 or more. At 260 labels, there are 5,784. At 300 labels, there are 9,126. Only 5,531 test candidate pairs are true pairs. So a fixed threshold of 0.5 suddenly calls thousands of extra pairs the same. The best threshold on the validation pairs moved from 0.59 to 0.72 in that round. A threshold chosen after every round follows the model.
What the simulation does not count
Section titled “What the simulation does not count”Be honest about what this costs in real work:
- A validation set needs labels too. Here, the threshold uses the true answers of all validation pairs. In real work, Ivana must also label a random sample of candidate pairs to choose the threshold and to measure. That is a fixed cost, not counted in the 140.
- Uncertain pairs are hard pairs. A reviewer may need more than 1.2 minutes for them, and may make more mistakes. We did not measure this.
- The chosen labels are not a fair sample. Never measure the model on them: they are the hardest pairs only. Module 8 shows what happens when review labels are used without care.
When to use active learning
Section titled “When to use active learning”| Use it when | Do not use it when | What it costs |
|---|---|---|
| labels are expensive, and you can label in rounds | you need a fair sample to measure: label random pairs for that | one training and one scoring per round: about 12 s on our laptop for 1.1 million pairs (logistic regression) |
| the model already gives useful scores, from a few starting labels | the reviewer cannot wait for rounds (one big batch only) | a reviewer’s time on hard pairs: more minutes per pair, more mistakes |
| most pairs are easy, and a few are hard (the usual case in matching) | the model is a poor judge of its own doubt | a separate labelled set to choose the threshold |
Common mistake
Measure the model on the labels you chose
Symptom: after 300 uncertain labels, TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. measures the model on those 300 pairs, with cross-validation. F1 is about 0.7. He concludes that the model is weak and buys more labels.
Cause: the uncertain pairs are the hardest pairs of all, by design. A score on them says nothing about the 1.1 million candidate pairs, where most pairs are easy. (The 0.7 here is a constructed example, not a run.) The same trap works the other way: labels sampled where a model is sure make it look too good.
Fix: keep a separate labelled sample, drawn at random from the candidate pairs of other customers, for the threshold and the measure. In the project, the validation and test splits play this role.
Consequence in AI work: any data that a model chose for itself is biased. This is true for active learning, for review queues and for user feedback.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does a label on a pair with score 0.48 teach the model more than a label on a pair with score 0.02?
Show the answer
At 0.02, the model is already sure that the pair is different, and it is almost always right. The label confirms what it knows and changes little. At 0.48, the model cannot decide. The label tells it which side of the line the pair belongs on, and similar pairs then move too. In matching, most candidate pairs are easy strangers, so random labels are mostly of the first kind.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
140 uncertain labels give F1 0.914; the 3,000 labelled pairs give 0.911 with logistic regression. Does this mean that labels 141 to 3,000 are worthless? What would you tell Omar?
Show the answer
No. It means that which pairs we label matters more than how many. The 3,000 were sampled by surname and region, not where the model is unsure, so many were easy. The curve still rises a little after 140 (about 0.92 from 220 labels). Tell Omar three things. Start with a few hundred labels that the model chooses, in rounds. Keep a separate random sample to measure. Stop when the curve stops rising on validation. That costs a few hours of review, not ten days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With the fixed threshold 0.5, F1 falls to 0.692 at 340 labels; with the threshold from validation it is 0.926. What changed between the rounds, and what is the rule?
Show the answer
Each round adds 40 hard pairs, and the model is trained again. This can move all its scores up or down, so the point where “same” begins moves too. At 260 labels, 5,784 test pairs score 0.5 or more; at 300 labels, 9,126. A fixed 0.5 is then sometimes far from the best threshold. The rule: choose the threshold again on the validation pairs every time the model is trained again.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Active learning: the model chooses which pairs a person labels next. Uncertainty sampling picks the pairs with a score closest to 0.5.
- In our simulation, the reviewer is always right. There, 140 uncertain labels give F1 0.914: better than all 3,000 labelled pairs (0.911 and 0.902). 140 random labels give 0.843.
- Even 20 labels beat Splink without labels (0.865 against 0.797).
- The curve is not smooth. Choose the threshold on validation after every round: with 0.5, F1 fell to 0.692.
- Keep a random, labelled sample to choose the threshold and to measure. Never measure on the pairs the model chose.
Next, we try a different kind of feature: embeddings from a small language model, for names, addresses and product titles.