A supervised pair classifier
- Lesson 18
- 40 min
- Module 6: Learned matching: classifiers, embeddings and language models
Where we are
Splink learned its weights without labels. Larkfield's reviewers have now labelled some pairs. In this lesson, we train a pair classifier on comparison vectors, and we handle the strong class imbalance of pairs.
Lesson 18 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?A supervised pair classifier
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Train a model that reads a pair’s comparison vector and says how likely the two records are one customer. Train it on labelled pairs: logistic regression first, then gradient boosting with LightGBM. Handle the class imbalance of pairs: the threshold comes from the validation pairs, never from 0.5. Add both models to the comparison table.
NotebookTrain a pair classifier in a notebook
The notebook turns the 3,000 labelled pairs into six features each. It trains logistic regression and asks you to train LightGBM. Then it measures both at the threshold 0.5 on the candidate pairs, so you see what the class imbalance does. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The comparison vector of 16 fields, from Compare by field, then combine.
- Match weights, Splink and the review band, from Learn the weights without labels and From weights to a decision.
- What a classifier is, from Classification intuition in the machine learning course.
What you need on your computer
- The project
larkfield-matchwith the files of Module 5. Or use the notebook only. - About 150 MB of free disk space for two new packages.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.
A new clerk learns from old decisions
Section titled “A new clerk learns from old decisions”A clinic hires a new clerk. Her job: decide if a new patient form belongs to a patient who is already in the files.
Nobody gives her a rule book. Instead, a senior clerk shows her a box of old decisions. Each card has two forms and one word on it: “same” or “different”. The new clerk reads hundreds of cards. Soon she sees patterns that no rule book says. “Same surname, same address, different first name: usually a brother or a sister.” “Same name, new address, same date of birth: usually a patient who moved.”
This is supervised learning. A model learns from examples that a person has already answered. The answers are labels.
The problem: our weights do not learn from answers
Section titled “The problem: our weights do not learn from answers”Every method so far chose its weights without looking at true answers:
- The hand-weighted score of Module 4 used our guesses.
- Splink, in Module 5, estimated its weights from the data alone, with no labels.
Splink weighs each field on its own and adds the weights. It cannot learn that combinations matter. The brothers Samuel and Nathan Green show the problem. They share a surname, an email, a landline, a postcode, a street and a town. Each agreement adds weight. Splink gives them a match weight of 37.9, a probability of 1.0. They are two people.
TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield now has 3,000 labelled pairs. A reviewer has said “same” or “different” for each one. In our synthetic data, the labels come from the true customers. In Larkfield’s real work, they would come from Ivana’s review queue. Can a model learn from them?
The idea: a pair classifier
Section titled “The idea: a pair classifier”A classifier is a model that puts each input into one of a few classes. A pair classifier reads one candidate pair and puts it into one of two classes: “same” or “different”.
- The input is the pair’s comparison vector: the 16 numbers of Module 4. In machine learning, these inputs are called features.
- The output is a number from 0 to 1: how likely the pair is one customer. A pair at or above a threshold is called “the same”.
- The training data is the labelled pairs: 3,000 vectors, each with its answer.
We try two kinds of classifier. The machine learning course explains both in detail.
- Logistic regression gives each feature a weight, adds the weighted features, and turns the sum into a number from 0 to 1. It is like the hand-weighted score, but it learns the weights from the labels.
- Gradient boosting builds many small decision trees, one after another. A decision tree asks yes-or-no questions: “is the first-name similarity below 0.8?”, “is the phone the same?”. Each new tree corrects the errors of the trees before it. It can learn combinations, such as “same household, different first name”. We use the library LightGBM. The machine learning course explains trees in Decision trees and boosting in Ensembles.
The diagram shows both models on our two example pairs.
Text description of the diagram
On the left, the five fields that matter most for the two pairs: first name, full name, email, phone and postcode. In the middle, the two models, each trained on the 3,000 labelled pairs. On the right, each model’s score and threshold (chosen on the validation pairs) and the decision. Valérie’s pair is called the same by both models. The brothers are called the same by logistic regression and different by LightGBM. Splink and the hand-weighted score called the brothers the same.
Look at the brothers. Logistic regression gives them 0.9784, just above its threshold of 0.9685: a false merge, but a close one. It adds up evidence like Splink, with better weights, so seven agreeing household fields still win.
LightGBM gives them 0.9646, below its threshold. Its labelled “different” pairs often share a surname but have different first names. Its trees learned a combination: when other fields agree but the first name does not, the pair is usually two people.
The class imbalance of pairs
Section titled “The class imbalance of pairs”Before we train, look at the classes. Most candidate pairs are not the same customer.
- Of the 1,108,591 candidate pairs of the five blocking keys, 27,503 are true pairs. That is about 1 in 40.
- Of the 3,000 labelled pairs, 1,050 are true pairs. That is 35%, about 1 in 3.
When one class is much rarer than the other, we call it class imbalance.
Text description of the diagram
The top row has 40 squares, with one coloured square: the candidate pairs, where about one pair in 40 is the same customer. The bottom row has 20 squares, with 7 coloured squares: the labelled pairs, where 35% are the same customer. The labelled pairs were chosen on purpose: 1,500 pairs with the same surname and region, 1,050 true pairs, and 450 random pairs.
Why are the labels so different? Because they were chosen on purpose. Random candidate pairs would give a reviewer about one true pair in 40: a waste of her time. So the labelled set has 1,500 pairs with the same surname and region (hard pairs), 1,050 true pairs, and 450 random pairs. All of them come from the training customers only.
This choice is good for learning, and it has a cost. The model learns that “same” is common. On real candidate pairs, it says “same” too often. Its numbers are not true probabilities.
What to do about it
Section titled “What to do about it”Three rules handle the imbalance:
- Never use 0.5 as the threshold. Choose the threshold with the best F1 on the validation pairs: candidate pairs, with the real balance. The logistic regression’s best threshold is 0.969. LightGBM’s is above 0.999.
- Never measure with accuracy. A model that says “different” for every candidate pair is right on 97.5% of them, and finds no customer at all. Use precision and recall, as in Measure a matcher.
- Do not read the score as a probability until you calibrate it. Lesson 4 does this, because it needs real probabilities.
The machine learning course shows the same problem in Thresholds and calibration.
Run it in the project
Section titled “Run it in the project”Add the classifier code to your project. The commands run in your project folder, with your TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active.
-
Copy the three new files from the snapshot
end-of-m06:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m06cp $S/larkfield_match/classifier.py larkfield_match/cp $S/larkfield_match/runs/learned.py larkfield_match/runs/cp $S/larkfield_match/commands/train.py larkfield_match/commands/Terminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m06"Copy-Item $S\larkfield_match\classifier.py larkfield_match\Copy-Item $S\larkfield_match\runs\learned.py larkfield_match\runs\Copy-Item $S\larkfield_match\commands\train.py larkfield_match\commands\ -
Install scikit-learn and LightGBM, in the course’s versions:
Terminal window python -m pip install scikit-learn==1.9.1 lightgbm==4.7.0Successfully installed cloudpickle-3.1.2 joblib-1.6.0 lightgbm-4.7.0 narwhals-2.28.0 scikit-learn-1.9.1 threadpoolctl-3.8.0On macOS, LightGBM needs the OpenMP library. If
match trainstops with an error aboutlibomp, install it with Homebrew:brew install libomp. We did not test a Mac without it. -
Train logistic regression on the labelled pairs:
Terminal window match train --model logregTrained logreg on 3,000 labelled pairs (1,050 same); threshold 0.969 (best F1 on validation)Test: precision 0.915, recall 0.907, F1 0.911Features (coefficient (scaled inputs)):name_tsr +2.484phone_eq +1.845email_eq +1.632dob_eq +1.290first_initial +1.241first_jw +1.004dob_year_eq +1.002postcode3_eq +0.942 -
Train LightGBM. It is the default model of
match train:Terminal window match trainTrained lgbm on 3,000 labelled pairs (1,050 same); threshold 1.000 (best F1 on validation)Test: precision 0.918, recall 0.886, F1 0.902Features (share of gain):first_jw +0.749email_local_jw +0.044phone_eq +0.041name_tsr +0.032postcode3_eq +0.029postcode_eq +0.028first_initial +0.028dob_eq +0.015The threshold prints as 1.000 because it is rounded. It is just below 1. LightGBM is very sure about most pairs, and the imbalance pushes the best threshold up.
Both commands train on the labelled pairs, score all 1,108,591 candidate pairs, and keep the model in outputs/models/. Then measure the two runs as usual:
match eval --run logregmatch eval --run lgbmRan logreg: 26,399 pairs called the same, in 44.9 slogreg on the test split: pair precision 0.915, recall 0.907, F1 0.911 5,514 pairs called the same: 5,048 right, 466 wrong; 516 of 5,564 true pairs missed entities after clustering (all records): 38,142 (true: 37,500)Ran lgbm: 25,851 pairs called the same, in 142.1 slgbm on the test split: pair precision 0.918, recall 0.886, F1 0.902 5,375 pairs called the same: 4,932 right, 443 wrong; 632 of 5,564 true pairs missed entities after clustering (all records): 38,514 (true: 37,500)match eval trains the model again, because match train keeps the model but not the scored pairs. Our laptop was busy with other work, so the times are long. Your times will differ; the counts will not.
To see where the two methods disagree, compare Splink with LightGBM:
match compare splink lgbm --show 3splink precision 0.817 recall 0.779 F1 0.797 (5,300 pairs)lgbm precision 0.918 recall 0.886 F1 0.902 (5,375 pairs)Only splink: 954 pairs (186 right, 768 wrong) wrong S-000008 'Mohammed Singh' / S-006895 'Mr Arjun Singh' wrong G-000830 'DELACROIX-MENSAH, FATIMA' / S-020042 'Fatima Delacroix-Mensah' wrong G-000869 'LECLERC, LUC' / S-022810 'Catherine Leclerc'Only lgbm: 1,029 pairs (786 right, 243 wrong) wrong L-006319 'Jessica Walker' / L-016797 'Jessica Miller' wrong G-003503 'NGUYEN, FATIMA' / S-001675 'Fatima Oyelaran' wrong G-003381 'MILLER, HANNAH' / L-015027 'Mrs. Hannah Miller'Most pairs that only Splink calls the same are wrong (768 of 954): family members, like Mohammed and Arjun Singh. Most pairs that only LightGBM calls the same are right (786 of 1,029). Some of LightGBM’s mistakes share a first name and have different surnames. And a pair like Fatima Delacroix-Mensah looks like one customer by name, but is two people in our data. No model is right on every pair.
How the code works
Section titled “How the code works”classifier.py makes the two models. Logistic regression gets a scaler first, so that every feature has the same range. Missing values become 0, with one extra “missing” flag per field. LightGBM reads the missing values as they are:
if kind == "logreg": return make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000, C=1.0))return LGBMClassifier(n_estimators=300, learning_rate=0.05, num_leaves=31, min_child_samples=20, random_state=seed, n_jobs=1, verbose=-1)runs/learned.py trains the model on the labelled pairs, scores every candidate pair, and chooses the threshold with thresholds.best_f1, on the validation pairs. n_jobs=1 keeps LightGBM on one CPU core, so your laptop stays usable.
What each model uses
Section titled “What each model uses”The two models reach almost the same F1, but they use the features in very different ways.
Text description of the chart
Eight dots. first_jw 0.749; email_local_jw 0.044; phone_eq 0.041; name_tsr 0.032; postcode3_eq 0.029; postcode_eq 0.028; first_initial 0.028; dob_eq 0.015.
Gain is how much a feature’s questions improve the trees, added up over all trees. LightGBM puts 75% of its gain on the first-name Jaro–Winkler similarity. Why the first name? In our labelled pairs, the 1,500 hard “different” pairs share a surname and a region. Only 4.5% of them share a postcode, so most are not family members: they are people with the same surname. The first name separates them best.
Text description of the chart
Eight dots, all counting for a match: name_tsr 2.48; phone_eq 1.85; email_eq 1.63; dob_eq 1.29; first_initial 1.24; first_jw 1.00; dob_year_eq 1.00; postcode3_eq 0.94. Two smaller coefficients outside the top 8 count against a match: last_sound −0.69 and last_jw −0.58.
A coefficient is the weight that logistic regression learned for a feature. The inputs are scaled first, so the sizes can be compared. Logistic regression spreads its weight over the full name, the phone, the email and the date of birth.
Two coefficients are negative: the surname similarity and the surname sound. That looks strange: should a similar surname count against a match? No. It is a side effect of the labels. Most hard “different” pairs share a surname, so a shared surname alone is not evidence. The full name already carries it. This is our reading of the coefficients; we did not test it. Do not read one coefficient alone.
The comparison table
Section titled “The comparison table”Both classifiers beat every earlier method, by a lot.
| Run | Module | Precision | Recall | F1 |
|---|---|---|---|---|
rules |
2 | 0.723 | 0.870 | 0.790 |
weighted |
4 | 0.751 | 0.766 | 0.759 |
fs-labels |
5 | 0.839 | 0.747 | 0.790 |
splink |
5 | 0.817 | 0.779 | 0.797 |
logreg |
6 | 0.915 | 0.907 | 0.911 |
lgbm |
6 | 0.918 | 0.886 | 0.902 |
One F1 per run hides a lot: it is one threshold. The chart shows every threshold. Each line is one run on the same test pairs: its precision (up) against its recall (along).
- lgbm
- logreg
- splink (no labels)
- weighted (hand weights)
Text description of the chart
Four lines. The two classifier lines stay above precision 0.9 up to recall about 0.85, then fall. The Splink line falls below 0.95 precision at recall about 0.47, and the hand-weighted line at about 0.42. No run reaches recall 1: blocking lost 33 of the 5,564 true test pairs, and a few pairs have no evidence at all.
The chart answers TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s question: “how many customers do we find, if a false merge is rare?”. At precision 0.95 (threshold chosen on validation), logistic regression finds 0.874 of the true pairs. LightGBM finds 0.837, Splink 0.474, the hand-weighted score 0.422. On the test pairs, these thresholds gave a precision between 0.925 and 0.940, a little below the target. That is normal: a threshold chosen on one set of pairs is not exact on another.
The simpler model wins here
Section titled “The simpler model wins here”Logistic regression is the better model on our records: F1 0.911 against 0.902, and more recall at high precision. It is also faster and easier to explain. The machine learning course discusses this in When a simple model is better.
But one F1 hides single cases: logistic regression merges the brothers, and LightGBM does not.
The project still uses LightGBM as its main model, lgbm. Why? Lesson 4 needs a probability for every pair to find the unsure band, and we built that on LightGBM. The comparison table keeps both, so you can always check the choice. In your own work, start with logistic regression and keep a more complex model only when it wins on your validation pairs.
When to use a pair classifier
Section titled “When to use a pair classifier”| Use it when | Do not use it when | What it costs |
|---|---|---|
| you have a few hundred labelled pairs or more, from review | you have no labels: use Splink (Module 5) | labels: 1.2 minutes a pair for a reviewer (our assumption), so 3,000 labels are 60 hours |
| fields work together (households, people who move) | the labels come from one kind of pair only: the model learns that kind | CPU: under US$0.000002 per 1,000 pairs (US$0.17 an hour, assumption) |
| you must reach a high precision with good recall | you cannot explain a decision to a customer, and must: prefer logistic regression | a new model when the data changes: keep the training script |
Labels are the main cost. Ivana reviews 300 pairs a day (our assumption), so 3,000 labels are ten of her working days. The next lesson asks: do we need all 3,000?
Common mistake
Trust the default threshold
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. trains LightGBM on the 3,000 labelled pairs and calls model.predict(). It uses the threshold 0.5. He reports “the model finds 93% of the true pairs!”. A week later, Ivana finds thousands of merged strangers.
Cause: the labelled pairs are 35% true pairs. The candidate pairs are about 2.5% true pairs. The model learned that “same” is common, so many candidate pairs of two people get more than 0.5. The best threshold on the validation pairs is above 0.999. The notebook of this lesson shows the effect with six features and the threshold 0.5. Logistic regression finds 0.935 of the true test pairs, but its precision is only 0.594. Of 8,765 pairs called the same, 3,562 are two people.
Fix: choose the threshold on validation candidate pairs, which have the real balance. In the project, thresholds.best_f1 does it for every run. If you need a probability that means what it says, calibrate it, as in lesson 4.
Consequence in AI work: a model is trained on one mix of classes and used on another. This happens in fraud, in medical tests and in content review. Always check the threshold on data with the real mix.
Knowledge check
Section titled “Knowledge check”Check your understanding
Splink gives the brothers Samuel and Nathan Green a match weight of 37.9. LightGBM calls them different. What did LightGBM learn that Splink cannot learn?
Show the answer
Splink weighs each field on its own and adds the weights, so many shared household fields give a very high weight. LightGBM learned from labelled pairs that a combination matters: a shared surname with a different first name is usually two people. Its trees can ask about the first name after they ask about the other fields. LightGBM puts 75% of its gain on the first-name similarity for this reason. Logistic regression adds weighted evidence like Splink, with learned weights. It scores the brothers just above its threshold, so it still merges them, but only just.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The best threshold of LightGBM on the validation pairs is above 0.999. Why so high, and why must it come from the validation pairs, not the labelled pairs?
Show the answer
The model trained on labelled pairs where 35% are true pairs, so it gives high scores easily. On candidate pairs, only about 1 in 40 is true, so a very high threshold is needed to keep precision. The labelled pairs have the wrong balance, and the model was trained on them, so a threshold chosen there would be far too low. The validation pairs are candidate pairs of other customers, with the real balance and unseen by the model.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Logistic regression gives the surname similarity a negative coefficient. A colleague says: “so a matching surname is evidence against a match”. What do you answer?
Show the answer
No. The coefficient is the effect of the surname after the other features. Most hard “different” pairs in the labels share a surname, because we sampled them by surname and region. And the full name, email and phone already carry the surname’s evidence. So the model gives the surname a small negative weight to balance the others. Read the coefficients together, and test a model by its results on validation pairs, not by one coefficient.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A pair classifier reads a comparison vector and gives a score from 0 to 1. It learns from labelled pairs: our 3,000 synthetic labels, or a reviewer’s decisions.
- Logistic regression learns one weight per feature. Gradient boosting (LightGBM) builds many small trees, and can learn combinations.
- Class imbalance: about 1 candidate pair in 40 is true, against 35% of the labelled pairs. Choose the threshold on validation, never 0.5, and never measure with accuracy.
- On the test pairs: logistic regression F1 0.911, LightGBM 0.902, Splink 0.797. At precision 0.95, logistic regression finds 0.874 of the true pairs, Splink 0.474.
- The simpler model wins here. The project keeps LightGBM as
lgbmfor lesson 4, and the table shows both.
Next, we ask how many labels we really need, and which pairs a reviewer should label first.
Next: Active learning