Skip to content
Français

A supervised pair classifier

  • Lesson 18
  • 40 min
  • Module 6: Learned matching: classifiers, embeddings and language models

Where we are

Splink learned its weights without labels. Larkfield's reviewers have now labelled some pairs. In this lesson, we train a pair classifier on comparison vectors, and we handle the strong class imbalance of pairs.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Train a model that reads a pair’s comparison vector and says how likely the two records are one customer. Train it on labelled pairs: logistic regression first, then gradient boosting with LightGBM. Handle the class imbalance of pairs: the threshold comes from the validation pairs, never from 0.5. Add both models to the comparison table.

NotebookTrain a pair classifier in a notebookColab · Kaggle · GitHub

The notebook turns the 3,000 labelled pairs into six features each. It trains logistic regression and asks you to train LightGBM. Then it measures both at the threshold 0.5 on the candidate pairs, so you see what the class imbalance does. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of Module 5. Or use the notebook only.
  • About 150 MB of free disk space for two new packages.

Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.

A clinic hires a new clerk. Her job: decide if a new patient form belongs to a patient who is already in the files.

Nobody gives her a rule book. Instead, a senior clerk shows her a box of old decisions. Each card has two forms and one word on it: “same” or “different”. The new clerk reads hundreds of cards. Soon she sees patterns that no rule book says. “Same surname, same address, different first name: usually a brother or a sister.” “Same name, new address, same date of birth: usually a patient who moved.”

This is supervised learning. A model learns from examples that a person has already answered. The answers are labels.

The problem: our weights do not learn from answers

Section titled “The problem: our weights do not learn from answers”

Every method so far chose its weights without looking at true answers:

  • The hand-weighted score of Module 4 used our guesses.
  • Splink, in Module 5, estimated its weights from the data alone, with no labels.

Splink weighs each field on its own and adds the weights. It cannot learn that combinations matter. The brothers Samuel and Nathan Green show the problem. They share a surname, an email, a landline, a postcode, a street and a town. Each agreement adds weight. Splink gives them a match weight of 37.9, a probability of 1.0. They are two people.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield now has 3,000 labelled pairs. A reviewer has said “same” or “different” for each one. In our synthetic data, the labels come from the true customers. In Larkfield’s real work, they would come from Ivana’s review queue. Can a model learn from them?

A classifier is a model that puts each input into one of a few classes. A pair classifier reads one candidate pair and puts it into one of two classes: “same” or “different”.

  • The input is the pair’s comparison vector: the 16 numbers of Module 4. In machine learning, these inputs are called features.
  • The output is a number from 0 to 1: how likely the pair is one customer. A pair at or above a threshold is called “the same”.
  • The training data is the labelled pairs: 3,000 vectors, each with its answer.

We try two kinds of classifier. The machine learning course explains both in detail.

  1. Logistic regression gives each feature a weight, adds the weighted features, and turns the sum into a number from 0 to 1. It is like the hand-weighted score, but it learns the weights from the labels.
  2. Gradient boosting builds many small decision trees, one after another. A decision tree asks yes-or-no questions: “is the first-name similarity below 0.8?”, “is the phone the same?”. Each new tree corrects the errors of the trees before it. It can learn combinations, such as “same household, different first name”. We use the library LightGBM. The machine learning course explains trees in Decision trees and boosting in Ensembles.

The diagram shows both models on our two example pairs.

Two pairs go into two classifiers. Valérie Tremblay's two records, one customer: logistic regression gives 0.9999, LightGBM gives 1.0000; both are at or above their thresholds, so both say same, right. The brothers Samuel and Nathan Green: logistic regression gives 0.9784, above its threshold 0.9685, so it says same, wrong; LightGBM gives 0.9646, below its threshold 0.9997, so it says different, right. Splink gave the brothers a match weight of 37.9.
The same two pairs as in Module 4, scored by the two classifiers. Only LightGBM keeps the brothers apart.
Text description of the diagram

On the left, the five fields that matter most for the two pairs: first name, full name, email, phone and postcode. In the middle, the two models, each trained on the 3,000 labelled pairs. On the right, each model’s score and threshold (chosen on the validation pairs) and the decision. Valérie’s pair is called the same by both models. The brothers are called the same by logistic regression and different by LightGBM. Splink and the hand-weighted score called the brothers the same.

Look at the brothers. Logistic regression gives them 0.9784, just above its threshold of 0.9685: a false merge, but a close one. It adds up evidence like Splink, with better weights, so seven agreeing household fields still win.

LightGBM gives them 0.9646, below its threshold. Its labelled “different” pairs often share a surname but have different first names. Its trees learned a combination: when other fields agree but the first name does not, the pair is usually two people.

Before we train, look at the classes. Most candidate pairs are not the same customer.

  • Of the 1,108,591 candidate pairs of the five blocking keys, 27,503 are true pairs. That is about 1 in 40.
  • Of the 3,000 labelled pairs, 1,050 are true pairs. That is 35%, about 1 in 3.

When one class is much rarer than the other, we call it class imbalance.

Two rows of squares. Candidate pairs: 40 squares, 1 of them coloured: about 1 true pair in 40 (27,503 of 1,108,591). Labelled pairs: 20 squares, 7 of them coloured: 35% true pairs (1,050 of 3,000).
The labelled pairs have far more true pairs than the candidate pairs. A model trained on the labels expects too many matches.
Text description of the diagram

The top row has 40 squares, with one coloured square: the candidate pairs, where about one pair in 40 is the same customer. The bottom row has 20 squares, with 7 coloured squares: the labelled pairs, where 35% are the same customer. The labelled pairs were chosen on purpose: 1,500 pairs with the same surname and region, 1,050 true pairs, and 450 random pairs.

Why are the labels so different? Because they were chosen on purpose. Random candidate pairs would give a reviewer about one true pair in 40: a waste of her time. So the labelled set has 1,500 pairs with the same surname and region (hard pairs), 1,050 true pairs, and 450 random pairs. All of them come from the training customers only.

This choice is good for learning, and it has a cost. The model learns that “same” is common. On real candidate pairs, it says “same” too often. Its numbers are not true probabilities.

Three rules handle the imbalance:

  1. Never use 0.5 as the threshold. Choose the threshold with the best F1 on the validation pairs: candidate pairs, with the real balance. The logistic regression’s best threshold is 0.969. LightGBM’s is above 0.999.
  2. Never measure with accuracy. A model that says “different” for every candidate pair is right on 97.5% of them, and finds no customer at all. Use precision and recall, as in Measure a matcher.
  3. Do not read the score as a probability until you calibrate it. Lesson 4 does this, because it needs real probabilities.

The machine learning course shows the same problem in Thresholds and calibration.

Add the classifier code to your project. The commands run in your project folder, with your TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active.

  1. Copy the three new files from the snapshot end-of-m06:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m06
    cp $S/larkfield_match/classifier.py larkfield_match/
    cp $S/larkfield_match/runs/learned.py larkfield_match/runs/
    cp $S/larkfield_match/commands/train.py larkfield_match/commands/
  2. Install scikit-learn and LightGBM, in the course’s versions:

    Terminal window
    python -m pip install scikit-learn==1.9.1 lightgbm==4.7.0
    Successfully installed cloudpickle-3.1.2 joblib-1.6.0 lightgbm-4.7.0 narwhals-2.28.0 scikit-learn-1.9.1 threadpoolctl-3.8.0

    On macOS, LightGBM needs the OpenMP library. If match train stops with an error about libomp, install it with Homebrew: brew install libomp. We did not test a Mac without it.

  3. Train logistic regression on the labelled pairs:

    Terminal window
    match train --model logreg
    Trained logreg on 3,000 labelled pairs (1,050 same); threshold 0.969 (best F1 on validation)
    Test: precision 0.915, recall 0.907, F1 0.911
    Features (coefficient (scaled inputs)):
    name_tsr +2.484
    phone_eq +1.845
    email_eq +1.632
    dob_eq +1.290
    first_initial +1.241
    first_jw +1.004
    dob_year_eq +1.002
    postcode3_eq +0.942
  4. Train LightGBM. It is the default model of match train:

    Terminal window
    match train
    Trained lgbm on 3,000 labelled pairs (1,050 same); threshold 1.000 (best F1 on validation)
    Test: precision 0.918, recall 0.886, F1 0.902
    Features (share of gain):
    first_jw +0.749
    email_local_jw +0.044
    phone_eq +0.041
    name_tsr +0.032
    postcode3_eq +0.029
    postcode_eq +0.028
    first_initial +0.028
    dob_eq +0.015

    The threshold prints as 1.000 because it is rounded. It is just below 1. LightGBM is very sure about most pairs, and the imbalance pushes the best threshold up.

Both commands train on the labelled pairs, score all 1,108,591 candidate pairs, and keep the model in outputs/models/. Then measure the two runs as usual:

Terminal window
match eval --run logreg
match eval --run lgbm
Ran logreg: 26,399 pairs called the same, in 44.9 s
logreg on the test split: pair precision 0.915, recall 0.907, F1 0.911
5,514 pairs called the same: 5,048 right, 466 wrong; 516 of 5,564 true pairs missed
entities after clustering (all records): 38,142 (true: 37,500)
Ran lgbm: 25,851 pairs called the same, in 142.1 s
lgbm on the test split: pair precision 0.918, recall 0.886, F1 0.902
5,375 pairs called the same: 4,932 right, 443 wrong; 632 of 5,564 true pairs missed
entities after clustering (all records): 38,514 (true: 37,500)

match eval trains the model again, because match train keeps the model but not the scored pairs. Our laptop was busy with other work, so the times are long. Your times will differ; the counts will not.

To see where the two methods disagree, compare Splink with LightGBM:

Terminal window
match compare splink lgbm --show 3
splink precision 0.817 recall 0.779 F1 0.797 (5,300 pairs)
lgbm precision 0.918 recall 0.886 F1 0.902 (5,375 pairs)
Only splink: 954 pairs (186 right, 768 wrong)
wrong S-000008 'Mohammed Singh' / S-006895 'Mr Arjun Singh'
wrong G-000830 'DELACROIX-MENSAH, FATIMA' / S-020042 'Fatima Delacroix-Mensah'
wrong G-000869 'LECLERC, LUC' / S-022810 'Catherine Leclerc'
Only lgbm: 1,029 pairs (786 right, 243 wrong)
wrong L-006319 'Jessica Walker' / L-016797 'Jessica Miller'
wrong G-003503 'NGUYEN, FATIMA' / S-001675 'Fatima Oyelaran'
wrong G-003381 'MILLER, HANNAH' / L-015027 'Mrs. Hannah Miller'

Most pairs that only Splink calls the same are wrong (768 of 954): family members, like Mohammed and Arjun Singh. Most pairs that only LightGBM calls the same are right (786 of 1,029). Some of LightGBM’s mistakes share a first name and have different surnames. And a pair like Fatima Delacroix-Mensah looks like one customer by name, but is two people in our data. No model is right on every pair.

classifier.py makes the two models. Logistic regression gets a scaler first, so that every feature has the same range. Missing values become 0, with one extra “missing” flag per field. LightGBM reads the missing values as they are:

if kind == "logreg":
return make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000, C=1.0))
return LGBMClassifier(n_estimators=300, learning_rate=0.05, num_leaves=31, min_child_samples=20,
random_state=seed, n_jobs=1, verbose=-1)

runs/learned.py trains the model on the labelled pairs, scores every candidate pair, and chooses the threshold with thresholds.best_f1, on the validation pairs. n_jobs=1 keeps LightGBM on one CPU core, so your laptop stays usable.

The two models reach almost the same F1, but they use the features in very different ways.

LightGBM: the share of all its split gain that each feature brings (the top 8 of 16). Three quarters come from one feature: the first-name similarity.
Text description of the chart

Eight dots. first_jw 0.749; email_local_jw 0.044; phone_eq 0.041; name_tsr 0.032; postcode3_eq 0.029; postcode_eq 0.028; first_initial 0.028; dob_eq 0.015.

Gain is how much a feature’s questions improve the trees, added up over all trees. LightGBM puts 75% of its gain on the first-name Jaro–Winkler similarity. Why the first name? In our labelled pairs, the 1,500 hard “different” pairs share a surname and a region. Only 4.5% of them share a postcode, so most are not family members: they are people with the same surname. The first name separates them best.

Logistic regression: the size of each feature's coefficient, on scaled inputs (the top 8 of 16). It spreads its weight over the name, phone, email and date of birth.
Text description of the chart

Eight dots, all counting for a match: name_tsr 2.48; phone_eq 1.85; email_eq 1.63; dob_eq 1.29; first_initial 1.24; first_jw 1.00; dob_year_eq 1.00; postcode3_eq 0.94. Two smaller coefficients outside the top 8 count against a match: last_sound −0.69 and last_jw −0.58.

A coefficient is the weight that logistic regression learned for a feature. The inputs are scaled first, so the sizes can be compared. Logistic regression spreads its weight over the full name, the phone, the email and the date of birth.

Two coefficients are negative: the surname similarity and the surname sound. That looks strange: should a similar surname count against a match? No. It is a side effect of the labels. Most hard “different” pairs share a surname, so a shared surname alone is not evidence. The full name already carries it. This is our reading of the coefficients; we did not test it. Do not read one coefficient alone.

Both classifiers beat every earlier method, by a lot.

Run Module Precision Recall F1
rules 2 0.723 0.870 0.790
weighted 4 0.751 0.766 0.759
fs-labels 5 0.839 0.747 0.790
splink 5 0.817 0.779 0.797
logreg 6 0.915 0.907 0.911
lgbm 6 0.918 0.886 0.902

One F1 per run hides a lot: it is one threshold. The chart shows every threshold. Each line is one run on the same test pairs: its precision (up) against its recall (along).

  • lgbm
  • logreg
  • splink (no labels)
  • weighted (hand weights)
Precision against recall for four runs on the test split of our synthetic records. The two classifiers stay above 0.9 precision until about 0.85 recall. Splink and the hand-weighted score fall much earlier.
Text description of the chart

Four lines. The two classifier lines stay above precision 0.9 up to recall about 0.85, then fall. The Splink line falls below 0.95 precision at recall about 0.47, and the hand-weighted line at about 0.42. No run reaches recall 1: blocking lost 33 of the 5,564 true test pairs, and a few pairs have no evidence at all.

The chart answers TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s question: “how many customers do we find, if a false merge is rare?”. At precision 0.95 (threshold chosen on validation), logistic regression finds 0.874 of the true pairs. LightGBM finds 0.837, Splink 0.474, the hand-weighted score 0.422. On the test pairs, these thresholds gave a precision between 0.925 and 0.940, a little below the target. That is normal: a threshold chosen on one set of pairs is not exact on another.

Logistic regression is the better model on our records: F1 0.911 against 0.902, and more recall at high precision. It is also faster and easier to explain. The machine learning course discusses this in When a simple model is better.

But one F1 hides single cases: logistic regression merges the brothers, and LightGBM does not.

The project still uses LightGBM as its main model, lgbm. Why? Lesson 4 needs a probability for every pair to find the unsure band, and we built that on LightGBM. The comparison table keeps both, so you can always check the choice. In your own work, start with logistic regression and keep a more complex model only when it wins on your validation pairs.

Use it when Do not use it when What it costs
you have a few hundred labelled pairs or more, from review you have no labels: use Splink (Module 5) labels: 1.2 minutes a pair for a reviewer (our assumption), so 3,000 labels are 60 hours
fields work together (households, people who move) the labels come from one kind of pair only: the model learns that kind CPU: under US$0.000002 per 1,000 pairs (US$0.17 an hour, assumption)
you must reach a high precision with good recall you cannot explain a decision to a customer, and must: prefer logistic regression a new model when the data changes: keep the training script

Labels are the main cost. Ivana reviews 300 pairs a day (our assumption), so 3,000 labels are ten of her working days. The next lesson asks: do we need all 3,000?

Common mistake

Trust the default threshold

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. trains LightGBM on the 3,000 labelled pairs and calls model.predict(). It uses the threshold 0.5. He reports “the model finds 93% of the true pairs!”. A week later, Ivana finds thousands of merged strangers.

Cause: the labelled pairs are 35% true pairs. The candidate pairs are about 2.5% true pairs. The model learned that “same” is common, so many candidate pairs of two people get more than 0.5. The best threshold on the validation pairs is above 0.999. The notebook of this lesson shows the effect with six features and the threshold 0.5. Logistic regression finds 0.935 of the true test pairs, but its precision is only 0.594. Of 8,765 pairs called the same, 3,562 are two people.

Fix: choose the threshold on validation candidate pairs, which have the real balance. In the project, thresholds.best_f1 does it for every run. If you need a probability that means what it says, calibrate it, as in lesson 4.

Consequence in AI work: a model is trained on one mix of classes and used on another. This happens in fraud, in medical tests and in content review. Always check the threshold on data with the real mix.

Check your understanding

Splink gives the brothers Samuel and Nathan Green a match weight of 37.9. LightGBM calls them different. What did LightGBM learn that Splink cannot learn?

Show the answer

Splink weighs each field on its own and adds the weights, so many shared household fields give a very high weight. LightGBM learned from labelled pairs that a combination matters: a shared surname with a different first name is usually two people. Its trees can ask about the first name after they ask about the other fields. LightGBM puts 75% of its gain on the first-name similarity for this reason. Logistic regression adds weighted evidence like Splink, with learned weights. It scores the brothers just above its threshold, so it still merges them, but only just.

Check your understanding

The best threshold of LightGBM on the validation pairs is above 0.999. Why so high, and why must it come from the validation pairs, not the labelled pairs?

Show the answer

The model trained on labelled pairs where 35% are true pairs, so it gives high scores easily. On candidate pairs, only about 1 in 40 is true, so a very high threshold is needed to keep precision. The labelled pairs have the wrong balance, and the model was trained on them, so a threshold chosen there would be far too low. The validation pairs are candidate pairs of other customers, with the real balance and unseen by the model.

Check your understanding

Logistic regression gives the surname similarity a negative coefficient. A colleague says: “so a matching surname is evidence against a match”. What do you answer?

Show the answer

No. The coefficient is the effect of the surname after the other features. Most hard “different” pairs in the labels share a surname, because we sampled them by surname and region. And the full name, email and phone already carry the surname’s evidence. So the model gives the surname a small negative weight to balance the others. Read the coefficients together, and test a model by its results on validation pairs, not by one coefficient.

Recap

  • A pair classifier reads a comparison vector and gives a score from 0 to 1. It learns from labelled pairs: our 3,000 synthetic labels, or a reviewer’s decisions.
  • Logistic regression learns one weight per feature. Gradient boosting (LightGBM) builds many small trees, and can learn combinations.
  • Class imbalance: about 1 candidate pair in 40 is true, against 35% of the labelled pairs. Choose the threshold on validation, never 0.5, and never measure with accuracy.
  • On the test pairs: logistic regression F1 0.911, LightGBM 0.902, Splink 0.797. At precision 0.95, logistic regression finds 0.874 of the true pairs, Splink 0.474.
  • The simpler model wins here. The project keeps LightGBM as lgbm for lesson 4, and the table shows both.

Next, we ask how many labels we really need, and which pairs a reviewer should label first.

Next: Active learning