Skip to content
Français

From weights to a decision

  • Lesson 17
  • 35 min
  • Module 5: Probabilistic linkage

Where we are

Splink gives every pair a match weight. A business needs a decision for each pair. In this lesson, we turn weights into probabilities, set two thresholds, and send the unsure band between them to review.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn match weights into decisions. See why Splink’s probabilities are far too sure on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s records, and why the project scores with the weight instead. Then set two thresholds. Pairs above the upper one merge. Pairs below the lower one stay apart. The unsure band between them goes to a person. Choose the thresholds by what the reviewer can do.

NotebookSplit the pairs into bands in a notebookColab · Kaggle · GitHub

The notebook trains Splink again, checks how sure its probabilities are against the truth, and draws the weights of true pairs and other pairs. You count the review band for two thresholds and the days of work it gives a reviewer. Its setup cells install Splink and download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of Module 5 and a trained model (match splink). Or use the notebook only.

A teacher marks 300 exam papers. The pass mark is 50. A paper with 85 is a clear pass, and a paper with 12 is a clear fail. Her marks on these papers are final.

A paper with 49 or 51 is different. One mark more or less changes the result, and her marking is not that exact. So the school sends every paper between 45 and 55 to a second marker. That person reads them again and decides.

How wide is that band? The school cannot send all 300 papers. It asks: how many papers can the second marker read this week? The band is as wide as her time allows.

Matching has the same three parts. Sure matches merge. Sure non-matches stay apart. Pairs in between go to a person: at Larkfield, Ivana.

The problem: probabilities that are far too sure

Section titled “The problem: probabilities that are far too sure”

Splink gives each pair a match weight, and from it a match probability: 2w ÷ (1 + 2w). The probability looks like the perfect tool. Merge above 0.99, and you expect at most 1 error in 100.

Look at one pair. G-001688 and G-002894 are two brothers, Samuel and Nathan Green: a practice problem we made on purpose. They share a surname, a family email, a landline and an address.

Terminal window
match explain G-001688 G-002894
prior (two random records) -15.83 (total -15.83)
first -3.44 (total -19.28) samuel / nathan
last +6.46 (total -12.82) green / green
email +12.26 (total -0.55) thegreens@example.com / thegreens@example.com
phone +14.16 (total +13.61) 9055551835 / 9055551835
postcode +13.28 (total +26.89) M7K4L8 / M7K4L8
street +7.19 (total +34.07) kestrel ave / kestrel ave
town +3.84 (total +37.91) heronvale / heronvale
dob +0.00 (total +37.91) None / 1966-05-20
Match weight +37.91 -> match probability 1.0000
True answer: different
A waterfall of match weights for the brothers pair G-001688 / G-002894. The prior starts at −15.83. The first name (samuel / nathan) removes 3.44. The surname adds 6.46, the email 12.26, the phone 14.16, the postcode 13.28, the street 7.19 and the town 3.84. The date of birth adds 0, because one record has none. The total is +37.91, a probability of 1.0000. The true answer: two different people.
A waterfall of match weights: each field adds evidence (green) or removes it (red). Five shared household fields outweigh the different first names.
Text description of the diagram

Each row is one step, from the top. The prior, −15.83, is the weight of two random records. The first names differ (samuel and nathan), so the first name removes 3.44, to −19.28. Then the surname green adds 6.46, the family email 12.26, the landline 14.16, the postcode 13.28, the street 7.19 and the town 3.84. The date of birth adds nothing, because one record has none. The total is +37.91, and the probability is 1.0000. The true answer: two different people.

The brothers get a match weight of 37.9 and a probability of 1.0. Splink is “sure”, and it is wrong.

No. Compare Splink’s probabilities with the truth for all 391,666 candidate pairs of the test split:

Match probabilityTest pairsShare that are true pairs
0 to 0.5152,5710.001
0.5 to 0.933,7570.002
0.9 to 0.9930,0110.005
0.99 to 0.99999911,6210.099
0.999999 to 14,6190.855

Of the 4,619 pairs with a probability above 0.999999, only 85.5% are true pairs. And 15 test pairs get a probability of exactly 1.0. A probability that matches the truth is called calibrated. Splink’s probabilities here are far from calibrated.

Why: the same evidence, counted five times

Section titled “Why: the same evidence, counted five times”

The Fellegi–Sunter model adds the weight of each field as separate evidence. That is true for strangers: two strangers who share a postcode rarely share a phone too. It is not true in a family. Brothers who share an address also share the postcode, the street, the town, the landline and often the surname and an email. That is one fact, “same household”, and the model counts it five times.

The notebook of this lesson shows the effect. Its model has no street and no town: two household fields fewer. Of its test pairs above 0.999999, 97.4% are true pairs, against 85.5% for the project’s model. Still not calibrated, but much less wrong.

So the project uses the match weight only to rank pairs, and chooses thresholds on the weight, by measurement. The run splink of the last lesson chose a weight of 15.61 on the validation split, not a probability of 0.5. Thresholds and calibration in the machine learning course shows how to measure and repair calibration for a classifier.

One threshold forces a “yes” or “no” for every pair, even the doubtful ones. Two thresholds make three bands:

  • Match band: weight at or above the upper threshold. Merge without review.
  • Review band: between the two thresholds. A person decides.
  • Non-match band: below the lower threshold. Do not merge.
Terminal window
match bands
splink, test split, thresholds 10.0 and 20.0:
match band: 4,604 pairs, precision 0.856
review band: 4,262 pairs (795 true pairs, 3,467 not)
non-match band: 382,800 pairs, 795 true pairs lost there
If reviewers decide every review pair correctly: precision 0.877, recall 0.851
Three bands of Splink's match weight on the test split, with thresholds 10 and 20. Below 10, the non-match band: 382,800 pairs, 795 true pairs lost, not merged. From 10 to 20, the review band: 4,262 pairs, 795 true pairs and 3,467 not, Ivana decides. 20 or more, the match band: 4,604 pairs, precision 0.856, merged without review.
Two thresholds on the match weight make three bands. The counts are the test split of the synthetic records.
Text description of the diagram

A weight axis runs from low to high, with dashed lines at 10 and 20. The left band, below 10, is the non-match band: 382,800 pairs; 795 true pairs are lost there; these pairs are not merged. The middle band, from 10 to 20, is the review band: 4,262 pairs, 795 of them true pairs and 3,467 not; Ivana decides. The right band, 20 or more, is the match band: 4,604 pairs with a precision of 0.856; they merge without review.

The chart shows where the true pairs and the other pairs sit on the weight:

  • True pairs
  • Other candidate pairs
  • Thresholds 10 and 20 (dashed)
Match weights of the test-split candidate pairs: 5,531 true pairs and 386,135 other pairs, each as a share of its own group. Not drawn: the pairs that Splink left out (probability below 0.001): 23 true and 159,064 other. Weights above 50 are in the last bin. Measured.
Text description of the chart

Two lines over the match weight, from −10 to 50, and dashed vertical lines at 10 and 20. The other candidate pairs are mostly on the left. About 13% of them are in each of the bins −10 to −8 and −8 to −6. A second bump of about 7% is at 2 to 4. From 8 on, each bin has under 1%. The true pairs are mostly on the right, with two peaks: 9.7% at weights 22 to 24, and 9.2% at 34 to 36. Between 6 and 20, both lines are low: that is where the two groups overlap. Not drawn: 159,064 other pairs and 23 true pairs that Splink left out with a probability below 0.001.

Each line is a share of its own group, so the true pairs look as common as the others. In counts, they are not. From 6 to 20, the test split has 12,522 other pairs and 1,204 true pairs. The groups overlap there, and no single threshold separates them. The review band sits on that overlap.

The review band costs Ivana’s time. The course’s assumptions: Ivana reviews 300 pairs a day, 1.2 minutes a pair. Larkfield wants to review its whole backlog once: all splits, not only the test split. For this lesson, we assume one more thing: TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. gives Ivana one month, 20 working days, for the backlog.

Explore

Two thresholds, three bands, one reviewer

Match band
21,523 pairsprecision 0.916
Review band
13,378 pairs3,954 true pairs
Days of review
44.6 days
If every review is right
recall 0.855, precision 0.929

Thresholds 10 and 20: 21,523 pairs merge without review (precision 0.916). 13,378 pairs go to review, and 3,954 of them are true pairs. One reviewer at 300 pairs a day needs 44.6 working days. If every review is right: recall 0.855, precision 0.929. That is more than the 20 days of review that Ivana can give.

Days of review for every pair of thresholds (one reviewer, 300 pairs a day). Under each number: the pairs in the review band.
Lower ↓ / upper →1618202225
5178.453,524182.054,614187.556,264197.259,150212.363,688
868.320,49872.021,58877.523,23887.126,124102.230,662
1035.510,63839.111,72844.613,37854.216,26469.320,802
1211.33,39715.04,48720.56,13730.19,02345.213,561
143.91,1607.52,25013.03,90022.66,78637.711,324
Splink's match weights for every candidate pair of the 60,108 synthetic records (all splits), measured. Ivana's 300 pairs a day is the course's assumption; the 20 days are this lesson's assumption.
Text description of the explorer

The explorer has two sliders. The lower threshold takes the values 5, 8, 10, 12 and 14; the upper threshold 16, 18, 20, 22 and 25. They start at 10 and 20. Four tiles show the match band (21,523 pairs, precision 0.916), the review band (13,378 pairs, 3,954 of them true pairs), the days of review (44.6) and the result if every review is right (recall 0.855, precision 0.929). A sentence repeats these numbers and says whether the review fits in Ivana’s 20 days. A bar chart shows the days of review for each lower threshold at the chosen upper threshold, with a dashed line at 20 days. A table lists the days and the review pairs for all 25 pairs of thresholds. With an upper threshold of 20: lower 5 gives 187.5 days; 8, 77.5; 10, 44.6; 12, 20.5; 14, 13.0. With a lower threshold of 10: upper 16 gives 35.5 days; 18, 39.1; 20, 44.6; 22, 54.2; 25, 69.3.

Things to try:

  • Keep the upper threshold at 20 and move the lower one. Each step down adds thousands of pairs to review, and gains a little recall.
  • Keep the lower threshold at 10 and move the upper one. A higher upper threshold makes the match band more precise, from 0.894 at 16 to 0.953 at 25. Those pairs move to review: more days.
  • Find the pairs of thresholds that fit in 20 days. Which of them loses the fewest true pairs?

With thresholds 10 and 20, the review takes 44.6 days: more than twice Ivana’s month. Thresholds 14 and 20 take 13.0 days and fit. The price: recall if every review is right falls from 0.855 to 0.796.

The course’s costs help to judge it (assumptions). A false merge costs 150 dollars: a stranger sees another customer’s orders and address. A missed match costs 3 dollars a year. One review costs 1.2 minutes at 40 dollars an hour: 0.80 dollars. So reviewing a doubtful pair is cheap, and a false merge is very expensive. The match band at 20 has a precision of 0.916 on all splits. So about 1,800 of its 21,523 pairs are false merges. Raising the upper threshold moves some of them to Ivana, and that is usually worth her time.

Try one more pair of thresholds in the project. The output is for the test split only:

Terminal window
match bands --lower 12 --upper 20
splink, test split, thresholds 12.0 and 20.0:
match band: 4,604 pairs, precision 0.856
review band: 1,820 pairs (581 true pairs, 1,239 not)
non-match band: 385,242 pairs, 1,009 true pairs lost there
If reviewers decide every review pair correctly: precision 0.872, recall 0.813

When to use a review band, and what it costs

Section titled “When to use a review band, and what it costs”
  • Use it whenever an error is expensive and a person can decide better than the model. At Larkfield, one false merge costs as much as 50 years of one missed match.
  • Choose the thresholds on validation pairs, then check them on the test pairs, as the explorer’s numbers were measured.
  • Size the band by the reviewer’s capacity, not by the model alone. Ask how many pairs a day, and for how many days.
  • Do not trust a probability that you have not checked. Measure its calibration first. For Splink on these records, use the weight.
  • A better model shrinks the overlap, and so the review band. A supervised pair classifier in Module 6 does that. Human review in Module 8 runs the review queue every day.

Common mistake

Merge every pair above a probability of 0.99

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads the probabilities. “We merge everything above 0.99. That is at most 1% wrong.” On the test split, 16,240 pairs have a probability above 0.99. Only 5,104 of them are true pairs: a precision of 0.314. Two of every three merges join different people.

Cause: the probabilities are not calibrated. Splink adds the household fields as separate evidence. So families get probabilities close to 1.0. “0.99” does not mean “99 in 100 are right” here.

Fix: never read a model’s probability as a promise. Measure what share of pairs is true at each score, as the table above does. Choose the threshold by measured precision and recall on validation pairs. Here, the threshold 15.61 on the weight calls 5,300 test pairs the same, with a precision of 0.817.

Consequence in AI work: many models give scores that look like probabilities but are not calibrated. Classifiers, rankers and language models that say how sure they are can all do this. A threshold set on an uncalibrated score gives a different error rate than you expect.

Check your understanding

A pair has a match weight of 0. What is its match probability? And a pair with a weight of 10?

Show the answer

Weight 0: 2⁰ ÷ (1 + 2⁰) = 1 ÷ 2 = 0.5. Weight 10: 2¹⁰ = 1,024, and 1,024 ÷ 1,025 ≈ 0.999. Each +1 doubles the odds. But on Larkfield’s records, these probabilities are not calibrated. Of the test pairs with probabilities from 0.99 to 0.999999, only 9.9% are true pairs.

Check your understanding

With thresholds 10 and 20, the review band of the whole backlog holds 13,378 pairs. How many working days does that take Ivana, and what does it cost at the course's assumptions?

Show the answer

13,378 ÷ 300 = 44.6 working days. Each review takes 1.2 minutes at 40 dollars an hour, so 0.80 dollars a pair: about 10,700 dollars in all. That is the cost of about 71 false merges at 150 dollars each. The match band at 20 still has about 1,800 false merges, so the review is cheap by comparison. The real limit is time: 44.6 days do not fit in Ivana’s 20.

Check your understanding

Omar asks to raise the lower threshold from 10 to 14, so that the review fits in a month. What does Larkfield lose?

Show the answer

The review band shrinks from 13,378 to 3,900 pairs: 13.0 days, which fits. But every pair with a weight from 10 to 14 now stays apart, without review. If every review is right, recall falls from 0.855 to 0.796: about 6% of the true pairs become missed matches. At 3 dollars a year each, that is cheap compared with a false merge. So the choice is reasonable. An alternative: keep the lower threshold and give Ivana more time, or improve the model so fewer pairs are unsure.

Recap

  • Match probability = 2w ÷ (1 + 2w). On Larkfield’s records, Splink’s probabilities are far too sure. Of the test pairs above 0.999999, only 85.5% are true. The brothers get 1.0.
  • The cause: the model adds household fields (postcode, phone, street, surname, email) as separate evidence. The project ranks pairs by the match weight and chooses thresholds by measurement.
  • Two thresholds make a match band, a review band and a non-match band. With 10 and 20 on the test split: 4,604 pairs merge (precision 0.856), 4,262 go to review.
  • On the whole backlog, the review band of 10 to 20 holds 13,378 pairs: 44.6 days for one reviewer at 300 a day. Size the band by the reviewer’s capacity and the cost of each error.
  • Splink without labels reaches F1 0.797. A classifier with labels does better.

Module 5 is done. Splink learned readable weights without labels, but it counts a household’s shared fields many times. Next, in Module 6, a classifier learns from labelled pairs how much each field really adds.

Next: A supervised pair classifier