From weights to a decision
- Lesson 17
- 35 min
- Module 5: Probabilistic linkage
Where we are
Splink gives every pair a match weight. A business needs a decision for each pair. In this lesson, we turn weights into probabilities, set two thresholds, and send the unsure band between them to review.
Lesson 17 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?From weights to a decision
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Turn match weights into decisions. See why Splink’s probabilities are far too sure on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s records, and why the project scores with the weight instead. Then set two thresholds. Pairs above the upper one merge. Pairs below the lower one stay apart. The unsure band between them goes to a person. Choose the thresholds by what the reviewer can do.
NotebookSplit the pairs into bands in a notebook
The notebook trains Splink again, checks how sure its probabilities are against the truth, and draws the weights of true pairs and other pairs. You count the review band for two thresholds and the days of work it gives a reviewer. Its setup cells install Splink and download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Match weights and match probability, from Evidence from each field.
- The Splink model of the project, from Learn the weights without labels.
- The cost of a false merge and of a missed match, from False merges and missed matches.
What you need on your computer
- The project
larkfield-matchwith the files of Module 5 and a trained model (match splink). Or use the notebook only.
Second marking
Section titled “Second marking”A teacher marks 300 exam papers. The pass mark is 50. A paper with 85 is a clear pass, and a paper with 12 is a clear fail. Her marks on these papers are final.
A paper with 49 or 51 is different. One mark more or less changes the result, and her marking is not that exact. So the school sends every paper between 45 and 55 to a second marker. That person reads them again and decides.
How wide is that band? The school cannot send all 300 papers. It asks: how many papers can the second marker read this week? The band is as wide as her time allows.
Matching has the same three parts. Sure matches merge. Sure non-matches stay apart. Pairs in between go to a person: at Larkfield, Ivana.
The problem: probabilities that are far too sure
Section titled “The problem: probabilities that are far too sure”Splink gives each pair a match weight, and from it a match probability: 2w ÷ (1 + 2w). The probability looks like the perfect tool. Merge above 0.99, and you expect at most 1 error in 100.
Look at one pair. G-001688 and G-002894 are two brothers, Samuel and Nathan Green: a practice problem we made on purpose. They share a surname, a family email, a landline and an address.
match explain G-001688 G-002894 prior (two random records) -15.83 (total -15.83) first -3.44 (total -19.28) samuel / nathan last +6.46 (total -12.82) green / green email +12.26 (total -0.55) thegreens@example.com / thegreens@example.com phone +14.16 (total +13.61) 9055551835 / 9055551835 postcode +13.28 (total +26.89) M7K4L8 / M7K4L8 street +7.19 (total +34.07) kestrel ave / kestrel ave town +3.84 (total +37.91) heronvale / heronvale dob +0.00 (total +37.91) None / 1966-05-20Match weight +37.91 -> match probability 1.0000True answer: differentText description of the diagram
Each row is one step, from the top. The prior, −15.83, is the weight of two random records. The first names differ (samuel and nathan), so the first name removes 3.44, to −19.28. Then the surname green adds 6.46, the family email 12.26, the landline 14.16, the postcode 13.28, the street 7.19 and the town 3.84. The date of birth adds nothing, because one record has none. The total is +37.91, and the probability is 1.0000. The true answer: two different people.
The brothers get a match weight of 37.9 and a probability of 1.0. Splink is “sure”, and it is wrong.
Is it only one pair?
Section titled “Is it only one pair?”No. Compare Splink’s probabilities with the truth for all 391,666 candidate pairs of the test split:
| Match probability | Test pairs | Share that are true pairs |
|---|---|---|
| 0 to 0.5 | 152,571 | 0.001 |
| 0.5 to 0.9 | 33,757 | 0.002 |
| 0.9 to 0.99 | 30,011 | 0.005 |
| 0.99 to 0.999999 | 11,621 | 0.099 |
| 0.999999 to 1 | 4,619 | 0.855 |
Of the 4,619 pairs with a probability above 0.999999, only 85.5% are true pairs. And 15 test pairs get a probability of exactly 1.0. A probability that matches the truth is called calibrated. Splink’s probabilities here are far from calibrated.
Why: the same evidence, counted five times
Section titled “Why: the same evidence, counted five times”The Fellegi–Sunter model adds the weight of each field as separate evidence. That is true for strangers: two strangers who share a postcode rarely share a phone too. It is not true in a family. Brothers who share an address also share the postcode, the street, the town, the landline and often the surname and an email. That is one fact, “same household”, and the model counts it five times.
The notebook of this lesson shows the effect. Its model has no street and no town: two household fields fewer. Of its test pairs above 0.999999, 97.4% are true pairs, against 85.5% for the project’s model. Still not calibrated, but much less wrong.
So the project uses the match weight only to rank pairs, and chooses thresholds on the weight, by measurement. The run splink of the last lesson chose a weight of 15.61 on the validation split, not a probability of 0.5. Thresholds and calibration in the machine learning course shows how to measure and repair calibration for a classifier.
Two thresholds, three bands
Section titled “Two thresholds, three bands”One threshold forces a “yes” or “no” for every pair, even the doubtful ones. Two thresholds make three bands:
- Match band: weight at or above the upper threshold. Merge without review.
- Review band: between the two thresholds. A person decides.
- Non-match band: below the lower threshold. Do not merge.
match bandssplink, test split, thresholds 10.0 and 20.0: match band: 4,604 pairs, precision 0.856 review band: 4,262 pairs (795 true pairs, 3,467 not) non-match band: 382,800 pairs, 795 true pairs lost thereIf reviewers decide every review pair correctly: precision 0.877, recall 0.851Text description of the diagram
A weight axis runs from low to high, with dashed lines at 10 and 20. The left band, below 10, is the non-match band: 382,800 pairs; 795 true pairs are lost there; these pairs are not merged. The middle band, from 10 to 20, is the review band: 4,262 pairs, 795 of them true pairs and 3,467 not; Ivana decides. The right band, 20 or more, is the match band: 4,604 pairs with a precision of 0.856; they merge without review.
The chart shows where the true pairs and the other pairs sit on the weight:
- True pairs
- Other candidate pairs
- Thresholds 10 and 20 (dashed)
Text description of the chart
Two lines over the match weight, from −10 to 50, and dashed vertical lines at 10 and 20. The other candidate pairs are mostly on the left. About 13% of them are in each of the bins −10 to −8 and −8 to −6. A second bump of about 7% is at 2 to 4. From 8 on, each bin has under 1%. The true pairs are mostly on the right, with two peaks: 9.7% at weights 22 to 24, and 9.2% at 34 to 36. Between 6 and 20, both lines are low: that is where the two groups overlap. Not drawn: 159,064 other pairs and 23 true pairs that Splink left out with a probability below 0.001.
Each line is a share of its own group, so the true pairs look as common as the others. In counts, they are not. From 6 to 20, the test split has 12,522 other pairs and 1,204 true pairs. The groups overlap there, and no single threshold separates them. The review band sits on that overlap.
How wide can the review band be?
Section titled “How wide can the review band be?”The review band costs Ivana’s time. The course’s assumptions: Ivana reviews 300 pairs a day, 1.2 minutes a pair. Larkfield wants to review its whole backlog once: all splits, not only the test split. For this lesson, we assume one more thing: TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. gives Ivana one month, 20 working days, for the backlog.
Explore
Two thresholds, three bands, one reviewer
Predict first: With thresholds 10 and 20, how many working days of review does Ivana need for the whole backlog?
- Match band
- 21,523 pairsprecision 0.916
- Review band
- 13,378 pairs3,954 true pairs
- Days of review
- 44.6 days
- If every review is right
- recall 0.855, precision 0.929
Thresholds 10 and 20: 21,523 pairs merge without review (precision 0.916). 13,378 pairs go to review, and 3,954 of them are true pairs. One reviewer at 300 pairs a day needs 44.6 working days. If every review is right: recall 0.855, precision 0.929. That is more than the 20 days of review that Ivana can give.
| Lower ↓ / upper → | 16 | 18 | 20 | 22 | 25 |
|---|---|---|---|---|---|
| 5 | 178.453,524 | 182.054,614 | 187.556,264 | 197.259,150 | 212.363,688 |
| 8 | 68.320,498 | 72.021,588 | 77.523,238 | 87.126,124 | 102.230,662 |
| 10 | 35.510,638 | 39.111,728 | 44.613,378 | 54.216,264 | 69.320,802 |
| 12 | 11.33,397 | 15.04,487 | 20.56,137 | 30.19,023 | 45.213,561 |
| 14 | 3.91,160 | 7.52,250 | 13.03,900 | 22.66,786 | 37.711,324 |
Text description of the explorer
The explorer has two sliders. The lower threshold takes the values 5, 8, 10, 12 and 14; the upper threshold 16, 18, 20, 22 and 25. They start at 10 and 20. Four tiles show the match band (21,523 pairs, precision 0.916), the review band (13,378 pairs, 3,954 of them true pairs), the days of review (44.6) and the result if every review is right (recall 0.855, precision 0.929). A sentence repeats these numbers and says whether the review fits in Ivana’s 20 days. A bar chart shows the days of review for each lower threshold at the chosen upper threshold, with a dashed line at 20 days. A table lists the days and the review pairs for all 25 pairs of thresholds. With an upper threshold of 20: lower 5 gives 187.5 days; 8, 77.5; 10, 44.6; 12, 20.5; 14, 13.0. With a lower threshold of 10: upper 16 gives 35.5 days; 18, 39.1; 20, 44.6; 22, 54.2; 25, 69.3.
Things to try:
- Keep the upper threshold at 20 and move the lower one. Each step down adds thousands of pairs to review, and gains a little recall.
- Keep the lower threshold at 10 and move the upper one. A higher upper threshold makes the match band more precise, from 0.894 at 16 to 0.953 at 25. Those pairs move to review: more days.
- Find the pairs of thresholds that fit in 20 days. Which of them loses the fewest true pairs?
Read the trade-off with costs
Section titled “Read the trade-off with costs”With thresholds 10 and 20, the review takes 44.6 days: more than twice Ivana’s month. Thresholds 14 and 20 take 13.0 days and fit. The price: recall if every review is right falls from 0.855 to 0.796.
The course’s costs help to judge it (assumptions). A false merge costs 150 dollars: a stranger sees another customer’s orders and address. A missed match costs 3 dollars a year. One review costs 1.2 minutes at 40 dollars an hour: 0.80 dollars. So reviewing a doubtful pair is cheap, and a false merge is very expensive. The match band at 20 has a precision of 0.916 on all splits. So about 1,800 of its 21,523 pairs are false merges. Raising the upper threshold moves some of them to Ivana, and that is usually worth her time.
Try one more pair of thresholds in the project. The output is for the test split only:
match bands --lower 12 --upper 20splink, test split, thresholds 12.0 and 20.0: match band: 4,604 pairs, precision 0.856 review band: 1,820 pairs (581 true pairs, 1,239 not) non-match band: 385,242 pairs, 1,009 true pairs lost thereIf reviewers decide every review pair correctly: precision 0.872, recall 0.813When to use a review band, and what it costs
Section titled “When to use a review band, and what it costs”- Use it whenever an error is expensive and a person can decide better than the model. At Larkfield, one false merge costs as much as 50 years of one missed match.
- Choose the thresholds on validation pairs, then check them on the test pairs, as the explorer’s numbers were measured.
- Size the band by the reviewer’s capacity, not by the model alone. Ask how many pairs a day, and for how many days.
- Do not trust a probability that you have not checked. Measure its calibration first. For Splink on these records, use the weight.
- A better model shrinks the overlap, and so the review band. A supervised pair classifier in Module 6 does that. Human review in Module 8 runs the review queue every day.
Common mistake
Merge every pair above a probability of 0.99
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads the probabilities. “We merge everything above 0.99. That is at most 1% wrong.” On the test split, 16,240 pairs have a probability above 0.99. Only 5,104 of them are true pairs: a precision of 0.314. Two of every three merges join different people.
Cause: the probabilities are not calibrated. Splink adds the household fields as separate evidence. So families get probabilities close to 1.0. “0.99” does not mean “99 in 100 are right” here.
Fix: never read a model’s probability as a promise. Measure what share of pairs is true at each score, as the table above does. Choose the threshold by measured precision and recall on validation pairs. Here, the threshold 15.61 on the weight calls 5,300 test pairs the same, with a precision of 0.817.
Consequence in AI work: many models give scores that look like probabilities but are not calibrated. Classifiers, rankers and language models that say how sure they are can all do this. A threshold set on an uncalibrated score gives a different error rate than you expect.
Knowledge check
Section titled “Knowledge check”Check your understanding
A pair has a match weight of 0. What is its match probability? And a pair with a weight of 10?
Show the answer
Weight 0: 2⁰ ÷ (1 + 2⁰) = 1 ÷ 2 = 0.5. Weight 10: 2¹⁰ = 1,024, and 1,024 ÷ 1,025 ≈ 0.999. Each +1 doubles the odds. But on Larkfield’s records, these probabilities are not calibrated. Of the test pairs with probabilities from 0.99 to 0.999999, only 9.9% are true pairs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With thresholds 10 and 20, the review band of the whole backlog holds 13,378 pairs. How many working days does that take Ivana, and what does it cost at the course's assumptions?
Show the answer
13,378 ÷ 300 = 44.6 working days. Each review takes 1.2 minutes at 40 dollars an hour, so 0.80 dollars a pair: about 10,700 dollars in all. That is the cost of about 71 false merges at 150 dollars each. The match band at 20 still has about 1,800 false merges, so the review is cheap by comparison. The real limit is time: 44.6 days do not fit in Ivana’s 20.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks to raise the lower threshold from 10 to 14, so that the review fits in a month. What does Larkfield lose?
Show the answer
The review band shrinks from 13,378 to 3,900 pairs: 13.0 days, which fits. But every pair with a weight from 10 to 14 now stays apart, without review. If every review is right, recall falls from 0.855 to 0.796: about 6% of the true pairs become missed matches. At 3 dollars a year each, that is cheap compared with a false merge. So the choice is reasonable. An alternative: keep the lower threshold and give Ivana more time, or improve the model so fewer pairs are unsure.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Match probability = 2w ÷ (1 + 2w). On Larkfield’s records, Splink’s probabilities are far too sure. Of the test pairs above 0.999999, only 85.5% are true. The brothers get 1.0.
- The cause: the model adds household fields (postcode, phone, street, surname, email) as separate evidence. The project ranks pairs by the match weight and chooses thresholds by measurement.
- Two thresholds make a match band, a review band and a non-match band. With 10 and 20 on the test split: 4,604 pairs merge (precision 0.856), 4,262 go to review.
- On the whole backlog, the review band of 10 to 20 holds 13,378 pairs: 44.6 days for one reviewer at 300 a day. Size the band by the reviewer’s capacity and the cost of each error.
- Splink without labels reaches F1 0.797. A classifier with labels does better.
Module 5 is done. Splink learned readable weights without labels, but it counts a household’s shared fields many times. Next, in Module 6, a classifier learns from labelled pairs how much each field really adds.