Evidence from each field
- Lesson 15
- 40 min
- Module 5: Probabilistic linkage
Where we are
Our hand weights worked, but we guessed them. A better weight says how much each agreement tells us. In this lesson, we meet the Fellegi–Sunter idea and its m and u probabilities, and we see why a rare surname counts more.
Lesson 15 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Evidence from each field
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Measure how much each field tells you about a pair. Count how often a field agrees for the same customer and how often it agrees by chance. Turn the two numbers into a match weight, add the weights, and see why a rare surname counts more than a common one.
NotebookCompute the evidence in a notebook
The notebook counts m and u for six fields on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s 3,000 labelled pairs. You compare the weight of Smith with the weight of Oyelaran, and you add the evidence of each field for one pair. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Comparison vectors and the hand-weighted score, from Compare by field, then combine.
- Pair precision, recall and F1, from Measure a matcher.
- Conditional probability helps, but the lesson explains what it needs. Conditional probability in Essential Mathematics and Statistics for AI explains it in full.
What you need on your computer
- The project
larkfield-matchat the end of Module 4, with its virtual environment active. Or use the notebook only.
Two cards at the clinic
Section titled “Two cards at the clinic”A receptionist at a busy clinic finds two patient cards. Both say Smith. Is it one patient filed twice? She shrugs. The clinic has hundreds of Smiths. Two cards with that name tell her very little.
The next day, she finds two cards that both say Oyelaran. Now she looks closely. The clinic has only a handful of patients with that name. Two cards with it are probably the same person.
Then she notices a third thing. Two cards with the same town tell her almost nothing: most patients live in the same town. Two cards with the same date of birth tell her a lot: few patients share one.
Without numbers, she asks one question about each clue: how often would I see this agreement by chance? A clue that is rare among strangers is strong evidence. A clue that strangers share all the time is weak.
The problem: weights that we guessed
Section titled “The problem: weights that we guessed”In Compare by field, then combine, we chose a weight for each field by feel. Email got 3.0, phone 2.5, surname 2.0 and town 0.5. The score adds weight × similarity, and pairs above 8 are “the same”. On the test pairs, it reached F1 0.759, below the four rules of Module 2 (0.790).
The guessed weights have three faults that the receptionist would see at once:
- Smith and Oyelaran count the same. Agreement on a surname adds 2.0 points, whatever the surname is.
- A disagreement counts zero. Two different dates of birth are strong evidence of two people. The score ignores it.
- The numbers mean nothing. Is a score of 8 “very likely” or “a little likely”? Nobody can say.
We need weights that come from the data, with a clear meaning.
The idea: two questions for each field
Section titled “The idea: two questions for each field”For each field, ask two questions:
- m: among pairs of records of the same customer, how often does this field agree?
- u: among pairs of records of different customers, how often does it agree?
The letters come from “match” and “unmatched”. A good field has a high m and a low u. The date of birth agrees for almost every true pair, and for very few pairs of strangers.
The ratio m ÷ u says how many times more likely the agreement is for a true pair than for a pair of strangers. If m is 0.80 and u is 0.01, an agreement is 80 times more likely for a true pair.
We take the logarithm in base 2 of that ratio, and call it the match weight of the agreement:
match weight of agreement = log2(m ÷ u)
On this scale, +1 means “twice as likely for a true pair”, +2 means four times, +10 about a thousand times. A weight of 0 means “no evidence”. A negative weight is evidence against a match. The logarithm lets us add weights, where the ratios would have to be multiplied.
A disagreement is evidence too. Its weight uses the other side of each probability:
match weight of disagreement = log2((1 − m) ÷ (1 − u))
And a missing value adds 0. As in Module 4, an empty field is no evidence either way.
This is the Fellegi–Sunter model, named after Ivan Fellegi and Alan Sunter, who published it in 1969. Almost every probabilistic record matcher today, Splink included, uses it.
A tiny example
Section titled “A tiny example”Imagine 1,000 candidate pairs whose answers we know: 100 pairs of the same customer and 900 pairs of different customers. The numbers are made up, to keep the arithmetic easy.
Text description of the diagram
The picture has three columns. The first column counts the pairs of the same customer: the email agrees in 80 of 100, so m = 0.80; the surname agrees in 90 of 100, so m = 0.90. The second column counts the pairs of different customers: the email agrees in 9 of 900, so u = 0.01; the surname agrees in 45 of 900, so u = 0.05. The third column gives the weights. Email: agree +6.32, disagree -2.31. Surname: agree +4.17, disagree -3.25. A note under the columns gives the prior weight: log2(100 ÷ 900) = -3.17.
Before we look at any field, a random candidate pair is the same customer 100 times in 1,000. The odds are 100 to 900. That starting point is the prior, and its weight is log2(100 ÷ 900) = -3.17. We add each field’s weight to the prior:
| Pair | Prior | Surname | Total weight | Probability | |
|---|---|---|---|---|---|
| Both agree | -3.17 | +6.32 | +4.17 | +7.32 | 0.994 |
| Email only | -3.17 | +6.32 | -3.25 | -0.10 | 0.483 |
| Surname only | -3.17 | -2.31 | +4.17 | -1.31 | 0.287 |
| Neither | -3.17 | -2.31 | -3.25 | -8.73 | 0.002 |
From a weight to a probability
Section titled “From a weight to a probability”The total weight is the logarithm of the odds. To get a probability back, undo the logarithm:
match probability = 2w ÷ (1 + 2w), where w is the total weight.
For the first row: 27.32 is about 160, and 160 ÷ 161 = 0.994. A total of 0 gives 1 ÷ 2 = 0.5: even odds.
This is Bayes’ rule, written with odds: the prior odds times the evidence of each field. Adding the weights treats each field as separate evidence. That is an assumption, and it is not always true. A family shares a surname, an address and a phone. Lesson 3 of this module shows what that assumption costs on Larkfield’s records.
Rare names count more
Section titled “Rare names count more”The receptionist’s question was about the value, not only the field. Agreement on “smith” is common among strangers. Agreement on “oyelaran” is rare.
We can compute u for each value. Say a share s of all records have a surname. Then two random records both have it with a chance of about s × s. Smith is on 3.9% of Larkfield’s records, so u ≈ 0.039 × 0.039 = 0.0015. Oyelaran is on 0.1% of them, so u ≈ 0.000001.
Keep m the same for every surname: 0.95, an assumption. The project computes the weight of agreement for any surname:
match weights --surnames Smith Tremblay Martin Nguyen OyelaranAgreeing on one surname (m = 0.95, an assumption; u from how common the surname is): smith 2,350 records (3.9097%) u 1.53e-03 weight +9.28 tremblay 1,949 records (3.2426%) u 1.05e-03 weight +9.82 martin 1,103 records (1.8351%) u 3.37e-04 weight +11.46 nguyen 730 records (1.2145%) u 1.48e-04 weight +12.65 oyelaran 60 records (0.0998%) u 9.96e-07 weight +19.86- Common surname (more than 500 records)
- Rare surname
Text description of the chart
A dot chart with six rows, from common to rare surnames. smith 9.28; tremblay 9.82; martin 11.46; nguyen 12.65; oyelaran 19.86; laframboise 21.34. The four common surnames have between 730 and 2,350 records each. Oyelaran has 60 and Laframboise 36.
Agreement on “oyelaran” is worth about 10 more than agreement on “smith”. In ratio terms, that is 210, about a thousand times stronger. This is a term frequency adjustment: the weight depends on how frequent the value is. Splink uses it for the surname in the next lesson.
Count m and u on Larkfield’s records
Section titled “Count m and u on Larkfield’s records”The project of this module has a file fs.py that counts m and u for every field. A field can have several levels, not only “agree” or “disagree”. For the first name, the levels are “same”, “close” (Jaro–Winkler 0.92 or more), “similar” (0.8 or more) and “different”. Each level gets its own m, u and weight.
Where do the labels come from? Our records are synthetic, so we know the true customer behind every record. fs.py counts on the candidate pairs of the training split and uses their true answers. In real work you rarely have so many labels. The next lesson removes this need.
Add the files of Module 5
Section titled “Add the files of Module 5”-
In your project folder, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files of the snapshot
end-of-m05. This module has three lessons, and you copy all their files now. Copy itsrequirements.txtandREADME.mdtoo:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m05cp $S/larkfield_match/fs.py $S/larkfield_match/splinkmodel.py larkfield_match/cp $S/larkfield_match/runs/probabilistic.py larkfield_match/runs/cp $S/larkfield_match/commands/weights.py $S/larkfield_match/commands/splink.py larkfield_match/commands/cp $S/larkfield_match/commands/explain.py $S/larkfield_match/commands/bands.py larkfield_match/commands/cp $S/tests/test_probabilistic.py tests/cp $S/requirements.txt $S/README.md .Terminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m05"Copy-Item $S\larkfield_match\fs.py, $S\larkfield_match\splinkmodel.py larkfield_match\Copy-Item $S\larkfield_match\runs\probabilistic.py larkfield_match\runs\Copy-Item $S\larkfield_match\commands\weights.py, $S\larkfield_match\commands\splink.py larkfield_match\commands\Copy-Item $S\larkfield_match\commands\explain.py, $S\larkfield_match\commands\bands.py larkfield_match\commands\Copy-Item $S\tests\test_probabilistic.py tests\Copy-Item $S\requirements.txt, $S\README.md . -
Install the new packages. Splink and DuckDB are for the next lesson:
Terminal window python -m pip install -r requirements.txtThe last line names the new packages:
Successfully installed duckdb-2.0.0 pyarrow-26.0.1 splink-5.0.0 sqlglot-30.30.0 -
Run the tests. There are 29 now:
Terminal window python -m pytest -q............................. [100%]29 passed in 57.64s
Read the weights
Section titled “Read the weights”match weightsCounted on 406,500 training candidate pairs (16,383 true pairs): field level m u weight first same 0.8138 0.4804 0.7605 first close (JW >= 0.92) 0.0505 0.0352 0.5191 first similar (JW >= 0.8) 0.0508 0.1989 -1.9701 first different 0.0850 0.2855 -1.7479 last same 0.8853 0.8608 0.0404 last close (JW >= 0.92) 0.0678 0.0514 0.3997 last different 0.0469 0.0878 -0.9034 email same 0.7142 0.0018 8.6183 email different 0.2858 0.9982 -1.8043 phone same 0.7561 0.0087 6.4359 phone different 0.2439 0.9913 -2.0228postcode same 0.8541 0.0602 3.8255postcode different 0.1459 0.9398 -2.6869 street same 0.8672 0.0649 3.7404 street similar (>= 0.9) 0.0000 0.0002 -7.7952 street different 0.1328 0.9349 -2.8153 town same 0.8796 0.1124 2.9687 town different 0.1204 0.8876 -2.8817 dob same 0.9701 0.0076 6.9995 dob different 0.0299 0.9924 -5.0505Read one row. For the email, m is 0.7142: the email agrees for 71% of the true pairs. u is 0.0018: it agrees for 0.18% of the other candidate pairs. The ratio is about 400, and log2(400) ≈ 8.62.
Text description of the chart
A dot chart with one row per field, from the strongest to the weakest. Email 8.62; date of birth 7.00; phone 6.44; postcode 3.83; street 3.74; town 2.97; first name 0.76; surname 0.04.
Three things stand out:
- Email, date of birth and phone are strong. Strangers rarely share them.
- A different date of birth is strong evidence against a match: −5.05. A different email is weak evidence against: −1.80. Many customers have two emails.
- The surname is worth almost nothing: +0.04. That looks wrong. The failure case below explains it.
Score every pair with these weights
Section titled “Score every pair with these weights”The run fs-labels adds the weights of each candidate pair, starting from the prior. It chooses the threshold on the validation split, as every run of the course does:
match eval --run fs-labelsRan fs-labels: 23,078 pairs called the same, in 242.8 sfs-labels on the test split: pair precision 0.839, recall 0.747, F1 0.790 4,951 pairs called the same: 4,155 right, 796 wrong; 1,409 of 5,564 true pairs missed entities after clustering (all records): 40,151 (true: 37,500)| Run | Precision | Recall | F1 |
|---|---|---|---|
rules |
0.723 | 0.870 | 0.790 |
weighted |
0.751 | 0.766 | 0.759 |
fs-labels |
0.839 | 0.747 | 0.790 |
The measured weights beat our guessed weights: F1 0.790 against 0.759. They tie with the four rules, with a different balance: much higher precision, lower recall. And now every weight has a meaning that we can check.
When to use it, and what it costs
Section titled “When to use it, and what it costs”- Use it when the fields are few and clear, and you can explain a decision field by field. A data steward can read a weight table.
- It needs labels to count m and u. Here the synthetic truth gave us 16,383 true pairs for free. A real project must label pairs by hand. The next lesson learns the weights without labels.
- It assumes the fields are separate evidence. Families break that assumption. Lesson 3 measures the damage.
- It is cheap. Counting is fast, and scoring is one addition per field.
Common mistake
Count u on the candidate pairs, and the surname looks useless
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads the table above. “Agreement on the surname is worth +0.04,” he says. “Let us drop the surname.”
Cause: fs.py counts u on the candidate pairs, not on random pairs. Blocking made those candidates. One of the five keys uses the first 4 letters of the surname. So most candidates share a surname already, and u for “same surname” is 0.86. Among the candidates, a shared surname is normal, so it tells us little. Between two random records, it is rare. The weight answers “how much does this agreement tell us among these pairs?”
Fix: keep the surname. Count u on random pairs of records, or on pairs that blocking did not shape. Splink does this in the next lesson. Its weight for the same surname is about +6. And always read a weight together with the pairs it was counted on.
Consequence in AI work: every learned number depends on the data it was learned from. A model trained on filtered data learns the filter. Machine Learning: From Problem to Reliable Model shows the same effect with a training set that does not look like the real cases.
Knowledge check
Section titled “Knowledge check”Check your understanding
The phone agrees for 76% of the true pairs and for 0.87% of the other candidate pairs. Compute the weight of agreement and the weight of disagreement.
Show the answer
Agreement: log2(0.7561 ÷ 0.0087) = log2(86.9) ≈ +6.44. A shared phone is about 87 times more likely for a true pair. Disagreement: log2((1 − 0.7561) ÷ (1 − 0.0087)) = log2(0.246) ≈ −2.02. A different phone is about 4 times more likely for two different customers. These are the two phone rows of match weights.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two records share the surname Oyelaran, and two other records share the surname Smith. Everything else is the same. Which pair gets the higher total weight, and why?
Show the answer
The Oyelaran pair, by about 10.6 (+19.86 against +9.28, with m = 0.95). Two random records share “smith” much more often, because 3.9% of the records have it. Two random records almost never share “oyelaran” (0.1% of the records). A rare agreement is stronger evidence. Splink applies this with a term frequency adjustment on the surname.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A pair has a total weight of 0. Ivana says: “So there is no evidence.” Is she right?
Show the answer
Not quite. A total of 0 means even odds: a match probability of 2⁰ ÷ (1 + 2⁰) = 0.5. The prior is far below 0: −3.17 in the tiny example, and much lower at Larkfield. The fields moved the pair from the prior up to 0. So the fields gave strong evidence for a match, but not enough to be sure. A field weight of 0 means “no evidence”; a total weight of 0 means “a coin toss”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- m is how often a field agrees for the same customer. u is how often it agrees by chance, for different customers.
- The match weight of agreement is log2(m ÷ u); of disagreement, log2((1 − m) ÷ (1 − u)). A missing value adds 0. Add the weights to the prior for the total.
- The match probability is 2w ÷ (1 + 2w). Adding weights assumes the fields are separate evidence.
- Rare values count more: agreeing on “oyelaran” (+19.86) is worth about a thousand times more than agreeing on “smith” (+9.28).
- Weights counted on labelled training pairs (
fs-labels) reach F1 0.790, above the guessed weights (0.759). u counted on candidate pairs makes the surname look useless.
The weights needed 16,383 labelled true pairs. Next, we let Splink learn them from the records alone.