Skip to content
Français

Evidence from each field

  • Lesson 15
  • 40 min
  • Module 5: Probabilistic linkage

Where we are

Our hand weights worked, but we guessed them. A better weight says how much each agreement tells us. In this lesson, we meet the Fellegi–Sunter idea and its m and u probabilities, and we see why a rare surname counts more.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Measure how much each field tells you about a pair. Count how often a field agrees for the same customer and how often it agrees by chance. Turn the two numbers into a match weight, add the weights, and see why a rare surname counts more than a common one.

NotebookCompute the evidence in a notebookColab · Kaggle · GitHub

The notebook counts m and u for six fields on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s 3,000 labelled pairs. You compare the weight of Smith with the weight of Oyelaran, and you add the evidence of each field for one pair. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-match at the end of Module 4, with its virtual environment active. Or use the notebook only.

A receptionist at a busy clinic finds two patient cards. Both say Smith. Is it one patient filed twice? She shrugs. The clinic has hundreds of Smiths. Two cards with that name tell her very little.

The next day, she finds two cards that both say Oyelaran. Now she looks closely. The clinic has only a handful of patients with that name. Two cards with it are probably the same person.

Then she notices a third thing. Two cards with the same town tell her almost nothing: most patients live in the same town. Two cards with the same date of birth tell her a lot: few patients share one.

Without numbers, she asks one question about each clue: how often would I see this agreement by chance? A clue that is rare among strangers is strong evidence. A clue that strangers share all the time is weak.

In Compare by field, then combine, we chose a weight for each field by feel. Email got 3.0, phone 2.5, surname 2.0 and town 0.5. The score adds weight × similarity, and pairs above 8 are “the same”. On the test pairs, it reached F1 0.759, below the four rules of Module 2 (0.790).

The guessed weights have three faults that the receptionist would see at once:

  • Smith and Oyelaran count the same. Agreement on a surname adds 2.0 points, whatever the surname is.
  • A disagreement counts zero. Two different dates of birth are strong evidence of two people. The score ignores it.
  • The numbers mean nothing. Is a score of 8 “very likely” or “a little likely”? Nobody can say.

We need weights that come from the data, with a clear meaning.

For each field, ask two questions:

  • m: among pairs of records of the same customer, how often does this field agree?
  • u: among pairs of records of different customers, how often does it agree?

The letters come from “match” and “unmatched”. A good field has a high m and a low u. The date of birth agrees for almost every true pair, and for very few pairs of strangers.

The ratio m ÷ u says how many times more likely the agreement is for a true pair than for a pair of strangers. If m is 0.80 and u is 0.01, an agreement is 80 times more likely for a true pair.

We take the logarithm in base 2 of that ratio, and call it the match weight of the agreement:

match weight of agreement = log2(m ÷ u)

On this scale, +1 means “twice as likely for a true pair”, +2 means four times, +10 about a thousand times. A weight of 0 means “no evidence”. A negative weight is evidence against a match. The logarithm lets us add weights, where the ratios would have to be multiplied.

A disagreement is evidence too. Its weight uses the other side of each probability:

match weight of disagreement = log2((1 − m) ÷ (1 − u))

And a missing value adds 0. As in Module 4, an empty field is no evidence either way.

This is the Fellegi–Sunter model, named after Ivan Fellegi and Alan Sunter, who published it in 1969. Almost every probabilistic record matcher today, Splink included, uses it.

Imagine 1,000 candidate pairs whose answers we know: 100 pairs of the same customer and 900 pairs of different customers. The numbers are made up, to keep the arithmetic easy.

m and u counted on 1,000 labelled pairs, a made-up example. Email agrees in 80 of 100 pairs of the same customer, so m = 0.80, and in 9 of 900 pairs of different customers, so u = 0.01. Agreement weight +6.32, disagreement weight -2.31. Surname agrees in 90 of 100: m = 0.90; in 45 of 900: u = 0.05. Agreement weight +4.17, disagreement weight -3.25. Prior weight log2(100 ÷ 900) = -3.17.
m and u for two fields, counted on 1,000 pairs whose answers we know (a made-up example).
Text description of the diagram

The picture has three columns. The first column counts the pairs of the same customer: the email agrees in 80 of 100, so m = 0.80; the surname agrees in 90 of 100, so m = 0.90. The second column counts the pairs of different customers: the email agrees in 9 of 900, so u = 0.01; the surname agrees in 45 of 900, so u = 0.05. The third column gives the weights. Email: agree +6.32, disagree -2.31. Surname: agree +4.17, disagree -3.25. A note under the columns gives the prior weight: log2(100 ÷ 900) = -3.17.

Before we look at any field, a random candidate pair is the same customer 100 times in 1,000. The odds are 100 to 900. That starting point is the prior, and its weight is log2(100 ÷ 900) = -3.17. We add each field’s weight to the prior:

Pair Prior Email Surname Total weight Probability
Both agree -3.17 +6.32 +4.17 +7.32 0.994
Email only -3.17 +6.32 -3.25 -0.10 0.483
Surname only -3.17 -2.31 +4.17 -1.31 0.287
Neither -3.17 -2.31 -3.25 -8.73 0.002

The total weight is the logarithm of the odds. To get a probability back, undo the logarithm:

match probability = 2w ÷ (1 + 2w), where w is the total weight.

For the first row: 27.32 is about 160, and 160 ÷ 161 = 0.994. A total of 0 gives 1 ÷ 2 = 0.5: even odds.

This is Bayes’ rule, written with odds: the prior odds times the evidence of each field. Adding the weights treats each field as separate evidence. That is an assumption, and it is not always true. A family shares a surname, an address and a phone. Lesson 3 of this module shows what that assumption costs on Larkfield’s records.

The receptionist’s question was about the value, not only the field. Agreement on “smith” is common among strangers. Agreement on “oyelaran” is rare.

We can compute u for each value. Say a share s of all records have a surname. Then two random records both have it with a chance of about s × s. Smith is on 3.9% of Larkfield’s records, so u ≈ 0.039 × 0.039 = 0.0015. Oyelaran is on 0.1% of them, so u ≈ 0.000001.

Keep m the same for every surname: 0.95, an assumption. The project computes the weight of agreement for any surname:

Terminal window
match weights --surnames Smith Tremblay Martin Nguyen Oyelaran
Agreeing on one surname (m = 0.95, an assumption; u from how common the surname is):
smith 2,350 records (3.9097%) u 1.53e-03 weight +9.28
tremblay 1,949 records (3.2426%) u 1.05e-03 weight +9.82
martin 1,103 records (1.8351%) u 3.37e-04 weight +11.46
nguyen 730 records (1.2145%) u 1.48e-04 weight +12.65
oyelaran 60 records (0.0998%) u 9.96e-07 weight +19.86
  • Common surname (more than 500 records)
  • Rare surname
The weight of agreeing on one surname, with m = 0.95 (an assumption) and u from how common the surname is in Larkfield's 60,108 synthetic records. Measured.
Text description of the chart

A dot chart with six rows, from common to rare surnames. smith 9.28; tremblay 9.82; martin 11.46; nguyen 12.65; oyelaran 19.86; laframboise 21.34. The four common surnames have between 730 and 2,350 records each. Oyelaran has 60 and Laframboise 36.

Agreement on “oyelaran” is worth about 10 more than agreement on “smith”. In ratio terms, that is 210, about a thousand times stronger. This is a term frequency adjustment: the weight depends on how frequent the value is. Splink uses it for the surname in the next lesson.

The project of this module has a file fs.py that counts m and u for every field. A field can have several levels, not only “agree” or “disagree”. For the first name, the levels are “same”, “close” (Jaro–Winkler 0.92 or more), “similar” (0.8 or more) and “different”. Each level gets its own m, u and weight.

Where do the labels come from? Our records are synthetic, so we know the true customer behind every record. fs.py counts on the candidate pairs of the training split and uses their true answers. In real work you rarely have so many labels. The next lesson removes this need.

  1. In your project folder, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files of the snapshot end-of-m05. This module has three lessons, and you copy all their files now. Copy its requirements.txt and README.md too:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m05
    cp $S/larkfield_match/fs.py $S/larkfield_match/splinkmodel.py larkfield_match/
    cp $S/larkfield_match/runs/probabilistic.py larkfield_match/runs/
    cp $S/larkfield_match/commands/weights.py $S/larkfield_match/commands/splink.py larkfield_match/commands/
    cp $S/larkfield_match/commands/explain.py $S/larkfield_match/commands/bands.py larkfield_match/commands/
    cp $S/tests/test_probabilistic.py tests/
    cp $S/requirements.txt $S/README.md .
  2. Install the new packages. Splink and DuckDB are for the next lesson:

    Terminal window
    python -m pip install -r requirements.txt

    The last line names the new packages:

    Successfully installed duckdb-2.0.0 pyarrow-26.0.1 splink-5.0.0 sqlglot-30.30.0
  3. Run the tests. There are 29 now:

    Terminal window
    python -m pytest -q
    ............................. [100%]
    29 passed in 57.64s
Terminal window
match weights
Counted on 406,500 training candidate pairs (16,383 true pairs):
field level m u weight
first same 0.8138 0.4804 0.7605
first close (JW >= 0.92) 0.0505 0.0352 0.5191
first similar (JW >= 0.8) 0.0508 0.1989 -1.9701
first different 0.0850 0.2855 -1.7479
last same 0.8853 0.8608 0.0404
last close (JW >= 0.92) 0.0678 0.0514 0.3997
last different 0.0469 0.0878 -0.9034
email same 0.7142 0.0018 8.6183
email different 0.2858 0.9982 -1.8043
phone same 0.7561 0.0087 6.4359
phone different 0.2439 0.9913 -2.0228
postcode same 0.8541 0.0602 3.8255
postcode different 0.1459 0.9398 -2.6869
street same 0.8672 0.0649 3.7404
street similar (>= 0.9) 0.0000 0.0002 -7.7952
street different 0.1328 0.9349 -2.8153
town same 0.8796 0.1124 2.9687
town different 0.1204 0.8876 -2.8817
dob same 0.9701 0.0076 6.9995
dob different 0.0299 0.9924 -5.0505

Read one row. For the email, m is 0.7142: the email agrees for 71% of the true pairs. u is 0.0018: it agrees for 0.18% of the other candidate pairs. The ratio is about 400, and log2(400) ≈ 8.62.

The weight of agreement for each field, counted on 406,500 candidate pairs of the training split of the synthetic records, with their true answers. Measured.
Text description of the chart

A dot chart with one row per field, from the strongest to the weakest. Email 8.62; date of birth 7.00; phone 6.44; postcode 3.83; street 3.74; town 2.97; first name 0.76; surname 0.04.

Three things stand out:

  • Email, date of birth and phone are strong. Strangers rarely share them.
  • A different date of birth is strong evidence against a match: −5.05. A different email is weak evidence against: −1.80. Many customers have two emails.
  • The surname is worth almost nothing: +0.04. That looks wrong. The failure case below explains it.

The run fs-labels adds the weights of each candidate pair, starting from the prior. It chooses the threshold on the validation split, as every run of the course does:

Terminal window
match eval --run fs-labels
Ran fs-labels: 23,078 pairs called the same, in 242.8 s
fs-labels on the test split: pair precision 0.839, recall 0.747, F1 0.790
4,951 pairs called the same: 4,155 right, 796 wrong; 1,409 of 5,564 true pairs missed
entities after clustering (all records): 40,151 (true: 37,500)
Run Precision Recall F1
rules 0.723 0.870 0.790
weighted 0.751 0.766 0.759
fs-labels 0.839 0.747 0.790

The measured weights beat our guessed weights: F1 0.790 against 0.759. They tie with the four rules, with a different balance: much higher precision, lower recall. And now every weight has a meaning that we can check.

  • Use it when the fields are few and clear, and you can explain a decision field by field. A data steward can read a weight table.
  • It needs labels to count m and u. Here the synthetic truth gave us 16,383 true pairs for free. A real project must label pairs by hand. The next lesson learns the weights without labels.
  • It assumes the fields are separate evidence. Families break that assumption. Lesson 3 measures the damage.
  • It is cheap. Counting is fast, and scoring is one addition per field.

Common mistake

Count u on the candidate pairs, and the surname looks useless

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads the table above. “Agreement on the surname is worth +0.04,” he says. “Let us drop the surname.”

Cause: fs.py counts u on the candidate pairs, not on random pairs. Blocking made those candidates. One of the five keys uses the first 4 letters of the surname. So most candidates share a surname already, and u for “same surname” is 0.86. Among the candidates, a shared surname is normal, so it tells us little. Between two random records, it is rare. The weight answers “how much does this agreement tell us among these pairs?”

Fix: keep the surname. Count u on random pairs of records, or on pairs that blocking did not shape. Splink does this in the next lesson. Its weight for the same surname is about +6. And always read a weight together with the pairs it was counted on.

Consequence in AI work: every learned number depends on the data it was learned from. A model trained on filtered data learns the filter. Machine Learning: From Problem to Reliable Model shows the same effect with a training set that does not look like the real cases.

Check your understanding

The phone agrees for 76% of the true pairs and for 0.87% of the other candidate pairs. Compute the weight of agreement and the weight of disagreement.

Show the answer

Agreement: log2(0.7561 ÷ 0.0087) = log2(86.9) ≈ +6.44. A shared phone is about 87 times more likely for a true pair. Disagreement: log2((1 − 0.7561) ÷ (1 − 0.0087)) = log2(0.246) ≈ −2.02. A different phone is about 4 times more likely for two different customers. These are the two phone rows of match weights.

Check your understanding

Two records share the surname Oyelaran, and two other records share the surname Smith. Everything else is the same. Which pair gets the higher total weight, and why?

Show the answer

The Oyelaran pair, by about 10.6 (+19.86 against +9.28, with m = 0.95). Two random records share “smith” much more often, because 3.9% of the records have it. Two random records almost never share “oyelaran” (0.1% of the records). A rare agreement is stronger evidence. Splink applies this with a term frequency adjustment on the surname.

Check your understanding

A pair has a total weight of 0. Ivana says: “So there is no evidence.” Is she right?

Show the answer

Not quite. A total of 0 means even odds: a match probability of 2⁰ ÷ (1 + 2⁰) = 0.5. The prior is far below 0: −3.17 in the tiny example, and much lower at Larkfield. The fields moved the pair from the prior up to 0. So the fields gave strong evidence for a match, but not enough to be sure. A field weight of 0 means “no evidence”; a total weight of 0 means “a coin toss”.

Recap

  • m is how often a field agrees for the same customer. u is how often it agrees by chance, for different customers.
  • The match weight of agreement is log2(m ÷ u); of disagreement, log2((1 − m) ÷ (1 − u)). A missing value adds 0. Add the weights to the prior for the total.
  • The match probability is 2w ÷ (1 + 2w). Adding weights assumes the fields are separate evidence.
  • Rare values count more: agreeing on “oyelaran” (+19.86) is worth about a thousand times more than agreeing on “smith” (+9.28).
  • Weights counted on labelled training pairs (fs-labels) reach F1 0.790, above the guessed weights (0.759). u counted on candidate pairs makes the surname look useless.

The weights needed 16,383 labelled true pairs. Next, we let Splink learn them from the records alone.

Next: Learn the weights without labels