What blocking loses
- Lesson 11
- 35 min
- Module 3: Blocking: do not compare everything
Where we are
We now have several ways to make candidate pairs. Each way can lose true pairs. In this lesson, we measure pair completeness and the reduction ratio, and we choose keys that also catch the customers who moved.
Lesson 11 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?What blocking loses
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Measure what a blocking loses, with pair completeness and the reduction ratio. Find which true pairs a key loses, and choose keys that also catch TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s customers who moved.
NotebookFind what a blocking loses in a notebook
The notebook measures the five keys on Larkfield’s records. You list the true pairs that the postcode loses, find what they have in common, and measure a blocking that catches them. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Blocking keys, keys together, and sorted neighbourhood, from Blocking keys and sorted neighbourhood.
- MinHash LSH for product titles, from MinHash and LSH.
- Recall, from Measure a matcher.
What you need on your computer
- The project
larkfield-matchwith the blocking files of this module. Or use the notebook only.
The fishing net
Section titled “The fishing net”A fisher chooses the mesh of her net. A wide mesh lets the small fish swim out: the net is light, and she sorts the catch fast. A fine mesh keeps almost every fish, but the net is heavy with weed and stones, and sorting takes all day.
She cannot judge the net by the weight of the catch alone. She must also know which fish got away. If every fish that got away is of one kind, the net has a hole in one place. She can mend that place.
Blocking is the net. The candidate pairs are the catch. The true pairs that the blocking removes are the fish that got away, and they never come back.
The problem: “blocking is only about speed”
Section titled “The problem: “blocking is only about speed””TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. has a fast matcher in mind. “Blocking is only about speed,” he says. “It removes pairs that are clearly different. It cannot lose matches.”
He blocks on the postcode. The candidate pairs fall from 1.8 billion to 89,630. A later model scores every candidate pair very well. Then TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. counts the customers. Many customers who moved still have two records. Every model that Tomás tries gives the same missed matches.
The model is not the problem. The postcode blocking removed those pairs before any model saw them. A missed match in blocking is a missed match of the whole matcher.
Two numbers, one more time
Section titled “Two numbers, one more time”You met both numbers in Why all pairs is impossible. Pair completeness is the share of true pairs that the blocking keeps: the recall of the blocking. No later step can have a higher pair recall, because a model can only say “same” for a pair that it sees. The reduction ratio is the share of all pairs that the blocking removes: the saved work. A reduction ratio of 0.999 still leaves 1.8 million of Larkfield’s pairs; 0.9999 leaves 180,000.
A tiny example
Section titled “A tiny example”Go back to the six records of Why all pairs is impossible: 15 pairs, 6 true pairs.
| Blocking | Candidates | True pairs kept | Pair completeness | Reduction ratio |
|---|---|---|---|---|
| Postcode | 6 | 3 | 3 / 6 = 0.50 | 1 − 6/15 = 0.60 |
| Surname key | 6 | 6 | 6 / 6 = 1.00 | 1 − 6/15 = 0.60 |
| Both keys | 9 | 6 | 6 / 6 = 1.00 | 1 − 9/15 = 0.40 |
| Window of 3, sorted by name | 9 | 5 | 5 / 6 = 0.83 | 1 − 9/15 = 0.40 |
On six records, every blocking removes few pairs. On 60,108 records, the same keys remove more than 99.9%.
A third number is sometimes useful: pair quality, the share of candidates that are true pairs. For the five keys it is 27,503 ÷ 1,108,591 = 0.025. Most candidates are still different people. That is normal: the comparison step must sort them.
Explore every blocking
Section titled “Explore every blocking”The explorer shows all the blockings of this module, measured on Larkfield’s synthetic records. Each dot is one blocking: the number of candidate pairs (on a log scale, where each step is ten times more) against the pair completeness. Choose a blocking to see its exact numbers.
Explore
What each blocking keeps and loses
Predict first: The postcode alone keeps 84.5% of the true pairs. Which true pairs does it lose?
- Candidate pairs
- 89,630
- Pair completeness
- 0.845
- True pairs lost
- 4,289
- Reduction ratio
- 0.999950
postcode (same postcode): 89,630 candidate pairs out of 1,806,455,778. It keeps 23,388 of 27,677 true pairs and loses 4,289. True pairs with two different postcodes (mostly customers who moved): postcode keeps 0 of 4,289.
- One key
- Several keys
- Sorted neighbourhood
| Blocking | Candidate pairs | True pairs kept | Pair completeness | Reduction ratio |
|---|---|---|---|---|
postcode | 89,630 | 23,388 | 0.845 | 0.999950 |
surname | 1,035,498 | 23,489 | 0.849 | 0.999427 |
email | 13,952 | 12,472 | 0.451 | 0.999992 |
phone | 19,873 | 13,678 | 0.494 | 0.999989 |
dob | 8,757 | 6,495 | 0.235 | 0.999995 |
postcode+surname | 1,103,620 | 27,019 | 0.976 | 0.999389 |
keys | 1,108,591 | 27,503 | 0.994 | 0.999386 |
keys-wide | 2,351,382 | 27,546 | 0.995 | 0.998698 |
sorted-name | 540,927 | 15,023 | 0.543 | 0.999701 |
sorted-name-50 | 2,944,067 | 22,144 | 0.800 | 0.998370 |
Text description of the explorer
The explorer has one button for each blocking: postcode, surname, email, phone, dob, postcode+surname, keys, keys-wide, sorted-name and sorted-name-50. It starts on postcode. Four tiles show the candidate pairs, the pair completeness, the true pairs lost and the reduction ratio. A sentence says what the blocking is. For postcode and keys, it also says how many of the 4,289 true pairs with two different postcodes it keeps. A chart shows every blocking as a dot. The candidate pairs run from 1,000 to 10 million on a log scale, and the pair completeness from 0.2 to 1. The chosen blocking is a larger dark dot with its name. A table below lists every blocking. Postcode: 89,630 candidates, completeness 0.845, 4,289 true pairs lost, and 0 of the 4,289 movers kept. Surname: 1,035,498, 0.849. Email: 13,952, 0.451. Phone: 19,873, 0.494. Date of birth: 8,757, 0.235. Postcode or surname: 1,103,620, 0.976. The five keys: 1,108,591, 0.994, 174 lost, and 4,115 of the 4,289 movers kept. The five keys with a wider surname key: 2,351,382, 0.995. Sorted neighbourhood, window 10: 540,927, 0.543; window 50: 2,944,067, 0.800.
Things to try:
- Choose
postcode. It loses exactly 4,289 true pairs. Then choosekeys. Read the sentence about the customers who moved for both. - Compare
surnameandpostcode+surname. One extra key adds almost no candidates, and keeps 3,530 more true pairs. - Compare
sorted-name-50andkeys. Which one is better on both numbers?
Which pairs does the postcode lose?
Section titled “Which pairs does the postcode lose?”The prediction asked which true pairs the postcode loses. The answer is exact. The postcode loses 4,289 true pairs. Larkfield’s records have exactly 4,289 true pairs whose two records have different postcodes. Most of them, 3,776, are pairs of customers who moved. One record has the old address, and one has the new. Moving is a practice problem that we made on purpose. The other 513 pairs differ for another reason, such as a typing error in one postcode. Valérie Tremblay’s shop record has H6P 3H1, and her loyalty record H6P 3H7. In this lesson, we call all 4,289 pairs the movers’ pairs, for short.
The lost pairs have one thing in common, so we can mend the net in one place. Look at one of them.
Text description of the diagram
Jacques Beaulieu has a garden-centre record, G-009906, and a shop record, S-005041. The table compares their five keys. The postcodes differ: J8Y1L3 and J0C6V5, because he moved. The garden-centre record has no email, and the shop record has no phone, so neither key can join them. Neither record has a date of birth. Only the surname key, beau|ja, is the same in both. So the postcode alone never compares his two records, and the five keys together do.
Jacques is typical. A customer who moves changes the address and the postcode, and often the phone. The name stays. So the keys that catch movers are the name keys.
Count the movers in the project
Section titled “Count the movers in the project”The project has no command for this, so write a small script. Make a file movers.py in your project folder:
from larkfield_match import blocking, evaluate, prepared
postcode = prepared.customers().set_index("record_id").postcodetrue = evaluate.true_pairs()moved = true[true.record_id_l.map(postcode) != true.record_id_r.map(postcode)]print(f"{len(moved):,} of {len(true):,} true pairs have two different postcodes")for name in ["postcode", "surname", "keys", "keys-wide"]: kept = moved.merge(blocking.candidates(name), on=["record_id_l", "record_id_r"]) print(f" {name:<10} keeps {len(kept):,} of them")The script reads the prepared records and the true pairs. It keeps the true pairs whose postcodes differ. Then it checks which of them are candidates of each blocking. blocking.candidates reads the pairs that match block saved, or makes them the first time. Run it:
python movers.py4,289 of 27,677 true pairs have two different postcodes postcode keeps 0 of them surname keeps 3,631 of them keys keeps 4,115 of them keys-wide keeps 4,158 of themThe chart shows the same measure for more blockings:
- One key or sorted names
- Five keys
Text description of the chart
A dot chart with eight rows, one dot per blocking, on an axis from 0 to 1. Postcode 0.00; date of birth 0.22; email 0.43; phone 0.50; sorted names with window 50: 0.79; surname key 0.85; the five keys 0.96; the five keys with the wider surname key 0.97.
The postcode keeps none of them, by definition. The surname key keeps 85%. The five keys keep 96%. Every key adds a few movers that the others miss: a mover with a nickname may still have the same phone or email.
What the five keys still lose
Section titled “What the five keys still lose”The five keys lose 174 true pairs: 27,677 − 27,503. And the five keys lose exactly 174 of the movers’ pairs: 4,289 − 4,115. So every true pair that keys loses has two different postcodes: 156 of the 174 belong to customers who moved. These pairs differ in the postcode, and also in each of the other four keys. They share no email and no phone. The date of birth is empty or different. And the surname key differs: a nickname, a typing error in the first letters, or an empty name.
How could we catch more? Two options, both measured:
| Blocking | Candidates | True pairs kept | Pair completeness | Movers kept |
|---|---|---|---|---|
keys |
1,108,591 | 27,503 | 0.9937 | 4,115 |
keys+street (adds the street number and street) |
1,120,529 | 27,517 | 0.9942 | 4,129 |
keys-wide (surname key with 1 letter of the first name) |
2,351,382 | 27,546 | 0.9953 | 4,158 |
keys-wide catches 43 more true pairs, but makes 2.1 times more candidates. keys+street catches 14 more, with 1% more candidates. A street key cannot help most movers: their street changed too.
match block --blocking keys-widematch block --blocking keys+streetkeys-wide: 2,351,382 candidate pairs out of 1,806,455,778 possible pairs (11.16 s) pair completeness 0.995 (27,546 of 27,677 true pairs kept); reduction ratio 0.998698keys+street: 1,120,529 candidate pairs out of 1,806,455,778 possible pairs (6.31 s) pair completeness 0.994 (27,517 of 27,677 true pairs kept); reduction ratio 0.999380Choose by cost, not by feel
Section titled “Choose by cost, not by feel”Which to choose? Ask what each option costs.
- More candidates cost compute time. In our bench on the course data, the comparison step of Module 4 took 2.5 seconds for the 1,108,591 candidates of
keys. Twice the candidates take about twice the time: a few seconds more. On Larkfield’s data, that is cheap. - A lost true pair costs a missed match. The course’s assumed cost of a missed match is 3 dollars a year: double emails and split loyalty points.
At 60,108 records, keys-wide would be a fair choice. The project keeps keys for one reason: growth. A wider key grows much faster with more records, because its blocks are bigger. Scale in Module 8 shows that at 1.1 million records, the five keys already make about 340 million candidate pairs. A wider surname key would make that much worse.
The ceiling on recall
Section titled “The ceiling on recall”Every run from Module 4 on compares only the candidates of keys. On the test split, keys keeps 5,531 of the 5,564 true pairs: a pair completeness of 0.9941. So no later matcher can reach a pair recall above 0.9941 on the test split. When a model in Module 6 misses a true pair, first check whether the pair was a candidate at all.
When to measure blocking
Section titled “When to measure blocking”- Every time you change a key, a window or a threshold. Measure the pair completeness and the reduction ratio together.
- By group, not only in total. Total pair completeness of 0.845 hides that the postcode loses all movers. Measure the groups that matter to the business: movers, each source, each region. The project’s
match eval --bydoes this for runs in Module 2; the notebook does it for blocking. - On all records, with true pairs that you trust. Here the synthetic records have known true customers. On real data, you need labelled pairs, and the pairs that blocking removes are exactly the ones a labeller never sees. Label a few random pairs that blocking removes, too.
Common mistake
Judge a blocking by its total alone
Symptom: Tomás compares the postcode (pair completeness 0.845) and the surname key (0.849). “They are the same,” he says, “so choose the faster one.” He chooses the postcode: 89,630 candidates against 1,035,498.
Cause: the two totals hide different losses. The postcode loses all 4,289 true pairs with two different postcodes: 0 of them are candidates. Most of them are customers who moved. The surname key loses other pairs, mostly names with errors in their first letters, and keeps 3,631 of the movers’ pairs. Larkfield’s movers are not a random group. They are customers whose address changed: the people who most need their records joined, so that the old address gets no more catalogues.
Fix: measure what each blocking loses by group, as the script above does. Then combine keys that lose different groups. The postcode and the surname key together keep 0.976 of all true pairs and 3,631 of the movers.
Consequence in AI work: an average can hide a group that a system always fails. A fair evaluation looks at slices. Machine Learning: From Problem to Reliable Model teaches the same lesson: one class of a classifier can fail while the accuracy looks good.
Knowledge check
Section titled “Knowledge check”Check your understanding
A blocking makes 180,000 candidate pairs from Larkfield's 60,108 records and keeps 22,000 of the 27,677 true pairs. Compute its pair completeness and its reduction ratio. Is it better than the postcode?
Show the answer
Pair completeness = 22,000 / 27,677 = 0.795. Reduction ratio = 1 − 180,000 / 1,806,455,778 = 0.999900. The postcode keeps 23,388 true pairs (0.845) with only 89,630 candidates (0.999950). So this blocking is worse on both numbers: more candidates and fewer true pairs. Before you discard it, check which true pairs it keeps. If it keeps the movers that the postcode loses, it can still be useful as a second key.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Every true pair that the five keys lose has two different postcodes. What must be true of such a pair for all five keys to miss it?
Show the answer
The two records differ in every key, or the key is empty in one of them. The postcode differs, mostly because the customer moved. The email is different or empty: for example a family email in one list and an empty email in the other. The phone is different or empty: a new phone after the move, or no phone in the shop record. The date of birth is empty or different. And the surname key differs. The cause is a nickname with other first letters, a typing error in the first four letters of the surname, or an empty name. The notebook shows one such pair: Jim Walker (S-000281) and James Walker (S-010340). Their postcodes differ by one character, and their emails differ. One has no phone, and neither has a date of birth. The surname keys are walk|ji and walk|ja.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks: “Why not use keys-wide? It loses fewer customers.” Give him a short, honest answer.
Show the answer
keys-wide keeps 43 more true pairs (27,546 against 27,503), and 43 more of the movers’ pairs. It makes 2.1 times more candidate pairs, which costs a few seconds more on today’s 60,108 records. So today it is a fair choice. But its surname key has bigger blocks, and bigger blocks grow with the square of their size. When Larkfield grows to a million records, the five keys already make about 340 million candidates; a wider key makes the comparison step far slower. The project keeps the narrower key, and measures the cost of the lost pairs instead.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Pair completeness is the recall of the blocking: no later step can have a higher pair recall. The reduction ratio is the work saved. Measure both, every time.
- The postcode keeps 0.845 of the true pairs. It loses exactly the 4,289 true pairs with two different postcodes, mostly customers who moved: it keeps none of them.
- Name keys catch movers. The five keys keep 0.9937 of all true pairs and 4,115 of the 4,289 movers’ pairs. Every true pair that they lose has two different postcodes.
keys-widekeeps 43 more true pairs with 2.1 times more candidates. Choose by the cost of a missed match against the cost of the candidates, and think about growth.- On the test split,
keyskeeps 0.9941 of the true pairs: the ceiling for the pair recall of every later run.
You now have 1,108,591 candidate pairs that keep almost every true pair. Next, Module 4 compares the two records of each pair, field by field: how similar are two names, two addresses, two emails?
Next: String similarity