Measure a matcher
- Lesson 7
- 35 min
- Module 2: Clean first, then match exactly
Where we are
Our rules found many pairs. We do not know yet how many are right. In this lesson, we measure a matcher with pair precision and recall against the true entities, and we see why pairs and entities give different numbers.
Lesson 7 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Measure a matcher
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Measure a matcher against the true customers: count its right merges, false merges and missed matches, and turn them into pair precision and recall. Then see why the same matcher gets other numbers when you count customers instead of pairs. Last, break the result down by list and by practice problem.
NotebookDo this lesson in a notebook
The notebook builds the rule set and counts its right merges, false merges and missed matches with pandas. You compute pair precision and recall without a helper. Then you join the pairs into customers and measure the pairs that the clusters make. It downloads the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The rule set
rulesand its numbers, from Exact and rule-based matching. - True pairs, pair precision and pair recall, from False merges and missed matches. The machine learning course explains precision and recall for any classifier in Classification metrics.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) and Linux commands are written the same way, but we did not test them.
The clinic’s audit
Section titled “The clinic’s audit”After a month, the clinic asks an auditor to check the clerk’s work. The auditor has an answer key: for a set of patients, the true identity is confirmed by their health card.
The auditor does not ask “Is the work good?”. She asks two exact questions. Of all the merges that the clerk made, how many were right? Of all the duplicates that exist, how many did the clerk merge? Then she counts. She also asks a third question: how many patients does the clinic have now? She knows that this count can look right when the work is wrong.
The problem: one number can hide the errors
Section titled “The problem: one number can hide the errors”The rule set of the last lesson makes 36,756 customers from 60,108 records. The truth is 37,500. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. is pleased: “We are only 744 customers below the truth. That is 2% off. Can we use it?”
But the same rule set made 1,853 false merges and 724 missed matches on the test split alone. False merges make the count smaller: two people become one. Missed matches make it larger: one person stays two. The two errors cancel each other in the count. A matcher can get the count exactly right and still mix up thousands of people.
So the count of customers is not a measure of quality. You need to count the errors themselves.
The idea: count pairs against the true entities
Section titled “The idea: count pairs against the true entities”The data has an answer key: truth.csv gives the true customer, the entity, of every record. From it, you know every true pair: two records of the same entity. Now compare the matcher’s pairs with the true pairs. Every pair of records falls into one of four boxes:
Text description of the diagram
The grid sorts every pair of the test split. The rules say SAME for 6,693 pairs: 4,840 are right merges and 1,853 are false merges. The rules say DIFFERENT for all other pairs. 724 of them are missed matches. The other 651,254,150 are pairs of two customers, and it is right to keep them apart. Pair precision is 4,840 divided by 6,693: 0.723. Pair recall is 4,840 divided by 5,564, the number of true pairs: 0.870.
This table is called a confusion matrix. Its four boxes have names from statistics. Right merges are true positives, and false merges are false positives. Missed matches are false negatives. Pairs that are right to keep apart are true negatives.
- Pair precision = right merges / all pairs called the same = 4,840 / 6,693 = 0.723.
- Pair recall = right merges / all true pairs = 4,840 / 5,564 = 0.870.
Look at the bottom-right box. Almost every pair of records is two different people. This box is why accuracy is useless for matching, as you saw in the failure case of False merges and missed matches. Precision and recall ignore this box on purpose.
Which pairs are “test” pairs?
Section titled “Which pairs are “test” pairs?”Every true customer is in one split: train, validation or test. A pair is in the test split when at least one of its two records belongs to a test customer. So a wrong merge of a test record with a training record counts as a test error. The test split has 12,041 records and 5,564 true pairs. Every module measures on these same pairs, so that the rows of the comparison table can be compared.
Pairs against entities
Section titled “Pairs against entities”A matcher returns pairs. But TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield needs customers. To get customers, the project clusters the pairs. If A matches B, and B matches C, then A, B and C are one customer. This is true even when nobody compared A and C. Module 7 teaches clustering. Here, the important point is that clustering changes the pairs.
A tiny example with every pair visible
Section titled “A tiny example with every pair visible”The diagram uses letters for records. This is a constructed example, not records from our data.
Text description of the diagram
Panel 1 has one customer with four records, A to D. The matcher finds three pairs in a chain: A–B, B–C and C–D. The customer has six true pairs, so pair recall is 3 out of 6. After clustering, the four records are one customer, and all six pairs are joined. This fixes the missed pairs A–C, A–D and B–D. Panel 2 has two customers: E, F and G, and H and I. The matcher finds three right pairs and one wrong pair, G–H: precision 3 out of 4. After clustering, all five records are one customer. That joins 10 pairs: the 4 true pairs, and 6 pairs of two different people. Precision falls to 4 out of 10.
So clustering helps recall and can hurt precision a lot. One wrong pair between two groups joins every record of one group with every record of the other.
The rules before and after clustering
Section titled “The rules before and after clustering”Now the real data. The project clusters the pairs of rules, and then counts every pair inside each cluster, as in the tiny example.
- Pairs the rules found
- Pairs inside the clusters
Text description of the chart
A column chart with four columns. Precision: 0.723 for the pairs that the rules found, 0.524 for the pairs inside the clusters. Recall: 0.870 before, 0.899 after.
| Pairs on the test split | Called the same | Right | Wrong | Precision | Recall |
|---|---|---|---|---|---|
| found by the rules | 6,693 | 4,840 | 1,853 | 0.723 | 0.870 |
| inside the clusters | 9,538 | 5,000 | 4,538 | 0.524 | 0.899 |
Clustering finds 160 more true pairs, as in panel 1. But it adds 2,685 false merges, as in panel 2. The largest cluster of rules has 14 records; no true customer has more than 5. These large clusters are families and shared phones, chained together by “or”.
Which number is right? Both, for different questions. The pairs that the rules found say how good each decision is. The pairs inside the clusters say how good the final customer list is. Larkfield’s customers live in the final list, so the second number matters for Omar. This course reports the first one in the comparison table, the same way for every method, and Module 7 measures the clusters.
Pair measures also weigh large customers more. A customer with 5 records has 10 pairs; a customer with 2 records has 1 pair. Module 7 adds measures that count each record once.
Measure in the project
Section titled “Measure in the project”This lesson adds the command match compare, the module’s tests, and the module’s README. After it, your project matches the snapshot end-of-m02.
-
Copy the three files:
Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m02for f in larkfield_match/commands/compare.py tests/test_clean.py README.md; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m02"foreach ($f in 'larkfield_match\commands\compare.py', 'tests\test_clean.py', 'README.md') { Copy-Item "$S\$f" $f } -
Run the tests.
tests/test_clean.pyadds five tests for normalising, parsing and the email rule:Terminal window python -m pytest -q................ [100%]16 passed in 4.93s -
Show the rules’ results for each pair of lists.
--by sourcenames the two lists of each pair:garden-shopis a garden-centre record with a shop record.Terminal window match eval --run rules --by sourcerules on the test split: pair precision 0.723, recall 0.870, F1 0.7906,693 pairs called the same: 4,840 right, 1,853 wrong; 724 of 5,564 true pairs missedentities after clustering (all records): 36,756 (true: 37,500)source true found recall predicted precisiongarden-garden 78 62 0.795 135 0.459garden-loyalty 1089 918 0.843 1104 0.832garden-shop 1647 1371 0.832 1648 0.832loyalty-loyalty 40 40 1.000 526 0.076loyalty-shop 2386 2200 0.922 2644 0.832shop-shop 324 249 0.769 636 0.392
Results by list
Section titled “Results by list”One total can hide a weak part. The chart shows the precision of the rules for each pair of lists.
- Two different lists
- Two records in one list
Text description of the chart
A column chart with six columns. The precision is 0.832 for loyalty with shop, for garden centres with shop, and for garden centres with loyalty. It is 0.392 for shop with shop. It is 0.459 for garden centres with garden centres. It is 0.076 for loyalty with loyalty.
Between two lists, the rules have precision 0.832. Inside one list, they are much worse. The worst is loyalty with loyalty: 526 pairs called the same, and only 40 right. Most of the others are family members. Each one has a loyalty card, and they all give the family email. Rule 1 merges them.
Results by practice problem
Section titled “Results by practice problem”--by problem groups the pairs by the practice problems that we made on purpose:
match eval --run rules --by problem problem true found recall predicted precision brothers 39 24 0.615 57 0.421 family_email 487 378 0.776 879 0.430 moved 1277 1051 0.823 1388 0.757 none 3468 3129 0.902 4056 0.771placeholder_phone 13 11 0.846 16 0.688 saint_town 280 247 0.882 297 0.832(The first three lines of the output are the same as above, and are not shown.) The family emails and the brothers have precision 0.430 and 0.421: more than half of the merges there are wrong. The movers keep a recall of 0.823, because rules 1, 2 and 3 do not use the address. Compare with the email as typed: run match eval --run exact --by problem. Its recall is 0.026 on the brothers’ true pairs, and 0.062 on the family emails.
Which pairs does a rule add?
Section titled “Which pairs does a rule add?”match compare shows two runs side by side, and the pairs that only one of them finds. Compare rule 4 alone with the four rules:
match compare rule-name-postcode rulesrule-name-postcode precision 0.921 recall 0.621 F1 0.742 (3,754 pairs)rules precision 0.723 recall 0.870 F1 0.790 (6,693 pairs)Only rule-name-postcode: 0 pairs (0 right, 0 wrong)Only rules: 2,939 pairs (1,382 right, 1,557 wrong) wrong S-000065 'Michael Anderson' / S-002479 'Michael Anderson' wrong L-005295 'Catherine Bélanger' / S-002438 'Josee Belanger' wrong L-005225 'Chloé Fournier' / S-011989 'Julie Fournier'The three other rules add 2,939 pairs: 1,382 right and 1,557 wrong. So they raise recall from 0.621 to 0.870, and lower precision from 0.921 to 0.723. The wrong examples show why. Catherine Bélanger and Josée Bélanger share a phone and a surname, so rule 2 merges them. Rule 2 also merges Chloé and Julie Fournier, who were born 26 years apart. The two Michael Andersons are two people with the same name and the same email address, michael.anderson@example.com, so rule 1 merges them. Use match show with a record ID to see all of their records.
--show 10 prints more examples. Read them before you trust a rule.
Common mistake
Precision measured on chosen pairs
Symptom: you measure the rules on the 3,000 labelled pairs in data/labelled_pairs.csv. You get precision 0.990 and recall 0.875. On the test split, precision is 0.723. Which one is true?
Cause: the labelled pairs were chosen, not taken from all pairs. They hold 1,050 true pairs and only 1,950 pairs of two people. Of these, 1,500 have the same surname in one region, and 450 are random. On all pairs, the rules see many more chances for a false merge: families, shared phones, people with the same name. On these 1,950 pairs, they made only 9 false merges. So precision on the labelled pairs says little about precision on all pairs. The recall is close (0.875 against 0.870), because the 1,050 true pairs are a fair sample of true pairs.
Fix: measure precision on all the pairs that a matcher calls the same, against the true entities, as match eval does. Use labelled pairs to train a model, not to report its precision. If you only have a sample, sample from the pairs that the matcher calls the same, and check them.
Consequence in AI work: a measure is only as good as the pairs it counts. Before you trust a number, ask: which pairs are in it, and how were they chosen? In real work, you rarely have a full answer key. Module 8 builds one, pair by pair, from human review.
When to use each measure
Section titled “When to use each measure”- Pair precision and recall, on the pairs the matcher found: to compare methods, and to tune a threshold. This is the comparison table of the course.
- The same measures on the pairs inside the clusters: to judge the final customer list. Module 7 adds cluster measures.
- By list and by problem: to find the weak part of a matcher, before you change it.
- The customer count: to report the result to the business, never to judge quality.
What it costs. The measures need an answer key. Our synthetic data has a complete one, because our code made the records from known people. Real data does not. There, you label a sample of pairs by hand, which costs a reviewer’s time: Module 8 does this with Ivana’s review queue.
Knowledge check
Section titled “Knowledge check”Check your understanding
A matcher calls 2,000 test pairs the same: 1,500 are right. The test split has 5,564 true pairs. Fill in the four boxes you can, and compute pair precision and recall.
Show the answer
- Right merges (true positives): 1,500.
- False merges (false positives): 2,000 − 1,500 = 500.
- Missed matches (false negatives): 5,564 − 1,500 = 4,064.
- The true negatives are all other pairs of the test split; you do not need them.
Pair precision = 1,500 / 2,000 = 0.750. Pair recall = 1,500 / 5,564 = 0.270.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar says: “The rules give 36,756 customers. The truth is 37,500. So the rules are 98% right.” Explain what is wrong with this reasoning.
Show the answer
The customer count mixes two errors that cancel. False merges make the count smaller, and missed matches make it larger. On the test split alone, the rules make 1,853 false merges and miss 724 true pairs. After clustering, the pairs inside the clusters have precision 0.524. The count can be close to the truth while thousands of people are mixed up. Judge a matcher by its precision and recall, and use the count only to report the result.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Show one rule that raises recall and lowers precision, and name the pairs that cause it. Use the project.
Show the answer
Run match compare rule-phone-surname rule-phone. Dropping the surname from rule 2 gives “same phone” alone. It adds 1,600 pairs: 312 right and 1,288 wrong. Recall rises from 0.438 to 0.494, and precision falls from 0.696 to 0.539. The wrong pairs are people with another surname and the same phone. Some are family members. Others are the 40 garden-centre walk-ins with the shop’s own number 450-555-0100. match eval --run rule-phone --by problem shows that none of its 345 test pairs on the walk-in records is right.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Compare the matcher’s pairs with the true pairs: right merges, false merges, missed matches, and pairs that are right to keep apart. This is the confusion matrix.
- Pair precision = right merges / pairs called the same; pair recall = right merges / true pairs. The rules: 4,840 / 6,693 = 0.723, and 4,840 / 5,564 = 0.870. Accuracy is useless: almost every pair is two people.
- Clustering changes the pairs. For the rules, it adds 160 right merges and 2,685 false merges: precision 0.524, recall 0.899 inside the clusters. The customer count, 36,756, hides these errors.
- Break results down: inside one list, the rules are weak (loyalty with loyalty: precision 0.076, family emails).
- Precision on a chosen set of pairs (0.990 on the labelled pairs) is not precision on all pairs (0.723).
- Your project now matches the snapshot
end-of-m02. The 16 tests pass.
The rules find their pairs by joining on exact keys, so they never look at most pairs. A matcher that allows small differences must compare records one pair at a time. Next, you count how many pairs that would be for 60,108 records, and how long it would take.