Skip to content
Français

Measure a matcher

  • Lesson 7
  • 35 min
  • Module 2: Clean first, then match exactly

Where we are

Our rules found many pairs. We do not know yet how many are right. In this lesson, we measure a matcher with pair precision and recall against the true entities, and we see why pairs and entities give different numbers.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Measure a matcher against the true customers: count its right merges, false merges and missed matches, and turn them into pair precision and recall. Then see why the same matcher gets other numbers when you count customers instead of pairs. Last, break the result down by list and by practice problem.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds the rule set and counts its right merges, false merges and missed matches with pandas. You compute pair precision and recall without a helper. Then you join the pairs into customers and measure the pairs that the clusters make. It downloads the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

Tested on macOS with Python 3.12.13. The Windows (PowerShell) and Linux commands are written the same way, but we did not test them.

After a month, the clinic asks an auditor to check the clerk’s work. The auditor has an answer key: for a set of patients, the true identity is confirmed by their health card.

The auditor does not ask “Is the work good?”. She asks two exact questions. Of all the merges that the clerk made, how many were right? Of all the duplicates that exist, how many did the clerk merge? Then she counts. She also asks a third question: how many patients does the clinic have now? She knows that this count can look right when the work is wrong.

The problem: one number can hide the errors

Section titled “The problem: one number can hide the errors”

The rule set of the last lesson makes 36,756 customers from 60,108 records. The truth is 37,500. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. is pleased: “We are only 744 customers below the truth. That is 2% off. Can we use it?”

But the same rule set made 1,853 false merges and 724 missed matches on the test split alone. False merges make the count smaller: two people become one. Missed matches make it larger: one person stays two. The two errors cancel each other in the count. A matcher can get the count exactly right and still mix up thousands of people.

So the count of customers is not a measure of quality. You need to count the errors themselves.

The idea: count pairs against the true entities

Section titled “The idea: count pairs against the true entities”

The data has an answer key: truth.csv gives the true customer, the entity, of every record. From it, you know every true pair: two records of the same entity. Now compare the matcher’s pairs with the true pairs. Every pair of records falls into one of four boxes:

A grid of four boxes. Columns: truly the same customer, truly two customers. Rows: the rules say SAME, the rules say DIFFERENT. SAME and truly the same: 4,840 right merges, the true positives. SAME but truly two customers: 1,853 false merges, the false positives. DIFFERENT but truly the same: 724 missed matches, the false negatives. DIFFERENT and truly two customers: 651,254,150 pairs that are right to keep apart, the true negatives. Below: pair precision = 4,840 / (4,840 + 1,853) = 0.723; pair recall = 4,840 / (4,840 + 724) = 0.870. The test split has 651,261,567 pairs; only 5,564 of them are true pairs.
The rules' confusion counts on the test split (synthetic records). Precision uses the top row; recall uses the left column. The bottom-right box is huge, and neither measure uses it.
Text description of the diagram

The grid sorts every pair of the test split. The rules say SAME for 6,693 pairs: 4,840 are right merges and 1,853 are false merges. The rules say DIFFERENT for all other pairs. 724 of them are missed matches. The other 651,254,150 are pairs of two customers, and it is right to keep them apart. Pair precision is 4,840 divided by 6,693: 0.723. Pair recall is 4,840 divided by 5,564, the number of true pairs: 0.870.

This table is called a confusion matrix. Its four boxes have names from statistics. Right merges are true positives, and false merges are false positives. Missed matches are false negatives. Pairs that are right to keep apart are true negatives.

  • Pair precision = right merges / all pairs called the same = 4,840 / 6,693 = 0.723.
  • Pair recall = right merges / all true pairs = 4,840 / 5,564 = 0.870.

Look at the bottom-right box. Almost every pair of records is two different people. This box is why accuracy is useless for matching, as you saw in the failure case of False merges and missed matches. Precision and recall ignore this box on purpose.

Every true customer is in one split: train, validation or test. A pair is in the test split when at least one of its two records belongs to a test customer. So a wrong merge of a test record with a training record counts as a test error. The test split has 12,041 records and 5,564 true pairs. Every module measures on these same pairs, so that the rows of the comparison table can be compared.

A matcher returns pairs. But TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield needs customers. To get customers, the project clusters the pairs. If A matches B, and B matches C, then A, B and C are one customer. This is true even when nobody compared A and C. Module 7 teaches clustering. Here, the important point is that clustering changes the pairs.

The diagram uses letters for records. This is a constructed example, not records from our data.

Two panels with records drawn as circles. Panel 1, one customer with four records A, B, C and D, joined by green lines A–B, B–C and C–D. The text says: the matcher finds 3 pairs; the customer has 6 true pairs; pair recall is 3 / 6 = 0.50. After clustering, A, B, C and D are one customer and all 6 pairs are joined; entities: 1, the right answer. Panel 2, two customers: E, F and G are customer 1, H and I are customer 2. Green lines E–F, F–G and H–I, and a red dashed line G–H, a wrong pair. The text says: the matcher finds 4 pairs, 3 right and G–H wrong; pair precision is 3 / 4 = 0.75. After clustering, E to I are one customer: 10 pairs are joined, 4 right and 6 wrong; pair precision is 4 / 10 = 0.40; entities: 1, not 2.
Clustering adds the pairs between every two records of a cluster. In panel 1, it fixes three missed matches. In panel 2, one wrong pair becomes six false merges.
Text description of the diagram

Panel 1 has one customer with four records, A to D. The matcher finds three pairs in a chain: A–B, B–C and C–D. The customer has six true pairs, so pair recall is 3 out of 6. After clustering, the four records are one customer, and all six pairs are joined. This fixes the missed pairs A–C, A–D and B–D. Panel 2 has two customers: E, F and G, and H and I. The matcher finds three right pairs and one wrong pair, G–H: precision 3 out of 4. After clustering, all five records are one customer. That joins 10 pairs: the 4 true pairs, and 6 pairs of two different people. Precision falls to 4 out of 10.

So clustering helps recall and can hurt precision a lot. One wrong pair between two groups joins every record of one group with every record of the other.

Now the real data. The project clusters the pairs of rules, and then counts every pair inside each cluster, as in the tiny example.

  • Pairs the rules found
  • Pairs inside the clusters
The rules on the test split, before and after clustering (synthetic records). Clustering adds 160 right merges and 2,685 false merges.
Text description of the chart

A column chart with four columns. Precision: 0.723 for the pairs that the rules found, 0.524 for the pairs inside the clusters. Recall: 0.870 before, 0.899 after.

Pairs on the test split Called the same Right Wrong Precision Recall
found by the rules 6,693 4,840 1,853 0.723 0.870
inside the clusters 9,538 5,000 4,538 0.524 0.899

Clustering finds 160 more true pairs, as in panel 1. But it adds 2,685 false merges, as in panel 2. The largest cluster of rules has 14 records; no true customer has more than 5. These large clusters are families and shared phones, chained together by “or”.

Which number is right? Both, for different questions. The pairs that the rules found say how good each decision is. The pairs inside the clusters say how good the final customer list is. Larkfield’s customers live in the final list, so the second number matters for Omar. This course reports the first one in the comparison table, the same way for every method, and Module 7 measures the clusters.

Pair measures also weigh large customers more. A customer with 5 records has 10 pairs; a customer with 2 records has 1 pair. Module 7 adds measures that count each record once.

This lesson adds the command match compare, the module’s tests, and the module’s README. After it, your project matches the snapshot end-of-m02.

  1. Copy the three files:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m02
    for f in larkfield_match/commands/compare.py tests/test_clean.py README.md; do
    cp "$S/$f" "$f"
    done
  2. Run the tests. tests/test_clean.py adds five tests for normalising, parsing and the email rule:

    Terminal window
    python -m pytest -q
    ................ [100%]
    16 passed in 4.93s
  3. Show the rules’ results for each pair of lists. --by source names the two lists of each pair: garden-shop is a garden-centre record with a shop record.

    Terminal window
    match eval --run rules --by source
    rules on the test split: pair precision 0.723, recall 0.870, F1 0.790
    6,693 pairs called the same: 4,840 right, 1,853 wrong; 724 of 5,564 true pairs missed
    entities after clustering (all records): 36,756 (true: 37,500)
    source true found recall predicted precision
    garden-garden 78 62 0.795 135 0.459
    garden-loyalty 1089 918 0.843 1104 0.832
    garden-shop 1647 1371 0.832 1648 0.832
    loyalty-loyalty 40 40 1.000 526 0.076
    loyalty-shop 2386 2200 0.922 2644 0.832
    shop-shop 324 249 0.769 636 0.392

One total can hide a weak part. The chart shows the precision of the rules for each pair of lists.

  • Two different lists
  • Two records in one list
Pair precision of the rules by the lists of the two records, test split (synthetic records). L = loyalty, S = shop, G = garden centres. Pairs inside one list are the weak part.
Text description of the chart

A column chart with six columns. The precision is 0.832 for loyalty with shop, for garden centres with shop, and for garden centres with loyalty. It is 0.392 for shop with shop. It is 0.459 for garden centres with garden centres. It is 0.076 for loyalty with loyalty.

Between two lists, the rules have precision 0.832. Inside one list, they are much worse. The worst is loyalty with loyalty: 526 pairs called the same, and only 40 right. Most of the others are family members. Each one has a loyalty card, and they all give the family email. Rule 1 merges them.

--by problem groups the pairs by the practice problems that we made on purpose:

Terminal window
match eval --run rules --by problem
problem true found recall predicted precision
brothers 39 24 0.615 57 0.421
family_email 487 378 0.776 879 0.430
moved 1277 1051 0.823 1388 0.757
none 3468 3129 0.902 4056 0.771
placeholder_phone 13 11 0.846 16 0.688
saint_town 280 247 0.882 297 0.832

(The first three lines of the output are the same as above, and are not shown.) The family emails and the brothers have precision 0.430 and 0.421: more than half of the merges there are wrong. The movers keep a recall of 0.823, because rules 1, 2 and 3 do not use the address. Compare with the email as typed: run match eval --run exact --by problem. Its recall is 0.026 on the brothers’ true pairs, and 0.062 on the family emails.

match compare shows two runs side by side, and the pairs that only one of them finds. Compare rule 4 alone with the four rules:

Terminal window
match compare rule-name-postcode rules
rule-name-postcode precision 0.921 recall 0.621 F1 0.742 (3,754 pairs)
rules precision 0.723 recall 0.870 F1 0.790 (6,693 pairs)
Only rule-name-postcode: 0 pairs (0 right, 0 wrong)
Only rules: 2,939 pairs (1,382 right, 1,557 wrong)
wrong S-000065 'Michael Anderson' / S-002479 'Michael Anderson'
wrong L-005295 'Catherine Bélanger' / S-002438 'Josee Belanger'
wrong L-005225 'Chloé Fournier' / S-011989 'Julie Fournier'

The three other rules add 2,939 pairs: 1,382 right and 1,557 wrong. So they raise recall from 0.621 to 0.870, and lower precision from 0.921 to 0.723. The wrong examples show why. Catherine Bélanger and Josée Bélanger share a phone and a surname, so rule 2 merges them. Rule 2 also merges Chloé and Julie Fournier, who were born 26 years apart. The two Michael Andersons are two people with the same name and the same email address, michael.anderson@example.com, so rule 1 merges them. Use match show with a record ID to see all of their records.

--show 10 prints more examples. Read them before you trust a rule.

Common mistake

Precision measured on chosen pairs

Symptom: you measure the rules on the 3,000 labelled pairs in data/labelled_pairs.csv. You get precision 0.990 and recall 0.875. On the test split, precision is 0.723. Which one is true?

Cause: the labelled pairs were chosen, not taken from all pairs. They hold 1,050 true pairs and only 1,950 pairs of two people. Of these, 1,500 have the same surname in one region, and 450 are random. On all pairs, the rules see many more chances for a false merge: families, shared phones, people with the same name. On these 1,950 pairs, they made only 9 false merges. So precision on the labelled pairs says little about precision on all pairs. The recall is close (0.875 against 0.870), because the 1,050 true pairs are a fair sample of true pairs.

Fix: measure precision on all the pairs that a matcher calls the same, against the true entities, as match eval does. Use labelled pairs to train a model, not to report its precision. If you only have a sample, sample from the pairs that the matcher calls the same, and check them.

Consequence in AI work: a measure is only as good as the pairs it counts. Before you trust a number, ask: which pairs are in it, and how were they chosen? In real work, you rarely have a full answer key. Module 8 builds one, pair by pair, from human review.

  • Pair precision and recall, on the pairs the matcher found: to compare methods, and to tune a threshold. This is the comparison table of the course.
  • The same measures on the pairs inside the clusters: to judge the final customer list. Module 7 adds cluster measures.
  • By list and by problem: to find the weak part of a matcher, before you change it.
  • The customer count: to report the result to the business, never to judge quality.

What it costs. The measures need an answer key. Our synthetic data has a complete one, because our code made the records from known people. Real data does not. There, you label a sample of pairs by hand, which costs a reviewer’s time: Module 8 does this with Ivana’s review queue.

Check your understanding

A matcher calls 2,000 test pairs the same: 1,500 are right. The test split has 5,564 true pairs. Fill in the four boxes you can, and compute pair precision and recall.

Show the answer
  • Right merges (true positives): 1,500.
  • False merges (false positives): 2,000 − 1,500 = 500.
  • Missed matches (false negatives): 5,564 − 1,500 = 4,064.
  • The true negatives are all other pairs of the test split; you do not need them.

Pair precision = 1,500 / 2,000 = 0.750. Pair recall = 1,500 / 5,564 = 0.270.

Check your understanding

Omar says: “The rules give 36,756 customers. The truth is 37,500. So the rules are 98% right.” Explain what is wrong with this reasoning.

Show the answer

The customer count mixes two errors that cancel. False merges make the count smaller, and missed matches make it larger. On the test split alone, the rules make 1,853 false merges and miss 724 true pairs. After clustering, the pairs inside the clusters have precision 0.524. The count can be close to the truth while thousands of people are mixed up. Judge a matcher by its precision and recall, and use the count only to report the result.

Check your understanding

Show one rule that raises recall and lowers precision, and name the pairs that cause it. Use the project.

Show the answer

Run match compare rule-phone-surname rule-phone. Dropping the surname from rule 2 gives “same phone” alone. It adds 1,600 pairs: 312 right and 1,288 wrong. Recall rises from 0.438 to 0.494, and precision falls from 0.696 to 0.539. The wrong pairs are people with another surname and the same phone. Some are family members. Others are the 40 garden-centre walk-ins with the shop’s own number 450-555-0100. match eval --run rule-phone --by problem shows that none of its 345 test pairs on the walk-in records is right.

Recap

  • Compare the matcher’s pairs with the true pairs: right merges, false merges, missed matches, and pairs that are right to keep apart. This is the confusion matrix.
  • Pair precision = right merges / pairs called the same; pair recall = right merges / true pairs. The rules: 4,840 / 6,693 = 0.723, and 4,840 / 5,564 = 0.870. Accuracy is useless: almost every pair is two people.
  • Clustering changes the pairs. For the rules, it adds 160 right merges and 2,685 false merges: precision 0.524, recall 0.899 inside the clusters. The customer count, 36,756, hides these errors.
  • Break results down: inside one list, the rules are weak (loyalty with loyalty: precision 0.076, family emails).
  • Precision on a chosen set of pairs (0.990 on the labelled pairs) is not precision on all pairs (0.723).
  • Your project now matches the snapshot end-of-m02. The 16 tests pass.

The rules find their pairs by joining on exact keys, so they never look at most pairs. A matcher that allows small differences must compare records one pair at a time. Next, you count how many pairs that would be for 60,108 records, and how long it would take.

Next: Why all pairs is impossible