Cluster measures
- Lesson 25
- 35 min
- Module 7: From pairs to entities
Where we are
Our entities have golden records. We measured pairs, but Larkfield counts customers. In this lesson, we measure clusters with pair, cluster and B-cubed measures, and we see how giant clusters hide behind good pair numbers.
Lesson 25 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Cluster measures
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Measure entities, not only pairs. Learn three cluster measures: cluster-pair, exact-cluster and B-cubed precision and recall. See how a few large clusters hide many wrong pairs that the pair measures never show, and why no single number is enough.
NotebookMeasure clusters in a notebook
The notebook starts with six records of two people and one bad link. You write B-cubed precision and recall. Then you measure the clusters of Module 2’s rules, with and without safe clustering, and you count where the wrong pairs hide. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Pair precision and recall, from Measure a matcher.
- Connected components and safe clustering, from Clustering pairs and Safer clustering.
What you need on your computer
- The project
larkfield-matchwith the files of Module 7. Or use the notebook only.
Checking the project groups
Section titled “Checking the project groups”A teacher asks a helper to split 30 students into project groups of friends. How does she check his work?
She can check pairs: “He said Sam and Lee are friends. Are they?” That checks his decisions, one by one.
She can check groups: “Is this group exactly one set of friends?” That is strict. A group with one wrong student fails completely.
She can ask each student: “How much of your group are your friends? How many of your friends are in your group?” Then she averages the answers. A student in a good group says “all of them”. A student who was put with strangers says “only a few”.
Each check sees something that the others miss. Matching has the same three views.
The problem: the pair measures judge the matcher, not the entities
Section titled “The problem: the pair measures judge the matcher, not the entities”Measure a matcher showed this problem first. Here we measure it properly. Since Module 2, we measured pairs: of the pairs that the matcher calls the same, how many are true pairs? That measures the matcher’s decisions. But TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield uses the entities. And clustering adds pairs that the matcher never decided.
Here is a tiny example. Ann has three records: a1, a2, a3. Ben has three records: b1, b2, b3. The matcher calls 5 pairs the same: a1–a2, a2–a3, b1–b2, b2–b3, a3–b1. Only the last one is wrong. Connected components joins all six records into one cluster.
Text description of the diagram
The picture shows six small circles inside one dashed box: a1, a2 and a3 for Ann, and b1, b2 and b3 for Ben. Solid lines join a1 to a2, a2 to a3, b1 to b2, and b2 to b3. A dashed red line joins a3 to b1: the one wrong link. Four text boxes give the measures. Pair measures: 4 of the 5 pairs called the same are right, so precision is 0.8. 4 of the 6 true pairs are found, so recall is 0.667. Cluster pairs: the cluster holds 6 × 5 ÷ 2 = 15 pairs. 6 of them are right, so precision is 0.4. All 6 true pairs are inside, so recall is 1. B-cubed: for every record, 3 of the 6 records in its cluster are the same person. So precision is 0.5. All 3 of its person’s records are in its cluster, so recall is 1.0. Exact clusters: the one cluster is not exactly one person, so 0 of 1 is right.
The matcher’s pairs look good: precision 0.8. But the cluster is half wrong. Of the 15 pairs inside it, 9 are wrong. The matcher decided only one of them, a3–b1. Connected components added the other 8: a1–b1, a1–b2, a2–b3 and the rest. No pair measure of the matcher can see them.
The idea: three ways to measure clusters
Section titled “The idea: three ways to measure clusters”Each measure compares the clusters with the true entities. Each has a precision (“how much of what we joined is right?”) and a recall (“how much of what should be joined did we join?”).
Cluster-pair precision and recall
Section titled “Cluster-pair precision and recall”Take every pair of records inside one cluster, also the pairs that the matcher never compared. Count them as “called the same”. Then compute pair precision and recall as before.
In the tiny example: 15 pairs, 6 right, so precision is 0.4. All 6 true pairs are inside, so recall is 1.
This measure is easy to compare with the matcher’s own pair measures. But it counts big clusters many times: a cluster of n records holds n × (n − 1) ÷ 2 pairs. One cluster of 40 records holds 780 pairs.
Exact-cluster precision and recall
Section titled “Exact-cluster precision and recall”A cluster is right only when it holds exactly the records of one true entity: not one record more or less.
- Exact-cluster precision: of the clusters we made, the share that are exactly right.
- Exact-cluster recall: of the true entities, the share that we found exactly.
In the tiny example: 0 of 1 cluster is right. This measure is strict and easy to explain: “88% of our customers are exactly right”. But it gives no credit for a cluster that is almost right.
B-cubed precision and recall
Section titled “B-cubed precision and recall”B-cubed asks each record two questions, then takes the mean over all records:
- precision of a record: the share of its cluster that is the same entity as the record;
- recall of a record: the share of its entity’s records that are in its cluster.
In the tiny example, every record has precision 3 ÷ 6 = 0.5 and recall 3 ÷ 3 = 1.0. So B-cubed precision is 0.5 and recall is 1. Amit Bagga and Breck Baldwin proposed it in 1998.
B-cubed counts every record once. A big cluster does not count more than its records. So it is the fairest summary of “how right are our customers?”. But for the same reason, it can hide one giant cluster among thousands of good ones.
Measure Larkfield’s entities
Section titled “Measure Larkfield’s entities”From this module on, match eval prints all three measures for both clusterings. Run it for LightGBM. You did the undo of the last lesson, so safe clustering also keeps S-009232 apart:
match eval --run lgbmlgbm on the test split: pair precision 0.918, recall 0.886, F1 0.902 5,375 pairs called the same: 4,932 right, 443 wrong; 632 of 5,564 true pairs missed entities after clustering (all records): 38,514 (true: 37,500) connected components: 38,514 entities, largest 9 records, 75 clusters over 5 records B-cubed precision 0.979, recall 0.967; cluster pairs precision 0.838, recall 0.917; exact clusters precision 0.873, recall 0.913 safe clustering: 38,909 entities, largest 6 records, 11 clusters over 5 records B-cubed precision 0.987, recall 0.965; cluster pairs precision 0.911, recall 0.909; exact clusters precision 0.880, recall 0.923Safe clustering counts 38,909 entities here, the same as match golden after the undo. Ivana’s two cannot-link pairs come first: safe clustering removes the links between S-009232 and the other two records before any other check. The right link between G-000001 and L-010072 stays, so Valérie’s other two records stay one entity. The measures of the test split do not change: Valérie is in the training split.
The same numbers in a table (test split):
| Measure | Matcher’s pairs | Connected components | Safe clustering |
|---|---|---|---|
| precision (pairs) | 0.918 | 0.838 | 0.911 |
| recall (pairs) | 0.886 | 0.917 | 0.909 |
| B-cubed precision | 0.979 | 0.987 | |
| B-cubed recall | 0.967 | 0.965 | |
| exact-cluster precision | 0.873 | 0.88 | |
| exact-cluster recall | 0.913 | 0.923 |
Read it from left to right:
- Clustering adds wrong pairs. The matcher’s pairs have precision 0.918. The pairs inside the connected components have precision 0.838. Recall goes up, from 0.886 to 0.917: transitivity also finds true pairs that the matcher missed.
- Safe clustering removes most of the added errors. Cluster-pair precision rises to 0.911, and recall falls a little, to 0.909.
- About nine in ten customers are exactly right. Exact-cluster precision is 0.88 with safe clustering.
Every matcher: the pairs, then the clusters
Section titled “Every matcher: the pairs, then the clusters”- Matcher's pairs
- Cluster pairs, connected components
- Cluster pairs, safe clustering
Text description of the chart
A dot chart with three rows for each of three matchers. rules: the matcher's own pairs 0.723; the pairs inside the connected components 0.524; the pairs inside the safe clusters 0.832. splink: the matcher's own pairs 0.817; the pairs inside the connected components 0.685; the pairs inside the safe clusters 0.831. lgbm: the matcher's own pairs 0.918; the pairs inside the connected components 0.838; the pairs inside the safe clusters 0.911. For every matcher, connected components lower the precision, and safe clustering brings it back up.
The rules lose most: from 0.723 to 0.524. They make many links, so they make long chains. LightGBM loses least, because it makes fewer wrong links.
Where the wrong pairs hide
Section titled “Where the wrong pairs hide”Which clusters hold the wrong pairs? We counted the pairs inside LightGBM’s connected components that involve a record of the test split, by the size of the cluster.
Text description of the chart
A column chart of wrong pairs by cluster size. Clusters of 2 records: 42 wrong pairs of 2,460. Clusters of 3 records: 160 wrong pairs of 2,190. Clusters of 4 records: 286 wrong pairs of 743. Clusters of 5 records: 266 wrong pairs of 397. Clusters of 6 records: 134 wrong pairs of 177. Clusters of 7 or more records: 97 wrong pairs of 118. The clusters of more than 5 records are highlighted: they hold 23.5% of all wrong pairs.
Over all the splits, only 75 clusters have more than 5 records. Together they hold 493 of the 60,108 records. Yet on the test pairs, they hold 23.5% of the wrong pairs inside clusters. A cluster’s pairs grow with the square of its size. So a few large clusters hold many wrong pairs.
The matcher’s own pair measures never showed this. They count only the 5,375 test pairs that LightGBM called the same. Clustering added 710 more pairs, and 542 of them are wrong.
And B-cubed can hide them too
Section titled “And B-cubed can hide them too”Now look at the rule “same phone”, with its giant cluster of 40 people:
| Rule “same phone” | Connected components | Safe clustering |
|---|---|---|
| B-cubed precision | 0.941 | 0.969 |
| cluster-pair precision | 0.539 | 0.745 |
| largest cluster | 40 | 6 |
B-cubed precision is 0.941: it looks good. The giant cluster has only 40 records among 60,108, so it barely moves a mean over records. But for Larkfield, it is one false “customer” who receives the orders, addresses and loyalty points of 40 strangers.
So every measure hides something. Pair measures hide the pairs that clustering adds. B-cubed hides a few giant clusters. Exact clusters hide clusters that are almost right. Report several measures, and always the size of the largest clusters.
The comparison table, with entities
Section titled “The comparison table, with entities”The comparison table of the course gets new columns. For each run: the pair precision and recall of the matcher, then the entities with safe clustering.
| Run | Pair precision | Pair recall | Entities (safe) | Largest | B-cubed precision | B-cubed recall | Exact clusters, precision |
|---|---|---|---|---|---|---|---|
exact-norm | 0.819 | 0.438 | 48,601 | 4 | 0.995 | 0.791 | 0.545 |
rules | 0.723 | 0.87 | 39,877 | 6 | 0.974 | 0.937 | 0.798 |
rule-phone | 0.539 | 0.494 | 47,096 | 6 | 0.969 | 0.8 | 0.533 |
weighted | 0.751 | 0.766 | 40,228 | 6 | 0.962 | 0.918 | 0.737 |
splink | 0.817 | 0.779 | 40,516 | 6 | 0.975 | 0.924 | 0.767 |
lgbm | 0.918 | 0.886 | 38,908 | 6 | 0.987 | 0.965 | 0.88 |
The true number is 37,500 entities. LightGBM with safe clustering has the best B-cubed precision and recall together. The order of the runs is almost the same as with pair F1, but the gaps are different. Module 8 runs the best of them every day.
When to use each measure
Section titled “When to use each measure”- Cluster-pair precision and recall: to compare with the matcher’s own pair measures, and to see how much clustering adds or removes.
- B-cubed: the main summary of the entities. Each record counts once.
- Exact clusters: a number that TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. can say in one sentence: “nine in ten customers are exactly right”.
- The largest clusters and the clusters above a size limit: always. They find the errors that the averages hide.
- What it costs: you need the true entities for the measured records. Here our synthetic data gives them. In real work, label a sample of clusters by hand: a reviewer can judge a whole cluster in one look.
Common mistake
The customer count is almost right, so the matching is right
Symptom: the four rules of Module 2 make 36,756 entities. The true number is 37,500. TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. says: “Only 2% off. The rules are almost perfect.”
Cause: a count adds wrong merges and wrong splits together, and they cancel. The rules merge whole families (too few entities) and miss customers who changed their email (too many entities). The total is close, but many clusters are wrong. With connected components, the rules’ B-cubed precision is 0.917, and only 0.786 of their clusters are exactly one customer.
Fix: never judge entities by the count alone. Measure B-cubed precision and recall and exact clusters, and look at the largest clusters. Report the count together with these measures.
Consequence in AI work: a total that matches the truth can hide errors in both directions. A classifier can predict the right number of positives with many wrong ones. Classification metrics in Machine Learning: From Problem to Reliable Model shows why precision and recall are needed together.
Your task: fix the largest clusters
Section titled “Your task: fix the largest clusters”This is the practice task of Module 7. Use the project, or the notebooks of this module.
- Cluster the pairs of one run with connected components, for example
match cluster --run splink. Write down the entity count and the three largest clusters. - Look inside each of the three largest clusters. Use
match showon its records, or the notebook. Is each one person? If not, why were the records joined: a placeholder value, a family, a weak link? - Fix them: run safe clustering, or
match undoa wrong record. - Build the golden records with
match golden --run <your run>. - Show the entity count before and after your fix, with B-cubed precision and recall from
match eval --run <your run>.
Completion check: show your entity count before and after the fix, with B-cubed (cluster) precision and recall. Add one sentence on each of the three largest clusters.
Knowledge check
Section titled “Knowledge check”Check your understanding
Four records: r1 and r2 are one customer, r3 and r4 another. The clusters are {r1, r2, r3} and {r4}. Compute B-cubed precision and recall.
Show the answer
Precision per record: r1 2 ÷ 3; r2 2 ÷ 3; r3 1 ÷ 3; r4 1 ÷ 1. The mean is (0.667 + 0.667 + 0.333 + 1) ÷ 4 = 0.667. Recall per record: r1 2 ÷ 2; r2 2 ÷ 2; r3 1 ÷ 2; r4 1 ÷ 2. The mean is (1 + 1 + 0.5 + 0.5) ÷ 4 = 0.75. One record in the wrong cluster lowers both.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
LightGBM's own pairs have precision 0.918, but the pairs inside its connected components have precision 0.838. Where do the extra wrong pairs come from?
Show the answer
From transitivity. When the model links A–B and B–C, the cluster also contains A–C, a pair that the model never called the same. If one of the links is wrong, every record on one side is paired with every record on the other side. Clustering added 710 pairs with a test record, and 542 of them are wrong. Safe clustering cuts the weak links and brings the precision back to 0.911.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The rule "same phone" has B-cubed precision 0.941 with connected components, and a cluster of 40 different people. Why does B-cubed not show the giant cluster?
Show the answer
B-cubed takes the mean over records, and each record counts once. The giant cluster has 40 records among more than 12,000 test records. Even with a precision of 1 ÷ 40 each, they move the mean very little. Cluster-pair precision counts the 780 wrong pairs inside it, and the largest cluster size shows it at once. That is why you report several measures and the largest clusters.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Pair measures judge the matcher’s decisions. Clustering adds pairs that the matcher never decided.
- Cluster-pair measures count every pair inside a cluster. Exact-cluster measures count clusters that are exactly one entity. B-cubed scores each record and takes the mean.
- For LightGBM, clustering lowers pair precision from 0.918 to 0.838; safe clustering brings it to 0.911, with B-cubed precision 0.987 and recall 0.965.
- The 75 clusters of more than 5 records hold 23.5% of the wrong pairs inside LightGBM’s clusters. B-cubed can hide a giant cluster too.
- Report several measures, the count, and the largest clusters. A count close to the truth can hide errors in both directions.
Module 7 is done: Larkfield has 38,908 entities with golden records, safe clusters and a history. Next, in Module 8, new records arrive every day, and the entities must stay stable.
Next: New records every day