Skip to content
Français

Cluster measures

  • Lesson 25
  • 35 min
  • Module 7: From pairs to entities

Where we are

Our entities have golden records. We measured pairs, but Larkfield counts customers. In this lesson, we measure clusters with pair, cluster and B-cubed measures, and we see how giant clusters hide behind good pair numbers.

What you will be able to do

Measure entities, not only pairs. Learn three cluster measures: cluster-pair, exact-cluster and B-cubed precision and recall. See how a few large clusters hide many wrong pairs that the pair measures never show, and why no single number is enough.

NotebookMeasure clusters in a notebookColab · Kaggle · GitHub

The notebook starts with six records of two people and one bad link. You write B-cubed precision and recall. Then you measure the clusters of Module 2’s rules, with and without safe clustering, and you count where the wrong pairs hide. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of Module 7. Or use the notebook only.

A teacher asks a helper to split 30 students into project groups of friends. How does she check his work?

She can check pairs: “He said Sam and Lee are friends. Are they?” That checks his decisions, one by one.

She can check groups: “Is this group exactly one set of friends?” That is strict. A group with one wrong student fails completely.

She can ask each student: “How much of your group are your friends? How many of your friends are in your group?” Then she averages the answers. A student in a good group says “all of them”. A student who was put with strangers says “only a few”.

Each check sees something that the others miss. Matching has the same three views.

The problem: the pair measures judge the matcher, not the entities

Section titled “The problem: the pair measures judge the matcher, not the entities”

Measure a matcher showed this problem first. Here we measure it properly. Since Module 2, we measured pairs: of the pairs that the matcher calls the same, how many are true pairs? That measures the matcher’s decisions. But TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield uses the entities. And clustering adds pairs that the matcher never decided.

Here is a tiny example. Ann has three records: a1, a2, a3. Ben has three records: b1, b2, b3. The matcher calls 5 pairs the same: a1–a2, a2–a3, b1–b2, b2–b3, a3–b1. Only the last one is wrong. Connected components joins all six records into one cluster.

Six records, a1 to a3 for Ann and b1 to b3 for Ben, in one cluster. Links a1–a2, a2–a3, b1–b2, b2–b3 are right; a3–b1 is wrong. Pair measures: 4 of 5 called pairs are right, precision 0.8, recall 0.667. Cluster pairs: 15 pairs inside the cluster, 6 right, precision 0.4, recall 1. B-cubed: precision 3 of 6 for each record, mean 0.5; recall 3 of 3, mean 1. Exact clusters: 0 of 1 right.
Three ways to measure one cluster of six records of two people, joined by one wrong link (a made-up example).
Text description of the diagram

The picture shows six small circles inside one dashed box: a1, a2 and a3 for Ann, and b1, b2 and b3 for Ben. Solid lines join a1 to a2, a2 to a3, b1 to b2, and b2 to b3. A dashed red line joins a3 to b1: the one wrong link. Four text boxes give the measures. Pair measures: 4 of the 5 pairs called the same are right, so precision is 0.8. 4 of the 6 true pairs are found, so recall is 0.667. Cluster pairs: the cluster holds 6 × 5 ÷ 2 = 15 pairs. 6 of them are right, so precision is 0.4. All 6 true pairs are inside, so recall is 1. B-cubed: for every record, 3 of the 6 records in its cluster are the same person. So precision is 0.5. All 3 of its person’s records are in its cluster, so recall is 1.0. Exact clusters: the one cluster is not exactly one person, so 0 of 1 is right.

The matcher’s pairs look good: precision 0.8. But the cluster is half wrong. Of the 15 pairs inside it, 9 are wrong. The matcher decided only one of them, a3–b1. Connected components added the other 8: a1–b1, a1–b2, a2–b3 and the rest. No pair measure of the matcher can see them.

Each measure compares the clusters with the true entities. Each has a precision (“how much of what we joined is right?”) and a recall (“how much of what should be joined did we join?”).

Take every pair of records inside one cluster, also the pairs that the matcher never compared. Count them as “called the same”. Then compute pair precision and recall as before.

In the tiny example: 15 pairs, 6 right, so precision is 0.4. All 6 true pairs are inside, so recall is 1.

This measure is easy to compare with the matcher’s own pair measures. But it counts big clusters many times: a cluster of n records holds n × (n − 1) ÷ 2 pairs. One cluster of 40 records holds 780 pairs.

A cluster is right only when it holds exactly the records of one true entity: not one record more or less.

  • Exact-cluster precision: of the clusters we made, the share that are exactly right.
  • Exact-cluster recall: of the true entities, the share that we found exactly.

In the tiny example: 0 of 1 cluster is right. This measure is strict and easy to explain: “88% of our customers are exactly right”. But it gives no credit for a cluster that is almost right.

B-cubed asks each record two questions, then takes the mean over all records:

  • precision of a record: the share of its cluster that is the same entity as the record;
  • recall of a record: the share of its entity’s records that are in its cluster.

In the tiny example, every record has precision 3 ÷ 6 = 0.5 and recall 3 ÷ 3 = 1.0. So B-cubed precision is 0.5 and recall is 1. Amit Bagga and Breck Baldwin proposed it in 1998.

B-cubed counts every record once. A big cluster does not count more than its records. So it is the fairest summary of “how right are our customers?”. But for the same reason, it can hide one giant cluster among thousands of good ones.

From this module on, match eval prints all three measures for both clusterings. Run it for LightGBM. You did the undo of the last lesson, so safe clustering also keeps S-009232 apart:

Terminal window
match eval --run lgbm
lgbm on the test split: pair precision 0.918, recall 0.886, F1 0.902
5,375 pairs called the same: 4,932 right, 443 wrong; 632 of 5,564 true pairs missed
entities after clustering (all records): 38,514 (true: 37,500)
connected components: 38,514 entities, largest 9 records, 75 clusters over 5 records
B-cubed precision 0.979, recall 0.967; cluster pairs precision 0.838, recall 0.917; exact clusters precision 0.873, recall 0.913
safe clustering: 38,909 entities, largest 6 records, 11 clusters over 5 records
B-cubed precision 0.987, recall 0.965; cluster pairs precision 0.911, recall 0.909; exact clusters precision 0.880, recall 0.923

Safe clustering counts 38,909 entities here, the same as match golden after the undo. Ivana’s two cannot-link pairs come first: safe clustering removes the links between S-009232 and the other two records before any other check. The right link between G-000001 and L-010072 stays, so Valérie’s other two records stay one entity. The measures of the test split do not change: Valérie is in the training split.

The same numbers in a table (test split):

Measure Matcher’s pairs Connected components Safe clustering
precision (pairs) 0.918 0.838 0.911
recall (pairs) 0.886 0.917 0.909
B-cubed precision 0.979 0.987
B-cubed recall 0.967 0.965
exact-cluster precision 0.873 0.88
exact-cluster recall 0.913 0.923

Read it from left to right:

  • Clustering adds wrong pairs. The matcher’s pairs have precision 0.918. The pairs inside the connected components have precision 0.838. Recall goes up, from 0.886 to 0.917: transitivity also finds true pairs that the matcher missed.
  • Safe clustering removes most of the added errors. Cluster-pair precision rises to 0.911, and recall falls a little, to 0.909.
  • About nine in ten customers are exactly right. Exact-cluster precision is 0.88 with safe clustering.

Every matcher: the pairs, then the clusters

Section titled “Every matcher: the pairs, then the clusters”
  • Matcher's pairs
  • Cluster pairs, connected components
  • Cluster pairs, safe clustering
Pair precision of three matchers' own pairs, and of the pairs inside their clusters (connected components and safe clustering), on the test split of the synthetic records. Measured.
Text description of the chart

A dot chart with three rows for each of three matchers. rules: the matcher's own pairs 0.723; the pairs inside the connected components 0.524; the pairs inside the safe clusters 0.832. splink: the matcher's own pairs 0.817; the pairs inside the connected components 0.685; the pairs inside the safe clusters 0.831. lgbm: the matcher's own pairs 0.918; the pairs inside the connected components 0.838; the pairs inside the safe clusters 0.911. For every matcher, connected components lower the precision, and safe clustering brings it back up.

The rules lose most: from 0.723 to 0.524. They make many links, so they make long chains. LightGBM loses least, because it makes fewer wrong links.

Which clusters hold the wrong pairs? We counted the pairs inside LightGBM’s connected components that involve a record of the test split, by the size of the cluster.

Wrong pairs inside LightGBM's connected components, by the number of records in the cluster, for pairs with a record in the test split of the synthetic records. Measured. No true customer has more than 5 records.
Text description of the chart

A column chart of wrong pairs by cluster size. Clusters of 2 records: 42 wrong pairs of 2,460. Clusters of 3 records: 160 wrong pairs of 2,190. Clusters of 4 records: 286 wrong pairs of 743. Clusters of 5 records: 266 wrong pairs of 397. Clusters of 6 records: 134 wrong pairs of 177. Clusters of 7 or more records: 97 wrong pairs of 118. The clusters of more than 5 records are highlighted: they hold 23.5% of all wrong pairs.

Over all the splits, only 75 clusters have more than 5 records. Together they hold 493 of the 60,108 records. Yet on the test pairs, they hold 23.5% of the wrong pairs inside clusters. A cluster’s pairs grow with the square of its size. So a few large clusters hold many wrong pairs.

The matcher’s own pair measures never showed this. They count only the 5,375 test pairs that LightGBM called the same. Clustering added 710 more pairs, and 542 of them are wrong.

Now look at the rule “same phone”, with its giant cluster of 40 people:

Rule “same phone” Connected components Safe clustering
B-cubed precision 0.941 0.969
cluster-pair precision 0.539 0.745
largest cluster 40 6

B-cubed precision is 0.941: it looks good. The giant cluster has only 40 records among 60,108, so it barely moves a mean over records. But for Larkfield, it is one false “customer” who receives the orders, addresses and loyalty points of 40 strangers.

So every measure hides something. Pair measures hide the pairs that clustering adds. B-cubed hides a few giant clusters. Exact clusters hide clusters that are almost right. Report several measures, and always the size of the largest clusters.

The comparison table of the course gets new columns. For each run: the pair precision and recall of the matcher, then the entities with safe clustering.

RunPair precisionPair recallEntities (safe)LargestB-cubed precisionB-cubed recallExact clusters, precision
exact-norm0.8190.43848,60140.9950.7910.545
rules0.7230.8739,87760.9740.9370.798
rule-phone0.5390.49447,09660.9690.80.533
weighted0.7510.76640,22860.9620.9180.737
splink0.8170.77940,51660.9750.9240.767
lgbm0.9180.88638,90860.9870.9650.88

The true number is 37,500 entities. LightGBM with safe clustering has the best B-cubed precision and recall together. The order of the runs is almost the same as with pair F1, but the gaps are different. Module 8 runs the best of them every day.

  • Cluster-pair precision and recall: to compare with the matcher’s own pair measures, and to see how much clustering adds or removes.
  • B-cubed: the main summary of the entities. Each record counts once.
  • Exact clusters: a number that TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. can say in one sentence: “nine in ten customers are exactly right”.
  • The largest clusters and the clusters above a size limit: always. They find the errors that the averages hide.
  • What it costs: you need the true entities for the measured records. Here our synthetic data gives them. In real work, label a sample of clusters by hand: a reviewer can judge a whole cluster in one look.

Common mistake

The customer count is almost right, so the matching is right

Symptom: the four rules of Module 2 make 36,756 entities. The true number is 37,500. TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. says: “Only 2% off. The rules are almost perfect.”

Cause: a count adds wrong merges and wrong splits together, and they cancel. The rules merge whole families (too few entities) and miss customers who changed their email (too many entities). The total is close, but many clusters are wrong. With connected components, the rules’ B-cubed precision is 0.917, and only 0.786 of their clusters are exactly one customer.

Fix: never judge entities by the count alone. Measure B-cubed precision and recall and exact clusters, and look at the largest clusters. Report the count together with these measures.

Consequence in AI work: a total that matches the truth can hide errors in both directions. A classifier can predict the right number of positives with many wrong ones. Classification metrics in Machine Learning: From Problem to Reliable Model shows why precision and recall are needed together.

This is the practice task of Module 7. Use the project, or the notebooks of this module.

  1. Cluster the pairs of one run with connected components, for example match cluster --run splink. Write down the entity count and the three largest clusters.
  2. Look inside each of the three largest clusters. Use match show on its records, or the notebook. Is each one person? If not, why were the records joined: a placeholder value, a family, a weak link?
  3. Fix them: run safe clustering, or match undo a wrong record.
  4. Build the golden records with match golden --run <your run>.
  5. Show the entity count before and after your fix, with B-cubed precision and recall from match eval --run <your run>.

Completion check: show your entity count before and after the fix, with B-cubed (cluster) precision and recall. Add one sentence on each of the three largest clusters.

Check your understanding

Four records: r1 and r2 are one customer, r3 and r4 another. The clusters are {r1, r2, r3} and {r4}. Compute B-cubed precision and recall.

Show the answer

Precision per record: r1 2 ÷ 3; r2 2 ÷ 3; r3 1 ÷ 3; r4 1 ÷ 1. The mean is (0.667 + 0.667 + 0.333 + 1) ÷ 4 = 0.667. Recall per record: r1 2 ÷ 2; r2 2 ÷ 2; r3 1 ÷ 2; r4 1 ÷ 2. The mean is (1 + 1 + 0.5 + 0.5) ÷ 4 = 0.75. One record in the wrong cluster lowers both.

Check your understanding

LightGBM's own pairs have precision 0.918, but the pairs inside its connected components have precision 0.838. Where do the extra wrong pairs come from?

Show the answer

From transitivity. When the model links A–B and B–C, the cluster also contains A–C, a pair that the model never called the same. If one of the links is wrong, every record on one side is paired with every record on the other side. Clustering added 710 pairs with a test record, and 542 of them are wrong. Safe clustering cuts the weak links and brings the precision back to 0.911.

Check your understanding

The rule "same phone" has B-cubed precision 0.941 with connected components, and a cluster of 40 different people. Why does B-cubed not show the giant cluster?

Show the answer

B-cubed takes the mean over records, and each record counts once. The giant cluster has 40 records among more than 12,000 test records. Even with a precision of 1 ÷ 40 each, they move the mean very little. Cluster-pair precision counts the 780 wrong pairs inside it, and the largest cluster size shows it at once. That is why you report several measures and the largest clusters.

Recap

  • Pair measures judge the matcher’s decisions. Clustering adds pairs that the matcher never decided.
  • Cluster-pair measures count every pair inside a cluster. Exact-cluster measures count clusters that are exactly one entity. B-cubed scores each record and takes the mean.
  • For LightGBM, clustering lowers pair precision from 0.918 to 0.838; safe clustering brings it to 0.911, with B-cubed precision 0.987 and recall 0.965.
  • The 75 clusters of more than 5 records hold 23.5% of the wrong pairs inside LightGBM’s clusters. B-cubed can hide a giant cluster too.
  • Report several measures, the count, and the largest clusters. A count close to the truth can hide errors in both directions.

Module 7 is done: Larkfield has 38,908 entities with golden records, safe clusters and a history. Next, in Module 8, new records arrive every day, and the entities must stay stable.

Next: New records every day