False merges and missed matches
- Lesson 2
- 30 min
- Module 1: The matching problem
Where we are
In the last lesson, we saw that one Larkfield customer can appear in two or three lists, and that a join on the email still counts far too many customers. A matcher that joins records can be wrong in two ways. In this lesson, we name the two errors, false merges and missed matches, and we see who pays for each one. We go back to the clinic first.
Lesson 2 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?False merges and missed matches
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Name the two ways a matcher can be wrong, and put a price on each one. Then you can say, for any use of the customer list, which error the business can accept more of.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s three lists, entities, records and true pairs, from Three lists, one customer.
- It helps to know precision and recall from Classification metrics in Machine Learning: From Problem to Reliable Model. This lesson explains them again for pairs of records.
What you need on your computer
Nothing. This lesson has no code. You run the same measures yourself in Meet the records.
Two mistakes at the clinic
Section titled “Two mistakes at the clinic”Go back to the clinic of the last lesson. The clerk now tries to clean up the files. She can make two kinds of mistake.
She can join two files that belong to two different people. Two brothers, Samuel and Nathan, live at the same address and share one phone. Their files become one. Now Samuel’s file shows Nathan’s allergy and Nathan’s blood test. A doctor reads it and gives Samuel the wrong treatment. And when Samuel asks for a copy of his file, he sees his brother’s private medical data.
She can leave two files of one person apart. Then the patient gets two reminder letters, and a test may be done twice. That costs time and money, and it annoys the patient. But nobody sees someone else’s private data, and nobody gets the wrong treatment because of it.
Both are mistakes. But they are not equally bad. The first one hurts people and breaks trust. The second one wastes money. A careful clerk joins two files only when she is quite sure. That is the whole idea of this lesson.
Two ways to be wrong
Section titled “Two ways to be wrong”A matcher looks at a pair of records and decides: same or different. The truth is also same or different. So every decision falls into one of four boxes:
| The truth: same entity | The truth: different entities | |
|---|---|---|
| The matcher says: same | a right merge | a false merge |
| The matcher says: different | a missed match | a right separation |
- A false merge joins two records that belong to different entities. Two people become one customer.
- A missed match keeps apart two records of the same entity. One person stays two customers.
Here are both errors in TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s real records. The matcher is the email join from the last lesson: same email, exactly as typed, means the same customer.
Text description of the diagram
The picture has two panels. On the left, two garden-centre records: G-001688 is GREEN, SAMUEL, and G-002894 is GREEN, NATHAN. They have the same email, thegreens@example.com, the same phone and the same address on Kestrel Avenue. Only Nathan’s record has a date of birth. The email join says “same”, so they become one customer. The truth, in a dashed box, is that they are two brothers who share a family email and a landline. On the right, a loyalty record, L-000482, says “Thibault Helene”, with the email thibault.helene93@example.net. A shop record, S-028040, says “Hélène Thibault”, with Thibault.Helene93@Example.net. The emails differ only in capitals, so the email join says “different”. The truth is that they are one woman. Her names are swapped in the loyalty list, and the two postcodes differ because she moved.
The brothers are a practice problem that we made on purpose: the course data has 80 pairs of brothers like them. Hélène is an ordinary customer of the data, with ordinary mistakes.
A tiny example with every pair visible
Section titled “A tiny example with every pair visible”Take only these four records: Samuel (G-001688), Nathan (G-002894), Hélène in the loyalty list (L-000482) and Hélène in the shop (S-028040). Four records make six pairs. The email join decides each pair:
| Pair | Same email as typed? | The matcher says | The truth | Result |
|---|---|---|---|---|
| Samuel – Nathan | yes | same | different | false merge |
| Hélène (L) – Hélène (S) | no, other capitals | different | same | missed match |
| Samuel – Hélène (L) | no | different | different | right separation |
| Samuel – Hélène (S) | no | different | different | right separation |
| Nathan – Hélène (L) | no | different | different | right separation |
| Nathan – Hélène (S) | no | different | different | right separation |
So the matcher made one false merge, one missed match, four right separations and no right merge. Four of six decisions are right. That sounds acceptable, but it did nothing useful. Notice where the right answers came from: most pairs of records are different people, so “different” is easy to get right.
Two measures: precision and recall
Section titled “Two measures: precision and recall”You know two measures from the machine learning course. Here they count pairs of records:
- Pair precision is the share of the pairs called “same” that really are the same. It answers: “When the matcher merges, how often is it right?” False merges lower it.
- Pair recall is the share of the true pairs that the matcher finds. It answers: “Of all the pairs that should be merged, how many did it merge?” Missed matches lower it.
In the tiny example, the matcher called one pair “same”, and it was wrong: precision 0 of 1. There is one true pair, and the matcher missed it: recall 0 of 1.
Now the same email join on the course’s test split, the 12,041 records that the course keeps for measuring. They hold 5,564 true pairs. The matcher calls 2,300 pairs “same”:
- 1,879 of them are right, and 421 are false merges. Precision = 1,879 / 2,300 = 0.817.
- It finds 1,879 of the 5,564 true pairs, so it makes 3,685 missed matches. Recall = 1,879 / 5,564 = 0.338.
So about one merge in five is wrong, and two of every three true pairs are missed. Precision and recall pull against each other. A strict matcher merges little: few false merges, many missed matches. A loose matcher merges a lot: the opposite. Measure a matcher in Module 2 builds these measures in the project.
Who pays for each error
Section titled “Who pays for each error”Precision and recall count errors. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. wants to know what they cost. The two errors hurt different people.
A false merge: a stranger’s data
Section titled “A false merge: a stranger’s data”When Samuel and Nathan become one customer, Larkfield shows one account page with both brothers’ orders and addresses. Samuel signs in and sees what his brother bought and where it went. Larkfield may send Nathan’s parcel to Samuel’s address. Each brother’s emails go to the other one too.
This is a privacy problem, not only an annoyance. One person sees another person’s personal data. Support must find the error, split the account, apologise and give a credit. The data team may need to report the incident. The course uses this assumption: a false merge costs 150 dollars.
A missed match: a little waste every year
Section titled “A missed match: a little waste every year”When Hélène stays two customers, she gets every marketing email twice and two catalogues. Her loyalty points are split over two accounts. Larkfield’s customer count is a little too high. This costs a little, and it costs it every year. The course’s assumption: a missed match costs 3 dollars a year.
Both numbers are assumptions of the course, written in the data file costs.json, not measured prices. Your business has its own numbers. But the ratio matters: here, one false merge costs as much as 50 years of one missed match.
The cost of four matchers
Section titled “The cost of four matchers”TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. puts a price on four simple matchers, measured on the test split, and on doing nothing at all. Three of them are rules that compare values after cleaning them, for example without capitals and accents. You build them in Module 2; here, only their results matter:
| Matcher | What it calls “same” | Precision | Recall | False merges | Missed matches | Cost in the first year |
|---|---|---|---|---|---|---|
| Nothing | no pair | – | 0 | 0 | 5,564 | 16,692 dollars |
| Strict | same name and same date of birth | 1.000 | 0.170 | 0 | 4,618 | 13,854 dollars |
| Middle | same name and same postcode | 0.921 | 0.621 | 296 | 2,106 | 50,718 dollars |
| same email as typed | 0.817 | 0.338 | 421 | 3,685 | 74,205 dollars | |
| Loose | four rules: same email, or same phone and surname, or same name and date of birth, or same name and postcode | 0.723 | 0.870 | 1,853 | 724 | 280,122 dollars |
The cost is false merges × 150 dollars + missed matches × 3 dollars. For the email join: 421 × 150 = 63,150, and 3,685 × 3 = 11,055, so 74,205 dollars.
Text description of the chart
A column chart with five columns, in thousands of dollars, on an axis from 0 to 300. Doing nothing costs 16.7 thousand dollars: only missed matches. The strict rule, same name and same date of birth, costs 13.9 thousand. The middle rule, same name and same postcode, costs 50.7 thousand. The email join costs 74.2 thousand. The loose matcher, four rules together, costs 280.1 thousand dollars, more than three times the email join, although it finds the most true pairs.
Read the table from the bottom. The loose matcher has the best recall: it finds 87% of the true pairs. But its 1,853 false merges cost 277,950 dollars, almost all of its cost. The strict rule finds only 17% of the true pairs, and it is the cheapest: it never merged two different people here.
TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. is surprised: “So the best matcher is the one that finds the fewest pairs?” Priya answers: “With these costs, a false merge is expensive, so precision matters most. But the strict rule still leaves 4,618 true pairs apart. We want a matcher with the precision of the strict rule and much more recall. That is what the rest of the course builds.”
Why most teams accept more missed matches
Section titled “Why most teams accept more missed matches”The table is one example, but the pattern is common. Most teams accept more missed matches than false merges, for four reasons:
- A false merge harms a person. Someone sees a stranger’s data, or gets their parcel. A missed match only wastes some money.
- A false merge spreads. Once two records are one, other systems use the merged record. Emails go out, orders ship and reports count it. Each of these must be fixed.
- A missed match can wait. A later run, with a better matcher, can still find the pair. Nothing was broken in the meantime.
- The law cares about personal data. Showing one person’s data to another can be a reportable incident. The AI Security, Privacy and Responsible Design course covers this in Impact and error costs.
So matching teams often add a third answer: unsure. The matcher merges only the pairs that it is very sure about. It leaves apart the pairs that it is sure are different. A person checks the pairs in between. At Larkfield, that person is Ivana, the customer-data steward. The course assumes that she can check 300 pairs a day, so the unsure pairs must stay few. Modules 5 and 8 build this review step.
Which error is worse depends on the use
Section titled “Which error is worse depends on the use”The same customer list serves several teams, and they do not weigh the errors in the same way:
| Use | A false merge means | A missed match means | Usually worse |
|---|---|---|---|
| Marketing emails | one person gets another’s name in an email, or one sibling stops getting emails | the same email twice, two catalogues | the false merge, but both are mild |
| Fraud checks | an honest customer is linked to a fraud ring and blocked | a ring of accounts stays hidden | the missed match, if a person checks each alert before acting |
| Legal requests, for example “delete my data” or “send me my data” | you send or delete another person’s data | you keep data that the person asked you to delete | the false merge; but a missed match also breaks the request |
For fraud, the team wants high recall: a missed link hides the ring. That is safe only because a person checks each alert before an account is blocked. For legal requests, both errors can break privacy rules, so the team uses its strictest matcher and asks a person about every unsure pair. There is no single best matcher. There is a best matcher for a use and its costs.
Common mistake
Judge a matcher by its accuracy
Symptom: Tomás reports a new matcher: “It is right on 99.998% of all pairs.” Then Omar finds that it merged nobody at all.
Cause: almost every pair of records is two different people. The course data has 1,806,455,778 possible pairs and only 27,677 true pairs. A matcher that says “different” for every pair is right on 99.998% of them, and it is useless. Accuracy, the share of right decisions, hides both errors behind the easy “different” pairs. This is the class imbalance that you know from the machine learning course, in an extreme form.
Fix: never report accuracy for matching. Report precision and recall, with the counts behind them: pairs called “same”, false merges and missed matches. Then multiply the two errors by their costs for the use.
Consequence in AI work: a measure must count the errors that matter. With rare true pairs, accuracy rewards a matcher for doing nothing.
Knowledge check
Section titled “Knowledge check”Check your understanding
For each case, say whether it is a false merge, a missed match or a right decision. (a) Two records of Valérie Tremblay are joined. (b) Marc Fortin and Marie Fortin are joined. (c) Hélène Thibault's two records stay apart.
Show the answer
(a) A right merge: one person, joined. (b) A false merge: two different people, joined, so precision falls and each one may see the other’s data. (c) A missed match: one person, kept apart, so recall falls and she gets everything twice.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
For marketing emails, fraud checks and a request to delete a customer's data, which error is worse, and why?
Show the answer
- Marketing emails: usually the false merge, because one person may get another’s name or stop getting emails. A missed match only sends the same email twice. Both are mild.
- Fraud checks: the missed match, because a ring of linked accounts stays hidden. A false merge is acceptable only because a person checks each alert before an account is blocked.
- A deletion request: the false merge, because you would delete another person’s data, or send it to the wrong person. But a missed match also fails the request, because some of the person’s data stays. So use the strictest matcher, and ask a person about every unsure pair.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A matcher calls 1,000 pairs “same” on the test split: 900 are right. The test split has 5,564 true pairs. Compute its precision, its recall and its first-year cost with the course's costs. Is it better than the email join?
Show the answer
Precision = 900 / 1,000 = 0.900. Recall = 900 / 5,564 = 0.162. It makes 100 false merges and 5,564 − 900 = 4,664 missed matches. Cost = 100 × 150 + 4,664 × 3 = 15,000 + 13,992 = 28,992 dollars. The email join costs 74,205 dollars, so this matcher is cheaper with these costs, although its recall is lower. It is still worse than the strict rule (13,854 dollars), which has no false merges. With other costs, the order could change.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A false merge joins two different entities. A missed match keeps one entity apart. Pair precision falls with false merges; pair recall falls with missed matches.
- The email join on the test split has precision 0.817: 421 of its 2,300 merges are false. Its recall is 0.338: it misses 3,685 of 5,564 true pairs.
- The course’s assumed costs: 150 dollars for a false merge and 3 dollars a year for a missed match. With them, the loose matcher costs 280,122 dollars in the first year, and the strict rule costs 13,854.
- Most teams accept more missed matches, because a false merge harms a person, spreads to other systems and can be a privacy incident. Unsure pairs go to a person.
- Which error is worse depends on the use: marketing, fraud and legal requests weigh them differently. Never judge a matcher by accuracy.
You can now name the two errors, measure them and price them. But so far, you have only read results that someone else ran. Next, you set up the project, open the three lists yourself, and run the email join on your own computer.
Next: Meet the records