Skip to content
Français

False merges and missed matches

  • Lesson 2
  • 30 min
  • Module 1: The matching problem

Where we are

In the last lesson, we saw that one Larkfield customer can appear in two or three lists, and that a join on the email still counts far too many customers. A matcher that joins records can be wrong in two ways. In this lesson, we name the two errors, false merges and missed matches, and we see who pays for each one. We go back to the clinic first.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Name the two ways a matcher can be wrong, and put a price on each one. Then you can say, for any use of the customer list, which error the business can accept more of.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. You run the same measures yourself in Meet the records.

Go back to the clinic of the last lesson. The clerk now tries to clean up the files. She can make two kinds of mistake.

She can join two files that belong to two different people. Two brothers, Samuel and Nathan, live at the same address and share one phone. Their files become one. Now Samuel’s file shows Nathan’s allergy and Nathan’s blood test. A doctor reads it and gives Samuel the wrong treatment. And when Samuel asks for a copy of his file, he sees his brother’s private medical data.

She can leave two files of one person apart. Then the patient gets two reminder letters, and a test may be done twice. That costs time and money, and it annoys the patient. But nobody sees someone else’s private data, and nobody gets the wrong treatment because of it.

Both are mistakes. But they are not equally bad. The first one hurts people and breaks trust. The second one wastes money. A careful clerk joins two files only when she is quite sure. That is the whole idea of this lesson.

A matcher looks at a pair of records and decides: same or different. The truth is also same or different. So every decision falls into one of four boxes:

The truth: same entity The truth: different entities
The matcher says: same a right merge a false merge
The matcher says: different a missed match a right separation
  • A false merge joins two records that belong to different entities. Two people become one customer.
  • A missed match keeps apart two records of the same entity. One person stays two customers.

Here are both errors in TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s real records. The matcher is the email join from the last lesson: same email, exactly as typed, means the same customer.

Two panels. Left, false merge: two people become one. Garden-centre record G-001688, GREEN, SAMUEL, and record G-002894, GREEN, NATHAN, have the same email thegreens@example.com, the same phone 9055551835 and the same address on Kestrel Avenue in Heronvale. The matcher compares the emails as typed and says same: one customer. The truth, in a dashed box: two brothers who share a family email and a landline; each one can now see the other's orders and address. Right, missed match: one person stays two. Loyalty record L-000482, Thibault Helene, with the email thibault.helene93@example.net, and shop record S-028040, Hélène Thibault, with the email Thibault.Helene93@Example.net. The emails differ in capitals, so the matcher says different: two customers. The truth: one woman, with her names swapped in one list and two postcodes because she moved; she gets every email twice, and her points are split over two accounts.
A false merge and a missed match in the synthetic records, both made by the email join. The brothers are a practice problem that we made on purpose.
Text description of the diagram

The picture has two panels. On the left, two garden-centre records: G-001688 is GREEN, SAMUEL, and G-002894 is GREEN, NATHAN. They have the same email, thegreens@example.com, the same phone and the same address on Kestrel Avenue. Only Nathan’s record has a date of birth. The email join says “same”, so they become one customer. The truth, in a dashed box, is that they are two brothers who share a family email and a landline. On the right, a loyalty record, L-000482, says “Thibault Helene”, with the email thibault.helene93@example.net. A shop record, S-028040, says “Hélène Thibault”, with Thibault.Helene93@Example.net. The emails differ only in capitals, so the email join says “different”. The truth is that they are one woman. Her names are swapped in the loyalty list, and the two postcodes differ because she moved.

The brothers are a practice problem that we made on purpose: the course data has 80 pairs of brothers like them. Hélène is an ordinary customer of the data, with ordinary mistakes.

Take only these four records: Samuel (G-001688), Nathan (G-002894), Hélène in the loyalty list (L-000482) and Hélène in the shop (S-028040). Four records make six pairs. The email join decides each pair:

Pair Same email as typed? The matcher says The truth Result
Samuel – Nathan yes same different false merge
Hélène (L) – Hélène (S) no, other capitals different same missed match
Samuel – Hélène (L) no different different right separation
Samuel – Hélène (S) no different different right separation
Nathan – Hélène (L) no different different right separation
Nathan – Hélène (S) no different different right separation

So the matcher made one false merge, one missed match, four right separations and no right merge. Four of six decisions are right. That sounds acceptable, but it did nothing useful. Notice where the right answers came from: most pairs of records are different people, so “different” is easy to get right.

You know two measures from the machine learning course. Here they count pairs of records:

  • Pair precision is the share of the pairs called “same” that really are the same. It answers: “When the matcher merges, how often is it right?” False merges lower it.
  • Pair recall is the share of the true pairs that the matcher finds. It answers: “Of all the pairs that should be merged, how many did it merge?” Missed matches lower it.

In the tiny example, the matcher called one pair “same”, and it was wrong: precision 0 of 1. There is one true pair, and the matcher missed it: recall 0 of 1.

Now the same email join on the course’s test split, the 12,041 records that the course keeps for measuring. They hold 5,564 true pairs. The matcher calls 2,300 pairs “same”:

  • 1,879 of them are right, and 421 are false merges. Precision = 1,879 / 2,300 = 0.817.
  • It finds 1,879 of the 5,564 true pairs, so it makes 3,685 missed matches. Recall = 1,879 / 5,564 = 0.338.

So about one merge in five is wrong, and two of every three true pairs are missed. Precision and recall pull against each other. A strict matcher merges little: few false merges, many missed matches. A loose matcher merges a lot: the opposite. Measure a matcher in Module 2 builds these measures in the project.

Precision and recall count errors. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. wants to know what they cost. The two errors hurt different people.

When Samuel and Nathan become one customer, Larkfield shows one account page with both brothers’ orders and addresses. Samuel signs in and sees what his brother bought and where it went. Larkfield may send Nathan’s parcel to Samuel’s address. Each brother’s emails go to the other one too.

This is a privacy problem, not only an annoyance. One person sees another person’s personal data. Support must find the error, split the account, apologise and give a credit. The data team may need to report the incident. The course uses this assumption: a false merge costs 150 dollars.

When Hélène stays two customers, she gets every marketing email twice and two catalogues. Her loyalty points are split over two accounts. Larkfield’s customer count is a little too high. This costs a little, and it costs it every year. The course’s assumption: a missed match costs 3 dollars a year.

Both numbers are assumptions of the course, written in the data file costs.json, not measured prices. Your business has its own numbers. But the ratio matters: here, one false merge costs as much as 50 years of one missed match.

TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. puts a price on four simple matchers, measured on the test split, and on doing nothing at all. Three of them are rules that compare values after cleaning them, for example without capitals and accents. You build them in Module 2; here, only their results matter:

Matcher What it calls “same” Precision Recall False merges Missed matches Cost in the first year
Nothing no pair – 0 0 5,564 16,692 dollars
Strict same name and same date of birth 1.000 0.170 0 4,618 13,854 dollars
Middle same name and same postcode 0.921 0.621 296 2,106 50,718 dollars
Email same email as typed 0.817 0.338 421 3,685 74,205 dollars
Loose four rules: same email, or same phone and surname, or same name and date of birth, or same name and postcode 0.723 0.870 1,853 724 280,122 dollars

The cost is false merges × 150 dollars + missed matches × 3 dollars. For the email join: 421 × 150 = 63,150, and 3,685 × 3 = 11,055, so 74,205 dollars.

The cost of each matcher on the test split in the first year: false merges at 150 dollars and missed matches at 3 dollars a year (both assumptions). The loose matcher finds the most true pairs and costs the most.
Text description of the chart

A column chart with five columns, in thousands of dollars, on an axis from 0 to 300. Doing nothing costs 16.7 thousand dollars: only missed matches. The strict rule, same name and same date of birth, costs 13.9 thousand. The middle rule, same name and same postcode, costs 50.7 thousand. The email join costs 74.2 thousand. The loose matcher, four rules together, costs 280.1 thousand dollars, more than three times the email join, although it finds the most true pairs.

Read the table from the bottom. The loose matcher has the best recall: it finds 87% of the true pairs. But its 1,853 false merges cost 277,950 dollars, almost all of its cost. The strict rule finds only 17% of the true pairs, and it is the cheapest: it never merged two different people here.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. is surprised: “So the best matcher is the one that finds the fewest pairs?” Priya answers: “With these costs, a false merge is expensive, so precision matters most. But the strict rule still leaves 4,618 true pairs apart. We want a matcher with the precision of the strict rule and much more recall. That is what the rest of the course builds.”

The table is one example, but the pattern is common. Most teams accept more missed matches than false merges, for four reasons:

  • A false merge harms a person. Someone sees a stranger’s data, or gets their parcel. A missed match only wastes some money.
  • A false merge spreads. Once two records are one, other systems use the merged record. Emails go out, orders ship and reports count it. Each of these must be fixed.
  • A missed match can wait. A later run, with a better matcher, can still find the pair. Nothing was broken in the meantime.
  • The law cares about personal data. Showing one person’s data to another can be a reportable incident. The AI Security, Privacy and Responsible Design course covers this in Impact and error costs.

So matching teams often add a third answer: unsure. The matcher merges only the pairs that it is very sure about. It leaves apart the pairs that it is sure are different. A person checks the pairs in between. At Larkfield, that person is Ivana, the customer-data steward. The course assumes that she can check 300 pairs a day, so the unsure pairs must stay few. Modules 5 and 8 build this review step.

The same customer list serves several teams, and they do not weigh the errors in the same way:

Use A false merge means A missed match means Usually worse
Marketing emails one person gets another’s name in an email, or one sibling stops getting emails the same email twice, two catalogues the false merge, but both are mild
Fraud checks an honest customer is linked to a fraud ring and blocked a ring of accounts stays hidden the missed match, if a person checks each alert before acting
Legal requests, for example “delete my data” or “send me my data” you send or delete another person’s data you keep data that the person asked you to delete the false merge; but a missed match also breaks the request

For fraud, the team wants high recall: a missed link hides the ring. That is safe only because a person checks each alert before an account is blocked. For legal requests, both errors can break privacy rules, so the team uses its strictest matcher and asks a person about every unsure pair. There is no single best matcher. There is a best matcher for a use and its costs.

Common mistake

Judge a matcher by its accuracy

Symptom: Tomás reports a new matcher: “It is right on 99.998% of all pairs.” Then Omar finds that it merged nobody at all.

Cause: almost every pair of records is two different people. The course data has 1,806,455,778 possible pairs and only 27,677 true pairs. A matcher that says “different” for every pair is right on 99.998% of them, and it is useless. Accuracy, the share of right decisions, hides both errors behind the easy “different” pairs. This is the class imbalance that you know from the machine learning course, in an extreme form.

Fix: never report accuracy for matching. Report precision and recall, with the counts behind them: pairs called “same”, false merges and missed matches. Then multiply the two errors by their costs for the use.

Consequence in AI work: a measure must count the errors that matter. With rare true pairs, accuracy rewards a matcher for doing nothing.

Check your understanding

For each case, say whether it is a false merge, a missed match or a right decision. (a) Two records of Valérie Tremblay are joined. (b) Marc Fortin and Marie Fortin are joined. (c) Hélène Thibault's two records stay apart.

Show the answer

(a) A right merge: one person, joined. (b) A false merge: two different people, joined, so precision falls and each one may see the other’s data. (c) A missed match: one person, kept apart, so recall falls and she gets everything twice.

Check your understanding

For marketing emails, fraud checks and a request to delete a customer's data, which error is worse, and why?

Show the answer
  • Marketing emails: usually the false merge, because one person may get another’s name or stop getting emails. A missed match only sends the same email twice. Both are mild.
  • Fraud checks: the missed match, because a ring of linked accounts stays hidden. A false merge is acceptable only because a person checks each alert before an account is blocked.
  • A deletion request: the false merge, because you would delete another person’s data, or send it to the wrong person. But a missed match also fails the request, because some of the person’s data stays. So use the strictest matcher, and ask a person about every unsure pair.

Check your understanding

A matcher calls 1,000 pairs “same” on the test split: 900 are right. The test split has 5,564 true pairs. Compute its precision, its recall and its first-year cost with the course's costs. Is it better than the email join?

Show the answer

Precision = 900 / 1,000 = 0.900. Recall = 900 / 5,564 = 0.162. It makes 100 false merges and 5,564 − 900 = 4,664 missed matches. Cost = 100 × 150 + 4,664 × 3 = 15,000 + 13,992 = 28,992 dollars. The email join costs 74,205 dollars, so this matcher is cheaper with these costs, although its recall is lower. It is still worse than the strict rule (13,854 dollars), which has no false merges. With other costs, the order could change.

Recap

  • A false merge joins two different entities. A missed match keeps one entity apart. Pair precision falls with false merges; pair recall falls with missed matches.
  • The email join on the test split has precision 0.817: 421 of its 2,300 merges are false. Its recall is 0.338: it misses 3,685 of 5,564 true pairs.
  • The course’s assumed costs: 150 dollars for a false merge and 3 dollars a year for a missed match. With them, the loose matcher costs 280,122 dollars in the first year, and the strict rule costs 13,854.
  • Most teams accept more missed matches, because a false merge harms a person, spreads to other systems and can be a privacy incident. Unsure pairs go to a person.
  • Which error is worse depends on the use: marketing, fraud and legal requests weigh them differently. Never judge a matcher by accuracy.

You can now name the two errors, measure them and price them. But so far, you have only read results that someone else ran. Next, you set up the project, open the three lists yourself, and run the email join on your own computer.

Next: Meet the records