Skip to content
Français

Find duplicate people and companies

  • Case study 3
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, our people were Larkfield's synthetic customers. In this case study, we find duplicate people in a list of simulated people, then duplicate companies in a real, open register, and build one record for each.

What you will be able to do

Find the duplicates inside one list, from the raw file to a decision. Do it two times: first in a list of simulated people, then in a real register of companies. You will compare rules, Splink without labels and two classifiers under one measure. You will turn pairs into entities, build golden records, and choose a design with its thresholds and its review queue.

Case studyPortfolio piece

Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Normalise names and legal forms
  • Blocking and its limits
  • Comparison vectors
  • Splink, logistic regression and LightGBM
  • Clusters and golden records
  • Incomplete labels
  • A review queue and its cost

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. Its setup cells install the course’s packages and download about 2 MB of data. It needs no account and no key. After the install, it runs in about a minute and a half.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • A virtual environment with the course’s packages from Module 6 and networkx 3.7 from Module 7. Add Splink 5.0.0, DuckDB 2.0.0 and pyarrow 26.0.1 from Module 5. The environment of the larkfield-match project has all of them. This case study does not change the project’s files.
  • The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell makes a new folder duplicates next to it. To start again, delete that folder.

Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Windows and Linux are not tested.

Think of a school with a list of its old students. Over twenty years, one student signs up three times: once as “Robert”, once as “Rob”, and once with a typing error in the surname. The school sends him three letters. Or worse, it joins him with another Robert, and a stranger gets his details.

Every list of people or companies has this problem. In this case study, two people meet it. (Nadia, Kofi and the bank are invented; the data is not.)

  • Nadia is a data scientist. She must remove the duplicates from a list of people. A real list of people cannot go into a public course, so she starts with a list of simulated people.
  • Kofi runs the reference data of a small bank. When the bank reports its trades, it names each client company by its LEI. One company must have one LEI. Kofi wants to find the companies that have two.

Each error has a price. A false merge joins two different people or companies. A person sees another person’s details, or the bank reports a trade for the wrong company. A missed duplicate splits one person or one company in two. Both teams set the same rule: merge without a person only when at least 99 in 100 such merges are right. A person checks the unsure pairs. (The rule and the reviewer’s time are our assumptions.)

You will build two matchers for one list each: they find duplicates inside one table, not between two tables. Each one sorts every candidate pair into merge, review or no match. Then it joins the pairs into entities and keeps one record for each entity.

The steps come from Modules 2 to 8. One idea is new: incomplete labels, in part 2.

The list is fake_1000 from the Splink team’s datasets repository: 1,000 records of 251 simulated people. A program made the people up. Then it wrote each one 1 to 7 times with errors: typing errors, nicknames, swapped first and last names, changed dates and empty values. No real person is in this data.

Why not another list? FEBRL, a well-known list of simulated people, is under the ANU Open Source Licence (see its project page), which is not on the course’s list of licences. A list of real people would need the care of Privacy and matching.

Column What it means Use
unique_id The record’s number Key
first_name, surname The name, as written Normalised, compared
dob The date of birth Compared as text and as a number of days
city, email The city and the email (often empty) Compared
cluster The true person Label only, never an input
repeated IDs: 0
empty values: {'first_name': 169, 'surname': 181, 'dob': 0, 'city': 187, 'email': 211}
people: 251 | records per person: {1: 43, 2: 30, 3: 37, 4: 34, 5: 33, 6: 33, 7: 41}

What this tells us. About one value in five is empty, except the date of birth. Most people have several records: 208 of the 251 people have two or more.

Records 0 to 3 of one simulated person. Record 0: Robert Alan, 1971-06-24, no city, robert255@smith.net. Record 1: Robert Allen, 1971-05-24, no city, roberta25@smith.net. Record 2: Rob Allen, 1971-06-24, London, roberta25@smith.net. Record 3: Robert Alen, 1971-06-24, Lonon, no email. The golden record: robert, allen, 1971-06-24, london, roberta25@smith.net.
One simulated person written four ways. The golden record takes the most common value of each field.
Text description of the diagram

A table with five columns (first name, surname, date of birth, city, email) and four rows, one per record. Empty values are grey. An arrow points down to one green row: the golden record.

  • The unit is a pair of records in the same list. The target is “same person or not”, from the cluster column.
  • The split puts each person in train (50%), validation (25%) or test (25%). A pair belongs to test if one of its records is in test. So a wrong link from a test person to anybody counts as an error. The test split has 498 true pairs. The shares and the seed (39) are our choice.
  • The measures. Pair precision, recall and F1 on test, with every threshold chosen on validation. For the entities, B-cubed and exact-cluster precision and recall, from Cluster measures.

1,000 records make 499,500 pairs. A laptop compares them all in about 3 seconds. But first we try four blocking keys from Module 3: the date of birth, the email, two letters of each name, and three letters of the surname.

candidates true kept pair completeness
date of birth 873 856 0.421
email 682 682 0.336
2 letters of first + last name 691 538 0.265
first 3 letters of last name 3015 818 0.403
all four together 3875 1601 0.788

What this tells us. The four keys together keep only 79% of the true pairs; on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s records, five keys kept 99.4%. So we compare all 499,500 pairs. With a million records, you could not. You would need looser keys, as in Why all pairs is impossible.

Each pair gets a comparison vector of seven similarities, as in Compare by field, then combine:

  • Jaro-Winkler on the first names, the surnames and the cities;
  • Jaro-Winkler of each first name against the other surname, for swapped names;
  • two for the date: a similarity of the text, and the number of days between the dates;
  • Jaro-Winkler on the emails.

A missing value stays missing. The five methods:

  • Rules: the same first name, surname and date of birth; or the same email.
  • Splink without labels, with comparisons like those of Splink’s own tutorials.
  • Logistic regression and LightGBM on the comparison vector, trained on the 125,250 train pairs (1,045 true).
precision recall f1 called right
exact name + date of birth 1.000 0.050 0.096 25.0 25.0
rule: same email 1.000 0.325 0.491 162.0 162.0
Splink, no labels 0.958 0.867 0.910 451.0 432.0
logistic regression 0.899 0.861 0.880 477.0 429.0
LightGBM 0.979 0.934 0.956 475.0 465.0
F1 of five methods on the same 498 test pairs of simulated people. LightGBM wins, and Splink without labels beats logistic regression.
Text description of the chart

Five dots on a scale from 0 to 1. Exact name and date of birth 0.096; same email 0.491; Splink without labels 0.91; logistic regression 0.88; LightGBM 0.956.

What this tells us. The exact rule finds 25 of 498 pairs. LightGBM reaches F1 0.956, and Splink without any label (0.910) beats logistic regression with labels (0.880). On Larkfield’s customers, it was the other way (logistic regression 0.911, Splink 0.797), because families share an address, a phone and a surname. These simulated people have no families.

LightGBM puts 54% of its gain on the email similarity and 29% on the days between the dates. The names count less: the errors are mostly in the names.

In the notebook, you train LightGBM without the swapped-name field. That field has only 1.4% of the gain, but the F1 falls from 0.956 to 0.936. A small field can decide the few hard pairs.

As in Human review, we cut LightGBM’s probability into three bands. The lower threshold is 0.05. On validation, we tried four upper thresholds (0.5, 0.8, 0.95 and 0.99). Only 0.99 meets the 99% rule. On test, used once:

accepted accepted right precision accepted to review true in review recall if reviews are right
validation 452.0 449.0 0.993 72.0 35.0 0.992
test 440.0 440.0 1.000 84.0 48.0 0.980

What this tells us. On test, all 440 merges are right. 48 of the 84 review pairs are true. If the person decides correctly, the matcher finds 98% of the true pairs.

Connected components, from Clustering pairs, join every chain of links into one cluster. With LightGBM’s links, the test people get B-cubed precision 0.942 and recall 0.986. The matcher finds 64 clusters for 63 true people. The largest cluster has 10 records, so it joins two people. A few wrong links chain many records together, so B-cubed precision is lower than pair precision (0.979).

Common mistake

A rule that worked on Larkfield

Symptom: Nadia reuses Module 7’s safe-clustering check: “two different dates of birth cannot be one person”. B-cubed recall falls from 0.986 to 0.568. The 63 test people break into 146 clusters.

Cause: in this list, 58% of the true pairs have two different dates of birth. 178 of the 208 people with several records have two or more dates. The program that made the records also makes errors in dates. Larkfield’s generator changed few dates. It swapped the day and the month in a few records, which the check allows. It swapped two digits of the year in about 1 record in 100.

Fix: keep the date as evidence in the score, not as a hard rule. The comparison vector has the number of days between the two dates, so a small difference still counts as evidence.

Consequence in AI work: a rule is a claim about the data. Before you reuse it on new data, measure how often it is true there, as in Safer clustering.

Golden records, and the design on new people

Section titled “Golden records, and the design on new people”

We use one merge rule per field, as in The golden record. It keeps the most common non-empty value; a tie goes to the oldest record. The four records of the first person become robert allen, 1971-06-24, london, roberta25@smith.net.

The notebook’s find_duplicate_people() runs every step on a new list. We run it on the 253 records of the test people only, as if they were new:

253 records of new people, 0.4 s -> 65 golden records: {'bcubed precision': 0.981, 'bcubed recall': 0.986,
'exact precision': 0.892, 'exact recall': 0.921, 'entities found': 65, 'true entities': 63, 'largest': 10}

What this tells us. 65 golden records for 63 people, with B-cubed precision and recall near 0.98. The design works on records it has never seen.

A library gives each book one call number, so that two copies of one book stand together. The LEI (legal entity identifier) does this for legal entities: one 20-character code for one company, fund or public body. Banks use it to report trades. GLEIF, the Global Legal Entity Identifier Foundation, runs the system. Local issuers give out the codes.

Sometimes one company gets two LEIs, for example from two issuers. When GLEIF or an issuer finds it, the second LEI gets the status DUPLICATE, with the LEI that stays. Kofi’s question: from names, legal forms and addresses, can a matcher find those duplicates?

The data is real: every LEI record with a legal address in Ireland, 38,650 records, from GLEIF’s public API. Ireland has many companies and many investment funds. 180 records are marked DUPLICATE. Records joined by those links are one company: 173 companies have two or more LEIs, with 191 true pairs.

Column What it means Use
legal_name, other_names The legal name; other names, often a previous name Normalised, compared
legal_form A code for the legal form, such as “Designated Activity Company” Compared
category GENERAL, FUND, BRANCH … Compared
address, city, postcode The legal address Compared
registration_authority, registration_number The register and the number in it Compared
managing_lou, first_registered The LEI issuer, and the year of the first registration Compared
registration_status, duplicate_of DUPLICATE and the LEI that stays Labels only, never inputs
records that GLEIF marks DUPLICATE: 180 | their surviving LEI is in the file: 180
categories: {'GENERAL': 24068, 'FUND': 14538, 'BRANCH': 39, 'RESIDENT_GOVERNMENT_ENTITY': 5}
empty values: {'legal_form': 0, 'address': 0, 'postcode': 6444, 'registration_number': 4902, 'other_names': 32325}
most common postcode: D02 Y512 with 2181 records

What this tells us. 38% of the records are funds. Fund administrators register thousands of funds at one office: 2,181 records share one postcode. So the address says little. An umbrella fund also writes “UMBRELLA PLC - SUB-FUND”, and all its sub-funds share the umbrella’s registration number.

Left: one company with two LEIs. Compass COSS Master ICAV, Ten Earlsfort Terrace, postcode D02 T380, number C567887, marked DUPLICATE, and COMPASS COSS MASTER ICAV, TEN EARLSFORT TERRACE, DUBLIN 2, D02 T380, C567887. Same name, address, postcode and number. Right: three funds named GLOBAL EQUITY FUND, at three addresses (Townsend Street D02 R156, Charlemont Place D02 F985, Sir John Rogerson's Quay D02 RK57) with three registration numbers (C70223, C129688, C92746). Same name, other address and number.
The same name can be one company with two LEIs, or three different funds.
Text description of the diagram

Two columns. On the left, two record cards of one company; the green box says that the matcher should find this pair. On the right, three record cards with the same name and different addresses and numbers; the red box says that a rule on the name alone merges them.

As in Normalise, company() writes the legal form in one short form: “public limited company” and “p.l.c.” become plc, and “limited” becomes ltd. core() then drops the legal-form words, so “AVOCA CLO XLII Designated Activity Company” becomes avoca clo xlii. sub_fund() keeps the part after the last dash.

The exact rules fail badly:

exact name: {'precision': 0.198, 'recall': 0.455, 'f1': 0.276, 'called': 101, 'right': 20, 'true': 44}
rule: same registration number: {'precision': 0.23, 'recall': 0.455, 'f1': 0.305, 'called': 87, 'right': 20, 'true': 44}

What this tells us. Four in five “same name” pairs are not GLEIF duplicates. For people, the exact rule was always right. For companies, many funds share one name, and all sub-funds of one umbrella share one registration number.

38,650 records make about 747 million pairs. We use five blockings together:

  • the same core name, and the same sub-fund name;
  • the same registration number (at most 20 records for one number);
  • an “other name” that is another record’s name;
  • TF-IDF top 10: for each name, the 10 most similar names, with the same idea as in Match products between two shops.
candidates true kept pair completeness reduction ratio
same name 308.0 92.0 0.482 1.00000
same sub-fund name 926.0 114.0 0.597 1.00000
same registration number 193.0 69.0 0.361 1.00000
other name = name 105.0 25.0 0.131 1.00000
TF-IDF top 10 292030.0 160.0 0.838 0.99961
all five together 292389.0 174.0 0.911 0.99961

What this tells us. 292,389 candidates keep 174 of the 191 true pairs. Most of the 17 lost pairs are funds with a new name, for example “OLD MUTUAL … US EQUITY INCOME FUND” and “JUPITER MERIAN US EQUITY INCOME FUND”. One link even joins a private equity company to a concrete company with another registration number. A register can hold errors too.

Each candidate gets 11 similarities. Four compare the names and the other names. The others compare the legal form, the category, the address, the postcode, the registration number and the LEI issuer, and count the years between the two first registrations.

Only 77 true pairs are in the training split. So the LightGBM here is small: 4 leaves, at least 30 pairs in each leaf, and a penalty on large weights. The logistic regression gives the rare true pairs more weight (class_weight="balanced").

precision recall f1 called right true
exact name 0.198 0.455 0.276 101.0 20.0 44.0
rule: same registration number 0.230 0.455 0.305 87.0 20.0 44.0
Splink, no labels 0.476 0.227 0.308 21.0 10.0 44.0
logistic regression 0.204 0.455 0.282 98.0 20.0 44.0
LightGBM (small) 0.404 0.523 0.455 57.0 23.0 44.0
F1 of five methods on the same 44 test pairs of Irish LEI records. Every method is far below the people's results; the small LightGBM is best.
Text description of the chart

Five dots on a scale from 0 to 1. Exact name 0.276; same registration number 0.305; Splink without labels 0.308; logistic regression 0.282; small LightGBM 0.455.

What this tells us. The best F1 is 0.455, against 0.956 for the simulated people. Splink without labels is fairly precise but finds few pairs (recall 0.227). Without labels, it probably cannot learn that a shared name and address are weak evidence among thousands of funds. The small LightGBM puts most of its gain on the registration number, the sub-fund name and the name.

Common mistake

A big model on few labels

Symptom: Kofi first used the people’s LightGBM: 15 leaves and 300 trees. Its F1 on test is 0.103. It calls 266 pairs to find 16.

Cause: with 77 true pairs, a big model learns the training pairs by heart. On validation, its best threshold is 0.997, and it still calls 184 pairs to find 30.

Fix: a small model with fewer leaves, more pairs per leaf and a penalty on large weights. The validation split chose it: F1 0.248 for the big model, 0.699 for the small one.

Consequence in AI work: count your positive labels before you choose a model. With a few dozen, choose a simple model, or label more pairs where the model is unsure, as in Active learning.

A teacher marks an exam with an answer key. If the key has no answer for question 7, a right answer to question 7 counts as wrong. The student’s score is then lower than it should be.

GLEIF’s DUPLICATE links are such a key, with gaps. GLEIF marks only the duplicates that someone found. So a pair that the matcher calls “same”, and GLEIF does not, is not always a mistake. We looked at the 34 test pairs that LightGBM called “same” against GLEIF’s links:

{'wrong matches': 34, 'same name and registration number': 8, 'same name, one number missing': 15}

What this tells us. 8 of the 34 “wrong” matches have the same name and the same registration number. For example, “BLACKROCK EUROPEAN CLO XVI DESIGNATED ACTIVITY COMPANY” has two LEIs with the number 784613 and one address. These are probably one company with two LEIs that nobody has marked yet. One of the 8 is the private equity company again: its real twin has the same name and number. 15 more have the same name and one number missing. They may be one fund, or two sub-funds of two umbrellas.

“Same name and same number” is our rule, not GLEIF’s, and we did not check the pairs one by one. So the true precision is probably higher than 0.404, but we do not know by how much.

When labels are incomplete, do not trust the measured precision alone. Read the top “false” matches by hand. Send the real duplicates back to the register: that helps everybody.

The test split has only 44 true pairs, so one pair changes the F1 by about 0.02. The notebook repeats the split with ten other seeds, on the same candidates.

With ten different splits, the small LightGBM's F1 moves from about 0.36 to 0.57, and the exact name's from 0.18 to 0.36. LightGBM wins on every split.
Text description of the chart

Two rows of ten dots on a scale from 0 to 1. Exact name: 0.286, 0.22, 0.182, 0.364, 0.283, 0.306, 0.254, 0.336, 0.303, 0.24. Small LightGBM: 0.505, 0.508, 0.364, 0.571, 0.563, 0.526, 0.5, 0.52, 0.539, 0.505. The test split has 31 to 59 true pairs.

What this tells us. The F1 moves by about 0.2 between splits. So read these results to one decimal, not three. The order of the two methods does not change.

On validation, no upper threshold reaches the 99% rule:

accepted accepted right precision accepted to review true in review recall if reviews are right
0.50 38.0 33.0 0.868 137.0 14.0 0.81
0.70 33.0 29.0 0.879 142.0 18.0 0.81
0.90 18.0 16.0 0.889 157.0 31.0 0.81
0.95 7.0 6.0 0.857 168.0 41.0 0.81

So no pair merges without a person. Every pair with a score of 0.05 or more goes to a reviewer, the highest score first. On test:

to review GLEIF duplicates probably the same, not marked (our rule) recall if reviews are right
validation 175.0 47.0 3.0 0.810
test 255.0 30.0 8.0 0.682

What this tells us. The order of the 255 test pairs matters: in the notebook’s task, the first 30 pairs hold 18 duplicates. After that, the reviewer finds few. Kofi can stop when the yield is too low for an hour of work.

One LEI per company, and the design on new records

Section titled “One LEI per company, and the design on new records”

For each company with two LEIs, the merge rule keeps one LEI (our choice): the best registration status (ISSUED first), then the oldest. The status is used only here, after matching. The reviewed pairs join 133 companies. For 132 of them, the kept LEI is GLEIF’s own surviving LEI.

The cluster measures say little here. Almost every company has one LEI, so B-cubed is near 1.0 whatever the matcher does. Count the merges instead.

The notebook’s find_duplicate_companies() runs every step on a new list. On the 9,661 records of the test companies only, in 2 seconds:

count sum
decision
review 47 9
review first 24 21

What this tells us. 24 pairs score 0.5 or more (“review first”), and 21 of them are GLEIF duplicates. Below 0.5, only 9 of 47 are. Together, the queue finds 30 of the 44 true pairs.

Two designs with test counts. People (simulated): all pairs, 7 similarities, LightGBM, links, clusters, golden records. Score 0.99 or more: merge, 440 pairs, 440 right. 0.05 to 0.99: a person reviews, 84 pairs, 48 true. Below 0.05: no match. Companies (GLEIF, Ireland): 5 blockings, 11 similarities, small LightGBM, review queue, one LEI. No score is safe enough to merge without a person. Score 0.05 or more: a person reviews, highest score first, 255 pairs, 30 GLEIF duplicates and 8 more probably the same. First 30 reviews: 18 duplicates.
The same steps, two decisions: the people's matcher merges most pairs alone; the companies' matcher only sorts a queue for a person.
Text description of the diagram

Two columns. Each has a header, a box with its steps, and three rows of bands with their counts on the test split.

The reviewer’s time is an assumption from the course: 1.2 minutes a pair at US$40 an hour. The computer is almost free: the whole notebook runs in about 80 seconds on a laptop.

Pairs to review on test Reviewer time Per 1,000 records (rough)
People 84, for 253 test records 1.7 hours, about US$67 about US$265
Companies 255, for 9,661 test records 5.1 hours, about US$204 about US$21

What this tells us. A noisy list of people costs much more review per record. The company register costs little per record, because most companies have one LEI. But only about one review pair in eight is a GLEIF duplicate, so most of Kofi’s cost buys a “no”.

Nadia writes, for a real list of people:

  • Use the people’s design. Merge at 0.99 or more; review from 0.05 to 0.99. First train it again on labels from the real list: simulated errors are not real errors.
  • Keep the date of birth as evidence, not as a hard rule, until you know how often dates differ in the real list.

Kofi writes, for the bank:

  • Never merge LEIs automatically. No threshold met the 99% rule on validation.
  • Review the queue in order of score, and stop when the yield is too low for the reviewer’s time.
  • Map the bank’s clients to the surviving LEI.
  • Send likely duplicates to the LEI issuer. The register’s labels are incomplete, and a correction helps everyone.

What not to do:

  • Do not merge on the name alone. Four in five same-name pairs were different entities.
  • Do not reuse a threshold, a rule or a model size from another list without a new validation. The people’s model failed on the companies.
  • Simulated people are harder in one way, easier in another. Their errors are heavy, but they have no families. Neither they nor Larkfield tell you how a real list behaves.
  • The companies’ labels are incomplete, and some are wrong. The measured precision is a lower bound.
  • One country, one day. Another country, or the same register next month, gives other numbers.
  • Small test sets. Another split moves the companies’ F1 by up to 0.2.
  • No language model and no embeddings here. The names are short, and Module 6 showed that embeddings did not help on names and addresses.
  • The prices are assumptions. Check the real costs of your own team.
  • Credit the sources. The MIT licence asks you to keep its copyright notice with the people’s file. CC0 asks for nothing, but say where the LEI data came from.
  • No private persons. The people are simulated. We left the sole proprietors out of the LEI data, because a sole proprietor is a natural person.
  • The companies are real. A DUPLICATE status is GLEIF’s data. “Probably the same” is our rule. Neither is a claim about any company. Report a suspected duplicate to the LEI issuer; do not publish it as a fact.
  • A false merge of people is a privacy breach. On a real list, keep the source records, log every merge and make every merge possible to undo, as in The golden record.

This case study shows that you can take a real register, with gaps in its labels, to a decision. Make it yours with one change, such as one from Go further below.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • The two comparison tables as CSV files: every method, its threshold, precision, recall and F1.
  • DECISION.md: the thresholds, the review queue, its size and its cost, and the assumptions.
  • The data sources and licences: the people’s file with its MIT notice, and the LEI extract with its filter, its date and its checksum.

A README template:

# Finding duplicate people and companies
## Problem
One list of people and one register of companies hold duplicates.
Merge without a person only when 99 in 100 merges are right; a person checks the rest.
## Data and licence
People: Splink's fake_1000 (MIT), 1,000 simulated records of 251 people.
Companies: GLEIF LEI records with a legal address in Ireland (CC0), 38,650 records;
labels: GLEIF's 180 DUPLICATE links (191 true pairs).
## Approach
Normalised names and legal forms; all pairs for people, five blockings for companies
(pair completeness 0.911); comparison vectors; rules, Splink without labels,
logistic regression and LightGBM compared; clusters and golden records.
## Results (test split)
People: LightGBM F1 0.956; merge >= 0.99: 440 pairs, all right.
Companies: small LightGBM F1 0.455 (exact name 0.276); no safe merge threshold;
review queue of 255 pairs, 18 duplicates in the first 30.
## Limits
Simulated people; incomplete register labels; one country; small test sets.

How to describe it in an interview, in two sentences: “I built two duplicate finders with the same steps: on simulated people, LightGBM reached F1 0.96 and merged 440 test pairs with no error. On real LEI records, the best F1 was 0.46 because thousands of funds share names and numbers; many ‘false’ matches were probably duplicates that the register had not marked, so I designed a review queue instead of automatic merges.”

Go further:

  • Add the umbrella name (the part before the dash) as a field, and measure again.
  • Review the 34 “wrong” matches by hand, label them, and measure the precision with your labels.
  • Download another country’s LEI records with the same API filter, and run the same design without new training.

Check your understanding

The exact rule “same first name, surname and date of birth” had a precision of 1.0 on people, but “same name” had a precision of only 0.198 on companies. Explain the difference, with an example from the GLEIF records.

Show the answer

For people, three equal fields together are rare by chance: two different simulated people almost never share all three. For companies, one field (the name) is often shared by different entities. Many funds have generic names, such as three different “GLOBAL EQUITY FUND” records with three registration numbers and three addresses. Sub-funds with the same name also exist in different umbrellas. So the name alone joins different funds, and the matcher needs the registration number, the sub-fund name and other fields as well.

Check your understanding

LightGBM called 34 test pairs of companies “same” that GLEIF does not link. Kofi says: “So 34 of our matches are wrong.” What do you answer?

Show the answer

Not necessarily. GLEIF’s labels are incomplete: it marks only the duplicates that someone found. 8 of the 34 pairs have the same name and the same registration number, so they are probably one entity with two LEIs. 15 more have the same name with one number missing. So the measured precision is a lower bound. Check the top pairs by hand, and report the real duplicates to the LEI issuer. Say clearly which labels are GLEIF’s and which are yours.

Check your understanding

Module 7's check “two different dates of birth cannot be one person” improved Larkfield's clusters. Why did it break the simulated people into pieces, and what should Nadia do instead?

Show the answer

In the simulated list, 58% of the true pairs have two different dates of birth, because the program also makes errors in dates. The check cut those true links, so B-cubed recall fell from 0.986 to 0.568. Larkfield’s generator changed few dates: mostly a swapped day and month, which the check allows. Nadia should keep the date as evidence in the score (for example, the number of days between the dates). Before she uses any rule as a hard rule, she should measure how often it is true on the new data.

Recap

  • On simulated people, four blocking keys kept only 79% of the true pairs, so we compared all 499,500 pairs.
  • LightGBM (F1 0.956) beat Splink without labels (0.910) and logistic regression (0.880). Merges at 0.99 or more were all right on test.
  • A Larkfield rule (two dates of birth cannot be one person) broke the people into pieces: measure a rule on new data first.
  • On real LEI records, four in five same-name pairs were different entities. A small LightGBM won (F1 0.455), because there were only 77 true training pairs.
  • GLEIF’s labels are incomplete: the measured precision is a lower bound. The design is a review queue sorted by score, with no automatic merge.

Next, the final assignment: merge Larkfield’s customers with a second company’s list, with everything from the course.

Next: Final assignment