Find duplicate people and companies
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In eight modules, our people were Larkfield's synthetic customers. In this case study, we find duplicate people in a list of simulated people, then duplicate companies in a real, open register, and build one record for each.
Case study 3 of 3 · Entity Resolution and Record Matching: Is This the Same Thing?Find duplicate people and companies
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Find the duplicates inside one list, from the raw file to a decision. Do it two times: first in a list of simulated people, then in a real register of companies. You will compare rules, Splink without labels and two classifiers under one measure. You will turn pairs into entities, build golden records, and choose a design with its thresholds and its review queue.
The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. Its setup cells install the course’s packages and download about 2 MB of data. It needs no account and no key. After the install, it runs in about a minute and a half.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- Normalising names, from Normalise.
- Pair precision, recall and F1, from Measure a matcher.
- Pair completeness, from What blocking loses.
- The comparison vector, from Compare by field, then combine.
- Splink without labels, from Learn the weights without labels.
- The pair classifier, from A supervised pair classifier.
- Clusters and golden records, from Clustering pairs, Cluster measures and The golden record.
- The review band, from Human review.
What you need on your computer (only for the local route; the notebook needs none of it)
- A virtual environment with the course’s packages from Module 6 and networkx 3.7 from Module 7. Add Splink 5.0.0, DuckDB 2.0.0 and pyarrow 26.0.1 from Module 5. The environment of the
larkfield-matchproject has all of them. This case study does not change the project’s files. - The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell makes a new folder
duplicatesnext to it. To start again, delete that folder.
Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Windows and Linux are not tested.
The situation
Section titled “The situation”Think of a school with a list of its old students. Over twenty years, one student signs up three times: once as “Robert”, once as “Rob”, and once with a typing error in the surname. The school sends him three letters. Or worse, it joins him with another Robert, and a stranger gets his details.
Every list of people or companies has this problem. In this case study, two people meet it. (Nadia, Kofi and the bank are invented; the data is not.)
- Nadia is a data scientist. She must remove the duplicates from a list of people. A real list of people cannot go into a public course, so she starts with a list of simulated people.
- Kofi runs the reference data of a small bank. When the bank reports its trades, it names each client company by its LEI. One company must have one LEI. Kofi wants to find the companies that have two.
Each error has a price. A false merge joins two different people or companies. A person sees another person’s details, or the bank reports a trade for the wrong company. A missed duplicate splits one person or one company in two. Both teams set the same rule: merge without a person only when at least 99 in 100 such merges are right. A person checks the unsure pairs. (The rule and the reviewer’s time are our assumptions.)
What you will build
Section titled “What you will build”You will build two matchers for one list each: they find duplicates inside one table, not between two tables. Each one sorts every candidate pair into merge, review or no match. Then it joins the pairs into entities and keeps one record for each entity.
The steps come from Modules 2 to 8. One idea is new: incomplete labels, in part 2.
Part 1: simulated people
Section titled “Part 1: simulated people”The data
Section titled “The data”The list is fake_1000 from the Splink team’s datasets repository: 1,000 records of 251 simulated people. A program made the people up. Then it wrote each one 1 to 7 times with errors: typing errors, nicknames, swapped first and last names, changed dates and empty values. No real person is in this data.
Why not another list? FEBRL, a well-known list of simulated people, is under the ANU Open Source Licence (see its project page), which is not on the course’s list of licences. A list of real people would need the care of Privacy and matching.
| Column | What it means | Use |
|---|---|---|
unique_id |
The record’s number | Key |
first_name, surname |
The name, as written | Normalised, compared |
dob |
The date of birth | Compared as text and as a number of days |
city, email |
The city and the email (often empty) | Compared |
cluster |
The true person | Label only, never an input |
First checks
Section titled “First checks”repeated IDs: 0empty values: {'first_name': 169, 'surname': 181, 'dob': 0, 'city': 187, 'email': 211}people: 251 | records per person: {1: 43, 2: 30, 3: 37, 4: 34, 5: 33, 6: 33, 7: 41}What this tells us. About one value in five is empty, except the date of birth. Most people have several records: 208 of the 251 people have two or more.
Text description of the diagram
A table with five columns (first name, surname, date of birth, city, email) and four rows, one per record. Empty values are grey. An arrow points down to one green row: the golden record.
Frame the problem
Section titled “Frame the problem”- The unit is a pair of records in the same list. The target is “same person or not”, from the
clustercolumn. - The split puts each person in train (50%), validation (25%) or test (25%). A pair belongs to test if one of its records is in test. So a wrong link from a test person to anybody counts as an error. The test split has 498 true pairs. The shares and the seed (39) are our choice.
- The measures. Pair precision, recall and F1 on test, with every threshold chosen on validation. For the entities, B-cubed and exact-cluster precision and recall, from Cluster measures.
Blocking: is it needed here?
Section titled “Blocking: is it needed here?”1,000 records make 499,500 pairs. A laptop compares them all in about 3 seconds. But first we try four blocking keys from Module 3: the date of birth, the email, two letters of each name, and three letters of the surname.
candidates true kept pair completenessdate of birth 873 856 0.421email 682 682 0.3362 letters of first + last name 691 538 0.265first 3 letters of last name 3015 818 0.403all four together 3875 1601 0.788What this tells us. The four keys together keep only 79% of the true pairs; on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s records, five keys kept 99.4%. So we compare all 499,500 pairs. With a million records, you could not. You would need looser keys, as in Why all pairs is impossible.
Five methods on the same pairs
Section titled “Five methods on the same pairs”Each pair gets a comparison vector of seven similarities, as in Compare by field, then combine:
- Jaro-Winkler on the first names, the surnames and the cities;
- Jaro-Winkler of each first name against the other surname, for swapped names;
- two for the date: a similarity of the text, and the number of days between the dates;
- Jaro-Winkler on the emails.
A missing value stays missing. The five methods:
- Rules: the same first name, surname and date of birth; or the same email.
- Splink without labels, with comparisons like those of Splink’s own tutorials.
- Logistic regression and LightGBM on the comparison vector, trained on the 125,250 train pairs (1,045 true).
precision recall f1 called rightexact name + date of birth 1.000 0.050 0.096 25.0 25.0rule: same email 1.000 0.325 0.491 162.0 162.0Splink, no labels 0.958 0.867 0.910 451.0 432.0logistic regression 0.899 0.861 0.880 477.0 429.0LightGBM 0.979 0.934 0.956 475.0 465.0Text description of the chart
Five dots on a scale from 0 to 1. Exact name and date of birth 0.096; same email 0.491; Splink without labels 0.91; logistic regression 0.88; LightGBM 0.956.
What this tells us. The exact rule finds 25 of 498 pairs. LightGBM reaches F1 0.956, and Splink without any label (0.910) beats logistic regression with labels (0.880). On Larkfield’s customers, it was the other way (logistic regression 0.911, Splink 0.797), because families share an address, a phone and a surname. These simulated people have no families.
LightGBM puts 54% of its gain on the email similarity and 29% on the days between the dates. The names count less: the errors are mostly in the names.
In the notebook, you train LightGBM without the swapped-name field. That field has only 1.4% of the gain, but the F1 falls from 0.956 to 0.936. A small field can decide the few hard pairs.
Decide: merge, review or no match
Section titled “Decide: merge, review or no match”As in Human review, we cut LightGBM’s probability into three bands. The lower threshold is 0.05. On validation, we tried four upper thresholds (0.5, 0.8, 0.95 and 0.99). Only 0.99 meets the 99% rule. On test, used once:
accepted accepted right precision accepted to review true in review recall if reviews are rightvalidation 452.0 449.0 0.993 72.0 35.0 0.992test 440.0 440.0 1.000 84.0 48.0 0.980What this tells us. On test, all 440 merges are right. 48 of the 84 review pairs are true. If the person decides correctly, the matcher finds 98% of the true pairs.
From pairs to people
Section titled “From pairs to people”Connected components, from Clustering pairs, join every chain of links into one cluster. With LightGBM’s links, the test people get B-cubed precision 0.942 and recall 0.986. The matcher finds 64 clusters for 63 true people. The largest cluster has 10 records, so it joins two people. A few wrong links chain many records together, so B-cubed precision is lower than pair precision (0.979).
Common mistake
A rule that worked on Larkfield
Symptom: Nadia reuses Module 7’s safe-clustering check: “two different dates of birth cannot be one person”. B-cubed recall falls from 0.986 to 0.568. The 63 test people break into 146 clusters.
Cause: in this list, 58% of the true pairs have two different dates of birth. 178 of the 208 people with several records have two or more dates. The program that made the records also makes errors in dates. Larkfield’s generator changed few dates. It swapped the day and the month in a few records, which the check allows. It swapped two digits of the year in about 1 record in 100.
Fix: keep the date as evidence in the score, not as a hard rule. The comparison vector has the number of days between the two dates, so a small difference still counts as evidence.
Consequence in AI work: a rule is a claim about the data. Before you reuse it on new data, measure how often it is true there, as in Safer clustering.
Golden records, and the design on new people
Section titled “Golden records, and the design on new people”We use one merge rule per field, as in The golden record. It keeps the most common non-empty value; a tie goes to the oldest record. The four records of the first person become robert allen, 1971-06-24, london, roberta25@smith.net.
The notebook’s find_duplicate_people() runs every step on a new list. We run it on the 253 records of the test people only, as if they were new:
253 records of new people, 0.4 s -> 65 golden records: {'bcubed precision': 0.981, 'bcubed recall': 0.986, 'exact precision': 0.892, 'exact recall': 0.921, 'entities found': 65, 'true entities': 63, 'largest': 10}What this tells us. 65 golden records for 63 people, with B-cubed precision and recall near 0.98. The design works on records it has never seen.
Part 2: real companies
Section titled “Part 2: real companies”What an LEI is
Section titled “What an LEI is”A library gives each book one call number, so that two copies of one book stand together. The LEI (legal entity identifier) does this for legal entities: one 20-character code for one company, fund or public body. Banks use it to report trades. GLEIF, the Global Legal Entity Identifier Foundation, runs the system. Local issuers give out the codes.
Sometimes one company gets two LEIs, for example from two issuers. When GLEIF or an issuer finds it, the second LEI gets the status DUPLICATE, with the LEI that stays. Kofi’s question: from names, legal forms and addresses, can a matcher find those duplicates?
The data
Section titled “The data”The data is real: every LEI record with a legal address in Ireland, 38,650 records, from GLEIF’s public API. Ireland has many companies and many investment funds. 180 records are marked DUPLICATE. Records joined by those links are one company: 173 companies have two or more LEIs, with 191 true pairs.
| Column | What it means | Use |
|---|---|---|
legal_name, other_names |
The legal name; other names, often a previous name | Normalised, compared |
legal_form |
A code for the legal form, such as “Designated Activity Company” | Compared |
category |
GENERAL, FUND, BRANCH … | Compared |
address, city, postcode |
The legal address | Compared |
registration_authority, registration_number |
The register and the number in it | Compared |
managing_lou, first_registered |
The LEI issuer, and the year of the first registration | Compared |
registration_status, duplicate_of |
DUPLICATE and the LEI that stays | Labels only, never inputs |
First checks
Section titled “First checks”records that GLEIF marks DUPLICATE: 180 | their surviving LEI is in the file: 180categories: {'GENERAL': 24068, 'FUND': 14538, 'BRANCH': 39, 'RESIDENT_GOVERNMENT_ENTITY': 5}empty values: {'legal_form': 0, 'address': 0, 'postcode': 6444, 'registration_number': 4902, 'other_names': 32325}most common postcode: D02 Y512 with 2181 recordsWhat this tells us. 38% of the records are funds. Fund administrators register thousands of funds at one office: 2,181 records share one postcode. So the address says little. An umbrella fund also writes “UMBRELLA PLC - SUB-FUND”, and all its sub-funds share the umbrella’s registration number.
Text description of the diagram
Two columns. On the left, two record cards of one company; the green box says that the matcher should find this pair. On the right, three record cards with the same name and different addresses and numbers; the red box says that a rule on the name alone merges them.
Normalise company names
Section titled “Normalise company names”As in Normalise, company() writes the legal form in one short form: “public limited company” and “p.l.c.” become plc, and “limited” becomes ltd. core() then drops the legal-form words, so “AVOCA CLO XLII Designated Activity Company” becomes avoca clo xlii. sub_fund() keeps the part after the last dash.
The exact rules fail badly:
exact name: {'precision': 0.198, 'recall': 0.455, 'f1': 0.276, 'called': 101, 'right': 20, 'true': 44}rule: same registration number: {'precision': 0.23, 'recall': 0.455, 'f1': 0.305, 'called': 87, 'right': 20, 'true': 44}What this tells us. Four in five “same name” pairs are not GLEIF duplicates. For people, the exact rule was always right. For companies, many funds share one name, and all sub-funds of one umbrella share one registration number.
Blocking
Section titled “Blocking”38,650 records make about 747 million pairs. We use five blockings together:
- the same core name, and the same sub-fund name;
- the same registration number (at most 20 records for one number);
- an “other name” that is another record’s name;
- TF-IDF top 10: for each name, the 10 most similar names, with the same idea as in Match products between two shops.
candidates true kept pair completeness reduction ratiosame name 308.0 92.0 0.482 1.00000same sub-fund name 926.0 114.0 0.597 1.00000same registration number 193.0 69.0 0.361 1.00000other name = name 105.0 25.0 0.131 1.00000TF-IDF top 10 292030.0 160.0 0.838 0.99961all five together 292389.0 174.0 0.911 0.99961What this tells us. 292,389 candidates keep 174 of the 191 true pairs. Most of the 17 lost pairs are funds with a new name, for example “OLD MUTUAL … US EQUITY INCOME FUND” and “JUPITER MERIAN US EQUITY INCOME FUND”. One link even joins a private equity company to a concrete company with another registration number. A register can hold errors too.
Five methods on the same pairs
Section titled “Five methods on the same pairs”Each candidate gets 11 similarities. Four compare the names and the other names. The others compare the legal form, the category, the address, the postcode, the registration number and the LEI issuer, and count the years between the two first registrations.
Only 77 true pairs are in the training split. So the LightGBM here is small: 4 leaves, at least 30 pairs in each leaf, and a penalty on large weights. The logistic regression gives the rare true pairs more weight (class_weight="balanced").
precision recall f1 called right trueexact name 0.198 0.455 0.276 101.0 20.0 44.0rule: same registration number 0.230 0.455 0.305 87.0 20.0 44.0Splink, no labels 0.476 0.227 0.308 21.0 10.0 44.0logistic regression 0.204 0.455 0.282 98.0 20.0 44.0LightGBM (small) 0.404 0.523 0.455 57.0 23.0 44.0Text description of the chart
Five dots on a scale from 0 to 1. Exact name 0.276; same registration number 0.305; Splink without labels 0.308; logistic regression 0.282; small LightGBM 0.455.
What this tells us. The best F1 is 0.455, against 0.956 for the simulated people. Splink without labels is fairly precise but finds few pairs (recall 0.227). Without labels, it probably cannot learn that a shared name and address are weak evidence among thousands of funds. The small LightGBM puts most of its gain on the registration number, the sub-fund name and the name.
Common mistake
A big model on few labels
Symptom: Kofi first used the people’s LightGBM: 15 leaves and 300 trees. Its F1 on test is 0.103. It calls 266 pairs to find 16.
Cause: with 77 true pairs, a big model learns the training pairs by heart. On validation, its best threshold is 0.997, and it still calls 184 pairs to find 30.
Fix: a small model with fewer leaves, more pairs per leaf and a penalty on large weights. The validation split chose it: F1 0.248 for the big model, 0.699 for the small one.
Consequence in AI work: count your positive labels before you choose a model. With a few dozen, choose a simple model, or label more pairs where the model is unsure, as in Active learning.
Incomplete labels
Section titled “Incomplete labels”A teacher marks an exam with an answer key. If the key has no answer for question 7, a right answer to question 7 counts as wrong. The student’s score is then lower than it should be.
GLEIF’s DUPLICATE links are such a key, with gaps. GLEIF marks only the duplicates that someone found. So a pair that the matcher calls “same”, and GLEIF does not, is not always a mistake. We looked at the 34 test pairs that LightGBM called “same” against GLEIF’s links:
{'wrong matches': 34, 'same name and registration number': 8, 'same name, one number missing': 15}What this tells us. 8 of the 34 “wrong” matches have the same name and the same registration number. For example, “BLACKROCK EUROPEAN CLO XVI DESIGNATED ACTIVITY COMPANY” has two LEIs with the number 784613 and one address. These are probably one company with two LEIs that nobody has marked yet. One of the 8 is the private equity company again: its real twin has the same name and number. 15 more have the same name and one number missing. They may be one fund, or two sub-funds of two umbrellas.
“Same name and same number” is our rule, not GLEIF’s, and we did not check the pairs one by one. So the true precision is probably higher than 0.404, but we do not know by how much.
When labels are incomplete, do not trust the measured precision alone. Read the top “false” matches by hand. Send the real duplicates back to the register: that helps everybody.
How much does one split decide?
Section titled “How much does one split decide?”The test split has only 44 true pairs, so one pair changes the F1 by about 0.02. The notebook repeats the split with ten other seeds, on the same candidates.
Text description of the chart
Two rows of ten dots on a scale from 0 to 1. Exact name: 0.286, 0.22, 0.182, 0.364, 0.283, 0.306, 0.254, 0.336, 0.303, 0.24. Small LightGBM: 0.505, 0.508, 0.364, 0.571, 0.563, 0.526, 0.5, 0.52, 0.539, 0.505. The test split has 31 to 59 true pairs.
What this tells us. The F1 moves by about 0.2 between splits. So read these results to one decimal, not three. The order of the two methods does not change.
Decide: a review queue
Section titled “Decide: a review queue”On validation, no upper threshold reaches the 99% rule:
accepted accepted right precision accepted to review true in review recall if reviews are right0.50 38.0 33.0 0.868 137.0 14.0 0.810.70 33.0 29.0 0.879 142.0 18.0 0.810.90 18.0 16.0 0.889 157.0 31.0 0.810.95 7.0 6.0 0.857 168.0 41.0 0.81So no pair merges without a person. Every pair with a score of 0.05 or more goes to a reviewer, the highest score first. On test:
to review GLEIF duplicates probably the same, not marked (our rule) recall if reviews are rightvalidation 175.0 47.0 3.0 0.810test 255.0 30.0 8.0 0.682What this tells us. The order of the 255 test pairs matters: in the notebook’s task, the first 30 pairs hold 18 duplicates. After that, the reviewer finds few. Kofi can stop when the yield is too low for an hour of work.
One LEI per company, and the design on new records
Section titled “One LEI per company, and the design on new records”For each company with two LEIs, the merge rule keeps one LEI (our choice): the best registration status (ISSUED first), then the oldest. The status is used only here, after matching. The reviewed pairs join 133 companies. For 132 of them, the kept LEI is GLEIF’s own surviving LEI.
The cluster measures say little here. Almost every company has one LEI, so B-cubed is near 1.0 whatever the matcher does. Count the merges instead.
The notebook’s find_duplicate_companies() runs every step on a new list. On the 9,661 records of the test companies only, in 2 seconds:
count sumdecisionreview 47 9review first 24 21What this tells us. 24 pairs score 0.5 or more (“review first”), and 21 of them are GLEIF duplicates. Below 0.5, only 9 of 47 are. Together, the queue finds 30 of the 44 true pairs.
Two designs
Section titled “Two designs”Text description of the diagram
Two columns. Each has a header, a box with its steps, and three rows of bands with their counts on the test split.
What it costs
Section titled “What it costs”The reviewer’s time is an assumption from the course: 1.2 minutes a pair at US$40 an hour. The computer is almost free: the whole notebook runs in about 80 seconds on a laptop.
| Pairs to review on test | Reviewer time | Per 1,000 records (rough) | |
|---|---|---|---|
| People | 84, for 253 test records | 1.7 hours, about US$67 | about US$265 |
| Companies | 255, for 9,661 test records | 5.1 hours, about US$204 | about US$21 |
What this tells us. A noisy list of people costs much more review per record. The company register costs little per record, because most companies have one LEI. But only about one review pair in eight is a GLEIF duplicate, so most of Kofi’s cost buys a “no”.
Recommendation
Section titled “Recommendation”Nadia writes, for a real list of people:
- Use the people’s design. Merge at 0.99 or more; review from 0.05 to 0.99. First train it again on labels from the real list: simulated errors are not real errors.
- Keep the date of birth as evidence, not as a hard rule, until you know how often dates differ in the real list.
Kofi writes, for the bank:
- Never merge LEIs automatically. No threshold met the 99% rule on validation.
- Review the queue in order of score, and stop when the yield is too low for the reviewer’s time.
- Map the bank’s clients to the surviving LEI.
- Send likely duplicates to the LEI issuer. The register’s labels are incomplete, and a correction helps everyone.
What not to do:
- Do not merge on the name alone. Four in five same-name pairs were different entities.
- Do not reuse a threshold, a rule or a model size from another list without a new validation. The people’s model failed on the companies.
Insights and limits
Section titled “Insights and limits”- Simulated people are harder in one way, easier in another. Their errors are heavy, but they have no families. Neither they nor Larkfield tell you how a real list behaves.
- The companies’ labels are incomplete, and some are wrong. The measured precision is a lower bound.
- One country, one day. Another country, or the same register next month, gives other numbers.
- Small test sets. Another split moves the companies’ F1 by up to 0.2.
- No language model and no embeddings here. The names are short, and Module 6 showed that embeddings did not help on names and addresses.
- The prices are assumptions. Check the real costs of your own team.
Responsible use
Section titled “Responsible use”- Credit the sources. The MIT licence asks you to keep its copyright notice with the people’s file. CC0 asks for nothing, but say where the LEI data came from.
- No private persons. The people are simulated. We left the sole proprietors out of the LEI data, because a sole proprietor is a natural person.
- The companies are real. A DUPLICATE status is GLEIF’s data. “Probably the same” is our rule. Neither is a claim about any company. Report a suspected duplicate to the LEI issuer; do not publish it as a fact.
- A false merge of people is a privacy breach. On a real list, keep the source records, log every merge and make every merge possible to undo, as in The golden record.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can take a real register, with gaps in its labels, to a decision. Make it yours with one change, such as one from Go further below.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
- The two comparison tables as CSV files: every method, its threshold, precision, recall and F1.
DECISION.md: the thresholds, the review queue, its size and its cost, and the assumptions.- The data sources and licences: the people’s file with its MIT notice, and the LEI extract with its filter, its date and its checksum.
A README template:
# Finding duplicate people and companies
## ProblemOne list of people and one register of companies hold duplicates.Merge without a person only when 99 in 100 merges are right; a person checks the rest.
## Data and licencePeople: Splink's fake_1000 (MIT), 1,000 simulated records of 251 people.Companies: GLEIF LEI records with a legal address in Ireland (CC0), 38,650 records;labels: GLEIF's 180 DUPLICATE links (191 true pairs).
## ApproachNormalised names and legal forms; all pairs for people, five blockings for companies(pair completeness 0.911); comparison vectors; rules, Splink without labels,logistic regression and LightGBM compared; clusters and golden records.
## Results (test split)People: LightGBM F1 0.956; merge >= 0.99: 440 pairs, all right.Companies: small LightGBM F1 0.455 (exact name 0.276); no safe merge threshold;review queue of 255 pairs, 18 duplicates in the first 30.
## LimitsSimulated people; incomplete register labels; one country; small test sets.How to describe it in an interview, in two sentences: “I built two duplicate finders with the same steps: on simulated people, LightGBM reached F1 0.96 and merged 440 test pairs with no error. On real LEI records, the best F1 was 0.46 because thousands of funds share names and numbers; many ‘false’ matches were probably duplicates that the register had not marked, so I designed a review queue instead of automatic merges.”
Go further:
- Add the umbrella name (the part before the dash) as a field, and measure again.
- Review the 34 “wrong” matches by hand, label them, and measure the precision with your labels.
- Download another country’s LEI records with the same API filter, and run the same design without new training.
Knowledge check
Section titled “Knowledge check”Check your understanding
The exact rule “same first name, surname and date of birth” had a precision of 1.0 on people, but “same name” had a precision of only 0.198 on companies. Explain the difference, with an example from the GLEIF records.
Show the answer
For people, three equal fields together are rare by chance: two different simulated people almost never share all three. For companies, one field (the name) is often shared by different entities. Many funds have generic names, such as three different “GLOBAL EQUITY FUND” records with three registration numbers and three addresses. Sub-funds with the same name also exist in different umbrellas. So the name alone joins different funds, and the matcher needs the registration number, the sub-fund name and other fields as well.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
LightGBM called 34 test pairs of companies “same” that GLEIF does not link. Kofi says: “So 34 of our matches are wrong.” What do you answer?
Show the answer
Not necessarily. GLEIF’s labels are incomplete: it marks only the duplicates that someone found. 8 of the 34 pairs have the same name and the same registration number, so they are probably one entity with two LEIs. 15 more have the same name with one number missing. So the measured precision is a lower bound. Check the top pairs by hand, and report the real duplicates to the LEI issuer. Say clearly which labels are GLEIF’s and which are yours.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Module 7's check “two different dates of birth cannot be one person” improved Larkfield's clusters. Why did it break the simulated people into pieces, and what should Nadia do instead?
Show the answer
In the simulated list, 58% of the true pairs have two different dates of birth, because the program also makes errors in dates. The check cut those true links, so B-cubed recall fell from 0.986 to 0.568. Larkfield’s generator changed few dates: mostly a swapped day and month, which the check allows. Nadia should keep the date as evidence in the score (for example, the number of days between the dates). Before she uses any rule as a hard rule, she should measure how often it is true on the new data.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On simulated people, four blocking keys kept only 79% of the true pairs, so we compared all 499,500 pairs.
- LightGBM (F1 0.956) beat Splink without labels (0.910) and logistic regression (0.880). Merges at 0.99 or more were all right on test.
- A Larkfield rule (two dates of birth cannot be one person) broke the people into pieces: measure a rule on new data first.
- On real LEI records, four in five same-name pairs were different entities. A small LightGBM won (F1 0.455), because there were only 77 true training pairs.
- GLEIF’s labels are incomplete: the measured precision is a lower bound. The design is a review queue sorted by score, with no automatic merge.
Next, the final assignment: merge Larkfield’s customers with a second company’s list, with everything from the course.
Next: Final assignment