Final assignment
- After the case studies
- about 5 hr
Where we are
Each module added one part of matching and measured it: normalisation, blocking, scoring, clustering, review and privacy. In this assignment, you use them all to merge Larkfield's customers with a second company's list.
What you will be able to do
Use every module of the course on one new list, and deliver a matching pack that Omar, Ivana and Amira can act on. The new list has its own columns and its own habits, and nobody gives you the true customers. So you measure each choice again, on this data, and you plan for the mistakes that you cannot see.
Before you startWhat to know and to have ready
- You completed Modules 1 to 8. Each part below names the lessons that it uses.
- The certificate also asks for at least one case study: Match products between two shops, Link bibliographic records or Find duplicate people and companies. They are good practice for this assignment, but you do not need them for it.
- Your folder
larkfield-matchworks at the end of Module 8:python -m pytest -qprints42 passed, as in New records every day. The course data is inlarkfield-match/data, and the foldernextia-learning-labsis besidelarkfield-match, as in Meet the records. - About 4 to 6 hours. Most of it is reading records and writing decisions. No step runs for long. In the author’s run, the whole pipeline took about 2 minutes. It used about 5.4 GB of memory at its peak.
- No language model and no key are needed.
- Tested on macOS with Python 3.12.13 and the project’s pinned packages. The Windows commands are written, but they were not run.
The situation: Larkfield buys a hardware chain
Section titled “The situation: Larkfield buys a hardware chain”Larkfield has bought Fernhollow Home & Hardware, a chain of hardware shops in four provinces. Fernhollow has its own customer list, from its own till system. Some of its customers already shop at Larkfield. Some appear twice in Fernhollow’s own list.
The team has questions, as in the course:
- Omar, the product manager, wants the real number of customers after the merge. He remembers the costs: a false merge costs about 150 dollars, and a missed match about 3 dollars a year. Both are assumptions in
data/costs.json. - Ivana, the customer-data steward, will review the unsure pairs. She can check about 300 pairs a day (an assumption). She asks how many days the merge will take her.
- Amira, the developer, owns the matching job. She wants every merge written down, and a way to undo one.
- Tomás, the AI engineer, has an idea: “We tuned the matcher on Larkfield. Just run it on Fernhollow.”
Priya asks you to answer them with evidence.
Be honest about the data. Fernhollow’s records are synthetic. The course’s code made them from invented people, with the same error model as Larkfield’s records. Fernhollow is an invented company. The data has practice problems that we made on purpose. Some are like Larkfield’s, and some are new. The course team knows them, and you do not. Quote your numbers “on our synthetic records”.
Text description of the diagram
A flow from left to right. Fernhollow, 10,773 records with a new schema, goes to “prepare”: the project’s columns. Then “block”, together with Larkfield’s 60,108 records and your entities from the course: only pairs with a Fernhollow record. Then “score”: rules and three other families of methods. From “score”, three arrows go out. “link”: the record joins a customer. “review”: Ivana checks the pair, 300 pairs a day. “new customer”. All three go to “entities”: checks, golden records, history and undo. A dashed arrow goes from labelled_pairs.csv (600 pairs: choose on validation, test once) to “score”, for pair completeness and recall. Another dashed arrow goes from “your own labels” (100 or more of your links and review pairs) to “link”, for the precision of each band.
The data
Section titled “The data”The files are in the labs repository, in C39/final/. The course data, C39/data/, stays as it was.
| File | Rows | What one row is |
|---|---|---|
fernhollow_customers.csv |
10,773 | One customer account in Fernhollow’s till system: record_id, customer_name, phone, email, street, town, province, postal_code, birth_year |
labelled_pairs.csv |
600 | A pair with at least one Fernhollow record, labelled by the course team: record_id_l, record_id_r, label (1 = same customer), why_sampled, split |
check_submission.py |
Checks the shape of your pack. It knows no answer and gives no score |
The folder also has README.md, dataset-card.md and SHA256SUMS. It is about 1.3 MB.
Get the files, check them, and look at them. Run this in your larkfield-match folder, with .venv active:
git -C ../nextia-learning-labs pullF=../nextia-learning-labs/C39/final(cd "$F" && shasum -a 256 -c SHA256SUMS | grep -vc ': OK$')python -c "import pandas as pdfh = pd.read_csv('$F/fernhollow_customers.csv', dtype=str, keep_default_na=False)lab = pd.read_csv('$F/labelled_pairs.csv', dtype=str)print(len(fh), 'Fernhollow records;', len(lab), 'labelled pairs')print(lab.groupby(['split', 'why_sampled']).size().to_string())print(fh.head(3).to_string(index=False))"git -C ..\nextia-learning-labs pull$F = "..\nextia-learning-labs\C39\final"Get-FileHash "$F\fernhollow_customers.csv" -Algorithm SHA256Compare the hash with the line for the file in SHA256SUMS. Then save the Python lines of the macOS tab in a file look.py, with $F replaced by the folder, and run python look.py.
The first number is the number of wrong files. It must be 0. Then you see:
010773 Fernhollow records; 600 labelled pairssplit why_sampledtest random 23 same surname and region 139 true pair 116validation random 37 same surname and region 161 true pair 124record_id customer_name phone email street town province postal_code birth_yearFH-000001 Girard Sophie 819-555-2733 SOPHIEGIRARD@EXAMPLE.NET 2144 rang de la Colline Sainte-Odile-sur-Mer QC G2B 9W6 1949FH-000002 Bergeron Alexandre ALEXANDREBERGERON@POST.EXAMPLE 8949, rang Sainte-Cécile #3 Saint-Ambroise-des-Monts QC H1G 7C0 1977FH-000003 Okafor Leila 819-555-9234 LEILA.OKAFOR55@EXAMPLE.COM 114, r. de l'Église Lac-aux-Grives QC G3C 1T7What the labelled pairs can and cannot tell you
Section titled “What the labelled pairs can and cannot tell you”The 600 pairs were chosen as Larkfield’s 3,000 labelled pairs were, in A supervised pair classifier:
- 240 true pairs, drawn at random from all true pairs that have a Fernhollow record. They are a fair sample of the pairs that you must find. So the share of them that your blocking keeps is an estimate of pair completeness. The share that your matcher finds is an estimate of recall.
- 300 pairs with the same surname start and the same postcode region, and 60 random pairs. Almost all of them are two different people.
split puts each pair in validation or test. Choose your blocking, your matcher and your thresholds on validation. Report on test once, at the end. One Fernhollow record is never in both splits.
The labelled pairs cannot tell you the precision of your links. Four pairs in ten here are true pairs. Among your candidate pairs, true pairs are rare, and the hard different pairs are not in this sample. So precision measured here is far too high. You measure precision yourself, in Part 6.
The course team keeps the true customer of every Fernhollow record. It grades your links and your entities against them. You cannot tune on them, and you do not need to.
Make a folder match-pack/ in your larkfield-match folder. Put every file of the assignment in it. What to submit lists the files.
Keep the assignment’s outputs apart from the course’s. Set two variables in every terminal that you use for the assignment:
export LARKFIELD_OUTPUTS=outputs-final LARKFIELD_DUCKDB_TEMP=outputs-final/duckdb_tmp$env:LARKFIELD_OUTPUTS = "outputs-final"; $env:LARKFIELD_DUCKDB_TEMP = "outputs-final\duckdb_tmp"The project still caps DuckDB at 4 GB of memory and 4 GB of temporary files, as in Scale.
The seven parts
Section titled “The seven parts”The pack follows the course’s completion rules: one part for each required item.
Text description of the diagram
A box on top: match-pack/REPORT.md, with the decisions, the numbers and the commands that make them. Below it, seven boxes. 1 Prepare: fernhollow.py, with the parse and value checks. 2 Block: pair completeness and reduction ratio. 3 Compare: comparison.csv, with the rules and three other families, and their cost. 4 Decide: links.csv, with the thresholds, the review band and the days of review. 5 Entities: entities.csv, golden.csv, history.jsonl and an undo. 6 Your labels: my_labels.csv, with the precision of each band. 7 Privacy: the fields used, retention and deletion.
You write the code yourself, in one or more scripts in match-pack/. The project is your library: every step below names the functions of larkfield_match that do the work. larkfield_match/incremental.py, which runs match daily, has the same shape as this assignment: new records, keys, vectors, a model, thresholds and joins. Read it first.
Part 1: prepare Fernhollow’s records
Section titled “Part 1: prepare Fernhollow’s records”Uses Normalise and Parse into parts.
-
Look before you parse. Print 30 random rows. For each column, write down what you see: the order of the words, capitals, spaces, extra text and empty values. Also count the most common values of
phone,emailandstreet. A value on many records is a warning, as the garden centres’ own phone number was in Clustering pairs. -
Write
match-pack/fernhollow.py. Give it a function that turns Fernhollow’s rows into the project’s prepared columns:record_id,source,first,last,full,email,phone,number,street,unit,town,province,postcode,dob,created,parse_ok. Uselarkfield_match.normaliseandlarkfield_match.parse.prepare_source()inlarkfield_match/prepared.pyshows the pattern for Larkfield’s three sources. -
Decide what to do with
birth_year. The project compares full dates of birth. A year is not a date. Do not put a year where the project expects a date: think about whatdob_eqwould say then. Keep the year in a column of its own, and use it in a check later. -
Check your parser on the rows that you wrote down in step 1, one by one. Then measure the share of records that parse, and the share with a valid email, phone and postcode.
-
Put the two lists together: Larkfield’s prepared records from
prepared.customers(), and yours.
Part 2: block, and measure what blocking loses
Section titled “Part 2: block, and measure what blocking loses”Uses Why all pairs is impossible, Blocking keys and sorted neighbourhood and What blocking loses.
You need only the pairs with at least one Fernhollow record: Fernhollow against Larkfield, and Fernhollow against Fernhollow. The course already decided Larkfield’s own pairs.
-
Count all possible pairs with a Fernhollow record. The reduction ratio divides by this number.
-
Make candidate pairs with the project’s keys on the combined table. Keep the pairs with a Fernhollow record:
from larkfield_match import blockingkeys = blocking.customer_keys(both) # both: Larkfield's and Fernhollow's prepared recordscand = blocking.key_pairs(keys, ["postcode", "email", "phone", "surname4", "dob"])fh = set(fernhollow.record_id)cand = cand[cand.record_id_l.isin(fh) | cand.record_id_r.isin(fh)] -
Estimate pair completeness on the true pairs of the validation split: the share of them that are in
cand. Try at least three blockings, for example the postcode alone, the course’s keys, and the keys with one more key. Report the candidates, the reduction ratio and the pair completeness of each one. -
Give the size of the sample with the estimate. 124 true pairs is a small sample. If your blocking keeps all of them, that does not prove a pair completeness of 1.0. With 124 of 124, a 95% lower limit is about 0.976. Estimate uncertainty in Essential Mathematics and Statistics for AI explains confidence intervals.
-
Choose one blocking on validation. Write why, and what it costs in candidate pairs. Then check it once on the test split.
Part 3: compare matchers
Section titled “Part 3: compare matchers”Uses Exact and rule-based matching, Compare by field, then combine, Learn the weights without labels, A supervised pair classifier and A language model for the unsure pairs.
-
Make the comparison vectors of your candidates.
customer_vectorstakes the table to read from:from larkfield_match.vectors import customer_vectorsv = customer_vectors(cand, both) -
A rule-based baseline. Use the course’s four rules (
larkfield_match/runs/rules.py) on your candidates. One rule finds nothing on Fernhollow’s records: say which one, and why. -
At least three other families. A family is one idea. Examples: the hand-weighted score (
runs/weighted.py); Splink without labels (splinkmodel.train(both[["record_id"] + splinkmodel.COLUMNS])); a classifier (classifier.labelled_training_set(),classifier.fit(),classifier.predict()). Two settings of one family count as one family. The classifiers learn on Larkfield’s 3,000 labelled pairs. If you also train on Fernhollow’s validation pairs, say so, and keep the test pairs out. -
Score every method on the same labelled pairs. A labelled pair that blocking lost counts as “not found”, as in the course’s comparison table. Report precision, recall and F1 on validation, with the number of pairs.
-
Give the cost per 1,000 pairs of each method. Time it on your candidates, and use the course’s CPU price of US$0.17 an hour (an assumption in
data/costs.json). -
Choose your matcher on validation. Then run it once on the test pairs, with no change. Write the table in
match-pack/comparison.csv: one row per method, with its threshold, the number of links it makes, the validation and test measures, and its cost.
A language model is optional. The course’s recorded answers cover only Larkfield’s pairs, so there is none for Fernhollow. You can use a model with your own key, as in A language model for the unsure pairs. Then use it only on the unsure band, record every answer and its cost, and let a person check its “same” answers before any merge.
Part 4: thresholds and the review band
Section titled “Part 4: thresholds and the review band”Uses From weights to a decision and Human review.
-
Turn your scores into probabilities that mean what they say. The course calibrates LightGBM on Larkfield’s validation pairs:
calibrate.fit(pairs.get("lgbm")). The first time, it trains the course’s model inoutputs-final/. Say why a calibration made on Larkfield’s records may not hold for Fernhollow’s. -
Choose two thresholds: an upper one (link) and a lower one (review). The review band is between them. Try at least three bands. For each band, give the number of links, the number of review pairs and Ivana’s days at 300 pairs a day. Also give recall on the validation true pairs, with and without a perfect review.
-
Decide with the costs. A false merge costs about 150 dollars and a missed match about 3 dollars a year. A review takes about 1.2 minutes at 40 dollars an hour. All three are assumptions. Say what Ivana’s time costs, and how many false merges it must prevent to pay for itself.
-
Add the checks that the new data needs, before any link. One example: two records with different birth years. Measure how many links each check removes.
-
Write
match-pack/links.csv: one row for every pair with a Fernhollow record that you link or send to review. Columns:record_id_l,record_id_r,score,decision(linkorreview). -
Say how review decisions come back. A “same” merges, and a “different” goes to
cannot_link.csv, as in Decisions become labels. Module 8 measured what retraining on review labels does. Say whether you would retrain on Ivana’s labels.
Part 5: entities, golden records and undo
Section titled “Part 5: entities, golden records and undo”Uses Clustering pairs, Safer clustering, Cluster measures, The golden record and New records every day.
-
Start from Larkfield’s entities. Run
match goldenonce (inoutputs-final/). Itsentities.csvgives every Larkfield record an entity ID. A Fernhollow record that you link to a Larkfield record joins that record’s entity. -
Cluster your links. Then check every cluster before you accept it. At least: no cluster with two different Larkfield customers, no cluster over the course’s size limit, and no two birth years in one cluster. A new record must not merge two old customers alone, as in Module 8. Cut the weakest link until a cluster passes, as
larkfield_match/safecluster.pydoes. Send every cut link to review. -
Look at your largest clusters by hand, and give the sizes of the five largest. A giant cluster means one bad value or one bad link.
-
Cluster measures. You have no truth for all records, so measure on the labelled test pairs. Give the share of true pairs that end in one entity, and the share of different pairs that end in one entity. Also give the number of customers after the merge, and the number of Fernhollow records that joined a Larkfield customer.
-
Write
match-pack/entities.csv:record_idandentity_idfor every record, Larkfield’s and Fernhollow’s. Keep the Larkfield entity IDs thatmatch goldengave. A Fernhollow record that starts a new customer getsLK-and its own record ID, as in the daily run. -
Golden records. Use the merge rules of The golden record, and decide where Fernhollow comes in the order of trust.
golden.build(entities, records=both, trust=...)builds them from your combined table, with your order of trust. Writematch-pack/golden.csv: one row per entity, with the source record of each value (name_from,email_from…). -
History and undo. Write one line to
match-pack/history.jsonlfor every Fernhollow record: the entity that it joined, or that it started a new one. Then show one undo, as in Undo a merge. Take one Fernhollow record out of its entity withgolden.undo(). Paste the history line and thecannot_link.csvrow that it writes.
Part 6: label your own sample
Section titled “Part 6: label your own sample”Uses Measure a matcher and What the reviewer sees.
This part measures what the labelled pairs cannot: the precision of your links.
-
Draw at random at least 50 pairs from your links and 50 from your review band. Use a fixed seed, and say which.
-
Hide the score and the band while you label. Show only the two records, as Ivana’s screen does.
-
Label each pair “1” (same person), “0” (different people) or “unsure”. Add a short note when you are not sure. Write
match-pack/my_labels.csvwith the columnsrecord_id_l,record_id_r,band,my_labelandnote. -
Estimate the precision of each band, with its interval. For example: 49 “same” of 50 links. Count “unsure” apart. Estimate uncertainty in Essential Mathematics and Statistics for AI computes an interval for a share.
-
Turn it into a number for Omar: the expected false merges among all your links, and their cost under the course’s assumptions.
-
Say how much you trust your own labels. You read the same fields as the matcher. Which pairs were hard for you, and why?
Part 7: the report, with privacy
Section titled “Part 7: the report, with privacy”Uses Privacy and matching. The AI security course covers personal data more widely.
Write match-pack/REPORT.md with these sections:
# Matching pack: Fernhollow joins Larkfield
## Data checks (files, checksums, what you saw in the columns, values on many records)## Prepare (Part 1: your parser, parse and value checks)## Blocking (Part 2: the blockings tried, pair completeness with the sample size)## Comparison (Part 3: the table, the cost per 1,000 pairs, the choice and the one test run)## Decision (Part 4: thresholds, the review band, Ivana's days, the checks, the costs)## Entities (Part 5: clusters, checks, largest clusters, golden records, undo)## Your labels (Part 6: precision of each band with intervals, expected false merges)## Privacy (fields used, fields left out, retention, a deletion request on a merged entity)## Answers (Omar's count, Ivana's days, Amira's undo, and Tomás's idea)## Limits (what your numbers cannot see)## How to reproduce (commands, versions, run times, a fresh environment)The privacy section answers four questions:
- Fields used. Which fields does your matcher read, and why does it need each one?
- Fields left out. Which fields did you not use, and what did leaving them out cost? Module 8 measured the value of each field.
- Retention. How long do you keep Fernhollow’s raw records, the candidate pairs, the vectors and the review queue? Who can read them?
- A deletion request on a merged entity. A customer who is in both lists asks Larkfield to delete their data. Which records, golden values, history lines and pairs must go, and what stays? Use A deletion request on a merged customer, and show it on one entity of your pack.
This is not legal advice. Say which rules you assumed.
What to submit
Section titled “What to submit”The folder match-pack/ in your larkfield-match folder:
match-pack/ REPORT.md Parts 1 to 7, with the commands that make every number fernhollow.py Part 1: your parser comparison.csv Part 3: one row per method links.csv Part 4: every pair you link or send to review entities.csv Part 5: an entity for every Larkfield and Fernhollow record golden.csv Part 5: one golden record per entity, with the source of each value history.jsonl Part 5: one event per Fernhollow record, and your undo my_labels.csv Part 6: your own labels *.py every other script requirements.txt exact versions (pip freeze)Check the shape before you hand it in:
python ../nextia-learning-labs/C39/final/check_submission.py match-packpython ..\nextia-learning-labs\C39\final\check_submission.py match-packWith the author’s pack, it printed:
links: 7,229 pairs to link, 1,426 pairs to review = 4.8 days for one reviewer at 300 pairs a dayentities: 44,912 customers from 70,881 records; 4,469 of 10,773 Fernhollow records are in an entity with a Larkfield recordlargest entities: LK-G-001224 (6 records), LK-G-002621 (6 records), LK-G-002066 (6 records)your labelled sample: 100 pairs; by band: review 50, link 50history: 10,775 events
The pack has the right shape. This check gives no score: the reviewer grades the content.When something is missing, it ends with “Not ready yet:” and a list, for example my_labels.csv is missing. Your numbers will differ from the author’s.
Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. The course is complete only when all seven items are demonstrated. You can improve an item and send it again. A reviewer also runs your scripts in a fresh environment, so REPORT.md must say how.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. Prepare (Module 2) | A parser for Fernhollow’s columns, checked by hand on named rows. The parse rate, and the share of valid emails, phones and postcodes. Values on many records found and handled. | A parser that works, with no check of the name order or of common values. | Fernhollow’s records read with Larkfield’s parser, unchecked. |
| 2. Block (Module 3) | At least three blockings, with candidates, reduction ratio and pair completeness on the validation true pairs, and the sample size. One chosen on validation, checked once on test. | Pair completeness with no sample size, or only one blocking. | No pair completeness, or pair completeness measured on the pairs that the blocking made. |
| 3. Compare (Modules 2, 4, 5 and 6) | The rules and three other families on the same labelled pairs, with lost pairs counted. The cost per 1,000 pairs. A matcher chosen on validation and run once on test. | Two other families, or no cost. | No rule-based baseline, or the choice made on the test pairs. |
| 4. Decide (Modules 5 and 8) | Two thresholds and a review band, with the number of review pairs and Ivana’s days. At least three bands compared, and the costs used in the choice. The new checks measured. links.csv, and how review labels come back. |
A band with no days, or thresholds with no reason. | One threshold and no review band. |
| 5. Entities (Modules 7 and 8) | Clusters checked (no two Larkfield customers, a size limit, birth years). The largest clusters read, cluster measures on the test pairs, and the customer count. Golden records with merge rules and the source of each value. A history, and one undo shown. | Golden records with no source columns, or an undo described but not run. | No check for giant clusters, or no way to undo a merge. |
| 6. Your labels (Modules 2 and 8) | At least 50 links and 50 review pairs, drawn at random and labelled blind. The precision of each band with an interval. The expected false merges and their cost. | Fewer pairs, or no interval. | Precision quoted only from labelled_pairs.csv. |
| 7. Report and privacy (Module 8) | Every section of REPORT.md. The four privacy answers, with a deletion shown on one merged entity. Answers to Omar, Ivana, Amira and Tomás. |
A privacy note with no deletion on a merged entity. | No privacy note. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- Tuned on the test pairs: you changed a blocking, a matcher, a threshold or a check after the test run. Or you used the test pairs to choose. (Items 2, 3 and 4.)
- No review band: every pair is either linked or dropped. (Item 4.)
- Clusters not checked for giant components: no look at the largest clusters, and no check that stops one bad link from joining many people. (Item 5.)
- No way to undo a merge: no history, or a merge that changes or deletes source records. (Item 5.)
- Pair results with no cluster results: good pairs prove little about entities. (Item 5.)
- A language model that decides merges with no check. (Items 3 and 4.)
What a good pack looks like
Section titled “What a good pack looks like”A good pack is honest before it is impressive. It has three things:
- Numbers with their sizes. Each estimate says which pairs it comes from, and how many.
- A simple baseline in each comparison. If the rules do as well as the classifier on some measure, the pack says so.
- A decision with its cost. For example: “We link at this threshold. Our own labels say that about this many of our links are wrong, which costs about this much. Ivana needs this many days for the review band. We accept that because …”
The author’s results
Section titled “The author’s results”The author solved the assignment once, from the public files only, with the course’s code. Then the course team graded the author’s pack against the true customers. The results below come from that run. Your design can be different. Compare the method and the size of each result, not the exact number.
The case studies met some of the same results on real data. Read them next to this list: Match products between two shops and Link bibliographic records.
Preparing and blocking
Section titled “Preparing and blocking”- The parse was the biggest single choice. The author also ran the same matcher with Fernhollow’s names read the way the course reads Larkfield’s. Then it found only about 6 true pairs in 10. With the right word order, it found about 8 in 10.
- Blocking was easy to get right. The course’s keys, with one more key on the street, made about 430,000 candidate pairs out of about 706 million. That is a reduction ratio of about 0.9994. They kept every validation and test true pair. The postcode alone kept about 84% of them.
The comparison
Section titled “The comparison”- On the labelled pairs, every learned method looked almost perfect. Splink, logistic regression and LightGBM had validation precision and recall between about 0.99 and 1.0. The sample could not separate them well.
- The rules were the honest baseline. They had no wrong link among the labelled pairs either. Graded, about 96% of their links were right, and they found about 93% of the true pairs. But they made about three times as many false merges as the author’s design: about 360 against about 110. At 150 dollars a false merge, those cost far more than the 3 dollars a year of each missed match that the rules found.
- The author chose LightGBM with two new checks: a birth-year conflict or a Jr / Sr conflict makes a pair “not the same”. The checks removed only about 60 links, and about 8 in 10 of those were wrong links.
Thresholds and review
Section titled “Thresholds and review”- The author linked at a calibrated probability of 0.9 and reviewed from 0.2. After the cluster checks, the review queue had about 1,430 pairs: about 5 days for Ivana.
- Graded: about 98% of the links were right. They found about 82% of the true pairs. With a perfect review, they would find about 93%.
Entities
Section titled “Entities”- The cluster checks cut about 140 links. Most of them would have joined two Larkfield customers through one Fernhollow record. No cluster had more than 6 records.
- Graded: B-cubed precision about 0.99 and recall about 0.96, on Fernhollow’s records. The merge added no entity with two different Larkfield people.
The author’s own labels
Section titled “The author’s own labels”- The link band: the author labelled 49 of 50 links “same” and one “unsure”. The grade agreed: about 98% of the links were right.
- The review band was harder than it looked. The author said “same” for 39 of the 42 review pairs that were not “unsure”. The grade showed that only about 8 in 10 of those were the same person. Every “unsure” pair was two people. A person reads the same fields as the matcher, and the same things fool them. Here, they were mostly two people of one family with the same name and address, but a different email and phone. This is why Ivana’s decisions need a test of their own, as in Human review.
Speed and cost
Section titled “Speed and cost”- The whole pipeline took about 2 minutes on the author’s laptop. Blocking took under 1 second, the vectors about 1 second, and Splink’s training about 3 seconds. Every method cost far less than one cent per 1,000 pairs at the course’s CPU price.
Common mistakes
Section titled “Common mistakes”Common mistake
A threshold that makes more than a quarter of a million links
Symptom: you choose the threshold with the best F1 on the validation pairs. LightGBM’s threshold is close to 0. The labelled pairs show a precision near 1.0. But your matcher links about 280,000 pairs for 10,773 records.
Cause: the labelled pairs have no hard different pairs. Almost every “different” pair has a different surname or a different region. So a very low threshold loses nothing on them. On all candidates, it links most pairs. In the author’s graded run, only about 3% of these links were right, and one entity had about 50,000 records.
Fix: count your links before you trust a threshold. Choose the threshold on calibrated probabilities and on Ivana’s capacity. Then measure precision on your own random sample of links, as in Part 6.
Consequence in AI work: a test set that does not look like the real inputs gives a threshold that fails in production. Check how your test set was sampled.
Common mistake
Names in the wrong order
Symptom: your blocking keeps almost every true pair, but your matcher finds far fewer of them than on Larkfield’s records.
Cause: your parser reads the surname from the wrong word. The email, phone and postcode keys still find many pairs, so blocking looks fine. But the name features of the vector now compare a first name with a surname.
Fix: read 30 rows and decide which word is the surname. Then check the parse on records where you can tell the first name from the surname.
Consequence in AI work: a pipeline that runs with no error can still read every value wrongly. Check the data, not only the code.
- Tomás’s idea accepted. The Larkfield model runs on Fernhollow. But its date-of-birth features are always missing there, and the new list has its own problems. Measure it on Fernhollow’s pairs.
- A year in the date field. The project compares
1979with1979-05-20and says “different”. Keep the year apart, and use it in a check. - Same name, same address, same phone. That is not always one person: the brothers in From weights to a decision were two people. Look at the names, the birth years and anything else that two people of one family do not share.
- A value on many records. Count the most common phones and emails before you trust them. One shared value can make one giant cluster.
- Two old customers merged by one new record. A Fernhollow record that links to two different Larkfield customers must not merge them alone. Send those links to review.
- Precision quoted from the labelled pairs. It is too high. Quote it from your own sample.
- A test run used twice. Run the test pairs once. If you change anything after that, say so, and say that your test is now a second validation.
Show the public check values
Open this only after you have your own numbers. These values come from the public files, so you can check them. They are not the answers to the assignment.
- Files: 10,773 Fernhollow records. 600 labelled pairs: 322 in validation (124 true pairs) and 278 in test (116 true pairs). Every file in
SHA256SUMSis OK. - All possible pairs with a Fernhollow record: 705,566,862. That is 10,773 × 60,108 pairs with Larkfield, plus the pairs inside Fernhollow.
- The course’s keys (postcode, email, phone, surname start, date of birth) on the combined table, with Fernhollow’s names in the right order: 421,383 candidate pairs with a Fernhollow record. They keep all 124 validation true pairs.
- Speed on the author’s laptop (Apple M4, one thread for scikit-learn and LightGBM): the whole pipeline, with
match golden, took about 2 minutes. The peak memory was about 5.4 GB.
Recap
You can now merge a new customer list into an existing one and stand behind the result. You read a new schema before you parse it. You block with pair completeness estimated on a fair sample of true pairs. You compare a rule baseline with three other families on the same pairs, and you know what the labelled pairs can and cannot tell you. You set thresholds with a review band that a person can finish. You check clusters before they become customers, build golden records with their sources and history, and undo a merge. You measure precision on your own labels, and you know that your own labels can be wrong. And you write down which personal data you used, how long you keep it, and how you delete it from a merged customer.