New records every day
- Lesson 26
- 40 min
- Module 8: Run matching in production
Where we are
We matched all the records at once. Larkfield gets new records every day. In this lesson, we match new records against existing entities, keep entity IDs stable, and see what happens when one new record links two old entities.
Lesson 26 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?New records every day
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Match each day’s new records against the customers TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield already knows, without matching everything again. Give every entity an ID that does not change. See what the daily run gets right and wrong in a week of synthetic records. Learn why one new record must never merge two old entities on its own.
NotebookMatch a day of new records in a notebook
The notebook reads the 300 new records of day 1 and joins them to known customers by email. You count the joins, the right joins, and the new records that start a new entity although the customer is already known. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Entities and golden records with stable IDs, from The golden record.
- Safe clustering, from Safer clustering.
- The calibrated LightGBM model and its unsure band, from A language model for the unsure pairs.
What you need on your computer
- The project
larkfield-matchat the end of Module 7, with its virtual environment active, and a trained model (match train). Or use the notebook only.
New books at the library
Section titled “New books at the library”A library has 60,000 cards in its catalogue. Every morning, 300 new books arrive. The librarian does not sort the whole catalogue again. She takes each new book and looks for its card.
Sometimes she finds it: a second copy of a book the library already has. She files the new copy under the same shelf number. Sometimes there is no card: the book is new, and it gets a new shelf number. Sometimes she is not sure. Two cards look alike, so she puts the book on a desk for the head librarian.
One rule matters most: a shelf number never changes. Readers, loans and reminders all use it. If it changed every night, every loan would point to the wrong book.
Larkfield has the same morning. New shop accounts and loyalty sign-ups arrive every day. Each one is a new customer or a customer Larkfield already knows.
The problem: match everything again, every night?
Section titled “The problem: match everything again, every night?”Module 7 made entities from all 60,108 records: 38,908 of them, each with an ID such as LK-G-006348. The simple plan for new records is to add them to the pile and run the whole matcher again.
This plan has two problems.
It does work that is already done. Every night, it compares again more than a million candidate pairs that it compared yesterday. On Larkfield’s records, that takes less than a minute. But records grow every day, and Scale shows how fast that time grows.
The IDs move. An entity’s ID is LK- and the smallest record ID in it. New records have IDs such as D1-0176. In alphabetical order, D1-… comes before G-…, L-… and S-…. So a fresh run renames every entity that a new record joins. If D1-0176 joins LK-G-006348, a fresh run calls the entity LK-D1-0176. Orders, emails and deletion requests that point to LK-G-006348 now point to nothing.
A matcher that runs every day needs two things: compare only what is new, and keep the old names.
The idea: incremental matching with stable IDs
Section titled “The idea: incremental matching with stable IDs”Incremental matching means matching only the new records: against the records that are already matched, and against each other. Each entity keeps its ID for ever. We call that a stable ID.
For each new record, the daily run does four things:
- Find candidates. Make candidate pairs with the five blocking keys of Module 3: postcode, email, phone, the surname key and the date of birth key. Each pair has at least one new record.
- Score. Compare each pair field by field and score it with the LightGBM model of Module 6. Then calibrate the score, so that 0.3 means “about 3 in 10 such pairs are true pairs”.
- Decide. A pair with a probability of 0.9 or more is a link. A pair from 0.1 to 0.9 goes to the review queue: a list of pairs for a person to check. Below 0.1, nothing happens.
- Join or start. A record with a link joins the entity of the linked record, and that entity keeps its ID. A record without a link starts a new entity:
LK-and its own record ID.
Every step goes into a history file, outputs/history.jsonl, so every change can be traced and undone.
Text description of the diagram
The diagram is a flow from top to bottom. The top box is day 1: 300 new records. An arrow leads to the candidate pairs: each new record against every known record, with the five blocking keys. The next box scores each pair with LightGBM and calibrates the probability. Three arrows lead to three outcomes. Green: probability 0.9 or more, the record joins that entity, and the entity’s ID does not change; 143 records on day 1. Yellow: probability 0.1 to 0.9, the pair goes to the review queue; 74 pairs on day 1. Blue: no link, a new entity named LK and the record ID; 157 records on day 1. A yellow box below says that a record with links to two entities joins the strongest one. The other link goes to review (4 records on day 1). A note says that a new record never merges two old entities on its own.
A tiny example: four records of day 1
Section titled “A tiny example: four records of day 1”Here are four real new records of day 1, and what the daily run did with them. Like every record in this course, they are synthetic.
| New record | Known records it was compared with | What the run did | True answer |
|---|---|---|---|
| D1-0003, Marie Tremblay, (450) 555-4557 | S-030041, Marie Tremblay | joined LK-S-030041 | right: the same customer |
| D1-0001, Ahmed Novak, J7P 2A9 | no link | started LK-D1-0001 | right: a new customer |
| D1-0049, Mary Walker, P3Z 1G8, mwalker@post.example | D1-0047, Mary Walker, P3Z1G8, walker.mary87@post.example | joined LK-D1-0047 | wrong: two women with one name at one postcode |
| D1-0176, Melanie Roy, melanie.roy@example.com | two entities: LK-G-006348 and LK-S-019824 | joined LK-G-006348; the second link went to review | one person in all three |
The first two rows are the usual cases. The third row is a wrong join, a false merge. Both Mary Walkers arrived on day 1, and the second one joined the first one’s new entity: same name, same postcode, different emails. The fourth row needs its own section.
One record, two entities
Section titled “One record, two entities”Look at Melanie Roy’s new loyalty record, D1-0176. Its phone number is the phone of an old garden-centre record. Its email is the email of an old shop record. But those two old records were in two different entities. Module 7 did not join them: their phone numbers and their postcodes differ, and the garden-centre record has no email.
Text description of the diagram
At the top is the new loyalty record D1-0176: Melanie Roy, email melanie.roy@example.com, phone (819) 555-4445, postcode G1R 8V0, born 13/09/1970. Two arrows go down. The left arrow, “strongest link”, goes to the green entity LK-G-006348. It has the garden-centre record G-006348, ROY, MELANIE, phone 8195554445. The right arrow, “second link”, goes to the yellow entity LK-S-019824. It has the shop record S-019824, Mélanie Roy, email melanie.roy@example.com, phone (819) 555-4702. A note says: the new record joins LK-G-006348, the link to LK-S-019824 goes to the review queue, and the two old entities stay apart until a person decides.
What should the run do? If all three records are one person, the two old entities should merge. Here, the truth says they are: one Melanie Roy, with an old phone number in one record.
But a family can make the same picture. Think of a son who signs up with the family email and his own phone. The email links him to his mother’s entity. The phone links him to his own old entity. A merge would join a mother and her son into one customer.
The daily run can’t tell these two stories apart. So it follows a firm rule: a new record never merges two old entities on its own. It joins the entity of its strongest link. Every other link goes to the review queue, marked “would join two entities”. A person decides. On day 1, 4 records were in this case. In Melanie Roy’s case, the reviewer said “same”, and the two entities merged into {two.kept}.
Run it in the project
Section titled “Run it in the project”This module has four lessons. You copy all their files now, and each lesson uses its own commands.
-
If you ran
match resetafter Module 7, make the entities again first. The command uses the saved LightGBM pairs, or trains the model if they are missing:Terminal window match golden38,908 entities from 60,108 records (16,513 with more than one record); 0 with two dates of birthSaved outputs/entities.csv and outputs/golden.csv -
In your project folder, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files of the snapshot
end-of-m08. Itsrequirements.txthas no new package:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m08cp $S/larkfield_match/incremental.py $S/larkfield_match/review.py larkfield_match/cp $S/larkfield_match/bench.py $S/larkfield_match/privacy.py larkfield_match/cp $S/larkfield_match/runs/private.py larkfield_match/runs/cp $S/larkfield_match/commands/daily.py $S/larkfield_match/commands/review.py larkfield_match/commands/cp $S/larkfield_match/commands/bench.py $S/larkfield_match/commands/privacy.py larkfield_match/commands/cp $S/larkfield_match/commands/forget.py larkfield_match/commands/cp $S/tests/test_production.py tests/cp $S/requirements.txt $S/README.md .Terminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m08"Copy-Item $S\larkfield_match\incremental.py, $S\larkfield_match\review.py larkfield_match\Copy-Item $S\larkfield_match\bench.py, $S\larkfield_match\privacy.py larkfield_match\Copy-Item $S\larkfield_match\runs\private.py larkfield_match\runs\Copy-Item $S\larkfield_match\commands\daily.py, $S\larkfield_match\commands\review.py larkfield_match\commands\Copy-Item $S\larkfield_match\commands\bench.py, $S\larkfield_match\commands\privacy.py larkfield_match\commands\Copy-Item $S\larkfield_match\commands\forget.py larkfield_match\commands\Copy-Item $S\tests\test_production.py tests\Copy-Item $S\requirements.txt, $S\README.md . -
Run the tests. Module 8 adds four, for 42 in all:
Terminal window python -m pytest -q42 passed in 4.26s -
Match the new records of day 1:
Terminal window match daily --day 1Day 1: 300 new records: 143 joined an existing entity, 157 started a new entity74 pairs sent to review; 4 record(s) would join two entities (not merged alone)checked against the true customers: 139 right joins, 6 wrong joins, 28 new entities for customers we already knewThe last line uses the true answers, which only synthetic data has. A real daily run can’t print it.
-
Look at the review queue:
Terminal window match reviewReview queue: 74 pairs (unsure 70, would join two entities 4)
The next lesson, Human review, works through this queue and then runs days 2 to 7. Run day 2 only after the review of day 1, so that your outputs match the lesson.
match daily changes outputs/entities.csv, not the golden records. The golden records of Module 7 stay as they were, and the new records are not in them. A real system would also update the golden record of each entity that a new record joins.
A week of new records
Section titled “A week of new records”The course data has seven days of new records: 2,101 in all. The chart shows the whole week. After each day, the reviewer of the next lesson decided that day’s queue.
- Right joins
- New entities
- New entities for known customers
- Wrong joins (dashed)
Text description of the chart
Four lines over the days 1 to 7. Right joins: 139, 148, 142, 144, 149, 141, 130. New entities: 157, 149, 157, 155, 149, 158, 168. New entities for customers we already knew: 28, 25, 17, 26, 20, 25, 18. Wrong joins, a dashed line close to zero: 6, 3, 1, 2, 4, 1, 3. The lines are almost flat: each day looks like the others.
| Day | New records | Joined | Right joins | Wrong joins | New entities | Known, but a new entity | To review |
|---|---|---|---|---|---|---|---|
| 1 | 300 | 143 | 139 | 6 | 157 | 28 | 74 |
| 2 | 300 | 151 | 148 | 3 | 149 | 25 | 70 |
| 3 | 300 | 143 | 142 | 1 | 157 | 17 | 70 |
| 4 | 301 | 146 | 144 | 2 | 155 | 26 | 69 |
| 5 | 300 | 151 | 149 | 4 | 149 | 20 | 61 |
| 6 | 300 | 142 | 141 | 1 | 158 | 25 | 51 |
| 7 | 300 | 132 | 130 | 3 | 168 | 18 | 76 |
The check counts a record as a join when its entity has other records after the day. So a record that started a new entity, and that a later record of the same day joined, is counted too.
Read the week in three numbers:
- 993 right joins and 20 wrong joins. That is 98 right joins in 100. At the course’s assumed 150 dollars a false merge, the 20 wrong joins cost about 3,000 dollars.
- 159 missed matches. These records started a new entity, but Larkfield already knew their customer. They become double emails, at the assumed 3 dollars a year each.
- 471 pairs to review, from 51 to 76 a day. That is far below Ivana’s assumed capacity of 300 pairs a day. The daily run is a small job for her. The backlog of Module 5 was not.
Why the IDs stay stable
Section titled “Why the IDs stay stable”Open outputs/history.jsonl. Every new record has one line: new (it started an entity) or join (and the entity it joined). An entity ID changes only when a person merges two entities in review. Then the older ID stays, the other one retires, and the history says which ID replaced it.
The search and ranking course uses the same idea for a search index. It keeps the index up to date without building it again: see Keep the index fresh.
When to use incremental matching, and what it costs
Section titled “When to use incremental matching, and what it costs”- Use it when records arrive all the time and other systems use your entity IDs: orders, marketing lists, deletion requests.
- Keep a full run as a check. Once a month, match everything again in a copy, and compare it with the daily entities. The differences show drift: errors that pile up day after day. The full run only reports. It never replaces the IDs.
- Never merge two old entities without a person. One new record can be the bridge between two people, as a family email often is.
- It costs less time and more care. Each day compares only a few thousand pairs. But you must keep the model, the thresholds and the history file. Every change in them changes tomorrow’s results.
- Missed matches pile up. A record that starts a new entity by mistake stays apart until something brings it to review. Look for them from time to time: for example, review the new entities whose best pair has a probability just below 0.9.
Common mistake
Rebuild the entities from scratch every night
Symptom: TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses.’s first plan runs match golden on all the records every night, new records included. The next morning, some customers on TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s marketing list have IDs that no longer exist. Ivana’s deletion request of yesterday points to an entity that is gone.
Cause: the entity ID came from the smallest record ID. New records with smaller IDs rename entities. A cluster that gains or loses one link can rename them too. Nothing in the fresh run knew yesterday’s names.
Fix: match only the new records against the existing entities, and let each entity keep its ID. When you need a full run, give each new cluster the ID it had before. golden.entity_ids does this when it gets the previous entities. Write every join, merge and split to a history file.
Consequence in AI work: every system that gives out IDs for others to use has the same duty. Examples are topic clusters, groups of support tickets and product groups for search. Stable IDs and a history are part of the product.
Knowledge check
Section titled “Knowledge check”Check your understanding
A new record gets a probability of 0.95 with a record of entity LK-L-001200, and 0.92 with a record of entity LK-S-030001. What does the daily run do?
Show the answer
It joins the new record to LK-L-001200, the entity of its strongest link. The link to LK-S-030001 goes to the review queue, marked “would join two entities”. The two old entities stay apart. If a reviewer says that the pair is one customer, the two entities merge, and the older ID stays. If the reviewer says “different”, the pair goes to cannot_link.csv and never merges.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the week, 159 new records started a new entity although Larkfield already knew their customer. What do these errors cost, and why are they hard to find?
Show the answer
They are missed matches. At the course’s assumed 3 dollars a year each, they cost about 477 dollars a year in double emails and split loyalty points. They are hard to find because nothing points at them. A wrong join can show up when a customer complains, but a missed match is silent. You find them by measuring on labelled data, or by reviewing the best pairs below the link threshold from time to time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the project keep the ID LK-G-006348 when the new record D1-0176 joins it, although D1-0176 comes first in alphabetical order?
Show the answer
Other systems already use LK-G-006348: orders, emails and deletion requests point to it. If the ID changed every time a record joined, those links would break. So an entity keeps its ID for ever. Only a merge in review retires an ID, and the history file says which ID replaced it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Incremental matching compares only the new records, with the known ones and with each other. Each entity keeps a stable ID.
- The daily run scores each candidate pair. 0.9 or more joins the entity, 0.1 to 0.9 goes to the review queue, and no link starts a new entity.
- A new record never merges two old entities on its own. It joins its strongest link, and the other links go to review.
- A week of 2,101 synthetic records gave 993 right joins, 20 wrong joins, 159 missed matches and 471 pairs to review (51 to 76 a day).
- Every join, merge and split goes to a history file, so every change can be traced and undone.
Next, Ivana works through the review queue, and we see what her decisions do to the entities and to the model.
Next: Human review