Skip to content
Français

New records every day

  • Lesson 26
  • 40 min
  • Module 8: Run matching in production

Where we are

We matched all the records at once. Larkfield gets new records every day. In this lesson, we match new records against existing entities, keep entity IDs stable, and see what happens when one new record links two old entities.

What you will be able to do

Match each day’s new records against the customers TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield already knows, without matching everything again. Give every entity an ID that does not change. See what the daily run gets right and wrong in a week of synthetic records. Learn why one new record must never merge two old entities on its own.

NotebookMatch a day of new records in a notebookColab · Kaggle · GitHub

The notebook reads the 300 new records of day 1 and joins them to known customers by email. You count the joins, the right joins, and the new records that start a new entity although the customer is already known. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-match at the end of Module 7, with its virtual environment active, and a trained model (match train). Or use the notebook only.

A library has 60,000 cards in its catalogue. Every morning, 300 new books arrive. The librarian does not sort the whole catalogue again. She takes each new book and looks for its card.

Sometimes she finds it: a second copy of a book the library already has. She files the new copy under the same shelf number. Sometimes there is no card: the book is new, and it gets a new shelf number. Sometimes she is not sure. Two cards look alike, so she puts the book on a desk for the head librarian.

One rule matters most: a shelf number never changes. Readers, loans and reminders all use it. If it changed every night, every loan would point to the wrong book.

Larkfield has the same morning. New shop accounts and loyalty sign-ups arrive every day. Each one is a new customer or a customer Larkfield already knows.

The problem: match everything again, every night?

Section titled “The problem: match everything again, every night?”

Module 7 made entities from all 60,108 records: 38,908 of them, each with an ID such as LK-G-006348. The simple plan for new records is to add them to the pile and run the whole matcher again.

This plan has two problems.

It does work that is already done. Every night, it compares again more than a million candidate pairs that it compared yesterday. On Larkfield’s records, that takes less than a minute. But records grow every day, and Scale shows how fast that time grows.

The IDs move. An entity’s ID is LK- and the smallest record ID in it. New records have IDs such as D1-0176. In alphabetical order, D1-… comes before G-…, L-… and S-…. So a fresh run renames every entity that a new record joins. If D1-0176 joins LK-G-006348, a fresh run calls the entity LK-D1-0176. Orders, emails and deletion requests that point to LK-G-006348 now point to nothing.

A matcher that runs every day needs two things: compare only what is new, and keep the old names.

The idea: incremental matching with stable IDs

Section titled “The idea: incremental matching with stable IDs”

Incremental matching means matching only the new records: against the records that are already matched, and against each other. Each entity keeps its ID for ever. We call that a stable ID.

For each new record, the daily run does four things:

  1. Find candidates. Make candidate pairs with the five blocking keys of Module 3: postcode, email, phone, the surname key and the date of birth key. Each pair has at least one new record.
  2. Score. Compare each pair field by field and score it with the LightGBM model of Module 6. Then calibrate the score, so that 0.3 means “about 3 in 10 such pairs are true pairs”.
  3. Decide. A pair with a probability of 0.9 or more is a link. A pair from 0.1 to 0.9 goes to the review queue: a list of pairs for a person to check. Below 0.1, nothing happens.
  4. Join or start. A record with a link joins the entity of the linked record, and that entity keeps its ID. A record without a link starts a new entity: LK- and its own record ID.

Every step goes into a history file, outputs/history.jsonl, so every change can be traced and undone.

The daily run. Day 1 brings 300 new records. Candidate pairs are made for each new record against every known record with the five blocking keys. LightGBM scores each pair, and the score is calibrated. Then there are three outcomes. A probability of 0.9 or more: the record joins that entity, whose ID does not change; on day 1, 143 records. A probability from 0.1 to 0.9: the pair goes to the review queue; on day 1, 74 pairs. No link: the record starts a new entity named LK and its record ID; on day 1, 157 records. A note: a record with links to two entities joins the strongest one, and the other link goes to review; on day 1, 4 records. A new record never merges two old entities on its own.
The daily run of the project: candidates, scores, three outcomes. The counts are day 1 of the synthetic records.
Text description of the diagram

The diagram is a flow from top to bottom. The top box is day 1: 300 new records. An arrow leads to the candidate pairs: each new record against every known record, with the five blocking keys. The next box scores each pair with LightGBM and calibrates the probability. Three arrows lead to three outcomes. Green: probability 0.9 or more, the record joins that entity, and the entity’s ID does not change; 143 records on day 1. Yellow: probability 0.1 to 0.9, the pair goes to the review queue; 74 pairs on day 1. Blue: no link, a new entity named LK and the record ID; 157 records on day 1. A yellow box below says that a record with links to two entities joins the strongest one. The other link goes to review (4 records on day 1). A note says that a new record never merges two old entities on its own.

Here are four real new records of day 1, and what the daily run did with them. Like every record in this course, they are synthetic.

New recordKnown records it was compared withWhat the run didTrue answer
D1-0003, Marie Tremblay, (450) 555-4557S-030041, Marie Tremblayjoined LK-S-030041right: the same customer
D1-0001, Ahmed Novak, J7P 2A9no linkstarted LK-D1-0001right: a new customer
D1-0049, Mary Walker, P3Z 1G8, mwalker@post.exampleD1-0047, Mary Walker, P3Z1G8, walker.mary87@post.examplejoined LK-D1-0047wrong: two women with one name at one postcode
D1-0176, Melanie Roy, melanie.roy@example.comtwo entities: LK-G-006348 and LK-S-019824joined LK-G-006348; the second link went to reviewone person in all three

The first two rows are the usual cases. The third row is a wrong join, a false merge. Both Mary Walkers arrived on day 1, and the second one joined the first one’s new entity: same name, same postcode, different emails. The fourth row needs its own section.

Look at Melanie Roy’s new loyalty record, D1-0176. Its phone number is the phone of an old garden-centre record. Its email is the email of an old shop record. But those two old records were in two different entities. Module 7 did not join them: their phone numbers and their postcodes differ, and the garden-centre record has no email.

The new loyalty record D1-0176, Melanie Roy, email melanie.roy@example.com, phone (819) 555-4445, postcode G1R 8V0, born 13/09/1970. Its strongest link goes to entity LK-G-006348: the garden-centre record G-006348, ROY, MELANIE, phone 8195554445. Its second link goes to entity LK-S-019824: the shop record S-019824, Mélanie Roy, email melanie.roy@example.com, phone (819) 555-4702, postcode G9Y 7M2, born 1970-09-13. The new record joins LK-G-006348. The link to LK-S-019824 goes to the review queue. The two old entities stay apart until a person decides.
One new record links two old entities. The run joins the strongest link and sends the other one to a person.
Text description of the diagram

At the top is the new loyalty record D1-0176: Melanie Roy, email melanie.roy@example.com, phone (819) 555-4445, postcode G1R 8V0, born 13/09/1970. Two arrows go down. The left arrow, “strongest link”, goes to the green entity LK-G-006348. It has the garden-centre record G-006348, ROY, MELANIE, phone 8195554445. The right arrow, “second link”, goes to the yellow entity LK-S-019824. It has the shop record S-019824, Mélanie Roy, email melanie.roy@example.com, phone (819) 555-4702. A note says: the new record joins LK-G-006348, the link to LK-S-019824 goes to the review queue, and the two old entities stay apart until a person decides.

What should the run do? If all three records are one person, the two old entities should merge. Here, the truth says they are: one Melanie Roy, with an old phone number in one record.

But a family can make the same picture. Think of a son who signs up with the family email and his own phone. The email links him to his mother’s entity. The phone links him to his own old entity. A merge would join a mother and her son into one customer.

The daily run can’t tell these two stories apart. So it follows a firm rule: a new record never merges two old entities on its own. It joins the entity of its strongest link. Every other link goes to the review queue, marked “would join two entities”. A person decides. On day 1, 4 records were in this case. In Melanie Roy’s case, the reviewer said “same”, and the two entities merged into {two.kept}.

This module has four lessons. You copy all their files now, and each lesson uses its own commands.

  1. If you ran match reset after Module 7, make the entities again first. The command uses the saved LightGBM pairs, or trains the model if they are missing:

    Terminal window
    match golden
    38,908 entities from 60,108 records (16,513 with more than one record); 0 with two dates of birth
    Saved outputs/entities.csv and outputs/golden.csv
  2. In your project folder, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files of the snapshot end-of-m08. Its requirements.txt has no new package:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m08
    cp $S/larkfield_match/incremental.py $S/larkfield_match/review.py larkfield_match/
    cp $S/larkfield_match/bench.py $S/larkfield_match/privacy.py larkfield_match/
    cp $S/larkfield_match/runs/private.py larkfield_match/runs/
    cp $S/larkfield_match/commands/daily.py $S/larkfield_match/commands/review.py larkfield_match/commands/
    cp $S/larkfield_match/commands/bench.py $S/larkfield_match/commands/privacy.py larkfield_match/commands/
    cp $S/larkfield_match/commands/forget.py larkfield_match/commands/
    cp $S/tests/test_production.py tests/
    cp $S/requirements.txt $S/README.md .
  3. Run the tests. Module 8 adds four, for 42 in all:

    Terminal window
    python -m pytest -q
    42 passed in 4.26s
  4. Match the new records of day 1:

    Terminal window
    match daily --day 1
    Day 1: 300 new records: 143 joined an existing entity, 157 started a new entity
    74 pairs sent to review; 4 record(s) would join two entities (not merged alone)
    checked against the true customers: 139 right joins, 6 wrong joins, 28 new entities for customers we already knew

    The last line uses the true answers, which only synthetic data has. A real daily run can’t print it.

  5. Look at the review queue:

    Terminal window
    match review
    Review queue: 74 pairs (unsure 70, would join two entities 4)

The next lesson, Human review, works through this queue and then runs days 2 to 7. Run day 2 only after the review of day 1, so that your outputs match the lesson.

match daily changes outputs/entities.csv, not the golden records. The golden records of Module 7 stay as they were, and the new records are not in them. A real system would also update the golden record of each entity that a new record joins.

The course data has seven days of new records: 2,101 in all. The chart shows the whole week. After each day, the reviewer of the next lesson decided that day’s queue.

  • Right joins
  • New entities
  • New entities for known customers
  • Wrong joins (dashed)
What the daily run did with each day's new records, checked against the true customers of the synthetic data. Measured. Week totals: 993 right joins, 20 wrong joins, 1,093 new entities, 159 of them for customers that Larkfield already knew.
Text description of the chart

Four lines over the days 1 to 7. Right joins: 139, 148, 142, 144, 149, 141, 130. New entities: 157, 149, 157, 155, 149, 158, 168. New entities for customers we already knew: 28, 25, 17, 26, 20, 25, 18. Wrong joins, a dashed line close to zero: 6, 3, 1, 2, 4, 1, 3. The lines are almost flat: each day looks like the others.

DayNew recordsJoinedRight joinsWrong joinsNew entitiesKnown, but a new entityTo review
130014313961572874
230015114831492570
330014314211571770
430114614421552669
530015114941492061
630014214111582551
730013213031681876

The check counts a record as a join when its entity has other records after the day. So a record that started a new entity, and that a later record of the same day joined, is counted too.

Read the week in three numbers:

  • 993 right joins and 20 wrong joins. That is 98 right joins in 100. At the course’s assumed 150 dollars a false merge, the 20 wrong joins cost about 3,000 dollars.
  • 159 missed matches. These records started a new entity, but Larkfield already knew their customer. They become double emails, at the assumed 3 dollars a year each.
  • 471 pairs to review, from 51 to 76 a day. That is far below Ivana’s assumed capacity of 300 pairs a day. The daily run is a small job for her. The backlog of Module 5 was not.

Open outputs/history.jsonl. Every new record has one line: new (it started an entity) or join (and the entity it joined). An entity ID changes only when a person merges two entities in review. Then the older ID stays, the other one retires, and the history says which ID replaced it.

The search and ranking course uses the same idea for a search index. It keeps the index up to date without building it again: see Keep the index fresh.

When to use incremental matching, and what it costs

Section titled “When to use incremental matching, and what it costs”
  • Use it when records arrive all the time and other systems use your entity IDs: orders, marketing lists, deletion requests.
  • Keep a full run as a check. Once a month, match everything again in a copy, and compare it with the daily entities. The differences show drift: errors that pile up day after day. The full run only reports. It never replaces the IDs.
  • Never merge two old entities without a person. One new record can be the bridge between two people, as a family email often is.
  • It costs less time and more care. Each day compares only a few thousand pairs. But you must keep the model, the thresholds and the history file. Every change in them changes tomorrow’s results.
  • Missed matches pile up. A record that starts a new entity by mistake stays apart until something brings it to review. Look for them from time to time: for example, review the new entities whose best pair has a probability just below 0.9.

Common mistake

Rebuild the entities from scratch every night

Symptom: TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses.’s first plan runs match golden on all the records every night, new records included. The next morning, some customers on TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s marketing list have IDs that no longer exist. Ivana’s deletion request of yesterday points to an entity that is gone.

Cause: the entity ID came from the smallest record ID. New records with smaller IDs rename entities. A cluster that gains or loses one link can rename them too. Nothing in the fresh run knew yesterday’s names.

Fix: match only the new records against the existing entities, and let each entity keep its ID. When you need a full run, give each new cluster the ID it had before. golden.entity_ids does this when it gets the previous entities. Write every join, merge and split to a history file.

Consequence in AI work: every system that gives out IDs for others to use has the same duty. Examples are topic clusters, groups of support tickets and product groups for search. Stable IDs and a history are part of the product.

Check your understanding

A new record gets a probability of 0.95 with a record of entity LK-L-001200, and 0.92 with a record of entity LK-S-030001. What does the daily run do?

Show the answer

It joins the new record to LK-L-001200, the entity of its strongest link. The link to LK-S-030001 goes to the review queue, marked “would join two entities”. The two old entities stay apart. If a reviewer says that the pair is one customer, the two entities merge, and the older ID stays. If the reviewer says “different”, the pair goes to cannot_link.csv and never merges.

Check your understanding

In the week, 159 new records started a new entity although Larkfield already knew their customer. What do these errors cost, and why are they hard to find?

Show the answer

They are missed matches. At the course’s assumed 3 dollars a year each, they cost about 477 dollars a year in double emails and split loyalty points. They are hard to find because nothing points at them. A wrong join can show up when a customer complains, but a missed match is silent. You find them by measuring on labelled data, or by reviewing the best pairs below the link threshold from time to time.

Check your understanding

Why does the project keep the ID LK-G-006348 when the new record D1-0176 joins it, although D1-0176 comes first in alphabetical order?

Show the answer

Other systems already use LK-G-006348: orders, emails and deletion requests point to it. If the ID changed every time a record joined, those links would break. So an entity keeps its ID for ever. Only a merge in review retires an ID, and the history file says which ID replaced it.

Recap

  • Incremental matching compares only the new records, with the known ones and with each other. Each entity keeps a stable ID.
  • The daily run scores each candidate pair. 0.9 or more joins the entity, 0.1 to 0.9 goes to the review queue, and no link starts a new entity.
  • A new record never merges two old entities on its own. It joins its strongest link, and the other links go to review.
  • A week of 2,101 synthetic records gave 993 right joins, 20 wrong joins, 159 missed matches and 471 pairs to review (51 to 76 a day).
  • Every join, merge and split goes to a history file, so every change can be traced and undone.

Next, Ivana works through the review queue, and we see what her decisions do to the entities and to the model.

Next: Human review