Skip to content
Français

Human review

  • Lesson 27
  • 40 min
  • Module 8: Run matching in production

Where we are

The daily run leaves an unsure band of pairs. A person must check them. In this lesson, we build the review queue, show what Ivana sees, turn her decisions into labels, and let her capacity set the thresholds.

What you will be able to do

Give the unsure pairs to a person. See what the reviewer needs on her screen. See what each decision does to the entities: a merge, a “never merge”, or a split that gives a record a new ID. Then use the decisions as training labels, and see why they made TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s model worse at first.

NotebookReview pairs and retrain in a notebookColab · Kaggle · GitHub

The notebook trains a small LightGBM model, builds a review queue from its unsure pairs, and lets a simulated reviewer decide 471 of them. You add her labels to the training set and measure the model before and after, then with a smaller weight for the review labels. Its setup cells install the packages and download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project after match daily --day 1 of the last lesson, with its virtual environment active. Or use the notebook only.

The clinic’s list of possible duplicates

Section titled “The clinic’s list of possible duplicates”

Every evening, a clinic’s computer prints a short list: pairs of patient cards that may be the same patient. In the morning, a records clerk checks each pair. She looks at both cards side by side, and she can phone a patient if she must.

She can check about forty pairs before the clinic opens. If the list has four hundred pairs, she never finishes, and the last pairs wait for weeks.

Her decisions do two jobs. Today, they join two cards or keep them apart. Later, they also teach. Each decision is a labelled pair, and the computer can learn from it. But she only sees the hard pairs. The easy ones never reach her desk.

At Larkfield, the clerk is Ivana, the customer-data steward. The list is the review queue.

The problem: a queue that a person must be able to finish

Section titled “The problem: a queue that a person must be able to finish”

In From weights to a decision, the review band of Splink’s weights 10 to 20 held 13,378 pairs of the whole backlog. That is 44.6 working days for Ivana, at the assumed 300 pairs a day. The daily run is a different job: 51 to 76 new pairs a day. The size of the band decides whether a person can keep up.

A review band has two edges, and each edge has a price:

  • Widen it, and fewer errors pass without a person. But the queue grows.
  • Narrow it, and Ivana finishes early. But more false merges and missed matches pass through without anyone looking.

So the reviewer’s capacity is a limit for the thresholds, as real as the model’s precision. The explorer below is the one from Module 5, with the backlog of all 60,108 records. It starts at thresholds 12 and 20.

Explore

How many days of review do the thresholds make?

Match band
21,523 pairsprecision 0.916
Review band
6,137 pairs2,830 true pairs
Days of review
20.5 days
If every review is right
recall 0.814, precision 0.926

Thresholds 12 and 20: 21,523 pairs merge without review (precision 0.916). 6,137 pairs go to review, and 2,830 of them are true pairs. One reviewer at 300 pairs a day needs 20.5 working days. If every review is right: recall 0.814, precision 0.926. That is more than the 20 days of review that Ivana can give.

Days of review for every pair of thresholds (one reviewer, 300 pairs a day). Under each number: the pairs in the review band.
Lower ↓ / upper →1618202225
5178.453,524182.054,614187.556,264197.259,150212.363,688
868.320,49872.021,58877.523,23887.126,124102.230,662
1035.510,63839.111,72844.613,37854.216,26469.320,802
1211.33,39715.04,48720.56,13730.19,02345.213,561
143.91,1607.52,25013.03,90022.66,78637.711,324
Splink's match weights for every candidate pair of the 60,108 synthetic records (all splits), measured. Ivana's 300 pairs a day is the course's assumption; the 20 working days for the backlog are an assumption of Module 5.
Text description of the explorer

The explorer has two sliders. The lower threshold takes the values 5, 8, 10, 12 and 14; the upper threshold 16, 18, 20, 22 and 25. They start at 12 and 20. Four tiles show the match band (21,523 pairs, precision 0.916), the review band (6,137 pairs, 2,830 of them true pairs), the days of review (20.5) and the result if every review is right (recall 0.814, precision 0.926). A sentence repeats these numbers and says whether the review fits in 20 days: at 20.5 days, it just does not. A bar chart shows the days of review for each lower threshold at the chosen upper threshold, with a dashed line at 20 days. A table lists the days and the review pairs for all 25 pairs of thresholds. With an upper threshold of 20: lower 5 gives 187.5 days; 8, 77.5; 10, 44.6; 12, 20.5; 14, 13.0.

Thresholds 12 and 20 need 20.5 days: half a day too many. Thresholds 14 and 20 fit, at 13.0 days. The daily run uses the band 0.1 to 0.9 of LightGBM’s calibrated probability. It gives Ivana about 70 pairs a day, so she has room to spare. If the queue grows, the thresholds must change, or Larkfield needs a second reviewer.

A good review screen shows the two records side by side, field by field, with the differences marked. It also says why the pair is in the queue. Ivana should not need to search other systems for each pair.

  1. Show the three pairs at the top of the queue:

    Terminal window
    match review --show 3
    Review queue: 74 pairs (unsure 70, would join two entities 4)
    D1-0282 / S-006406 probability 1.00 (would join two entities)
    name Richard Johnson Richard Johnson
    email johnson.richard81@example.net johnson.richard81@example.net
    ≠ phone 4165555430 (416) 555-5430
    ≠ address 868 Hawthorn Pl. #22, Wrenhollow ON 868 Hawthorn Pl Apt 22, Wrenhollow ON
    postcode K4N 7N8 K4N 7N8
    ≠ date_of_birth - 1937-01-06
    D1-0209 / S-004695 probability 0.92 (would join two entities)
    ≠ name Emma Smith Emma Simth
    ≠ email emma.smith22@mail.example emma.smith36@post.example
    ≠ phone - 1-250-555-3426
    address 9785 Orchard Place, Alderholm BC 9785 Orchard Place, Alderholm BC
    postcode V9M 2A1 V9M 2A1
    ≠ date_of_birth - 2006-07-24
    D1-0176 / S-019824 probability 0.92 (would join two entities)
    ≠ name Melanie Roy Mélanie Roy
    email melanie.roy@example.com melanie.roy@example.com
    ≠ phone (819) 555-4445 (819) 555-4702
    ≠ address - 4609 boul. de la Colline, Val-des-Hérons QC
    ≠ postcode G1R 8V0 G9Y 7M2
    ≠ date_of_birth 13/09/1970 1970-09-13

    ≠ marks a field whose values differ as written. Many marks are only formats: the phone 4165555430 and (416) 555-5430 are one number.

  2. Let the simulated reviewer decide the day’s queue:

    Terminal window
    match review --decide
    Review queue: 74 pairs (unsure 70, would join two entities 4)
    Decided 74 pairs: 33 same, 41 different (1 reviewer mistakes, simulated); 0 left in the queue

The first pair shows why a person helps. Richard Johnson’s two old records were two entities, because their dates of birth are 06/01/1973 and 1937-01-06. Safe clustering of Safer clustering keeps two dates of birth apart. A person sees at once that the digits of the year are swapped. The model can’t.

In this course, the reviewer is simulated: she decides from the true answer, and she makes a mistake on 2% of the pairs. That rate is our assumption. A real reviewer’s error rate must be measured, for example by giving two reviewers the same pairs.

Each decision changes the entities, and it is also kept as a label.

The review loop. The review queue holds 471 pairs in the week, 51 to 76 a day. Ivana sees both records field by field, with the differences marked. A decision "same" merges the two entities, the older ID stays, and the merge is written to history.jsonl. A decision "different" writes the pair to cannot_link.csv, and it never merges again. Every decision is also a label in review_labels.csv: 231 same and 240 different in the week. A last box: use the labels with care, with a separate test and a smaller weight.
What a review decision does: it changes the entities today and becomes a training label later.
Text description of the diagram

The diagram starts with the review queue: 471 pairs in the week, 51 to 76 a day. An arrow leads to Ivana’s screen: both records, field by field, with the differences marked. Two arrows leave it. “Same” (green) merges the two entities; the older ID stays; the merge is written to history.jsonl. “Different” (red) writes the pair to cannot_link.csv, so it never merges again. Below, a blue box: every decision is also a label in review_labels.csv, 231 same and 240 different in the week. An arrow leads to a yellow box: use the labels with care, with a separate test and a smaller weight.

  • “Same” merges the two entities. The older entity keeps its ID, and the history says which ID retired.
  • “Different” goes to outputs/cannot_link.csv. Later runs read that file: safe clustering never joins the two records again, even if the model scores them high.
  • Every decision goes to outputs/review_labels.csv, with who decided.

Sometimes a reviewer finds a wrong merge that is already done. Then she splits it: one record leaves its entity. Here is a wrong join of day 1. The new shop record of Marie Ève Bérubé, D1-0184, joined the entity of Marie-Claude Bérubé through a shared family email, mberube@post.example. They are two women.

Terminal window
match undo D1-0184
D1-0184 is now entity LK-D1-0184; the other 2 record(s) stay in LK-G-002767.
Recorded in outputs/history.jsonl and outputs/cannot_link.csv (safe clustering will keep them apart).
Entity IDs before and after a split. Before: entity LK-G-002767 with the records D1-0184, G-002767, L-004561. The command match undo D1-0184. After: entity LK-G-002767 with the records G-002767, L-004561, and a new entity LK-D1-0184 with the record D1-0184. cannot_link.csv gets D1-0184 with each record of LK-G-002767. history.jsonl gets a split event from LK-G-002767 to LK-D1-0184.
A split: the record that was merged by mistake gets a new entity ID. The other records keep the old one.
Text description of the diagram

On the left, before: entity LK-G-002767 with the records D1-0184, G-002767, L-004561. An arrow labelled “match undo D1-0184” leads to the right side, after. Entity LK-G-002767 still has the records G-002767, L-004561. A new entity, LK-D1-0184, has only the record D1-0184. A note says that cannot_link.csv now has D1-0184 with each record of LK-G-002767, and that history.jsonl has a split event from LK-G-002767 to LK-D1-0184.

This is the rule for an entity ID in a split: the records that stay keep the old ID, and the record that leaves gets a new one. Other systems that point to {sp.old_after} still point to Marie-Claude Bérubé. One case is different: when the entity was named after the leaving record, the leaving record keeps the name. The others then get a new one, LK- and their smallest record ID. Either way, the history says which records got which ID.

The week of review gave 471 labelled pairs: 231 “same” and 240 “different”, with 12 simulated mistakes. They are free training data. TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails.’s plan sounds obvious: add them to the 3,000 labelled pairs and train again.

  1. Run the six other days. Each day, the reviewer decides that day’s queue:

    Terminal window
    for d in 2 3 4 5 6 7; do match daily --day $d; match review --decide; done
    Day 2: 300 new records: 151 joined an existing entity, 149 started a new entity
    70 pairs sent to review; 2 record(s) would join two entities (not merged alone)
    checked against the true customers: 148 right joins, 3 wrong joins, 25 new entities for customers we already knew
    Review queue: 70 pairs (unsure 68, would join two entities 2)
    Decided 70 pairs: 36 same, 34 different (3 reviewer mistakes, simulated); 0 left in the queue
    …
    Day 7: 300 new records: 132 joined an existing entity, 168 started a new entity
    76 pairs sent to review; 4 record(s) would join two entities (not merged alone)
    checked against the true customers: 130 right joins, 3 wrong joins, 18 new entities for customers we already knew
    Review queue: 76 pairs (unsure 72, would join two entities 4)
    Decided 76 pairs: 37 same, 39 different (1 reviewer mistakes, simulated); 0 left in the queue
  2. Retrain LightGBM with the review labels, and compare it with the model of the 3,000 labels on the test split:

    Terminal window
    match review --retrain
    Review queue: 0 pairs
    lgbm without review labels: test precision 0.918, recall 0.886, F1 0.902
    lgbm with 471 review labels: test precision 0.880, recall 0.875, F1 0.878
    Saved the retrained model in outputs/models/lgbm-review.pkl (outputs/models/lgbm.pkl is unchanged)

    The retrained model goes to a file of its own. The model of the 3,000 labels stays in outputs/models/lgbm.pkl, so match daily and the other commands still use it.

More labels, and the model is worse: F1 falls from 0.902 to 0.878, and precision falls most, from 0.918 to 0.880. This is what we found. It was not what we expected.

The review labels are not like the 3,000 labelled pairs. They are a different sample: a different choice of pairs.

  • They are only the hardest pairs. Every review pair was a pair the model was unsure about. The 3,000 pairs were sampled by surname and region, by random choice, and from the true pairs: easy and hard cases together.
  • They change the balance. The 3,000 pairs are 35% “same”. The review pairs are 49% “same”. Our reading, which we did not test: in the review pairs, “same” often looks like a family (shared address and surname, different first name). The model learns that this look often means “same”, and it merges more families. That fits the fall in precision.
  • They carry the reviewer’s mistakes. 12 of the 471 labels are wrong.

We tested each reason with new runs, on the same test split:

LightGBM trained on the 3,000 labelled pairs plus the 471 review labels with different weights, measured on the test split. Weight 1 is what match review --retrain does. The axis starts at 0.85, so small differences look large.
Text description of the chart

A column chart of F1 on an axis from 0.85 to 0.91. No review labels: 0.902. Review labels with weight 0.1: 0.904; weight 0.25: 0.9; weight 0.5: 0.904. Weight 1: 0.878. Weight 1 with the reviewer’s mistakes corrected: 0.888.

Two things stand out:

  • The mistakes explain only part of the fall. With every mistake corrected, F1 is 0.888: still below 0.902. The bigger part comes from which pairs the labels describe.
  • A smaller weight removes the harm. A weight says how much one label counts in training. With each review label counted as a quarter or a half of a normal label, F1 stays at 0.9 to 0.904.

So should we leave the review labels out? Not yet. The test split measures the common pairs. The review pairs are the pairs that matter most for Ivana. We need a separate test: hard pairs that the model has not seen.

We trained on the review labels of days 1 to 5 only, and tested on the 127 review pairs of days 6 and 7, with their true answers:

ModelRight on the hard pairs of days 6 and 7Called “same”
no review labels49.6%127 of 127 (true: 63)
review labels of days 1-5, weight 181.9%66 of 127 (true: 63)
review labels of days 1-5, weight 0.2581.9%70 of 127 (true: 63)

Without review labels, the model calls all 127 hard pairs “same”, and it is right on 49.6%: no better than a coin. With the review labels, it is right on 81.9%. The review labels teach exactly what the model did not know. They also bend its picture of the common pairs.

The lesson: review labels need their own weight and their own test. Measure every retrained model twice: on the test split, and on new review pairs it has not seen. Keep a model only if it is not worse on either. The direction is not fixed. The notebook has a smaller model, and its review batch is as balanced as the 3,000 labels. There, the same step raised F1 from 0.854 to 0.876. Measure it every time. The machine learning course explains why a sample that is not like the real data misleads a model, in Train, validation and test sets.

When to use human review, and what it costs

Section titled “When to use human review, and what it costs”
  • Use it where an error is expensive and a person can decide better than the model. A false merge at Larkfield costs an assumed 150 dollars; one review costs about 0.80 dollars (1.2 minutes at 40 dollars an hour).
  • Size the band by the reviewer’s capacity. Ask how many pairs a day, and how many days. Change the thresholds when the queue grows.
  • Design the screen. Both records, field by field, the differences marked, and the reason for the pair. Mark format-only differences less loudly than real ones.
  • Measure the reviewer too. Give some pairs to two reviewers, or put pairs with a known answer in the queue. A tired reviewer makes more mistakes late in the day.
  • Keep every decision. “Different” must stop the pair for ever, and a split must keep the old ID with the records that stay.

Common mistake

Retrain on every review label at full weight

Symptom: after a week, Tomás adds the 471 review labels to the training set and retrains. The test F1 falls from 0.902 to 0.878. More families are merged than before.

Cause: the review pairs are the hardest pairs only, and 49% of them are “same”, against 35% in the 3,000 labelled pairs. At full weight, they change what the model thinks a normal pair looks like. The reviewer’s mistakes add a little more harm.

Fix: give the review labels a smaller weight: 0.25 to 0.5 worked here. Test every retrained model on the test split, and on new review pairs that it has not seen.

Consequence in AI work: people who check a model’s unsure cases make a biased sample of labels. This is true in every field: flagged transactions, moderated posts, corrected answers of a language model. They are valuable, and they need their own weight and their own test.

Check your understanding

A reviewer finds that record S-001234 was merged by mistake into entity LK-G-000777, which has three other records. She splits it. What happens to the entity IDs, and where can you see it in the data?

Show the answer

The three other records stay in LK-G-000777: their ID does not change, so systems that point to it still point to the same customer. S-001234 gets a new entity, LK-S-001234. In the data, outputs/entities.csv shows the new entity. outputs/cannot_link.csv has S-001234 with each of the three records, so they never merge again. outputs/history.jsonl has a “split” event from LK-G-000777 to LK-S-001234. One case is different: if the entity had been named after S-001234, S-001234 would keep that name. The other three would get a new one, and the history would say so.

Check your understanding

The review queue grows to 450 pairs a day, and Ivana can review 300. Name two ways to fix it, and the price of each.

Show the answer

Narrow the review band: raise the lower threshold or lower the upper one. The price: more pairs are decided without a person, so more missed matches (below the band) or more false merges (above it). Or add review time: a second reviewer, or longer review days. The price is about 0.80 dollars a pair (assumed). A third way is a better model, which makes fewer pairs unsure; it costs development time.

Check your understanding

With 471 review labels, the test F1 fell from 0.902 to 0.878. But on new hard pairs, the retrained model was right far more often. Which model should Larkfield use?

Show the answer

Neither as it is. The retrained model at full weight is worse on the common pairs, which are most of the daily work. The old model fails the hard pairs, which go to Ivana anyway. Train with the review labels at a smaller weight, for example 0.25. Here, the test F1 stayed at 0.900, and the model was right on 81.9% of the new hard pairs. Measure both before you switch.

Recap

  • The review queue holds the pairs that the model is unsure about. Its size must fit the reviewer’s capacity: here, 51 to 76 pairs a day against an assumed 300.
  • The reviewer needs both records side by side, the differences marked, and the reason for the pair.
  • “Same” merges and the older ID stays. “Different” goes to cannot_link.csv. A split gives the leaving record a new ID, and the others keep theirs.
  • At full weight, the 471 review labels made LightGBM worse: F1 0.902 → 0.878. They are only the hardest pairs and change the balance. A weight of 0.25 to 0.5 kept F1 at 0.900 to 0.904.
  • Test a retrained model on the test split and on new review pairs.

Next, we leave Larkfield’s 60,000 records and ask what happens at 100,000 and 1 million.

Next: Scale