Human review
- Lesson 27
- 40 min
- Module 8: Run matching in production
Where we are
The daily run leaves an unsure band of pairs. A person must check them. In this lesson, we build the review queue, show what Ivana sees, turn her decisions into labels, and let her capacity set the thresholds.
Lesson 27 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Human review
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Give the unsure pairs to a person. See what the reviewer needs on her screen. See what each decision does to the entities: a merge, a “never merge”, or a split that gives a record a new ID. Then use the decisions as training labels, and see why they made TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s model worse at first.
NotebookReview pairs and retrain in a notebook
The notebook trains a small LightGBM model, builds a review queue from its unsure pairs, and lets a simulated reviewer decide 471 of them. You add her labels to the training set and measure the model before and after, then with a smaller weight for the review labels. Its setup cells install the packages and download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The daily run and its review queue, from New records every day.
- Two thresholds and the review band, from From weights to a decision.
- How a classifier learns from labelled pairs, from A supervised pair classifier.
What you need on your computer
- The project after
match daily --day 1of the last lesson, with its virtual environment active. Or use the notebook only.
The clinic’s list of possible duplicates
Section titled “The clinic’s list of possible duplicates”Every evening, a clinic’s computer prints a short list: pairs of patient cards that may be the same patient. In the morning, a records clerk checks each pair. She looks at both cards side by side, and she can phone a patient if she must.
She can check about forty pairs before the clinic opens. If the list has four hundred pairs, she never finishes, and the last pairs wait for weeks.
Her decisions do two jobs. Today, they join two cards or keep them apart. Later, they also teach. Each decision is a labelled pair, and the computer can learn from it. But she only sees the hard pairs. The easy ones never reach her desk.
At Larkfield, the clerk is Ivana, the customer-data steward. The list is the review queue.
The problem: a queue that a person must be able to finish
Section titled “The problem: a queue that a person must be able to finish”In From weights to a decision, the review band of Splink’s weights 10 to 20 held 13,378 pairs of the whole backlog. That is 44.6 working days for Ivana, at the assumed 300 pairs a day. The daily run is a different job: 51 to 76 new pairs a day. The size of the band decides whether a person can keep up.
A review band has two edges, and each edge has a price:
- Widen it, and fewer errors pass without a person. But the queue grows.
- Narrow it, and Ivana finishes early. But more false merges and missed matches pass through without anyone looking.
So the reviewer’s capacity is a limit for the thresholds, as real as the model’s precision. The explorer below is the one from Module 5, with the backlog of all 60,108 records. It starts at thresholds 12 and 20.
Explore
How many days of review do the thresholds make?
Predict first: With thresholds 12 and 20, does the backlog fit in Ivana's 20 days?
- Match band
- 21,523 pairsprecision 0.916
- Review band
- 6,137 pairs2,830 true pairs
- Days of review
- 20.5 days
- If every review is right
- recall 0.814, precision 0.926
Thresholds 12 and 20: 21,523 pairs merge without review (precision 0.916). 6,137 pairs go to review, and 2,830 of them are true pairs. One reviewer at 300 pairs a day needs 20.5 working days. If every review is right: recall 0.814, precision 0.926. That is more than the 20 days of review that Ivana can give.
| Lower ↓ / upper → | 16 | 18 | 20 | 22 | 25 |
|---|---|---|---|---|---|
| 5 | 178.453,524 | 182.054,614 | 187.556,264 | 197.259,150 | 212.363,688 |
| 8 | 68.320,498 | 72.021,588 | 77.523,238 | 87.126,124 | 102.230,662 |
| 10 | 35.510,638 | 39.111,728 | 44.613,378 | 54.216,264 | 69.320,802 |
| 12 | 11.33,397 | 15.04,487 | 20.56,137 | 30.19,023 | 45.213,561 |
| 14 | 3.91,160 | 7.52,250 | 13.03,900 | 22.66,786 | 37.711,324 |
Text description of the explorer
The explorer has two sliders. The lower threshold takes the values 5, 8, 10, 12 and 14; the upper threshold 16, 18, 20, 22 and 25. They start at 12 and 20. Four tiles show the match band (21,523 pairs, precision 0.916), the review band (6,137 pairs, 2,830 of them true pairs), the days of review (20.5) and the result if every review is right (recall 0.814, precision 0.926). A sentence repeats these numbers and says whether the review fits in 20 days: at 20.5 days, it just does not. A bar chart shows the days of review for each lower threshold at the chosen upper threshold, with a dashed line at 20 days. A table lists the days and the review pairs for all 25 pairs of thresholds. With an upper threshold of 20: lower 5 gives 187.5 days; 8, 77.5; 10, 44.6; 12, 20.5; 14, 13.0.
Thresholds 12 and 20 need 20.5 days: half a day too many. Thresholds 14 and 20 fit, at 13.0 days. The daily run uses the band 0.1 to 0.9 of LightGBM’s calibrated probability. It gives Ivana about 70 pairs a day, so she has room to spare. If the queue grows, the thresholds must change, or Larkfield needs a second reviewer.
What the reviewer sees
Section titled “What the reviewer sees”A good review screen shows the two records side by side, field by field, with the differences marked. It also says why the pair is in the queue. Ivana should not need to search other systems for each pair.
-
Show the three pairs at the top of the queue:
Terminal window match review --show 3Review queue: 74 pairs (unsure 70, would join two entities 4)D1-0282 / S-006406 probability 1.00 (would join two entities)name Richard Johnson Richard Johnsonemail johnson.richard81@example.net johnson.richard81@example.net≠ phone 4165555430 (416) 555-5430≠ address 868 Hawthorn Pl. #22, Wrenhollow ON 868 Hawthorn Pl Apt 22, Wrenhollow ONpostcode K4N 7N8 K4N 7N8≠ date_of_birth - 1937-01-06D1-0209 / S-004695 probability 0.92 (would join two entities)≠ name Emma Smith Emma Simth≠ email emma.smith22@mail.example emma.smith36@post.example≠ phone - 1-250-555-3426address 9785 Orchard Place, Alderholm BC 9785 Orchard Place, Alderholm BCpostcode V9M 2A1 V9M 2A1≠ date_of_birth - 2006-07-24D1-0176 / S-019824 probability 0.92 (would join two entities)≠ name Melanie Roy Mélanie Royemail melanie.roy@example.com melanie.roy@example.com≠ phone (819) 555-4445 (819) 555-4702≠ address - 4609 boul. de la Colline, Val-des-Hérons QC≠ postcode G1R 8V0 G9Y 7M2≠ date_of_birth 13/09/1970 1970-09-13≠marks a field whose values differ as written. Many marks are only formats: the phone4165555430and(416) 555-5430are one number. -
Let the simulated reviewer decide the day’s queue:
Terminal window match review --decideReview queue: 74 pairs (unsure 70, would join two entities 4)Decided 74 pairs: 33 same, 41 different (1 reviewer mistakes, simulated); 0 left in the queue
The first pair shows why a person helps. Richard Johnson’s two old records were two entities, because their dates of birth are 06/01/1973 and 1937-01-06. Safe clustering of Safer clustering keeps two dates of birth apart. A person sees at once that the digits of the year are swapped. The model can’t.
In this course, the reviewer is simulated: she decides from the true answer, and she makes a mistake on 2% of the pairs. That rate is our assumption. A real reviewer’s error rate must be measured, for example by giving two reviewers the same pairs.
What a decision does
Section titled “What a decision does”Each decision changes the entities, and it is also kept as a label.
Text description of the diagram
The diagram starts with the review queue: 471 pairs in the week, 51 to 76 a day. An arrow leads to Ivana’s screen: both records, field by field, with the differences marked. Two arrows leave it. “Same” (green) merges the two entities; the older ID stays; the merge is written to history.jsonl. “Different” (red) writes the pair to cannot_link.csv, so it never merges again. Below, a blue box: every decision is also a label in review_labels.csv, 231 same and 240 different in the week. An arrow leads to a yellow box: use the labels with care, with a separate test and a smaller weight.
- “Same” merges the two entities. The older entity keeps its ID, and the history says which ID retired.
- “Different” goes to
outputs/cannot_link.csv. Later runs read that file: safe clustering never joins the two records again, even if the model scores them high. - Every decision goes to
outputs/review_labels.csv, with who decided.
Splitting a wrong merge
Section titled “Splitting a wrong merge”Sometimes a reviewer finds a wrong merge that is already done. Then she splits it: one record leaves its entity. Here is a wrong join of day 1. The new shop record of Marie Ève Bérubé, D1-0184, joined the entity of Marie-Claude Bérubé through a shared family email, mberube@post.example. They are two women.
match undo D1-0184D1-0184 is now entity LK-D1-0184; the other 2 record(s) stay in LK-G-002767.Recorded in outputs/history.jsonl and outputs/cannot_link.csv (safe clustering will keep them apart).Text description of the diagram
On the left, before: entity LK-G-002767 with the records D1-0184, G-002767, L-004561. An arrow labelled “match undo D1-0184” leads to the right side, after. Entity LK-G-002767 still has the records G-002767, L-004561. A new entity, LK-D1-0184, has only the record D1-0184. A note says that cannot_link.csv now has D1-0184 with each record of LK-G-002767, and that history.jsonl has a split event from LK-G-002767 to LK-D1-0184.
This is the rule for an entity ID in a split: the records that stay keep the old ID, and the record that leaves gets a new one. Other systems that point to {sp.old_after} still point to Marie-Claude Bérubé. One case is different: when the entity was named after the leaving record, the leaving record keeps the name. The others then get a new one, LK- and their smallest record ID. Either way, the history says which records got which ID.
Decisions become labels
Section titled “Decisions become labels”The week of review gave 471 labelled pairs: 231 “same” and 240 “different”, with 12 simulated mistakes. They are free training data. TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails.’s plan sounds obvious: add them to the 3,000 labelled pairs and train again.
-
Run the six other days. Each day, the reviewer decides that day’s queue:
Terminal window for d in 2 3 4 5 6 7; do match daily --day $d; match review --decide; doneTerminal window foreach ($d in 2..7) { match daily --day $d; match review --decide }Day 2: 300 new records: 151 joined an existing entity, 149 started a new entity70 pairs sent to review; 2 record(s) would join two entities (not merged alone)checked against the true customers: 148 right joins, 3 wrong joins, 25 new entities for customers we already knewReview queue: 70 pairs (unsure 68, would join two entities 2)Decided 70 pairs: 36 same, 34 different (3 reviewer mistakes, simulated); 0 left in the queue…Day 7: 300 new records: 132 joined an existing entity, 168 started a new entity76 pairs sent to review; 4 record(s) would join two entities (not merged alone)checked against the true customers: 130 right joins, 3 wrong joins, 18 new entities for customers we already knewReview queue: 76 pairs (unsure 72, would join two entities 4)Decided 76 pairs: 37 same, 39 different (1 reviewer mistakes, simulated); 0 left in the queue -
Retrain LightGBM with the review labels, and compare it with the model of the 3,000 labels on the test split:
Terminal window match review --retrainReview queue: 0 pairslgbm without review labels: test precision 0.918, recall 0.886, F1 0.902lgbm with 471 review labels: test precision 0.880, recall 0.875, F1 0.878Saved the retrained model in outputs/models/lgbm-review.pkl (outputs/models/lgbm.pkl is unchanged)The retrained model goes to a file of its own. The model of the 3,000 labels stays in
outputs/models/lgbm.pkl, somatch dailyand the other commands still use it.
More labels, and the model is worse: F1 falls from 0.902 to 0.878, and precision falls most, from 0.918 to 0.880. This is what we found. It was not what we expected.
Why more labels hurt
Section titled “Why more labels hurt”The review labels are not like the 3,000 labelled pairs. They are a different sample: a different choice of pairs.
- They are only the hardest pairs. Every review pair was a pair the model was unsure about. The 3,000 pairs were sampled by surname and region, by random choice, and from the true pairs: easy and hard cases together.
- They change the balance. The 3,000 pairs are 35% “same”. The review pairs are 49% “same”. Our reading, which we did not test: in the review pairs, “same” often looks like a family (shared address and surname, different first name). The model learns that this look often means “same”, and it merges more families. That fits the fall in precision.
- They carry the reviewer’s mistakes. 12 of the 471 labels are wrong.
We tested each reason with new runs, on the same test split:
Text description of the chart
A column chart of F1 on an axis from 0.85 to 0.91. No review labels: 0.902. Review labels with weight 0.1: 0.904; weight 0.25: 0.9; weight 0.5: 0.904. Weight 1: 0.878. Weight 1 with the reviewer’s mistakes corrected: 0.888.
Two things stand out:
- The mistakes explain only part of the fall. With every mistake corrected, F1 is 0.888: still below 0.902. The bigger part comes from which pairs the labels describe.
- A smaller weight removes the harm. A weight says how much one label counts in training. With each review label counted as a quarter or a half of a normal label, F1 stays at 0.9 to 0.904.
A separate test for the hard pairs
Section titled “A separate test for the hard pairs”So should we leave the review labels out? Not yet. The test split measures the common pairs. The review pairs are the pairs that matter most for Ivana. We need a separate test: hard pairs that the model has not seen.
We trained on the review labels of days 1 to 5 only, and tested on the 127 review pairs of days 6 and 7, with their true answers:
| Model | Right on the hard pairs of days 6 and 7 | Called “same” |
|---|---|---|
| no review labels | 49.6% | 127 of 127 (true: 63) |
| review labels of days 1-5, weight 1 | 81.9% | 66 of 127 (true: 63) |
| review labels of days 1-5, weight 0.25 | 81.9% | 70 of 127 (true: 63) |
Without review labels, the model calls all 127 hard pairs “same”, and it is right on 49.6%: no better than a coin. With the review labels, it is right on 81.9%. The review labels teach exactly what the model did not know. They also bend its picture of the common pairs.
The lesson: review labels need their own weight and their own test. Measure every retrained model twice: on the test split, and on new review pairs it has not seen. Keep a model only if it is not worse on either. The direction is not fixed. The notebook has a smaller model, and its review batch is as balanced as the 3,000 labels. There, the same step raised F1 from 0.854 to 0.876. Measure it every time. The machine learning course explains why a sample that is not like the real data misleads a model, in Train, validation and test sets.
When to use human review, and what it costs
Section titled “When to use human review, and what it costs”- Use it where an error is expensive and a person can decide better than the model. A false merge at Larkfield costs an assumed 150 dollars; one review costs about 0.80 dollars (1.2 minutes at 40 dollars an hour).
- Size the band by the reviewer’s capacity. Ask how many pairs a day, and how many days. Change the thresholds when the queue grows.
- Design the screen. Both records, field by field, the differences marked, and the reason for the pair. Mark format-only differences less loudly than real ones.
- Measure the reviewer too. Give some pairs to two reviewers, or put pairs with a known answer in the queue. A tired reviewer makes more mistakes late in the day.
- Keep every decision. “Different” must stop the pair for ever, and a split must keep the old ID with the records that stay.
Common mistake
Retrain on every review label at full weight
Symptom: after a week, Tomás adds the 471 review labels to the training set and retrains. The test F1 falls from 0.902 to 0.878. More families are merged than before.
Cause: the review pairs are the hardest pairs only, and 49% of them are “same”, against 35% in the 3,000 labelled pairs. At full weight, they change what the model thinks a normal pair looks like. The reviewer’s mistakes add a little more harm.
Fix: give the review labels a smaller weight: 0.25 to 0.5 worked here. Test every retrained model on the test split, and on new review pairs that it has not seen.
Consequence in AI work: people who check a model’s unsure cases make a biased sample of labels. This is true in every field: flagged transactions, moderated posts, corrected answers of a language model. They are valuable, and they need their own weight and their own test.
Knowledge check
Section titled “Knowledge check”Check your understanding
A reviewer finds that record S-001234 was merged by mistake into entity LK-G-000777, which has three other records. She splits it. What happens to the entity IDs, and where can you see it in the data?
Show the answer
The three other records stay in LK-G-000777: their ID does not change, so systems that point to it still point to the same customer. S-001234 gets a new entity, LK-S-001234. In the data, outputs/entities.csv shows the new entity. outputs/cannot_link.csv has S-001234 with each of the three records, so they never merge again. outputs/history.jsonl has a “split” event from LK-G-000777 to LK-S-001234. One case is different: if the entity had been named after S-001234, S-001234 would keep that name. The other three would get a new one, and the history would say so.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The review queue grows to 450 pairs a day, and Ivana can review 300. Name two ways to fix it, and the price of each.
Show the answer
Narrow the review band: raise the lower threshold or lower the upper one. The price: more pairs are decided without a person, so more missed matches (below the band) or more false merges (above it). Or add review time: a second reviewer, or longer review days. The price is about 0.80 dollars a pair (assumed). A third way is a better model, which makes fewer pairs unsure; it costs development time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With 471 review labels, the test F1 fell from 0.902 to 0.878. But on new hard pairs, the retrained model was right far more often. Which model should Larkfield use?
Show the answer
Neither as it is. The retrained model at full weight is worse on the common pairs, which are most of the daily work. The old model fails the hard pairs, which go to Ivana anyway. Train with the review labels at a smaller weight, for example 0.25. Here, the test F1 stayed at 0.900, and the model was right on 81.9% of the new hard pairs. Measure both before you switch.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The review queue holds the pairs that the model is unsure about. Its size must fit the reviewer’s capacity: here, 51 to 76 pairs a day against an assumed 300.
- The reviewer needs both records side by side, the differences marked, and the reason for the pair.
- “Same” merges and the older ID stays. “Different” goes to
cannot_link.csv. A split gives the leaving record a new ID, and the others keep theirs. - At full weight, the 471 review labels made LightGBM worse: F1 0.902 → 0.878. They are only the hardest pairs and change the balance. A weight of 0.25 to 0.5 kept F1 at 0.900 to 0.904.
- Test a retrained model on the test split and on new review pairs.
Next, we leave Larkfield’s 60,000 records and ask what happens at 100,000 and 1 million.
Next: Scale