A language model for the unsure pairs
- Lesson 21
- 40 min
- Module 6: Learned matching: classifiers, embeddings and language models
Where we are
Every method so far leaves an unsure band of pairs. A language model can read both records and answer a question. In this lesson, we send only the unsure pairs to a model, measure its answers, and count the cost.
Lesson 21 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?A language model for the unsure pairs
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Send only the unsure pairs to a language model, with both records and one clear question. Read its recorded answers, and measure them against the true answers. Count the cost and the time per pair. Then decide what the model may decide alone, and what it may not.
NotebookMeasure the recorded answers in a notebook
The notebook reads the 3,712 recorded answers and shows one pair as the model saw it. You measure the answers in the unsure band against the true answers. Then it computes the cost and the time per pair. It needs no key and no model. Its setup cells download the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The review band of Splink, from From weights to a decision.
- LightGBM, its threshold and the class imbalance, from A supervised pair classifier.
- It helps to know how a chat model is called, from the LLM applications course.
What you need on your computer
- The project
larkfield-matchwith the files of the last three lessons. Or use the notebook only. - No key and no account: the model’s answers are recorded in the course data.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.
A second opinion for the hard cases
Section titled “A second opinion for the hard cases”A clinic’s clerk decides most patient forms in a few seconds: the same patient, or clearly another one. A few forms are hard. For those, she asks a senior colleague. The colleague is slower and costs more, so the clerk does not ask about every form. Only the hard ones.
The colleague is good, but not perfect. When she says “same patient” and the two files have different blood types, the clerk still checks. And a merge of two patients’ files always needs a signature.
This lesson gives TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s matcher a senior colleague: a language model.
The problem: the unsure band is too big for one person
Section titled “The problem: the unsure band is too big for one person”Every method so far leaves pairs it cannot decide. In Module 5, Splink’s review band (match weights 10 to 20) held 13,378 pairs: 44.6 working days for Ivana at 300 pairs a day (our assumption).
A language model can read two records as text and answer a question. It knows that “Kathy” is a short form of “Katherine”, that “r. des Hérons” is “rue des Hérons”, and that a family email does not make two people one. Can it decide the unsure pairs?
TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. has a bigger idea: “The model can decide every merge.” Keep it in mind. We test it at the end.
The idea: only the unsure band, with a clear question
Section titled “The idea: only the unsure band, with a clear question”We do not send every pair to the model. We send only the unsure band: the pairs where LightGBM is neither sure of “same” nor sure of “different”.
First, a probability that means what it says
Section titled “First, a probability that means what it says”LightGBM’s scores are not real probabilities. It learned from labelled pairs where 35% are true pairs, so its scores are far too high for candidate pairs (lesson 1). Its best threshold was above 0.999. On the validation pairs, only 14.3% of the pairs with a raw score from 0.9 to 1.0 are true pairs (37,792 pairs).
So we calibrate the scores first. Calibration turns a score into the share of true pairs among pairs with a similar score. We fit it on the validation pairs with isotonic regression: a step function that only goes up. After calibration, a probability of 0.3 means that about 3 of 10 such pairs are true pairs. The machine learning course explains calibration in Thresholds and calibration.
Then, three bands
Section titled “Then, three bands”We chose two limits, 0.1 and 0.9:
Text description of the diagram
A horizontal bar from probability 0 to 1, cut at 0.1 and 0.9. The left part is large: most candidate pairs are sure non-matches. The right part holds the sure matches. The middle part is the unsure band. An arrow goes from it to a box: the language model, with the question “Are A and B the same?” and three possible answers. Under each part, the number of candidate pairs and how many of them are true pairs.
The question we ask
Section titled “The question we ask”The model gets the two records as text, field by field, with (missing) for empty fields. It gets a short instruction (the system prompt) that says what matters in this task:
You compare customer records of a home and garden shop. Decide if two records describe the same person.Records come from different systems, so names, addresses, phone numbers and dates can be written in differentways, have typing errors or be missing. People in one family often share an address, a phone number and anemail; that alone does not make them the same person. People move, so a different address alone does not makethem different. Answer only with JSON: {"answer": "same" | "different" | "unsure", "reason": "<one short sentence>"}.Three choices matter:
- “Unsure” is allowed. A model that must choose often guesses. A guess that merges two people costs 150 dollars (our assumption); “unsure” costs nothing extra here.
- The known traps are named: families share fields, and people move.
- The answer is JSON with a fixed word. The code reads the word, and anything else counts as “unsure”.
Recorded answers
Section titled “Recorded answers”We sent every unsure pair of the validation and test splits to Azure’s Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs model (gpt-6-luna-2026-09-22) once, and recorded each answer. We also sent 200 control pairs that LightGBM is sure about. The course data has all 3,712 recorded answers, so the lessons need no key and cost nothing. The 3,712 answers cost US$0.28 (US$0.29 with the product answers and a first run that we stopped).
Every answer below is a recorded answer from that model. A model of another version can answer differently.
Run it in the project
Section titled “Run it in the project”-
Copy the last files of this module from the snapshot
end-of-m06: thellmcommand, the tests and the README:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m06cp $S/larkfield_match/commands/llm.py larkfield_match/commands/cp $S/tests/test_learned.py tests/cp $S/README.md .Terminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m06"Copy-Item $S\larkfield_match\commands\llm.py larkfield_match\commands\Copy-Item $S\tests\test_learned.py tests\Copy-Item $S\README.md .Your project now equals the snapshot
end-of-m06. -
Run the tests. There are 34 now:
Terminal window python -m pytest -q.................................. [100%]34 passed in 17.14s -
Show the recorded answers:
Terminal window match llm3,712 recorded answers from gpt-6-luna-2026-09-22; US$0.2783 in all, US$0.075 per 1,000 pairs; median 1.51 s per answercontrol: sure match 100 pairs: same 97, different 1, unsure 2; right on 98.0% of the decided pairscontrol: sure non-match 100 pairs: same 1, different 56, unsure 43; right on 98.2% of the decided pairsunsure band 3512 pairs: same 1697, different 371, unsure 1444; right on 82.4% of the decided pairs
What the answers say
Section titled “What the answers say”Text description of the chart
Three bars: same 1,697, different 371, unsure 1,444.
In the unsure band:
- The model said “unsure” for 1,444 pairs: 41%. These pairs stay “no match” in the run; in Module 8 they go to Ivana.
- Of the pairs it decided, it was right on 82.4%.
- It said “same” for 363 pairs of two different people: false merges.
- It said “different” for a true pair 0 times. Never.
So its mistakes all go one way: it merges too easily. That is the most expensive kind of mistake for Larkfield.
On the control pairs, it agrees with LightGBM almost always: 97 “same” of 100 sure matches. But on the 100 sure non-matches, it said “unsure” 43 times. It is careful where the answer is easy.
Look at one pair
Section titled “Look at one pair”match llm --pair shows what the model saw and what it answered:
match llm --pair S-000049 S-006358[system]You compare customer records of a home and garden shop. Decide if two records describe the same person. Records come from different systems, so names, addresses, phone numbers and dates can be written in different ways, have typing errors or be missing. People in one family often share an address, a phone number and an email; that alone does not make them the same person. People move, so a different address alone does not make them different. Answer only with JSON: {"answer": "same" | "different" | "unsure", "reason": "<one short sentence>"}.
[user]Record Aname: Dr Mateo Khanemail: khan.mateo72@example.netphone: (missing)address: 83 Cobble Court, Mossbank Harbour NBpostcode: E1X-9G5date_of_birth: (missing)
Record Bname: Mateo Khanemail: mateo.khan35@example.orgphone: 506.555.9565address: 83 Cobble Crt, Mossbank Harbour NBpostcode: E1X-9G5date_of_birth: (missing)
Are A and B the same?
[recorded answer]{"answer":"same","reason":"The names and matching street address and postcode strongly align; the different emails alone do not rule out the same person."}(true: same)Record A has a title and another email. Record B writes “Crt” for “Court” and has a phone. The model reads both addresses as one street, and the answer is right. The reason it gives is one short sentence, as the prompt asked.
Better than Splink, and different from it
Section titled “Better than Splink, and different from it”Where does the model help? We compared its answers with Splink’s decisions on the same band pairs. It was right where Splink was wrong 502 times. Splink was right where it was wrong 158 times. The diagram shows one of each, as recorded.
Text description of the diagram
Two panels. Each panel shows record A and record B with their name, email, phone, address, postcode and date of birth. Below them: Splink’s match weight and decision, the language model’s recorded answer and its one-sentence reason, and the true answer. In the first panel, the model reads the evidence correctly and Splink does not. In the second panel, the model merges two different people and gives a confident reason; Splink keeps them apart.
The second case is a warning. The reason sounds certain and sensible. It is still wrong. A model’s reason is a story about its answer, not proof.
Add it to the comparison table
Section titled “Add it to the comparison table”The run lgbm+llm uses LightGBM’s decision outside the band, and the model’s recorded answer inside it. “Same” makes a match. “Different” and “unsure” make no match.
match eval --run lgbm+llmRan lgbm+llm: 26,013 pairs called the same, in 11.8 slgbm+llm on the test split: pair precision 0.925, recall 0.905, F1 0.915 5,444 pairs called the same: 5,034 right, 410 wrong; 530 of 5,564 true pairs missed entities after clustering (all records): 38,385 (true: 37,500)| Run | Precision | Recall | F1 |
|---|---|---|---|
splink |
0.817 | 0.779 | 0.797 |
logreg |
0.915 | 0.907 | 0.911 |
lgbm |
0.918 | 0.886 | 0.902 |
lgbm+llm |
0.925 | 0.905 | 0.915 |
The best customer run of the course, by a little: 0.915 against 0.902 for LightGBM alone and 0.911 for logistic regression. Precision and recall both rise.
The products
Section titled “The products”On the products, the model’s answers are even better. On the 150 pairs of a single item and its pack of 2, it answered “different” 149 times. The system prompt for products says “A pack of 2 is a different product from one item”, and the model reads the title. Embeddings missed exactly this (lesson 3).
What it costs
Section titled “What it costs”| Per 1,000 pairs it reads | Per 1,000 candidate pairs | |
|---|---|---|
| Any method on the CPU (US$0.17 an hour, assumption) | under US$0.000002 | under US$0.000002 |
| chat-small, unsure band only | US$0.075 | US$0.00046 |
The model reads about 259 tokens and writes about 98 for a pair (measured). Chat-small costs US$0.10 per million tokens in and US$0.50 per million out (our assumption; check the provider’s page). That is US$0.075 per 1,000 pairs. Because it reads only the band, 6,773 of 1,108,591 candidate pairs, the cost per 1,000 candidate pairs is tiny.
Time matters more than money. The TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One answer took 1.51 seconds, and 1 answer in 20 took more than 2.53 seconds (recorded, from our laptop to the service). LightGBM scores a pair in microseconds.
Tomás’s idea: the model decides every merge
Section titled “Tomás’s idea: the model decides every merge”Now test it. Send all 1,108,591 candidate pairs to the model:
- Time: 1,108,591 × 1.51 s is about 19 days in one thread. Parallel calls shorten it, but the service limits them.
- Money: about US$83, small. But every new day of records adds pairs.
- Quality: on the 100 sure non-matches, it said “unsure” 43 times. Most candidate pairs are sure non-matches. So a reviewer would get tens of thousands of “unsure” pairs that LightGBM decides correctly.
- Privacy: every pair sends two customers’ personal data to an outside service. Send as little as you can, as the AI security course explains in Minimize and retain deliberately. Our records are synthetic, so we could record them.
So the model works as a second opinion for the band, not as the judge of every pair.
Why it is never the only judge of a merge
Section titled “Why it is never the only judge of a merge”Even in the band, the model does not merge alone in Larkfield’s design:
- Its errors are false merges: 363 in the band. A false merge shows a stranger another customer’s orders and address.
- Its reasons can be wrong and still sound right, as the second case shows.
- It can change. A new model version can answer differently on the same pair. Recordings make our results repeatable; a live service does not.
In Module 8, the daily run does not call the model at all. It sends the whole unsure band to Ivana, and it never lets one new record merge two existing customers on its own. A model like this one could sort her queue or suggest an answer. It does not replace her signature.
When to use a language model for matching
Section titled “When to use a language model for matching”| Use it when | Do not use it when | What it costs |
|---|---|---|
| a small unsure band remains after a cheap model | you would send every candidate pair | about 1.5 s and US$0.000075 a pair (recorded; price an assumption) |
| the evidence is in words a person would read: nicknames, abbreviations, packs | the records hold data you may not send to an outside service | a privacy review, and a data-processing agreement with the provider |
| you can record its answers and measure them on labelled pairs | nobody checks its “same” answers | a person’s time for the merges it proposes |
Common mistake
Trust the model's reason
Symptom: the matcher stores the model’s reason next to each merge. A customer complains that she sees another person’s orders. The log says: “The exact name and postcode match, while the differing phone numbers could reflect an error or a changed number.” The team reads it and agrees: the merge looked right.
Cause: the reason is text that the model writes with its answer. It sounds sure in both cases: when the answer is right and when it is wrong. This recorded reason belongs to a pair of two different people (L-001701 and S-015863 in our synthetic data). A reason is not evidence, and it does not say how likely the answer is.
Fix: measure the model on labelled pairs, as this lesson does, and decide by the measured error rate, not by the reason. Keep the reason only as a hint for the reviewer. Send “same” answers that join two existing customers to a person.
Consequence in AI work: language models explain their answers fluently, right or wrong. Calibrate a model judge against human ratings in the evaluations course and the search and ranking course’s Judgments from a language model measure model judges the same way: against human labels.
Knowledge check
Section titled “Knowledge check”Check your understanding
Give one case where the language model was right and Splink was wrong, and one where it was the other way round. What does each case tell you?
Show the answer
In the band, the model was right where Splink was wrong 502 times, and wrong where Splink was right 158 times. The first case in the diagram is one of the 502: the model read the evidence like a person and Splink did not. The second is one of the 158: the model said “same” for two different people with a confident reason, and Splink kept them apart. The first tells you that the model adds evidence that field weights miss. The second tells you that it merges too easily, and that its reason is no proof.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the project calibrate LightGBM's scores before it cuts the unsure band at 0.1 and 0.9?
Show the answer
LightGBM learned from labelled pairs with 35% true pairs. So its raw scores are far too high for candidate pairs, and its best threshold was above 0.999. Limits like 0.1 and 0.9 would then cut in the wrong place. Calibration on the validation pairs maps each score to the real share of true pairs at that score. Then “between 0.1 and 0.9” really means “LightGBM is unsure”, and the band holds the right pairs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “The model is right on 82.4% of the pairs it decides, and costs almost nothing. Let it decide every merge.” Give three reasons against this.
Show the answer
Time: 1.5 seconds a pair makes about 19 days for the 1.1 million candidate pairs in one thread. Quality: its errors are all false merges (363 in the band). On sure non-matches, it says “unsure” 43 times in 100, and that would flood the review queue. Privacy: every pair sends two customers’ personal data to an outside service. Also, a new model version can change the answers. Use it for the small unsure band, and keep a person for the merges it proposes.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Calibrate first: LightGBM’s scores become probabilities on the validation pairs. The unsure band is 0.1 to 0.9.
- Send only the band to the model, with both records, the known traps, and “unsure” as an allowed answer. The course uses recorded answers from chat-small.
- In the band, it said “unsure” for 41% and was right on 82.4% of the rest. It said “same” for 363 pairs of two people, and never “different” for a true pair.
lgbm+llmreaches F1 0.915, the best customer run, by a little. Cost: US$0.075 per 1,000 pairs it reads; time: 1.51 s a pair.- The model is a second opinion for the band, never the only judge of a merge.
This ends Module 6. We now have good pair decisions. Next, in Module 7, we turn pairs into customers, and see how one bad pair can join forty people.
Next: Clustering pairs