Skip to content
Français

A language model for the unsure pairs

  • Lesson 21
  • 40 min
  • Module 6: Learned matching: classifiers, embeddings and language models

Where we are

Every method so far leaves an unsure band of pairs. A language model can read both records and answer a question. In this lesson, we send only the unsure pairs to a model, measure its answers, and count the cost.

What you will be able to do

Send only the unsure pairs to a language model, with both records and one clear question. Read its recorded answers, and measure them against the true answers. Count the cost and the time per pair. Then decide what the model may decide alone, and what it may not.

NotebookMeasure the recorded answers in a notebookColab · Kaggle · GitHub

The notebook reads the 3,712 recorded answers and shows one pair as the model saw it. You measure the answers in the unsure band against the true answers. Then it computes the cost and the time per pair. It needs no key and no model. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of the last three lessons. Or use the notebook only.
  • No key and no account: the model’s answers are recorded in the course data.

Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.

A clinic’s clerk decides most patient forms in a few seconds: the same patient, or clearly another one. A few forms are hard. For those, she asks a senior colleague. The colleague is slower and costs more, so the clerk does not ask about every form. Only the hard ones.

The colleague is good, but not perfect. When she says “same patient” and the two files have different blood types, the clerk still checks. And a merge of two patients’ files always needs a signature.

This lesson gives TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s matcher a senior colleague: a language model.

The problem: the unsure band is too big for one person

Section titled “The problem: the unsure band is too big for one person”

Every method so far leaves pairs it cannot decide. In Module 5, Splink’s review band (match weights 10 to 20) held 13,378 pairs: 44.6 working days for Ivana at 300 pairs a day (our assumption).

A language model can read two records as text and answer a question. It knows that “Kathy” is a short form of “Katherine”, that “r. des Hérons” is “rue des Hérons”, and that a family email does not make two people one. Can it decide the unsure pairs?

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. has a bigger idea: “The model can decide every merge.” Keep it in mind. We test it at the end.

The idea: only the unsure band, with a clear question

Section titled “The idea: only the unsure band, with a clear question”

We do not send every pair to the model. We send only the unsure band: the pairs where LightGBM is neither sure of “same” nor sure of “different”.

First, a probability that means what it says

Section titled “First, a probability that means what it says”

LightGBM’s scores are not real probabilities. It learned from labelled pairs where 35% are true pairs, so its scores are far too high for candidate pairs (lesson 1). Its best threshold was above 0.999. On the validation pairs, only 14.3% of the pairs with a raw score from 0.9 to 1.0 are true pairs (37,792 pairs).

So we calibrate the scores first. Calibration turns a score into the share of true pairs among pairs with a similar score. We fit it on the validation pairs with isotonic regression: a step function that only goes up. After calibration, a probability of 0.3 means that about 3 of 10 such pairs are true pairs. The machine learning course explains calibration in Thresholds and calibration.

We chose two limits, 0.1 and 0.9:

The calibrated probability of LightGBM, from 0 to 1, cut into three bands. Below 0.1: LightGBM decides "different" (1,079,341 candidate pairs). From 0.1 to 0.9: the unsure band, where the language model decides (6,773 pairs; 3,512 of them in the validation and test splits, recorded). At 0.9 or more: LightGBM decides "same" (22,477 pairs). In the band, the model's answer "same" makes a match; "different" or "unsure" makes no match.
Who decides each candidate pair in the run lgbm+llm. The language model sees well under 1% of the pairs.
Text description of the diagram

A horizontal bar from probability 0 to 1, cut at 0.1 and 0.9. The left part is large: most candidate pairs are sure non-matches. The right part holds the sure matches. The middle part is the unsure band. An arrow goes from it to a box: the language model, with the question “Are A and B the same?” and three possible answers. Under each part, the number of candidate pairs and how many of them are true pairs.

The model gets the two records as text, field by field, with (missing) for empty fields. It gets a short instruction (the system prompt) that says what matters in this task:

You compare customer records of a home and garden shop. Decide if two records describe the same person.
Records come from different systems, so names, addresses, phone numbers and dates can be written in different
ways, have typing errors or be missing. People in one family often share an address, a phone number and an
email; that alone does not make them the same person. People move, so a different address alone does not make
them different. Answer only with JSON: {"answer": "same" | "different" | "unsure", "reason": "<one short sentence>"}.

Three choices matter:

  1. “Unsure” is allowed. A model that must choose often guesses. A guess that merges two people costs 150 dollars (our assumption); “unsure” costs nothing extra here.
  2. The known traps are named: families share fields, and people move.
  3. The answer is JSON with a fixed word. The code reads the word, and anything else counts as “unsure”.

We sent every unsure pair of the validation and test splits to Azure’s Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs model (gpt-6-luna-2026-09-22) once, and recorded each answer. We also sent 200 control pairs that LightGBM is sure about. The course data has all 3,712 recorded answers, so the lessons need no key and cost nothing. The 3,712 answers cost US$0.28 (US$0.29 with the product answers and a first run that we stopped).

Every answer below is a recorded answer from that model. A model of another version can answer differently.

  1. Copy the last files of this module from the snapshot end-of-m06: the llm command, the tests and the README:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m06
    cp $S/larkfield_match/commands/llm.py larkfield_match/commands/
    cp $S/tests/test_learned.py tests/
    cp $S/README.md .

    Your project now equals the snapshot end-of-m06.

  2. Run the tests. There are 34 now:

    Terminal window
    python -m pytest -q
    .................................. [100%]
    34 passed in 17.14s
  3. Show the recorded answers:

    Terminal window
    match llm
    3,712 recorded answers from gpt-6-luna-2026-09-22; US$0.2783 in all, US$0.075 per 1,000 pairs; median 1.51 s per answer
    control: sure match 100 pairs: same 97, different 1, unsure 2; right on 98.0% of the decided pairs
    control: sure non-match 100 pairs: same 1, different 56, unsure 43; right on 98.2% of the decided pairs
    unsure band 3512 pairs: same 1697, different 371, unsure 1444; right on 82.4% of the decided pairs
The recorded answers for the 3,512 unsure pairs of the validation and test splits.
Text description of the chart

Three bars: same 1,697, different 371, unsure 1,444.

In the unsure band:

  • The model said “unsure” for 1,444 pairs: 41%. These pairs stay “no match” in the run; in Module 8 they go to Ivana.
  • Of the pairs it decided, it was right on 82.4%.
  • It said “same” for 363 pairs of two different people: false merges.
  • It said “different” for a true pair 0 times. Never.

So its mistakes all go one way: it merges too easily. That is the most expensive kind of mistake for Larkfield.

On the control pairs, it agrees with LightGBM almost always: 97 “same” of 100 sure matches. But on the 100 sure non-matches, it said “unsure” 43 times. It is careful where the answer is easy.

match llm --pair shows what the model saw and what it answered:

Terminal window
match llm --pair S-000049 S-006358
[system]
You compare customer records of a home and garden shop. Decide if two records describe the same person. Records come from different systems, so names, addresses, phone numbers and dates can be written in different ways, have typing errors or be missing. People in one family often share an address, a phone number and an email; that alone does not make them the same person. People move, so a different address alone does not make them different. Answer only with JSON: {"answer": "same" | "different" | "unsure", "reason": "<one short sentence>"}.
[user]
Record A
name: Dr Mateo Khan
email: khan.mateo72@example.net
phone: (missing)
address: 83 Cobble Court, Mossbank Harbour NB
postcode: E1X-9G5
date_of_birth: (missing)
Record B
name: Mateo Khan
email: mateo.khan35@example.org
phone: 506.555.9565
address: 83 Cobble Crt, Mossbank Harbour NB
postcode: E1X-9G5
date_of_birth: (missing)
Are A and B the same?
[recorded answer]
{"answer":"same","reason":"The names and matching street address and postcode strongly align; the different emails alone do not rule out the same person."}
(true: same)

Record A has a title and another email. Record B writes “Crt” for “Court” and has a phone. The model reads both addresses as one street, and the answer is right. The reason it gives is one short sentence, as the prompt asked.

Where does the model help? We compared its answers with Splink’s decisions on the same band pairs. It was right where Splink was wrong 502 times. Splink was right where it was wrong 158 times. The diagram shows one of each, as recorded.

Two recorded cases. First case: the language model is right and Splink is wrong. Second case: Splink is right and the language model is wrong; the model says same for two different people. For each case: the two records field by field, Splink's decision, the model's recorded answer with its reason, and the true answer.
One case each way, with the model's own reason, as recorded. The records are synthetic.
Text description of the diagram

Two panels. Each panel shows record A and record B with their name, email, phone, address, postcode and date of birth. Below them: Splink’s match weight and decision, the language model’s recorded answer and its one-sentence reason, and the true answer. In the first panel, the model reads the evidence correctly and Splink does not. In the second panel, the model merges two different people and gives a confident reason; Splink keeps them apart.

The second case is a warning. The reason sounds certain and sensible. It is still wrong. A model’s reason is a story about its answer, not proof.

The run lgbm+llm uses LightGBM’s decision outside the band, and the model’s recorded answer inside it. “Same” makes a match. “Different” and “unsure” make no match.

Terminal window
match eval --run lgbm+llm
Ran lgbm+llm: 26,013 pairs called the same, in 11.8 s
lgbm+llm on the test split: pair precision 0.925, recall 0.905, F1 0.915
5,444 pairs called the same: 5,034 right, 410 wrong; 530 of 5,564 true pairs missed
entities after clustering (all records): 38,385 (true: 37,500)
Run Precision Recall F1
splink 0.817 0.779 0.797
logreg 0.915 0.907 0.911
lgbm 0.918 0.886 0.902
lgbm+llm 0.925 0.905 0.915

The best customer run of the course, by a little: 0.915 against 0.902 for LightGBM alone and 0.911 for logistic regression. Precision and recall both rise.

On the products, the model’s answers are even better. On the 150 pairs of a single item and its pack of 2, it answered “different” 149 times. The system prompt for products says “A pack of 2 is a different product from one item”, and the model reads the title. Embeddings missed exactly this (lesson 3).

Per 1,000 pairs it reads Per 1,000 candidate pairs
Any method on the CPU (US$0.17 an hour, assumption) under US$0.000002 under US$0.000002
chat-small, unsure band only US$0.075 US$0.00046

The model reads about 259 tokens and writes about 98 for a pair (measured). Chat-small costs US$0.10 per million tokens in and US$0.50 per million out (our assumption; check the provider’s page). That is US$0.075 per 1,000 pairs. Because it reads only the band, 6,773 of 1,108,591 candidate pairs, the cost per 1,000 candidate pairs is tiny.

Time matters more than money. The TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One answer took 1.51 seconds, and 1 answer in 20 took more than 2.53 seconds (recorded, from our laptop to the service). LightGBM scores a pair in microseconds.

Tomás’s idea: the model decides every merge

Section titled “Tomás’s idea: the model decides every merge”

Now test it. Send all 1,108,591 candidate pairs to the model:

  • Time: 1,108,591 × 1.51 s is about 19 days in one thread. Parallel calls shorten it, but the service limits them.
  • Money: about US$83, small. But every new day of records adds pairs.
  • Quality: on the 100 sure non-matches, it said “unsure” 43 times. Most candidate pairs are sure non-matches. So a reviewer would get tens of thousands of “unsure” pairs that LightGBM decides correctly.
  • Privacy: every pair sends two customers’ personal data to an outside service. Send as little as you can, as the AI security course explains in Minimize and retain deliberately. Our records are synthetic, so we could record them.

So the model works as a second opinion for the band, not as the judge of every pair.

Even in the band, the model does not merge alone in Larkfield’s design:

  1. Its errors are false merges: 363 in the band. A false merge shows a stranger another customer’s orders and address.
  2. Its reasons can be wrong and still sound right, as the second case shows.
  3. It can change. A new model version can answer differently on the same pair. Recordings make our results repeatable; a live service does not.

In Module 8, the daily run does not call the model at all. It sends the whole unsure band to Ivana, and it never lets one new record merge two existing customers on its own. A model like this one could sort her queue or suggest an answer. It does not replace her signature.

Use it when Do not use it when What it costs
a small unsure band remains after a cheap model you would send every candidate pair about 1.5 s and US$0.000075 a pair (recorded; price an assumption)
the evidence is in words a person would read: nicknames, abbreviations, packs the records hold data you may not send to an outside service a privacy review, and a data-processing agreement with the provider
you can record its answers and measure them on labelled pairs nobody checks its “same” answers a person’s time for the merges it proposes

Common mistake

Trust the model's reason

Symptom: the matcher stores the model’s reason next to each merge. A customer complains that she sees another person’s orders. The log says: “The exact name and postcode match, while the differing phone numbers could reflect an error or a changed number.” The team reads it and agrees: the merge looked right.

Cause: the reason is text that the model writes with its answer. It sounds sure in both cases: when the answer is right and when it is wrong. This recorded reason belongs to a pair of two different people (L-001701 and S-015863 in our synthetic data). A reason is not evidence, and it does not say how likely the answer is.

Fix: measure the model on labelled pairs, as this lesson does, and decide by the measured error rate, not by the reason. Keep the reason only as a hint for the reviewer. Send “same” answers that join two existing customers to a person.

Consequence in AI work: language models explain their answers fluently, right or wrong. Calibrate a model judge against human ratings in the evaluations course and the search and ranking course’s Judgments from a language model measure model judges the same way: against human labels.

Check your understanding

Give one case where the language model was right and Splink was wrong, and one where it was the other way round. What does each case tell you?

Show the answer

In the band, the model was right where Splink was wrong 502 times, and wrong where Splink was right 158 times. The first case in the diagram is one of the 502: the model read the evidence like a person and Splink did not. The second is one of the 158: the model said “same” for two different people with a confident reason, and Splink kept them apart. The first tells you that the model adds evidence that field weights miss. The second tells you that it merges too easily, and that its reason is no proof.

Check your understanding

Why does the project calibrate LightGBM's scores before it cuts the unsure band at 0.1 and 0.9?

Show the answer

LightGBM learned from labelled pairs with 35% true pairs. So its raw scores are far too high for candidate pairs, and its best threshold was above 0.999. Limits like 0.1 and 0.9 would then cut in the wrong place. Calibration on the validation pairs maps each score to the real share of true pairs at that score. Then “between 0.1 and 0.9” really means “LightGBM is unsure”, and the band holds the right pairs.

Check your understanding

Tomás says: “The model is right on 82.4% of the pairs it decides, and costs almost nothing. Let it decide every merge.” Give three reasons against this.

Show the answer

Time: 1.5 seconds a pair makes about 19 days for the 1.1 million candidate pairs in one thread. Quality: its errors are all false merges (363 in the band). On sure non-matches, it says “unsure” 43 times in 100, and that would flood the review queue. Privacy: every pair sends two customers’ personal data to an outside service. Also, a new model version can change the answers. Use it for the small unsure band, and keep a person for the merges it proposes.

Recap

  • Calibrate first: LightGBM’s scores become probabilities on the validation pairs. The unsure band is 0.1 to 0.9.
  • Send only the band to the model, with both records, the known traps, and “unsure” as an allowed answer. The course uses recorded answers from chat-small.
  • In the band, it said “unsure” for 41% and was right on 82.4% of the rest. It said “same” for 363 pairs of two people, and never “different” for a true pair.
  • lgbm+llm reaches F1 0.915, the best customer run, by a little. Cost: US$0.075 per 1,000 pairs it reads; time: 1.51 s a pair.
  • The model is a second opinion for the band, never the only judge of a merge.

This ends Module 6. We now have good pair decisions. Next, in Module 7, we turn pairs into customers, and see how one bad pair can join forty people.

Next: Clustering pairs