Skip to content
Français

Match products between two shops

  • Case study 1
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we matched Larkfield's synthetic records. In this case study, we match real products between two shops from an open benchmark, compare several methods, and report precision at a fixed recall and the cost.

What you will be able to do

Match real products between two online shops, from the raw files to a decision that a shop team can use. You will normalise titles, block, compare, classify and measure, as in the course. You will compare seven methods under one measure, and choose a design with its thresholds, its review band and its cost.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Normalise titles and model codes
  • Blocking with TF-IDF and embeddings
  • Comparison vectors
  • Splink and LightGBM
  • One-to-one matching
  • Thresholds and a review band
  • Cost per 1,000 pairs

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the course’s packages and download about 4 MB of data. The benchmark comes from its source page; the embeddings and recorded answers come from the course’s labs. It needs no account and no key. After the install, it runs in under a minute.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • A virtual environment with the course’s packages from Module 6, plus Splink 5.0.0, DuckDB 2.0.0 and pyarrow 26.0.1 from Module 5. The environment of the larkfield-match project has all of them. This case study does not change the project’s files.
  • The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell makes a new folder product-matching next to it. To start again, delete that folder.

Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Windows and Linux are not tested.

You want to buy a television. A price-comparison website shows the same model in two shops, with two prices. Someone had to decide that two offers with two different titles are one product.

Leila runs the catalogue team of a small price-comparison website. Ben is her data engineer. The site compares the offers of two online shops that sold electronics, Abt and Buy. (Leila, Ben and the website are invented. The two shops and their product titles are real.)

Each error has a price:

  • A false match shows two different products as one. A reader sees a wrong comparison, for example a camera case next to a camera. Leila says: “One wrong match on the page costs us more trust than ten missing ones.”
  • A missed match hides a comparison that the site could show. It costs a little traffic.

So Leila sets a rule: matches go live automatically only when at least 99 in 100 of them are right. A person checks the unsure ones.

You will build a matcher that reads the two shops’ product lists. It puts every candidate pair in one of three groups: accept, review or no match. You will compare seven methods on the same pairs with the same measures, and choose one design. Then you run that design on products that no step has seen, and on a second, harder pair of shops.

The skills come from the whole course, Modules 2 to 8. One idea is new here: one-to-one matching.

The data is a public benchmark: a dataset that researchers publish so that everyone can test methods on the same problem. It comes from the database group of the University of Leipzig. It has the product offers of Abt (1,081) and Buy (1,092), and a list of 1,097 true matches: pairs of offers that are the same product.

Real data, at last. Every record in Modules 1 to 8 was synthetic. This data is real: the shops typed these titles for their customers, and the Leipzig researchers made the list of true matches, not our code.

These are the columns that the case study uses.

Column What it means Use
id The offer’s number in its shop Key
name The product title, as the shop wrote it Normalised, compared, embedded
description A longer text (in Buy, often empty) Searched for model codes
manufacturer The brand (Buy only) Not used: Abt has no such column, so the first word of the title stands for the brand
price The price, such as $399.00 (often empty) Compared when both offers have one
idAbt, idBuy One true match per row (the third file) Labels only, never an input
print("repeated IDs:", int(abt.id.duplicated().sum()), int(buy.id.duplicated().sum()))
print("no price:", f"Abt {(abt.price == '').mean():.0%},", f"Buy {(buy.price == '').mean():.0%}")
print("no description in Buy:", f"{(buy.description == '').mean():.0%}")
print("products with a match:", f"Abt {abt.id.isin(matches.id_a).sum()} of {len(abt)},",
f"Buy {buy.id.isin(matches.id_b).sum()} of {len(buy)}")
print("products with two or more matches:", f"Abt {(matches.id_a.value_counts() > 1).sum()},",
f"Buy {(matches.id_b.value_counts() > 1).sum()}")

The output:

repeated IDs: 0 0
no price: Abt 61%, Buy 46%
no description in Buy: 40%
products with a match: Abt 1081 of 1081, Buy 1092 of 1092
products with two or more matches: Abt 16, Buy 5

What this tells us. Prices are often missing, so price can only be weak evidence. And every product has a partner in the other shop, the most important fact about this benchmark. In real work, a shop sells many products that the other shop does not have.

Four of the true matches:

Sony Black 9 Megapixel Cyber-Shot Digital Camera - DSCH50B | Sony Cyber-shot DSC-H50 Digital Camera - Black - DSCH50B
Panasonic 2-Line Integrated Telephone System - KXTS208W | Panasonic KX-TS208W Corded Phone
Speck SeeThru Clear Hard Shell Case For 15' MacBook - MB15CLRSEEV2 | Speck Products SeeThru Case for Apple 15' MacBook Pro - MB15-CLR-SEE-V2
VMware Fusion 2 For Mac - VMFM20BX2 | VMware Fusion 2 - VMFM20BX2

What this tells us. Abt ends almost every title with the model number. Buy writes it with hyphens (KX-TS208W), in another place, or not at all. The other words differ too.

  • The unit is a pair: one Abt offer and one Buy offer. The target is “same product or not”, from Leipzig’s list.
  • The split puts each product in train (50%), validation (25%) or test (25%), by group. Products joined by true matches stay in one split, so no answer leaks from training to test. A pair belongs to the split of its Abt product. The test split has 277 true matches. The split, its shares and its seed (39) are our choice.
  • The measures. Every method is measured on the test split, with a threshold chosen on validation. We report pair precision, recall and F1, as in Measure a matcher. Leila’s rule needs a second measure: precision at a fixed recall. We take the threshold that finds 80% of the validation matches, and report the precision at that threshold on test. A rule that cannot reach 80% recall gets no value.
  • The labels are Leipzig’s. We made no label ourselves. The 99% rule, the reviewer’s time and the prices are our assumptions.

The simplest matcher says “same product when the names are the same”. On the test split, it finds 1 of 277 true matches. Shops almost never write a title in the same way.

So we normalise the titles first, as in Normalise. Normalised titles are in lower case, with no accents or signs. The inch mark 42' becomes 42 in, and no hyphen or slash stays inside a word. Then a small function keeps the words that look like model codes: at least 4 characters, with both letters and digits.

One Panasonic television in two shops. Abt: Panasonic VIERA 42' Plasma Flat Panel HDTV In Black - TH42PX80U. Buy: Panasonic Viera TH-42PX80U 42' Plasma TV. After normalise: panasonic viera 42 in plasma flat panel hdtv in black th42px80u, and panasonic viera th42px80u 42 in plasma tv. The model code is th42px80u in both, so the codes are the same.
Normalising makes the model number the same in both shops, although the two titles still differ in length and order.
Text description of the diagram

Two columns, Abt and Buy, and three rows. Row 1: the title as each shop wrote it. Row 2: the title after normalise(). Row 3: the model codes that model_codes() found. At the bottom, a green box says that the codes are the same: th42px80u.

Exact normalised titles find 4 matches: still almost nothing. A rule on the codes works much better:

same model code: {'precision': 0.954, 'recall': 0.675, 'f1': 0.791, 'called': 196, 'right': 187, 'true': 277}

What this tells us. When two titles share a model code, they are the same product 95 times in 100. But a third of the true matches share no code. Some Buy titles have none (“LG 5.0 cu.ft. Freestanding Gas Range”). Others write a shorter code (“JFS516” for “JFS516NA”).

Common mistake

A size that looks like a model number

Symptom: Ben’s first version of the code rule had a precision of 0.519: one call in two was wrong.

Cause: his code function kept every word with letters and digits. The most common “codes” in Abt were 1080p (28 products), 60gb and 16gb. Those are a screen resolution and sizes, shared by many different products. So every 1080p television “matched” every other 1080p product.

Fix: refuse a word that is a number followed by a unit (in, gb, p, hz and others). The precision rose to 0.954.

Consequence in AI work: before you trust a key, look at its most common values, as in What blocking loses. One common value can join thousands of pairs. In Module 7, one garden centre’s phone number joined 40 different people.

The two shops make 1,081 × 1,092 = 1,180,452 pairs. A real catalogue has a million products, so we block, as in Module 3, and keep only candidate pairs.

The code rule is a precise blocking, but it keeps only 66% of the true matches. Two more blockings find titles that are similar, not equal:

  • TF-IDF top 5. Think of a librarian who looks for a book by its rare words: “Acoustimass” says more than “speaker”. TF-IDF does the same. It cuts each title into short pieces of 3 to 5 letters. It gives a piece more weight when few titles have it. Two titles that share rare pieces get a high TF-IDF similarity, from 0 to 1. For each product, we keep the 5 products of the other shop with the highest similarity, in both directions. The search and ranking course ranks documents with the same idea: see BM25.
  • Embeddings top 5. The same, with the cosine of the titles’ embeddings, from the course’s Granite model, as in Embeddings for names, addresses and products. The embeddings come with the notebook, so it does not run the model.
candidates true kept pair completeness reduction ratio
same model code 770.0 729.0 0.6645 0.99935
TF-IDF top 5 7308.0 1077.0 0.9818 0.99381
embeddings top 5 7374.0 1077.0 0.9818 0.99375
all three together 10205.0 1087.0 0.9909 0.99136

What this tells us. TF-IDF and embeddings keep the same share of true matches, but not the same matches. Together with the codes, they keep 1,087 of 1,097 with 10,205 candidates: less than 1% of all pairs. In the notebook, you try 3 neighbours instead of 5: 5,919 candidates, and 6 more true matches lost.

Each candidate pair gets nine similarities, as in Compare by field, then combine:

  • two string similarities of the titles (token set ratio and Jaro-Winkler), and the TF-IDF similarity;
  • “same code”, the best similarity between two codes, and “the Abt code appears in the Buy title or description”;
  • the share of numbers that the two titles have in common, the price ratio, and the embedding cosine.

A missing value (no code, no price) stays missing.

Every method below sees the same 10,205 candidates and is measured on the same 277 test matches.

  • Rules: the exact title, and the same model code.
  • One score alone: the TF-IDF similarity, or the embedding cosine, with the threshold that gave the best F1 on validation.
  • Splink without labels, as in Module 5. It compares the title words, the first word (the brand) and the codes, and learns its weights with expectation maximisation.
  • Logistic regression and LightGBM on the comparison vector, as in A supervised pair classifier. They learn from the 5,097 candidate pairs of the train split (539 true).
F1 of every method on the same candidate pairs of Abt–Buy. The classifiers win, and one-to-one matching (the last row) adds the most.
Text description of the chart

Nine dots, one per method, on a scale from 0 to 1. Exact title 0.028; rule same model code 0.791; embeddings alone 0.579; TF-IDF alone 0.707; Splink without labels 0.765; logistic regression 0.862; LightGBM 0.869; LightGBM with the embedding cosine 0.88; LightGBM with one-to-one matching 0.924.

What this tells us. The classifiers win clearly: LightGBM F1 0.869, logistic regression 0.862. The code rule (0.791) beats Splink without labels (0.765). The embedding cosine alone is the weakest score (0.579), and adds little to LightGBM (0.880). LightGBM puts 55% of its gain on one field: the Abt code found in the Buy text.

About 67% of the true candidate pairs share a model code. So the m probability of “share a code” should be about 0.67: the share of true matches that agree on it. Splink learned 0.048.

Expectation maximisation looks for two groups of pairs, and it does not know which group we mean. Our reading, not measured: it probably found pairs of one brand and one product line: they share many words, but rarely a code. Module 5 saw the same kind of problem with the date of birth. On product titles, the evidence is in free text, and labels help a lot.

Precision on the test split, at the threshold that reached recall 0.8 on validation. Only the classifiers come near Leila's 0.99, and only one-to-one matching passes 0.98.
Text description of the chart

Seven dots. TF-IDF alone 0.535; embeddings alone 0.378; Splink without labels 0.326; logistic regression 0.884; LightGBM 0.906; LightGBM with the cosine 0.92; LightGBM with one-to-one 0.983. The two rules have no value: they cannot reach recall 0.8.

At recall 0.8, about 1 of LightGBM’s matches in 10 is wrong (0.906); Leila needs 99 in 100. The next idea closes most of that gap.

At a dance class, each person dances with one partner. A product match is similar. An offer in one shop is the same product as at most one offer in the other shop. (In this benchmark, a few products have two matches, because one shop listed the same product twice.)

The classifier scores every pair on its own. So it can give 0.7 to “camera and camera case”, and 0.9 to “camera and camera”. It does not know that the camera already has a better partner.

One-to-one matching keeps a pair only when it is the best pair of both its products. It drops all other pairs. The pairs are the edges of a graph, as in Clustering pairs, but here each product may keep only one edge.

Two products in shop A (A1, A2) and two in shop B (B1, B2). Scores: A1–B1 0.90, A1–B2 0.60, A2–B1 0.70, A2–B2 0.40. A1–B1 is the best pair of A1 and of B1, so it is kept. A1–B2 and A2–B2 are dropped because A1 and A2 have better pairs. A2–B1 is dropped because B1 has a better pair. A2 and B2 get no partner.
One-to-one on a tiny example: only A1–B1 stays. A wrong pair with a high score can also take a partner away, so this step needs good scores.
Text description of the diagram

Four boxes: A1 and A2 on the left, B1 and B2 on the right, and four lines with their scores. The line A1–B1 (0.90) is thick and green: kept. The other three lines are dashed: dropped. A note lists each pair and the reason.

On the real candidates, one-to-one keeps 1,030 of the 10,205 pairs. LightGBM’s F1 rises from 0.869 to 0.924, and its precision from 0.930 to 0.976. Its precision at recall 0.8 rises from 0.906 to 0.983. On logistic regression (the notebook’s task), the F1 becomes 0.915.

When to use it: when each entity can appear at most once in each list, such as two shops’ catalogues or two registers. When not to use it: inside one list with duplicates, such as TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s customers with up to 5 records each. Also not when many products have no partner. Then a wrong pair with a low score can still be “the best” of two products. Keep a threshold as well.

As in Human review, we cut the score into three bands. On the validation split, we tried three lower and three upper thresholds. Six of the nine rows:

accepted accepted right precision accepted to review true in review recall if reviews are right
lower upper
0.01 0.50 221.0 213.0 0.964 25.0 21.0 0.854
0.80 201.0 200.0 0.995 45.0 34.0 0.854
0.95 182.0 181.0 0.995 64.0 53.0 0.854
0.05 0.50 221.0 213.0 0.964 20.0 17.0 0.839
0.80 201.0 200.0 0.995 40.0 30.0 0.839
0.95 182.0 181.0 0.995 59.0 49.0 0.839

The lowest upper threshold that meets Leila’s 99% is 0.8. For the lower threshold, we take 0.01. Compared with 0.05, it adds 5 pairs to review, and 4 of them are true matches. On the test split, used once:

accepted accepted right precision accepted to review true in review recall if reviews are right
validation 201.0 200.0 0.995 45.0 34.0 0.854
test 218.0 216.0 0.991 31.0 27.0 0.877

What this tells us. On test, 216 of the 218 automatic matches are right. 27 of the 31 review pairs are true. If the person decides them all correctly, the site shows 87.7% of the true matches.

Where do the other 34 true matches go? The blocking loses 1. One-to-one drops 7 because the product’s other true partner won, and 23 because a wrong pair won. 3 score below 0.01.

The chosen design. Normalise; block with the same code, TF-IDF top 5 and embeddings top 5 (10,205 of 1,180,452 pairs, 1,087 of 1,097 true kept); compare 9 similarities; classify with LightGBM; one-to-one. Then three bands on the test split: score 0.8 or more, accept, 218 pairs, 216 right; 0.01 to 0.8, a person reviews, 31 pairs, 27 true; below 0.01, no match.
The design that the case study recommends, with the counts of the test split.
Text description of the diagram

Five steps from left to right, each with its main number: normalise, block, compare, classify, one-to-one. Under the last step, three boxes for the bands: accept, review and no match, each with its count of pairs.

A language model for the review band (recorded)

Section titled “A language model for the review band (recorded)”

As in A language model for the unsure pairs, we asked a small chat model (Azure Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs) about every validation and test pair in the band. We asked once and recorded the answers. The model read both titles, the first 300 characters of each description, and the prices. Its recorded answers for the 76 band pairs:

The model said True match Not a match
same 49 0
different 3 5
unsure 9 10

What this tells us. When the model said “same”, it was right 49 times out of 49. For vague Buy titles, it said “unsure”: “Offer B does not specify the model or colour”. It also caught a colour variant, rightly: “A is white (WH) and B is black (BF)”.

One of its “mistakes” is worth a look. For “Canon 2GB SD Secure Digital Card - 3505B001” against “Simpletech 2GB SD Card - 3505B001AA”, the model said “different”. Its reason: “Offer A is branded Canon, while Offer B is branded Simpletech”. The benchmark says “same”, because the part number is the same. A person might agree with the model. A benchmark’s labels can be wrong too.

49 of 49 is a small sample: the true error rate could still be about 6% (a rough rule for zero errors: 3 divided by the number of answers). So the model cannot replace the person yet. It can sort the queue: its “same” pairs first, with its reason shown to the reviewer.

The prices are assumptions from the course’s price table; check the providers’ current pages.

  • A CPU server costs US$0.17 an hour.
  • The chat model costs US$0.10 per million input tokens and US$0.50 per million output tokens.
  • A reviewer takes 1.2 minutes a pair and costs US$40 an hour.
Step Measured Cost (assumption × measurement)
Compare and classify on a CPU under 0.5 s for 10,205 pairs under US$0.000002 per 1,000 candidate pairs
Language model, band pairs only 252 tokens in, 139 out; median 1.96 s a pair US$0.095 per 1,000 pairs it reads; only 1.5% of candidates are in the band
A person reviews the band 115 pairs per 1,000 Abt products 2.3 hours, about US$92 per 1,000 products

What this tells us. The computer is almost free; the person is the cost. The language model costs about one cent per 100 band pairs, so it pays for itself if it saves a reviewer a few seconds a pair. The whole recording (186 pairs, with a margin around the band and 20 sure pairs as a control) cost US$0.016.

A design must work on products that it has never seen. The notebook’s match_shops() runs every step, from normalising to the bands. We run it on the 270 Abt products of the test split, as if they were new, against all of Buy:

270 new products, 0.3 s: {'accept': 220, 'review': 47}
count true
decision
accept 220 216
review 47 30

What this tells us. The precision is 0.982, a little below the 0.991 of the evaluation, and the review queue is longer. The reason is one-to-one. A Buy product now chooses only among the 270 new products, so a weaker new product can become its “best” pair. In daily use, match new products against the whole catalogue, with the products that already have a partner. New records every day shows how.

The Leipzig page has a second product benchmark: Amazon (1,363 offers) against Google’s shopping service (3,226). It has 1,300 true matches, mostly software. Here, not every product has a partner: only 1,291 of the 3,226 Google offers have one. In the notebook’s last task, you run the whole design on it with one call. The result on its test split:

{'blocking': {'candidates': 28776, 'true kept': 1289, 'pair completeness': 0.9915, ...},
'threshold': 0.05, 'test': {'precision': 0.683, 'recall': 0.59, 'f1': 0.633, ...}}
The same kind of method on four problems: Larkfield's synthetic products and customers (from Module 6), and the two real benchmarks.
Text description of the chart

Four dots. Larkfield products, synthetic, p-lgbm: 0.996. Abt–Buy, LightGBM with one-to-one: 0.924. Larkfield customers, synthetic, lgbm: 0.902. Amazon–Google with the same design: 0.633.

What this tells us. The blocking still works (99% of true matches kept), but the F1 falls to 0.633. Software titles have few model numbers. Many offers differ only by a year or an edition. For example, the design matched “peachtree by sage premium accounting for nonprofits 2007” to the 2008 edition. Most offers have no partner, so one-to-one can keep a wrong one. Some “false matches” look right to a person, such as two Red Hat licences with the same part number rl296aa; we counted them as errors, as the benchmark says. A good next step is a version and edition field, parsed from the title as Module 2 parsed sizes and packs.

Compared with Larkfield. On Larkfield’s synthetic supplier feeds, LightGBM reached F1 0.996: our generator wrote sizes and packs in a few known ways. Real shops are less tidy. Abt–Buy is harder than our feeds, and Amazon–Google is harder than our synthetic customers.

Ben writes to Leila in her words:

  • Use the design above. Match automatically when the score is 0.8 or more.
  • Review the band from 0.01 to 0.8: about 2.3 hours for every 1,000 products, mostly true matches with a vague title.
  • Use the language model to sort the queue, not to decide. Show its reason to the reviewer, and measure it again after a few hundred reviews.
  • Match new products against the whole catalogue, never only against each other.
  • Ask the shops for model numbers or barcodes. A shared barcode would make most of this work unnecessary.

What not to do:

  • Do not use the 0.8 threshold for another pair of shops without new labels and a new validation.
  • Do not show a match as certain. Show “compare offers”, and let readers report a wrong match.
  • Every product has a partner here. That makes one-to-one look better than it will be in real work. In a real catalogue, measure the share of products without a partner first.
  • Old and small. The files date from 2010: two electronics shops, about 1,000 products each.
  • Labels can be wrong. The Canon and Simpletech pair shows that a benchmark’s truth is someone’s judgement.
  • The test split is small. One error moves the precision by about 0.4 points, so do not read the third decimal.
  • The recorded answers are from one model, once. A new model version, or a new run, gives other answers.
  • The prices are assumptions. Check the providers’ current pages.
  • Credit the source. CC BY 4.0 requires the attribution in the box above wherever you use or share the data.
  • The brands are real. This case study says nothing about any shop or brand. A wrong match on a public page can misrepresent a shop’s price. That is why Leila’s rule is strict, and why readers can report errors.
  • No personal data. These are product offers only. The same methods on customer records need the care of Privacy and matching.
  • A language model sees the text you send. Here it was public product text. Never send customer records to a model service without the right agreement.

This case study shows that you can take real, messy records to a decision. Make it yours with one change: add a version field for Amazon–Google, or a threshold for products without a partner, or try another open benchmark.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • The comparison table as a CSV: every method, its threshold, precision, recall, F1 and precision at recall 0.8.
  • DECISION.md: the thresholds, the review band, its size and its cost, and the assumptions.
  • The data source and licence, with the attribution text. CC BY 4.0 lets you share the data with credit, but a link and the checksums are enough.

A README template:

# Matching products between two online shops
## Problem
A price-comparison site must decide which offers in two shops are the same product.
Matches go live automatically only when 99 in 100 are right; a person checks the rest.
## Data and licence
Abt-Buy benchmark, University of Leipzig (Köpcke, Thor and Rahm, PVLDB 2010), CC BY 4.0,
https://dbs.uni-leipzig.de/research/projects/benchmark-datasets-for-entity-resolution.
1,081 + 1,092 offers, 1,097 true matches. Split 50/25/25 by group of products.
## Approach
Normalised titles and model codes; blocking with the same code, TF-IDF top 5 and
embeddings top 5 (pair completeness 0.991, 10,205 of 1.18 million pairs); a comparison
vector of 9 similarities; seven methods compared; LightGBM with one-to-one matching.
## Results (test split)
F1 0.924 (exact names 0.007, code rule 0.791, Splink without labels 0.765).
Accept >= 0.8: 218 matches, 216 right. Review 0.01-0.8: 31 pairs.
## Limits
Every product has a partner in this benchmark; old data, one category. On the harder
Amazon-Google benchmark, the same design reached only F1 0.633.

How to describe it in an interview, in two sentences: “I matched real products between two shops and compared seven methods on one test split; LightGBM with one-to-one matching reached F1 0.92 and 99% precision on the automatic matches. On a harder software benchmark, the same design fell to 0.63, because versions and editions need their own field.”

Go further:

  • Parse the edition and the year from software titles, add them to the comparison vector, and run Amazon–Google again.
  • Replace “best of both” with an optimal assignment (scipy.optimize.linear_sum_assignment), and compare.
  • Ask a chat model about the 31 test review pairs with your own key, and compare its answers with the recorded ones.

Check your understanding

The code rule has a precision of 0.954 but a recall of only 0.675. Give two reasons why a true match may share no model code, from the titles in this case study.

Show the answer

First, some Buy titles have no model code at all, for example “LG 5.0 cu.ft. Freestanding Gas Range”. Second, a shop may write a shorter or another form of the code: “JFS516” against “JFS516NA”, or “PMCII” against “PMC2”. Exact equality of codes misses both. That is why the comparison vector also has the best similarity between two codes, and “the Abt code appears in the Buy text”.

Check your understanding

One-to-one matching raised LightGBM's precision from 0.930 to 0.976 on Abt–Buy. Would you expect the same gain on Larkfield's customer records? Explain.

Show the answer

No. One-to-one assumes that each entity appears at most once in each list. Larkfield’s customers have up to 5 records of one person, often in the same source, so one person may need several partners. One-to-one would drop true pairs. It also helps most when every product has a partner, as in this benchmark. In lists where many records have no partner, keep a threshold as well.

Check your understanding

The recorded language model said “same” for 49 band pairs and was right every time. Leila asks: “Can we let it accept band pairs without a person?” What do you answer?

Show the answer

Not yet. 49 answers is a small sample: the true error rate could still be several percent, and Leila’s rule allows 1 error in 100. The answers are also from one model version, recorded once. Use the model to sort the queue and show its reason, and keep measuring its answers against the reviewers’ decisions. If it stays right on a few hundred more pairs, the team can decide again.

Recap

  • On real product data, exact names found 1 of 277 test matches. Normalised model codes were precise (0.954) but found only two thirds.
  • Three blockings together (codes, TF-IDF top 5, embeddings top 5) kept 99.1% of the true matches with less than 1% of the pairs.
  • LightGBM on the comparison vector (F1 0.869) beat the rules, Splink without labels (0.765) and the embeddings (0.579). Splink learned a wrong weight for the codes.
  • One-to-one matching raised the F1 to 0.924, because each product has at most one partner.
  • The decision: accept at 0.8 or more (216 of 218 right on test). Review from 0.01 to 0.8: about 115 pairs per 1,000 products. Let the language model sort the queue, not decide it.

Next, real bibliographic records: titles, author lists and venues from two sources.

Next: Link bibliographic records