Skip to content
Français

Embeddings for names, addresses and products

  • Lesson 20
  • 40 min
  • Module 6: Learned matching: classifiers, embeddings and language models

Where we are

Our features compare letters and sounds. A small language model can turn a whole record into numbers. In this lesson, we use embeddings to find and score pairs, and we see where they help and where they hurt.

What you will be able to do

Turn each record into an embedding: 384 numbers made by a small language model. Compare two records by the cosine of their embeddings. Use the nearest records as a blocking, and the cosine as a score. Measure both, and see why embeddings fail on households, sizes and packs.

NotebookCompare records by embeddings in a notebookColab · Kaggle · GitHub

The notebook reads the shipped embeddings and computes cosines: for two example pairs, for customer pairs at one postcode, and for product matches. Then you compute the cosine of each single item with its pack of 2. It does not run the model. Its setup cells download about 33 MB, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-match with the files of the last two lessons. Or use the notebook only.

Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are not tested.

You ask a shop assistant for a “garden hose, about 15 metres, green”. The label on the shelf says “Rubber Garden Hose, 50 ft, Green”. The words are different, and the unit is different. A good assistant still knows it is the same kind of thing.

Now you ask for “a pack of two of those hoses”. The assistant must also notice the number. One hose is not two hoses, and the price is different.

A matcher needs both skills: see that two texts mean the same thing, and see the small details that make them different.

Our comparison vector compares letters (Jaro–Winkler, token set ratio) and sounds (Metaphone). Some titles of one product are written in very different ways. Here is one true match from the two feeds:

Feed A (Northfield Supply) Feed B (Copperleaf Wholesale)
Hearthside Cedar Raised Garden Bed, 4 ft, Sand raised garden bed 1.2 m cedar sand - Hearthside

The words are in another order, the brand moved to the end, and “4 ft” became “1.2 m”. A letter measure on the whole title gives a lower score. A model that knows what the words mean could see one product.

An embedding is a list of numbers that a model makes from a text. The model was trained on a lot of text, so that texts with a similar meaning get similar lists. Each number on its own means nothing to a person. Together, they place the text at a point in a space with many directions.

We compare two embeddings with the cosine: 1 when they point the same way, lower when they differ. The search and ranking course explains this in Search by meaning.

Real embeddings have 384 numbers. Here are three made-up embeddings with 3 numbers each, already of length 1:

Text Embedding
A: “garden hose” (0.6, 0.8, 0.0)
B: “gdn hose” (0.8, 0.6, 0.0)
C: “lamp” (0.0, 0.0, 1.0)

The cosine of two embeddings of length 1 is the sum of the products of their numbers:

  • A and B: 0.6 × 0.8 + 0.8 × 0.6 + 0.0 × 0.0 = 0.48 + 0.48 + 0 = 0.96: very alike.
  • A and C: 0.6 × 0.0 + 0.8 × 0.0 + 0.0 × 1.0 = 0: nothing in common.
From record to embedding to cosine. A customer record becomes one text: name, then the address, for example “val tremblay, 6760 rue des herons st eloi du lac”. The model ibm-granite/granite-embedding-97m-multilingual-r2 turns the text into 384 numbers; the first 8 numbers are shown. The embeddings are stored as 8-bit numbers with one scale per row. Two embeddings give one cosine. The worked example: (0.6, 0.8, 0) and (0.8, 0.6, 0) give 0.96.
What the project does with each record: one text, 384 numbers, then one cosine per pair. The course data has every record's embedding already.
Text description of the diagram

Three steps from left to right. First, the text that the model reads: a customer’s name and address, or a product’s title as the feed wrote it. Second, the model and its output: 384 numbers per record, of which the first 8 are shown. Third, two embeddings and their cosine, with the worked example of the table above.

We use ibm-granite/granite-embedding-97m-multilingual-r2, the same small model as the search and ranking course. It has an Apache-2.0 licence (a vendor claim, from its model card) and makes 384 numbers per text. It knows French as well as English, which suits TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s names and towns.

What the model reads:

  • A customer: the name, then the address: number, street and town. If a record has no address, the postcode.
  • A product: the title, exactly as the feed wrote it, with its sizes and packs.

On our laptop’s CPU, it encoded 553 customer records a second: about 2 minutes for all 60,108. You do not need to wait. The course data has the embedding of every record and every product in data/embeddings/, 23 MB. They are stored as 8-bit numbers with one scale per row, a quarter of the normal size. The cosine changes by less than 0.001.

  1. As a blocking. For each record, find its nearest records by cosine. These become the candidate pairs. This is how search by meaning works: the record is the query.
  2. As a score. For each candidate pair, the cosine is one number. Call the pair “the same” when the cosine is above a threshold. Or add the cosine to the comparison vector, as one more feature for LightGBM.

We try both on the customers, then on the products.

  1. Copy this lesson’s files from the snapshot end-of-m06. blocking.py replaces your file: it adds the blocking embed-nn. calibrate.py and llm.py belong to the next lesson, but runs/embedding.py needs them:

    Terminal window
    S=../nextia-learning-labs/C39/snapshots/end-of-m06
    cp $S/larkfield_match/embed.py $S/larkfield_match/blocking.py larkfield_match/
    cp $S/larkfield_match/calibrate.py $S/larkfield_match/llm.py larkfield_match/
    cp $S/larkfield_match/runs/embedding.py larkfield_match/runs/
    cp $S/requirements.txt .

    The new requirements.txt adds scikit-learn and LightGBM (you have them), and sentence-transformers and PyTorch. You need the last two only to make new embeddings, for example for new records. They take about 700 MB. This lesson does not need them, so you can install them later with python -m pip install -r requirements.txt. On Linux, install the CPU version of PyTorch first, or pip also brings large GPU packages: python -m pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu. We did not test this on Linux.

  2. Make candidate pairs from embeddings: each record with its 10 nearest records:

    Terminal window
    match block --blocking embed-nn
    embed-nn: 425,396 candidate pairs out of 1,806,455,778 possible pairs (137.32 s)
    pair completeness 0.336 (9,294 of 27,677 true pairs kept); reduction ratio 0.999765

    It took more than 2 minutes on our busy laptop: every record is compared with all 60,108 others, in chunks of 1,000.

  3. Score the candidate pairs of the five keys by the cosine alone:

    Terminal window
    match eval --run embed
    Ran embed: 8,734 pairs called the same, in 31.1 s
    embed on the test split: pair precision 0.724, recall 0.261, F1 0.384
    2,008 pairs called the same: 1,453 right, 555 wrong; 4,111 of 5,564 true pairs missed
    entities after clustering (all records): 52,292 (true: 37,500)

The results are poor. As a blocking, the 10 nearest records keep only 0.336 of the true pairs, against 0.994 for the five keys. As a score, the cosine alone reaches F1 0.384. Why?

The text is the name and the address. People in one household share the address and often the surname. So their texts are almost the same: “samuel green, 3294 kestrel ave heronvale” and “nathan green, 3294 kestrel ave heronvale”. Their cosine is 0.975, almost as high as Valérie’s own two records (0.984). The model does not know that the first name is the part that matters.

  • one customer
  • two people
Cosine of the candidate pairs of the test split, in our synthetic records. One customer: median 0.863. Two people: median 0.849. Only a cosine near 1 (the same text) is a clear sign.
Text description of the chart

Two curves over cosines from 0.7 to 1. The two-person curve rises from about 0.75, peaks between 0.82 and 0.86, and is almost zero above 0.96. The one-customer curve has two parts. About 27% of true pairs are at 0.98 or more: records with the same text. The other true pairs lie between 0.75 and 0.95, on top of the two-person curve.

The medians are 0.863 for one customer and 0.849 for two people. There is almost no gap. About a quarter of the true pairs have a cosine of 0.98 or more: their two records have the same name and address text. The model finds those, and so do our rules. The other true pairs look just like pairs of two people at one address. That is why embed has a precision of 0.724 but a recall of only 0.261. The notebook of this lesson goes one step further. Among pairs with the same postcode, two people have a higher TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One cosine (0.909) than one customer’s two records (0.865). Two people at one address often have exactly the same address text. One customer’s records often write it in two ways, or one has an old address.

Maybe the cosine helps as one more feature, next to the 16 of the comparison vector? We measured it (our measurement script, measure/m06_learned.py): LightGBM with the cosine as a 17th feature.

Features Precision Recall F1
comparison vector (16 fields) 0.918 0.886 0.902
comparison vector + embedding cosine 0.942 0.794 0.862

F1 falls from 0.902 to 0.862. Precision rises a little, and recall falls a lot. Our reading, not measured: a high cosine is common for strangers in one household, so the feature adds more noise than evidence.

The supplier feeds are a different case. Product titles in the two feeds are written in very different ways, so meaning should help. And it does find the product: true matches have a median cosine of 0.952, other candidates 0.890.

But the scored run p-embed reaches only F1 0.383. LightGBM on the product comparison vector reaches 0.996. Run it:

Terminal window
match eval --run p-embed
Ran p-embed: 2,301 pairs called the same, in 30.9 s
p-embed on the test split: pair precision 0.351, recall 0.421, F1 0.383
678 pairs called the same: 238 right, 440 wrong; 327 of 565 true pairs missed
entities after clustering (all records): 5,952 (true: 5,211)

Compare it with LightGBM on the product comparison vector, match eval --run p-lgbm:

Ran p-lgbm: 2,748 pairs called the same, in 32.5 s
p-lgbm on the test split: pair precision 0.998, recall 0.995, F1 0.996
563 pairs called the same: 562 right, 1 wrong; 3 of 565 true pairs missed
entities after clustering (all records): 5,213 (true: 5,211)

p-embed wrongly calls 440 pairs the same on the test split. Of these, 294 differ in size, for example a 5-pattern hose nozzle against a 10-pattern one. And 46 are a single item against a pack of 2. The two groups can overlap.

Cosines of example pairs. Customers: Valérie Tremblay's two records, one customer, 0.984; the brothers Samuel and Nathan Green, two people, 0.975. Products: a true match between the feeds, the Hearthside raised garden bed, 0.959; a trellis and a rake against their pack of 2, 0.989 and 0.979; a 10-pattern hose nozzle against the 5-pattern and 8-pattern ones, 0.991 and 0.99. Every wrong product pair scores higher than the true match.
Cosines of example pairs from the shipped embeddings. The brothers score almost as high as one customer's own records. A pack of 2 and a nozzle of another size score higher than a true product match.
Text description of the diagram

A table of example pairs with their cosine. The customer pairs: Valérie Tremblay’s garden-centre and shop records, and the brothers’ records. The product pairs: one true match written two ways, two single items against their pack of 2, and one hose nozzle in other sizes. A colour marks each pair as one entity or two.

For the model, “(Pack of 2)” is a small change at the end of a long title. So is “5-pattern” instead of “10-pattern”. These wrong pairs even score higher than the true match, whose titles are written in two styles. The meaning of the title barely changes, but the product does. The notebook of this lesson takes the 79 single items in feed A that have their own pack of 2. Their median cosine is 0.989, above the median true match (0.952).

  • true match
  • a pack of 2 and another product
  • other candidates
Cosine of the product candidate pairs (all splits). True matches: median 0.952. Candidate pairs of a pack of 2 with another product: 0.888. Other candidates: 0.89.
Text description of the chart

Three curves over cosines from 0.7 to 1. True matches are mostly between 0.92 and 0.98. The pairs with a pack of 2 and the other candidates lie mostly between 0.84 and 0.94, almost on top of each other. Both overlap the lower part of the true matches.

The product comparison vector has fields for exactly these details: size_ratio, pack_eq and price_ratio (Module 4). That is why p-lgbm gets 0.996, and the cosine gets 0.383.

F1 of the embedding runs (green) against the field-by-field runs (blue), on our synthetic customers (lgbm, embed) and products (names that start with p-). Embeddings lose every time.
Text description of the chart

Five dots. Customers: lgbm 0.902; lgbm with the cosine as a feature 0.862; the cosine alone 0.384. Products: p-lgbm 0.996; p-embed 0.383.

On our records, embeddings did not help. That is a real result, not a failure of the lesson. In other data, they can help. We did not test the three cases below in this course; they are where embeddings have the best chance:

  • Free text with the same meaning in other words: product descriptions, company activities, job titles. Our records have short, structured fields instead.
  • Names in two scripts or two transliterations, such as a name in Arabic letters and in Latin letters. Our synthetic names are all in Latin letters.
  • As a fallback blocking for records that every key misses, together with the keys, never alone.

The search and ranking course found the same pattern for queries: see Where dense search fails.

Use them when Do not use them when What it costs
the texts say the same thing in other words the difference is a number, a unit, a size or a pack encoding: about 2 minutes for 60,108 records on a laptop CPU (measured)
names come in different scripts or transliterations people share most of the text (households, one address) storage: 384 numbers a record; 23 MB for 68,069 records in 8 bits
as a feature, after you measured that it helps you have not compared them with field comparisons on the same test pairs about 700 MB of packages (PyTorch) to make new embeddings

Common mistake

One embedding for the whole record

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. puts every field of a customer into one text: name, email, phone, address and date of birth. He takes the cosine of two records as the match score, as embed does with the name and address. Strangers at one address get high scores; a customer who moved gets a low one.

Cause: one cosine mixes all fields into one number. It cannot say “the address agrees, but the first name and the date of birth differ”. It weighs each field by how much text it has, not by how much evidence it gives. The address is the longest field, so it decides. In our measurement, the name-and-address cosine alone reached F1 0.384.

Fix: compare field by field, as the comparison vector does. If you use embeddings, embed one field (for example a product title or a company activity) and add its cosine as one feature. Then check on the validation pairs that it helps.

Consequence in AI work: an embedding is a summary. It is good for finding things that are about the same topic, and weak at exact facts: numbers, dates, sizes and names. Retrieval systems meet the same problem; see the RAG course.

Check your understanding

The brothers Samuel and Nathan Green get a cosine as high as one customer's own two records. Explain why, from what the model reads.

Show the answer

The model reads the name and the address as one text. The brothers share the surname, the house number, the street and the town, so most of the text is the same. Only the first name differs, and it is a small part of the text. The model has no idea that the first name is the field that separates two people in one household. So their embeddings point almost the same way.

Check your understanding

True product matches have a median cosine of 0.952, other candidates 0.890. So embeddings do see the product. Why does p-embed still reach only F1 0.383?

Show the answer

Many wrong candidates are the same product line in another size, colour or pack. Their titles differ by one number or a few words, so their cosine is also high. A threshold that keeps most true matches also keeps many of these. Of the 440 wrong matches, 294 differ in size, and 46 are a single item against a pack of 2. The comparison vector has fields for size, pack and price, so LightGBM on it reaches 0.996.

Check your understanding

Adding the cosine as a 17th feature lowered LightGBM's F1 from 0.902 to 0.862. Is “more features are always better” true? What should you do before adding a feature?

Show the answer

No. A feature that is high for many non-matches (strangers at one address) adds noise. The model can also learn the wrong balance from only 3,000 labels. Here, precision rose a little but recall fell from 0.886 to 0.794. Before you keep a feature, train with and without it and compare on the validation pairs, then confirm once on the test pairs.

Recap

  • An embedding is a list of numbers from a model; similar texts get similar lists. The cosine compares two embeddings.
  • The course data ships every record’s embedding (384 numbers, 8 bits), so no lesson runs the model.
  • Customers: households share most of the text. As a blocking, the 10 nearest records keep 0.336 of the true pairs. As a score, F1 is 0.384. As one more feature, F1 falls from 0.902 to 0.862.
  • Products: embeddings find the product line, but miss sizes and packs: p-embed F1 0.383 against 0.996 for p-lgbm.
  • Embeddings help with free text, other words and other scripts. Measure them against field comparisons before you use them.

Next, we try a bigger model for the hardest pairs only: a language model that reads both records and answers.

Next: A language model for the unsure pairs