Skip to content
Français

Exact matches and rules

  • Lesson 10
  • 30 min
  • Module 3: Lexical search: the baseline to beat

Where we are

BM25 is a strong baseline for most queries. Some queries are a product code, a model name or a size. In this lesson, we see where an exact-match rule beats any score, and we measure the queries with codes and numbers apart.

What you will be able to do

Find the one right product when a person types its code, and decide when a simple rule is safer than any score.

NotebookDo the rule of this lesson in a notebookColab · Kaggle · GitHub

The notebook shows why BM25 cannot find a code, writes the rule, measures it on the code queries, and finds the trap. You write a stricter rule yourself. Its setup cells download the data, so it needs no local setup. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • The project larkfield-search with this module’s files. You copied rules.py with the other files in the last lesson.

A customer comes back to the shop with an old receipt. She points at a line: “LK-91857. I want this one again.” The assistant does not ask her to describe the product. She does not walk the aisles and guess. She types the number into the stock list, and the product appears: an arlo twin over full bunk bed.

The number has one meaning and one answer. Describing it would be slower, and it could fail. And if the number is not in the stock list, the assistant does not invent a product. She says so and asks a question.

The problem: a code is not a word of the text

Section titled “The problem: a code is not a word of the text”

People type product codes. They copy them from a receipt, an email or an advert. In the course data, these are planted queries: a practice problem we made on purpose. The synthetic log has 301 such searches, and Larkfield’s search finds nothing for every one of them.

BM25 does no better. Try one:

Terminal window
search query "LK-91857" --run bm25
bm25: "LK-91857" -> 0 results (query C-002: judged results are marked)

The reason is in the index. The tokens of “LK-91857” are lk and 91857. No title and no description contains “91857”. The code lives in the column product_id, and that column is not in the index. BM25 can only find words that are in the text. So the 20 code queries of the test set score 0.000 with every run so far.

An exact-match rule recognises a pattern in the query and goes straight to the answer, without a score. Here, the pattern is a product code: “LK” and five digits. If a product with that ID exists, it goes first. The other results follow, in BM25’s order.

People write a code in several ways. The 40 code queries of the course use four forms:

What the person types Form Queries
91857 the number alone 15
LK-91857 as printed 11
lk91857 lower case, no dash 9
lk 91857 lower case, a space 5

The rule in larkfield_search/rules.py accepts all four with one regular expression, a small pattern language for text:

CODE = re.compile(r"\b(?:lk[\s-]?)?(\d{5})\b", re.IGNORECASE)

Read it from left to right. \b is the edge of a word. (?:lk[\s-]?)? is an optional “lk”, followed by an optional space or dash. (\d{5}) is exactly five digits, which the rule keeps. re.IGNORECASE makes “LK” and “lk” the same. For each match, the rule builds the ID LK- plus the five digits, and it keeps the ID only if it is in the catalogue.

A query leads to a question: does the query have a product code, written LK-91857, lk91857, lk 91857 or 91857, and is that product in the catalogue? Yes leads to: put that product first, then the BM25 results; example lk 91857 gives LK-91857, arlo twin over full solid wood bunk bed. No leads to: the BM25 results only, BM25 per field. A dashed box below says: the trap, 44.5% of all five-digit numbers are real codes; 12000 btu air conditioner gives LK-12000, which exists, so a closet door pivot rod comes first.
The exact-match rule for product codes. It answers without a score when the pattern has one meaning. The dashed box shows where the pattern has another meaning.
Text description of the diagram

A query enters at the top. The rule asks one question. Does the query contain a product code (LK-91857, lk91857, lk 91857 or 91857), and is that product in the catalogue? If yes, that product goes first, and the BM25 results follow. For example, “lk 91857” puts LK-91857, the arlo twin over full solid wood bunk bed, first. If no, the page shows the BM25 results per field only. A dashed box at the bottom shows the trap. 44.5% of all five-digit numbers are real product codes. So “12000 btu air conditioner” finds LK-12000, and a closet door pivot rod comes first.

Look at the tiny example in the notebook. Each text goes through the same two steps: find the pattern, then check the catalogue.

Text Pattern found In the catalogue? Result
LK-91857 91857 yes LK-91857 first
lk 91857 91857 yes LK-91857 first
bunk bed 91857 91857 yes LK-91857 first
LK-00001 00001 no the BM25 results only

The check against the catalogue matters. Without it, “LK-00001” would put a product that does not exist on the page.

The run bm25-fields+codes is the per-field BM25 of the last lesson, with the rule in front. Try the code from the receipt in another form:

Terminal window
search query "lk 91857" --run bm25-fields+codes -k 3
bm25-fields+codes: "lk 91857" -> 1 results
1. LK-91857 arlo twin over full solid wood standard bunk bed Kids Beds 208.99

The rule finds the bunk bed. BM25 adds nothing: no product text has “lk” or “91857”. So the page has one result, and it is the right one.

Now measure the run on all the test queries:

Terminal window
search eval --run bm25-fields+codes
bm25-fields+codes on 243 test queries: NDCG@10 0.506 (95% interval 0.458 to 0.552)
recall@100 0.480, MRR 0.546, zero results 4.5%, judged in the top 10 71.0%; 0.2 ms a query (p50), 0.7 ms (p95)

NDCG@10 rises from 0.423 to 0.506. That is the largest step of this module, and it comes from about ten lines of code.

An average can hide where a change helps and where it hurts. search compare with --by source splits the test queries by where they come from:

Terminal window
search compare bm25 bm25-fields+codes --by source
bm25-fields+codes minus bm25 on 241 test queries: ndcg@10 +0.077 (95% interval +0.037 to +0.119); better on 100, worse on 75, the same on 66
source queries bm25 bm25-fields+codes difference low high worse
code 20 0.000 1.000 1.000 1.000 1.000 0
planted 3 0.451 0.236 -0.215 -0.435 0.000 2
translated 59 0.177 0.178 0.001 -0.025 0.027 13
typo 40 0.357 0.362 0.005 -0.036 0.048 17
wands 119 0.650 0.641 -0.009 -0.039 0.020 43
The change in NDCG@10 from bm25 to bm25-fields+codes (bm25-fields+codes minus bm25), by source of the test queries, with 95% intervals. All the gain is on the product codes.
Text description of the chart

The chart shows one row per source of test queries, with the difference in NDCG@10 and its 95% interval. The axis goes from −0.5 to +1, and 0 means no change. Product codes (20 queries): +1.000, from 0 to 1, with no spread. Planted broad queries (3): −0.215 (−0.435 to 0.000). French (59): +0.001 (−0.025 to +0.027). Misspelt (40): +0.005 (−0.036 to +0.048). English WANDS queries (119): −0.009 (−0.039 to +0.020). Only the product-code row is far from zero.

Read the table row by row:

  • Codes: from 0.000 to 1.000 on all 20 queries. Each code query has one right product, and the rule puts it first every time.
  • English WANDS queries: −0.009, with an interval from −0.039 to +0.020. No real change. The rule never fires on these queries; the small change comes from the field weights.
  • Planted broad queries (“sofa”, “couch”, “canapé”): −0.215, but on 3 queries only, and the interval reaches 0. The fields cause it, not the rule.

To see the rule alone, compare the run with and without it:

Terminal window
search compare bm25-fields bm25-fields+codes
bm25-fields+codes minus bm25-fields on 241 test queries: ndcg@10 +0.083 (95% interval +0.050 to +0.120); better on 20, worse on 0, the same on 221

Better on 20, worse on 0, the same on 221. The rule touches exactly the 20 code queries, and nothing else on the test set.

From now on, every run of the course keeps this rule. So later modules compare their runs with bm25-fields+codes, not with bm25.

Here are the five runs of the first three modules, on the same 243 test queries:

NDCG@10 of every run so far on the test queries (mean of 241 of the 243), with 95% intervals. The code rule is the only change of this module with an interval clearly above BM25's.
Text description of the chart

The chart shows five runs, each with NDCG@10 and its 95% interval on the test queries (mean of 241 of the 243). current, Larkfield’s search today: 0.195 (0.155 to 0.237). bm25: 0.429 (0.381 to 0.476). bm25-title: 0.423 (0.374 to 0.471). bm25-fields: 0.423 (0.375 to 0.471). bm25-fields+codes: 0.506 (0.458 to 0.552), highlighted.

Run What it does NDCG@10 (95% interval) recall@100 MRR Zero results
current Larkfield today: every word must match 0.195 (0.155 to 0.237) 0.146 0.312 66.7%
bm25 BM25, title and description as one field 0.429 (0.381 to 0.476) 0.388 0.438 12.8%
bm25-title BM25 on the title only 0.423 (0.374 to 0.471) 0.351 0.409 14.4%
bm25-fields BM25 per field, weights tuned on validation 0.423 (0.375 to 0.471) 0.375 0.441 12.8%
bm25-fields+codes the same, and a product code puts its product first 0.506 (0.458 to 0.552) 0.480 0.546 4.5%

Codes are not the only exact things in queries. People type sizes (“48 in entry table”), capacities (“washer 4.5”), and model names (“zachary 72.5”). These are not product codes, so the rule does not fire. BM25 treats their numbers as ordinary tokens.

The tokenizer splits on every character that is not a letter or a digit. So:

Query Tokens
zachary 72.5 zachary, 72, 5
3 1/2 inch drawer pull 3, 1, 2, inch, drawer, pull
blk 18x18 seat cushions blk, 18x18, seat, cushions

Sometimes this works well. For “zachary 72.5”, the rare name “zachary” does most of the work, and the right bookcase comes first:

Terminal window
search query "zachary 72.5" --run bm25-fields -k 2
bm25-fields: "zachary 72.5" -> 2 results (query W-064: judged results are marked)
1. LK-29728 zachary 72.5 '' h x 24 '' w steel ladder bookcase Teen Bookcases 96.99 [exact]
2. LK-15434 slagelse 72 '' double bathroom vanity base only Vanity Bases 149.99 [not judged]

The second result shows the weakness. “72” matches a 72-inch vanity, because BM25 sees “72” as a word like any other. For “ge top loading washer 4.5”, BM25 finds washers of 4.4, 5 and 5.4 cubic feet. The tokens “4” and “5” appear in many sizes.

The test set has 10 queries with a number that are not codes. On them, bm25 scores 0.414 and bm25-fields+codes 0.368. Ten queries are too few for a firm conclusion. But a rule for sizes is harder than a rule for codes. “4.5” can mean cubic feet, inches or a rating. And the product text writes sizes in many ways. Module 4 detects some of them, such as a price limit, and turns them into filters.

Model and brand names are the other side of the IDF from the last lesson. A rare name has a high IDF. This helps when the name belongs to the product, as “zachary” does. It hurts when the name also belongs to other products, as “emma” did.

When a rule beats any model, and when it does not

Section titled “When a rule beats any model, and when it does not”

Use an exact-match rule when a pattern has one meaning and one answer that you can check. Examples are product codes, order numbers, an ISBN or a full model number. Here, no score can do better than 1.000. A rule also needs no training data, runs in microseconds, and is easy to explain.

Do not use a rule when the pattern has other meanings. Our rule accepts a five-digit number alone, because 15 of the 40 code queries are written that way. But Larkfield’s IDs run from LK-10000 to LK-99997. So 40,066 of the 90,000 five-digit numbers are real IDs: 44.5%. Here is a query that we wrote to test this. It is not in the course’s query set:

Terminal window
search query "12000 btu air conditioner" --run bm25-fields+codes -k 3
bm25-fields+codes: "12000 btu air conditioner" -> 3 results
1. LK-12000 bi-fold closet door bottom pivot rod Door Hardware Accessorie 42.99
2. LK-91211 quick fill external electric air pump Air Mattresses 244.99
3. LK-86558 bike/motorcycle hand air pump Air Mattresses 214.99

“12000” is the power of an air conditioner, in BTU, a unit of heat. The rule reads it as a code, and a closet door pivot rod comes first.

The measurement cannot see this risk. On the 624 queries with judgments, the rule never fires on a query that is not a code. In the synthetic log, it fires on exactly the 301 code searches. A test set shows only the problems that it contains. So test a rule on the queries where it should not fire, too.

What a rule costs. Running it costs almost nothing. The cost is keeping it right. Every rule is a promise about what a pattern means. When the catalogue or the people change, the promise can break. Keep each rule small, with tests for the queries where it must fire and where it must not.

Make the rule stricter. Accept a bare five-digit number only when it is the whole query, and accept “lk” with a code anywhere. In larkfield_search/rules.py, change two lines:

CODE = re.compile(r"\blk[\s-]?(\d{5})\b|^\s*(\d{5})\s*$", re.IGNORECASE)
pid = f"LK-{m.group(1) or m.group(2)}"

The pattern now has two alternatives, so a match keeps its digits in group 1 or in group 2. Before you run anything, predict: does NDCG@10 on the test queries change?

Then run search query "12000 btu air conditioner" --run bm25-fields+codes -k 3, search eval --run bm25-fields+codes --by source and python -m pytest -q.

What we got: the air pumps come first, and the closet door rod is gone. NDCG@10 stays 0.506, with 1.000 on the codes, and the 19 tests pass. When you are done, copy rules.py back from the snapshot end-of-m03, so your project stays the same as the course’s:

Terminal window
cp ../nextia-learning-labs/C38/snapshots/end-of-m03/larkfield_search/rules.py larkfield_search/rules.py

Common mistake

A rule that fires on the wrong query

Symptom: after a release with a new rule, a person searches “12000 btu air conditioner” and sees a closet door part first. The offline measurement did not change, so nobody expected a problem.

Cause: the pattern of the rule (five digits) has a second meaning (a power, a size or a year). The rule cannot tell the meanings apart. The test queries had no such query, so NDCG@10 could not show it.

Diagnosis: run the rule alone over the whole query log, and list the queries where it fires. Read the ones that are not planted codes. In the notebook, you do the same with three test queries of your own.

Fix: narrow the pattern (an “lk” before the digits, or the number alone), and keep the check against the catalogue. Add the wrong case as a test, so that a later change cannot bring it back.

Consequence in AI work: rules and models fail in different ways. A model is wrong a little on many queries; a rule is right on its pattern and badly wrong outside it. Test a rule where it must not fire, not only where it must.

Check your understanding

BM25 scores 0.000 on the 20 code queries of the test set. Explain why with the inverted index, and why the rule scores 1.000.

Show the answer

The tokens of “LK-91857” are lk and 91857. The index is built from the title and the description, and no product text contains “91857”. The code is in the product_id column, which is not indexed. So every postings list is empty and BM25 finds nothing. The rule does not use the index. It reads the code from the query, checks that the product exists, and puts it first. Each code query has one right product, so the first result is right every time, and NDCG@10 is 1.000.

Check your understanding

`search compare bm25 bm25-fields+codes --by source` gives +0.077 overall. Hana says: “Great, the new run is better for every customer.” What does the table say?

Show the answer

The gain comes from the 20 code queries: +1.000 on each. On the 119 English WANDS queries, the difference is −0.009, with an interval from −0.039 to +0.020: no measurable change. The French and misspelt queries do not change either. So the new run helps the people who type a code, and changes nothing measurable for the others. The average of +0.077 is real, but it comes from one slice.

Check your understanding

A colleague proposes a rule: “If the query has a number followed by ‘inch’, keep only the products whose title has that number.” Name one query where it helps and one where it hurts, and say how you would test it.

Show the answer

It can help “46 inch closet door”: products with “46” in the title come first. It can hurt when the title writes the size in another way, such as 46 ‘’ or 117 cm. It can also hurt when the number is another measure: a 48-inch-wide cabinet for “48 inch kitchen hood”. Test it as the code rule: measure with search compare on the queries with numbers (--by has_number). Then list the queries in the log where it fires, and read the ones where it should not.

Recap

  • BM25 cannot find a product code: the code is not a word of the indexed text. The 20 code queries score 0.000 with every run before this lesson.
  • An exact-match rule recognises a pattern, checks the catalogue and puts the product first. bm25-fields+codes scores 0.506 (0.458 to 0.552), against 0.423 without the rule.
  • The gain is all on the codes: +1.000 on 20 queries, no measurable change on the others. Every later run keeps the rule.
  • Numbers and sizes are not codes. BM25 treats them as ordinary tokens, which works for rare names and fails for sizes.
  • A rule is right on its pattern and badly wrong outside it. 44.5% of five-digit numbers are real codes. So “12000 btu air conditioner” shows a closet door part. Test a rule where it must not fire.

You can now measure a lexical search against a strong baseline: BM25 per field, with a rule for codes. But misspelt queries still score 0.362, and French queries 0.178. “outdor” and “canapé” are not words of the English text. Next, you clean up the query and the products the same way, and measure what it fixes and what it breaks.

Next: Normalise text