Skip to content
Français

Synonyms and query rewriting

  • Lesson 13
  • 40 min
  • Module 4: Understand the query

Where we are

In the last two lessons, we normalised text and fixed spelling, so the words that people type now match more products. Some queries are spelt right and still miss, because the person uses another word for the same thing. In this lesson, we add synonym rules, mine rules from the query log, try a language model that rewrites queries, and test every rule.

What you will be able to do

Find the right products when people use a different word from the catalogue: “couch” for “sofa”, French for English. You write synonym rules, test each rule alone, and weigh a language model that rewrites the whole query.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook adds one synonym rule to a small BM25 search and asks you to measure it on every validation query that it touches. Then it mines pairs of queries with the same clicks from the synthetic log, and compares the recorded rewrites of a language model by language. Its setup cells download the practice data, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the guided practice)

  • The project larkfield-search with this module’s files, from the guided practice of Normalise text. python -m pytest -q must say 24 passed.
  • A text editor (VS Code) to write one short Python file.

A shop assistant who knows that a couch is a sofa

Section titled “A shop assistant who knows that a couch is a sofa”

A customer asks the shop assistant for a couch. The shop’s labels say “sofa”. She knows that a couch and a sofa are the same thing, and she walks to the sofas.

She also knows the limits. A “sectional” is a sofa, but a sofa is not always a sectional. And for “light blue cushions”, she does not bring lamps. Some word rules go both ways, some go one way, and some are wrong in a context. This lesson writes such rules for the search, and measures each one.

The problem: the customer’s word is not the catalogue’s word

Section titled “The problem: the customer’s word is not the catalogue’s word”

The test set has a planted broad query, “couch”, a practice problem we made on purpose. The catalogue mostly says “sofa”. With bm25-norm+spell, the top 5 for “couch” are a folding chaise lounge, a kids’ foam couch and a floor couch for gaming. Then come two throw pillows with “couch” in their names. NDCG@10 is 0.278.

Spelling correction cannot help: “couch” is spelt right. Normalising cannot help: “couch” is not a form of “sofa”. Only a rule that says “couch means sofa” can join them. French queries have the same problem on a larger scale: “canapé” is “sofa” in French, and only 7,597 products have a French title.

A synonym is a word with the same meaning as another word. A synonym rule tells the search to add words to a query before it searches. This is called query expansion: the query grows. There are two kinds of rule.

  • A two-way rule, written couch, sofa, says the words are equal. A query with “couch” also searches “sofa”, and a query with “sofa” also searches “couch”.
  • A one-way rule, written sectional => sofa, says the left word is a kind of the right word. A query with “sectional” also searches “sofa”. A query with “sofa” gets nothing added.
Two kinds of synonym rule. Two-way rule couch, sofa: the query couch searches couch plus sofa, counting half; the query sofa searches sofa plus couch, counting half. One-way rule sectional => sofa: the query sectional searches sectional plus sofa, counting half; the query sofa searches sofa with nothing added.
A two-way rule adds words in both directions; a one-way rule only from left to right (the diagram draws => as ⇒). Added words count half.
Text description of the diagram

Two groups of boxes. “Two-way rule: couch, sofa”: “Query: couch” leads to “Search: couch + sofa (counts half)”; “Query: sofa” leads to “Search: sofa + couch (counts half)”. “One-way rule: sectional => sofa”: “Query: sectional” leads to “Search: sectional + sofa (counts half)”; “Query: sofa” leads to a box with a thick border: “Search: sofa, nothing added”.

In the project, an added word counts half. The search adds 0.5 × the BM25 score of the added words to the score of the person’s own words. Why half? The person’s own word is the best evidence of what they want. When the person typed “couch”, a product that says “couch” should rank above a product that only says “sofa”.

A tiny example: a product’s BM25 score for “couch” is 4.0, and its score for “sofa” is 3.0. Its score for the query “couch” becomes 4.0 + 0.5 × 3.0 = 5.5. A second product says only “sofa”: 0 + 0.5 × 3.0 = 1.5. Both are found now, and the product with the person’s own word stays first.

Larkfield’s rules are in data/synonyms.csv: 20 rules, one per line. 18 are two-way and 2 are one-way. The file starts like this:

rule,kind,note
"couch, sofa",two-way,
"grey, gray",two-way,
"rug, carpet",two-way,
"nightstand, bedside table, night stand",two-way,
"loveseat, love seat",two-way,

It ends with the two one-way rules and a rule that we planted on purpose:

sectional => sofa,one-way,a sectional is a kind of sofa
recliner => chair,one-way,
"light, lamp",two-way,"planted: breaks light blue, light wood"

The rule “light, lamp” is a practice problem we made on purpose. It looks reasonable, and it is the kind of rule that a person adds in five seconds.

The run bm25-norm+spell+syn uses all 20 rules. On the test queries, it reaches NDCG@10 0.620 (95% interval 0.574 to 0.662), against 0.615 without the rules. Each mean covers 241 of the 243 queries. The difference is +0.005 (95% interval −0.002 to +0.011): 13 queries better and 7 worse.

That one number hides 20 different rules. Some help, some hurt, and some do nothing. So you test each rule alone. Take every validation and test query that the rule changes, and compare NDCG@10 with and without that rule only.

Rule Queries it touches NDCG@10 without with Better Worse
couch, sofa 8 0.516 0.615 5 1
patio, outdoor 14 0.628 0.643 6 3
dresser, chest of drawers 2 0.644 0.743 2 0
sectional => sofa 2 0.503 0.577 2 0
light, lamp (planted) 10 0.784 0.778 2 4
wardrobe, armoire 4 0.049 0.046 0 1
tv, television 0

The full list has all 20 rules. Four rules touch no query at all: “tv, television”, “drapes, curtains”, “armchair, accent chair” and “bookcase, bookshelf”. They do nothing that you can measure, good or bad.

Read the table row by row.

  • “couch, sofa” is the best rule: 5 queries better, 1 worse. The query “couch” alone goes from 0.278 to 0.546.
  • “light, lamp”, the planted rule, makes 4 queries worse and 2 better. We planted it to break “light blue” and “light wood”. On these queries, those two did not change. Instead, “family lamp” fell from 0.724 to 0.687, because “light” is a very common word. A planted problem did not fail where we expected, but it failed.
  • “wardrobe, armoire” hurts French queries. In French, “armoire” means any cupboard: “armoire de rangement haute” is a tall storage cabinet. The rule adds “wardrobe”, which is wrong for most of them.

One rule that helps one query and hurts another

Section titled “One rule that helps one query and hurts another”

The rule “patio, outdoor” touches 14 queries. On average it helps: 0.628 to 0.643. But look at each query:

The rule “patio, outdoor” alone, on each validation and test query that it changes (spelling corrected first). The same rule makes “wooden chair outdoor” better and “patio design” much worse.
Text description of the chart

One dot per query, on an axis of differences from −0.5 to +0.5, with a line at zero. outdoor privacy wall: +0.049; cover set for outdoor furniture: −0.117; wooden chair outdoor: +0.109; flared outdoor sofa: +0.452; modern outdoor furniture: +0.057; outdoor waterproof chest: −0.063; outdoor sectional dining: +0.073; patio design: −0.421.

The rule makes “wooden chair outdoor” better: from 0.504 to 0.613. Chairs that say “patio” but not “outdoor” can now match.

The same rule makes “patio design” much worse: from 0.792 to 0.371. The person typed “patio”, and the WANDS judges graded patio chairs and patio sofas exact. The rule adds “outdoor”, a much more common word, and the order changes. Products with both words move up, and many of them were judged irrelevant or not judged at all. The exact patio chairs and sofas move down.

The two words are not equal. Patio furniture is outdoor furniture, but “outdoor” covers more than patios. So the rule should be one-way. You test outdoor => patio in Your turn below.

Choose rules on validation, confirm on test

Section titled “Choose rules on validation, confirm on test”

The table above uses the validation and the test queries together, because each rule touches only a few queries. That is fine for finding bad rules. But if you change a rule because of these queries, they no longer test it fairly. This is the same trap as tuning field weights on the test set, from BM25. Change rules by looking at validation queries and at the log, then confirm on new queries that you did not look at.

Writing rules by hand is slow. The query log can suggest rules. The idea: if two different queries lead people to click the same products, the queries probably mean the same thing.

To compare two sets of clicked products, use the Jaccard similarity. It is the number of products in both sets, divided by the number in either set. Here is a tiny example. Query A led to clicks on products 1, 2, 3 and 4. Query B led to clicks on 2, 3, 4 and 5. Three products are in both, and five are in either. Jaccard = 3 ÷ 5 = 0.6. Identical sets give 1, and sets with nothing in common give 0.

We mined Larkfield’s synthetic log this way. We kept every pair of different queries with at least 3 clicked products each and a Jaccard of 0.3 or more. It found 29 pairs. The first five are:

Query A Query B Jaccard
larkfield tension larkfield tension rod 1
outdoor movie outdoor movie screen 0.833
aloe vera aloe vera plant 0.75
large cushion outdoor large cushion outdoor chair 0.75
luau string luau string lights 0.75

These are not synonyms. In each of these five pairs, and in most of the 29, one query is the other without its last word. When a simulated person searches again, our code drops the last word of the query. So the mining found exactly the rule that made the synthetic log. Mined rules show how your log was made, and with real people they show real behaviour.

With a real log, the same method finds real pairs, such as a British and an American word. It also finds pairs that are related but not equal. So a mined rule is a suggestion for a person to check, and then to measure.

A synonym rule changes one word. A query rewrite changes the whole query. A language model reads the query and writes a better one: it fixes spelling, translates French into English and writes units one way. The course recorded the answers of chat-small, a small, low-cost chat model on Azure, for the 364 validation and test queries. The run llm-rewrite replays these recorded answers, so you need no key and pay nothing. Its instructions asked it to keep names, numbers and product codes, and not to add details that the person did not ask for.

Some recorded rewrites:

Typed Recorded rewrite
dnosaur dinosaur
outtdoor sectional doning outdoor sectional dining
blk 18x18 seat cushions black 18x18 seat cushions
canapé sofa
abeille bee
filaret outdoor sofa filaret outdoor sofa

The last row matters: the model kept the name “filaret”, where the spelling corrector of the last lesson changed it into “flared”.

NDCG@10 with its 95% interval on Larkfield's test queries: the best rules of this module against the language model's recorded rewrites. The rewrite wins most on French.
Text description of the chart

Three panels with two rows each. English WANDS queries: rules 0.703, recorded rewrite 0.721. Misspelt queries: rules 0.659, rewrite 0.681. French queries: rules 0.301, rewrite 0.69. Each value has its 95% interval as a line.

On all test queries, the recorded rewrite reaches NDCG@10 0.728 (95% interval 0.690 to 0.764), against 0.620 for the rules: +0.108 (+0.078 to +0.140). It is the best single change of this module. Most of the gain is French: from 0.301 to 0.690. On the real English queries, the gain is small but clear: +0.018 (+0.002 to +0.036).

Before Larkfield uses it, read the caveats. Each one is measured.

  1. It is slow. A live call to chat-small took 1.2 seconds per query (p50) and 1.6 seconds (p95), on 20 test calls. Amira’s budget for a whole search is 200 milliseconds. The rule-based search takes 1.1 ms. A live rewrite is about a thousand times slower than the search itself.
  2. It looks too good on French. Our French queries were written by a language model from English queries. The rewrite asks a model of the same family to translate them back. That is unrealistically easy. For 43 of the 60 French test queries, the rewrite is word for word the English query that the French query was made from. For example, “tables d’appoint stoneford blanches et bois” became “stoneford end tables white and wood”, exactly the original. Real French customers make typos, mix languages and use words that no translation produced. Measure again on real French queries before you trust 0.690.
  3. It works only where an answer was recorded: the validation and test queries. For any other query, llm-rewrite searches the query unchanged.
  4. The money is not the problem. A rewrite costs about US$0.04 per 1,000 queries at chat-small’s price. Time is the problem, not cost.

So the rewrite is strong, slow, and only fair to measure on real French queries. One practical way to use it: cache the rewrites. Compute them ahead of time for the most common query texts, and store them. In the synthetic log, the 100 most common texts make 51.1% of all searches, and those searches would get the rewrite with no delay. The others use the rules.

  1. In larkfield-search, with the virtual environment active, start Python with python, and try the expansion:

    >>> from larkfield_search import synonyms
    >>> synonyms.expansion("couch")
    'sofa'
    >>> synonyms.expansion("modern sofa")
    'couch'
    >>> synonyms.expansion("sectional")
    'sofa'
    >>> synonyms.expansion("light blue pillow")
    'lamp'

    The last line is the planted rule at work: “light” here is a colour, and the rule adds “lamp”. Type exit() to leave Python.

  2. Measure all 20 rules together, by kind of query:

    Terminal window
    search compare bm25-norm+spell bm25-norm+spell+syn --by source
    bm25-norm+spell+syn minus bm25-norm+spell on 241 test queries: ndcg@10 +0.005 (95% interval -0.002 to +0.011); better on 13, worse on 7, the same on 221
    source queries bm25-norm+spell bm25-norm+spell+syn difference low high worse
    code 20 1.000 1.000 0.000 0.000 0.000 0
    planted 3 0.384 0.515 0.132 0.000 0.269 0
    translated 59 0.301 0.301 -0.000 -0.001 0.000 1
    typo 40 0.648 0.659 0.011 0.000 0.024 1
    wands 119 0.701 0.703 0.002 -0.009 0.015 5
  3. Write a script that tests one rule alone. Create the file rule_test.py in the folder larkfield-search, with this code:

    """Test one synonym rule on the validation and test queries: python rule_test.py "couch, sofa" """
    import sys
    import pandas as pd
    from larkfield_search import compare, data, index, spell, synonyms
    from larkfield_search.runs.understand import norm_scores
    rule = sys.argv[1]
    queries = pd.concat([data.split("validation"), data.split("test")])
    judged = data.judgments()
    rows = []
    for qid, text in zip(queries["query_id"], queries["query"]):
    text = spell.correct(text)[0]
    if not synonyms.matches(text, rule):
    continue
    scores = norm_scores(text)
    added = synonyms.expansion(text, only=rule)
    with_rule = scores + synonyms.EXPANSION_WEIGHT * norm_scores(added) if added else scores
    before = compare.ndcg([p for p, _ in index.top(scores, 10)], judged[qid])
    after = compare.ndcg([p for p, _ in index.top(with_rule, 10)], judged[qid])
    rows.append((qid, text, added, round(before, 3), round(after, 3)))
    table = pd.DataFrame(rows, columns=["query_id", "query", "added", "without", "with"])
    print(table.to_string(index=False))
    better = (table["with"] > table["without"]).sum()
    worse = (table["with"] < table["without"]).sum()
    print(f"{len(table)} queries; mean NDCG@10 without {table['without'].mean():.3f}, "
    f"with {table['with'].mean():.3f}; better {better}, worse {worse}")

    For each query, the script corrects the spelling, as the run does. If the rule matches, it scores the products twice: without the rule, and with the rule’s added words at half weight. Then it computes NDCG@10 of both top 10s.

  4. Run it on the rule “patio, outdoor”:

    Terminal window
    python rule_test.py "patio, outdoor"
    query_id query added without with
    W-013 outdoor privacy wall patio 0.443 0.492
    W-105 outdoor movie screen patio 1.000 1.000
    W-287 cover set for outdoor furniture patio 0.728 0.611
    W-367 wooden chair outdoor patio 0.504 0.613
    W-404 large cushion outdoor chair patio 0.470 0.470
    T-042 outdoor movie screen patio 1.000 1.000
    W-090 flared outdoor sofa patio 0.438 0.890
    W-133 outdoor light fixtures patio 1.000 1.000
    W-164 modern outdoor furniture patio 0.296 0.353
    W-325 outdoor waterproof chest patio 0.109 0.046
    W-388 outdoor sectional dining patio 0.503 0.576
    W-443 patio design outdoor 0.792 0.371
    T-055 outdoor light fixtures patio 1.000 1.000
    T-059 outdoor sectional dining patio 0.503 0.576
    14 queries; mean NDCG@10 without 0.628, with 0.643; better 6, worse 3

    Some queries appear twice: a misspelt copy (T-) becomes the same text as its original (W-) after correction. W-090 shows “flared”, the corrector’s mistake from the last lesson.

  5. Measure the recorded rewrites, by language:

    Terminal window
    search eval --run llm-rewrite --by language
    llm-rewrite on 243 test queries: NDCG@10 0.728 (95% interval 0.690 to 0.764)
    recall@100 0.617, MRR 0.710, zero results 0.4%, judged in the top 10 88.8%; 0.2 ms a query (p50), 0.4 ms (p95)
    language queries ndcg@10 low high zero-result rate
    en 182 0.740 0.695 0.781 0.005
    fr 61 0.690 0.618 0.759 0.000

    The time, 0.2 ms, is only the search. The rewrite was recorded, so search eval does not wait for the model. Live, add about 1.2 seconds per query.

Change one thing: write “patio, outdoor” as a one-way rule. Before you run it, predict: which direction stops the harm to “patio design”, outdoor => patio or patio => outdoor? Which one keeps “wooden chair outdoor” better?

Terminal window
python rule_test.py "outdoor => patio"
The output, and what it shows
query_id query added without with
W-013 outdoor privacy wall patio 0.443 0.492
W-105 outdoor movie screen patio 1.000 1.000
W-287 cover set for outdoor furniture patio 0.728 0.611
W-367 wooden chair outdoor patio 0.504 0.613
W-404 large cushion outdoor chair patio 0.470 0.470
T-042 outdoor movie screen patio 1.000 1.000
W-090 flared outdoor sofa patio 0.438 0.890
W-133 outdoor light fixtures patio 1.000 1.000
W-164 modern outdoor furniture patio 0.296 0.353
W-325 outdoor waterproof chest patio 0.109 0.046
W-388 outdoor sectional dining patio 0.503 0.576
T-055 outdoor light fixtures patio 1.000 1.000
T-059 outdoor sectional dining patio 0.503 0.576
13 queries; mean NDCG@10 without 0.615, with 0.664; better 6, worse 2

“patio design” is gone from the list: the rule no longer adds “outdoor” to “patio”. The 6 better queries stay better, and only 2 get worse (“cover set for outdoor furniture” and “outdoor waterproof chest”). The mean gain grows from +0.015 to +0.049.

But remember the trap. You chose this direction by looking at these same queries. Before Larkfield keeps it, check it on queries that you did not look at.

Common mistake

A rule that touches nothing looks harmless

Symptom: Tomás adds a rule and tests it. He forgets the comma:

Terminal window
python rule_test.py "couch sofa"
Empty DataFrame
Columns: [query_id, query, added, without, with]
Index: []
0 queries; mean NDCG@10 without nan, with nan; better 0, worse 0

“better 0, worse 0”. He reads it as “no harm” and adds the rule to the file.

Cause: without a comma, the rule is one term: the phrase “couch sofa”. No query contains that phrase, so the rule never fires. The mean of zero queries is nan, “not a number”.

Fix: read the number of queries first. Zero queries means the test said nothing, not that the rule is safe. Correct the rule ("couch, sofa") and test again. Four of Larkfield’s real rules also touch no query, such as “tv, television”: test them on the log instead, or wait until queries need them.

Consequence in search work: a test that measures nothing looks like a test that passed. Always report how many queries a change touches, beside how much it changes them.

When to use synonyms and rewrites, and what they cost

Section titled “When to use synonyms and rewrites, and what they cost”

Use hand-written rules for the words on which your customers and your catalogue disagree: couch and sofa, French and English product words. A rule is cheap at query time. The whole rule-based search (normalise, spell, synonyms) takes 1.1 ms per query (p50) in the project’s timing.

Do not add a rule because it “makes sense”. “light, lamp” made sense, and it made 4 queries worse. Test each rule alone, count the queries that it touches, and prefer a one-way rule when one word is wider than the other. Treat mined rules as suggestions for a person to check.

Use a language-model rewrite when the gain is large and the delay is acceptable: offline, or for a cache of common queries. On Larkfield’s test queries, it is the strongest change of this module. But its gain on French is too optimistic, and live it takes 1.2 seconds, six times the whole budget.

Check your understanding

A product scores 2.0 for “sofa” and 6.0 for “couch”. With the rule “couch, sofa” and the added word at half weight, what is its score for the query “sofa”? And for the query “couch”?

Show the answer

For “sofa”: 2.0 + 0.5 × 6.0 = 5.0. For “couch”: 6.0 + 0.5 × 2.0 = 7.0. The person’s own word always counts in full, and the added word counts half.

Check your understanding

The rule “patio, outdoor” raises the mean NDCG@10 of its 14 queries from 0.628 to 0.643. Should Larkfield keep it as it is? Use the per-query results.

Show the answer

Not as a two-way rule. It helps “wooden chair outdoor” (0.504 → 0.613) but drops “patio design” from 0.792 to 0.371, because “outdoor” is wider than “patio”. As a one-way rule, outdoor => patio, it keeps the gains and stops that harm (13 queries, 0.615 → 0.664). Then confirm it on queries you did not look at.

Check your understanding

Omar reads: “language-model rewrite, NDCG@10 0.728, French 0.690”, and wants it live tomorrow. Give him the three measured caveats.

Show the answer

First, a live rewrite takes 1.2 s per query (p50), against a 200 ms budget for the whole search. Second, the French result is too easy. Our French queries were machine-translated from English. For 43 of the 60 French test queries, the rewrite is word for word the English original. Third, rewrites exist only for the recorded queries. A cache of rewrites for the most common texts is a safer first step.

Recap

  • A synonym rule adds words to a query (query expansion). A two-way rule (couch, sofa) adds in both directions; a one-way rule (sectional => sofa) only from left to right. Added words count half.
  • All 20 rules together: NDCG@10 0.615 → 0.620 on Larkfield’s test queries (+0.005, −0.002 to +0.011). Alone, “couch, sofa” helps 5 queries and hurts 1; the planted “light, lamp” hurts 4 and helps 2.
  • “patio, outdoor” helps “wooden chair outdoor” (+0.109) and hurts “patio design” (−0.421). As the one-way rule outdoor => patio, it stops that harm.
  • Rules mined from the synthetic log are a query and its shorter form: they show how the log was made.
  • The recorded language-model rewrite reaches 0.728 (0.690 to 0.764), French 0.690. It is slow live (1.2 s per query), and too easy on our machine-translated French.

You can now widen a query with other words. But some queries ask for more than words: a price limit, a colour, a kind of product. Next, you detect what the query asks for and turn it into filters.

Next: Detect what the query asks for