Synonyms and query rewriting
- Lesson 13
- 40 min
- Module 4: Understand the query
Where we are
In the last two lessons, we normalised text and fixed spelling, so the words that people type now match more products. Some queries are spelt right and still miss, because the person uses another word for the same thing. In this lesson, we add synonym rules, mine rules from the query log, try a language model that rewrites queries, and test every rule.
Lesson 13 of 30 · Search and Ranking: Find the Best MatchSynonyms and query rewriting
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Find the right products when people use a different word from the catalogue: “couch” for “sofa”, French for English. You write synonym rules, test each rule alone, and weigh a language model that rewrites the whole query.
NotebookDo this lesson in a notebook
The notebook adds one synonym rule to a small BM25 search and asks you to measure it on every validation query that it touches. Then it mines pairs of queries with the same clicks from the synthetic log, and compares the recorded rewrites of a language model by language. Its setup cells download the practice data, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The runs
bm25-normandbm25-norm+spell, from Normalise text and Fix spelling. - How a rare word gets a high BM25 weight, from BM25.
- Judgments made by a language model and why they are recorded, from Judgments from a language model.
What you need on your computer (only for the guided practice)
- The project
larkfield-searchwith this module’s files, from the guided practice of Normalise text.python -m pytest -qmust say24 passed. - A text editor (VS Code) to write one short Python file.
A shop assistant who knows that a couch is a sofa
Section titled “A shop assistant who knows that a couch is a sofa”A customer asks the shop assistant for a couch. The shop’s labels say “sofa”. She knows that a couch and a sofa are the same thing, and she walks to the sofas.
She also knows the limits. A “sectional” is a sofa, but a sofa is not always a sectional. And for “light blue cushions”, she does not bring lamps. Some word rules go both ways, some go one way, and some are wrong in a context. This lesson writes such rules for the search, and measures each one.
The problem: the customer’s word is not the catalogue’s word
Section titled “The problem: the customer’s word is not the catalogue’s word”The test set has a planted broad query, “couch”, a practice problem we made on purpose. The catalogue mostly says “sofa”. With bm25-norm+spell, the top 5 for “couch” are a folding chaise lounge, a kids’ foam couch and a floor couch for gaming. Then come two throw pillows with “couch” in their names. NDCG@10 is 0.278.
Spelling correction cannot help: “couch” is spelt right. Normalising cannot help: “couch” is not a form of “sofa”. Only a rule that says “couch means sofa” can join them. French queries have the same problem on a larger scale: “canapé” is “sofa” in French, and only 7,597 products have a French title.
The idea: synonym rules
Section titled “The idea: synonym rules”A synonym is a word with the same meaning as another word. A synonym rule tells the search to add words to a query before it searches. This is called query expansion: the query grows. There are two kinds of rule.
- A two-way rule, written
couch, sofa, says the words are equal. A query with “couch” also searches “sofa”, and a query with “sofa” also searches “couch”. - A one-way rule, written
sectional => sofa, says the left word is a kind of the right word. A query with “sectional” also searches “sofa”. A query with “sofa” gets nothing added.
Text description of the diagram
Two groups of boxes. “Two-way rule: couch, sofa”: “Query: couch” leads to “Search: couch + sofa (counts half)”; “Query: sofa” leads to “Search: sofa + couch (counts half)”. “One-way rule: sectional => sofa”: “Query: sectional” leads to “Search: sectional + sofa (counts half)”; “Query: sofa” leads to a box with a thick border: “Search: sofa, nothing added”.
Added words count half
Section titled “Added words count half”In the project, an added word counts half. The search adds 0.5 × the BM25 score of the added words to the score of the person’s own words. Why half? The person’s own word is the best evidence of what they want. When the person typed “couch”, a product that says “couch” should rank above a product that only says “sofa”.
A tiny example: a product’s BM25 score for “couch” is 4.0, and its score for “sofa” is 3.0. Its score for the query “couch” becomes 4.0 + 0.5 × 3.0 = 5.5. A second product says only “sofa”: 0 + 0.5 × 3.0 = 1.5. Both are found now, and the product with the person’s own word stays first.
Larkfield’s synonym file
Section titled “Larkfield’s synonym file”Larkfield’s rules are in data/synonyms.csv: 20 rules, one per line. 18 are two-way and 2 are one-way. The file starts like this:
rule,kind,note"couch, sofa",two-way,"grey, gray",two-way,"rug, carpet",two-way,"nightstand, bedside table, night stand",two-way,"loveseat, love seat",two-way,It ends with the two one-way rules and a rule that we planted on purpose:
sectional => sofa,one-way,a sectional is a kind of sofarecliner => chair,one-way,"light, lamp",two-way,"planted: breaks light blue, light wood"The rule “light, lamp” is a practice problem we made on purpose. It looks reasonable, and it is the kind of rule that a person adds in five seconds.
Test every rule alone
Section titled “Test every rule alone”The run bm25-norm+spell+syn uses all 20 rules. On the test queries, it reaches NDCG@10 0.620 (95% interval 0.574 to 0.662), against 0.615 without the rules. Each mean covers 241 of the 243 queries. The difference is +0.005 (95% interval −0.002 to +0.011): 13 queries better and 7 worse.
That one number hides 20 different rules. Some help, some hurt, and some do nothing. So you test each rule alone. Take every validation and test query that the rule changes, and compare NDCG@10 with and without that rule only.
| Rule | Queries it touches | NDCG@10 without | with | Better | Worse |
|---|---|---|---|---|---|
| couch, sofa | 8 | 0.516 | 0.615 | 5 | 1 |
| patio, outdoor | 14 | 0.628 | 0.643 | 6 | 3 |
| dresser, chest of drawers | 2 | 0.644 | 0.743 | 2 | 0 |
| sectional => sofa | 2 | 0.503 | 0.577 | 2 | 0 |
| light, lamp (planted) | 10 | 0.784 | 0.778 | 2 | 4 |
| wardrobe, armoire | 4 | 0.049 | 0.046 | 0 | 1 |
| tv, television | 0 |
The full list has all 20 rules. Four rules touch no query at all: “tv, television”, “drapes, curtains”, “armchair, accent chair” and “bookcase, bookshelf”. They do nothing that you can measure, good or bad.
Read the table row by row.
- “couch, sofa” is the best rule: 5 queries better, 1 worse. The query “couch” alone goes from 0.278 to 0.546.
- “light, lamp”, the planted rule, makes 4 queries worse and 2 better. We planted it to break “light blue” and “light wood”. On these queries, those two did not change. Instead, “family lamp” fell from 0.724 to 0.687, because “light” is a very common word. A planted problem did not fail where we expected, but it failed.
- “wardrobe, armoire” hurts French queries. In French, “armoire” means any cupboard: “armoire de rangement haute” is a tall storage cabinet. The rule adds “wardrobe”, which is wrong for most of them.
One rule that helps one query and hurts another
Section titled “One rule that helps one query and hurts another”The rule “patio, outdoor” touches 14 queries. On average it helps: 0.628 to 0.643. But look at each query:
Text description of the chart
One dot per query, on an axis of differences from −0.5 to +0.5, with a line at zero. outdoor privacy wall: +0.049; cover set for outdoor furniture: −0.117; wooden chair outdoor: +0.109; flared outdoor sofa: +0.452; modern outdoor furniture: +0.057; outdoor waterproof chest: −0.063; outdoor sectional dining: +0.073; patio design: −0.421.
The rule makes “wooden chair outdoor” better: from 0.504 to 0.613. Chairs that say “patio” but not “outdoor” can now match.
The same rule makes “patio design” much worse: from 0.792 to 0.371. The person typed “patio”, and the WANDS judges graded patio chairs and patio sofas exact. The rule adds “outdoor”, a much more common word, and the order changes. Products with both words move up, and many of them were judged irrelevant or not judged at all. The exact patio chairs and sofas move down.
The two words are not equal. Patio furniture is outdoor furniture, but “outdoor” covers more than patios. So the rule should be one-way. You test outdoor => patio in Your turn below.
Choose rules on validation, confirm on test
Section titled “Choose rules on validation, confirm on test”The table above uses the validation and the test queries together, because each rule touches only a few queries. That is fine for finding bad rules. But if you change a rule because of these queries, they no longer test it fairly. This is the same trap as tuning field weights on the test set, from BM25. Change rules by looking at validation queries and at the log, then confirm on new queries that you did not look at.
Rules mined from the query log
Section titled “Rules mined from the query log”Writing rules by hand is slow. The query log can suggest rules. The idea: if two different queries lead people to click the same products, the queries probably mean the same thing.
To compare two sets of clicked products, use the Jaccard similarity. It is the number of products in both sets, divided by the number in either set. Here is a tiny example. Query A led to clicks on products 1, 2, 3 and 4. Query B led to clicks on 2, 3, 4 and 5. Three products are in both, and five are in either. Jaccard = 3 ÷ 5 = 0.6. Identical sets give 1, and sets with nothing in common give 0.
We mined Larkfield’s synthetic log this way. We kept every pair of different queries with at least 3 clicked products each and a Jaccard of 0.3 or more. It found 29 pairs. The first five are:
| Query A | Query B | Jaccard |
|---|---|---|
| larkfield tension | larkfield tension rod | 1 |
| outdoor movie | outdoor movie screen | 0.833 |
| aloe vera | aloe vera plant | 0.75 |
| large cushion outdoor | large cushion outdoor chair | 0.75 |
| luau string | luau string lights | 0.75 |
These are not synonyms. In each of these five pairs, and in most of the 29, one query is the other without its last word. When a simulated person searches again, our code drops the last word of the query. So the mining found exactly the rule that made the synthetic log. Mined rules show how your log was made, and with real people they show real behaviour.
With a real log, the same method finds real pairs, such as a British and an American word. It also finds pairs that are related but not equal. So a mined rule is a suggestion for a person to check, and then to measure.
A language model that rewrites the query
Section titled “A language model that rewrites the query”A synonym rule changes one word. A query rewrite changes the whole query. A language model reads the query and writes a better one: it fixes spelling, translates French into English and writes units one way. The course recorded the answers of chat-small, a small, low-cost chat model on Azure, for the 364 validation and test queries. The run llm-rewrite replays these recorded answers, so you need no key and pay nothing. Its instructions asked it to keep names, numbers and product codes, and not to add details that the person did not ask for.
Some recorded rewrites:
| Typed | Recorded rewrite |
|---|---|
| dnosaur | dinosaur |
| outtdoor sectional doning | outdoor sectional dining |
| blk 18x18 seat cushions | black 18x18 seat cushions |
| canapé | sofa |
| abeille | bee |
| filaret outdoor sofa | filaret outdoor sofa |
The last row matters: the model kept the name “filaret”, where the spelling corrector of the last lesson changed it into “flared”.
Text description of the chart
Three panels with two rows each. English WANDS queries: rules 0.703, recorded rewrite 0.721. Misspelt queries: rules 0.659, rewrite 0.681. French queries: rules 0.301, rewrite 0.69. Each value has its 95% interval as a line.
On all test queries, the recorded rewrite reaches NDCG@10 0.728 (95% interval 0.690 to 0.764), against 0.620 for the rules: +0.108 (+0.078 to +0.140). It is the best single change of this module. Most of the gain is French: from 0.301 to 0.690. On the real English queries, the gain is small but clear: +0.018 (+0.002 to +0.036).
Strong, slow, and too easy on French
Section titled “Strong, slow, and too easy on French”Before Larkfield uses it, read the caveats. Each one is measured.
- It is slow. A live call to chat-small took 1.2 seconds per query (p50) and 1.6 seconds (p95), on 20 test calls. Amira’s budget for a whole search is 200 milliseconds. The rule-based search takes 1.1 ms. A live rewrite is about a thousand times slower than the search itself.
- It looks too good on French. Our French queries were written by a language model from English queries. The rewrite asks a model of the same family to translate them back. That is unrealistically easy. For 43 of the 60 French test queries, the rewrite is word for word the English query that the French query was made from. For example, “tables d’appoint stoneford blanches et bois” became “stoneford end tables white and wood”, exactly the original. Real French customers make typos, mix languages and use words that no translation produced. Measure again on real French queries before you trust 0.690.
- It works only where an answer was recorded: the validation and test queries. For any other query,
llm-rewritesearches the query unchanged. - The money is not the problem. A rewrite costs about US$0.04 per 1,000 queries at chat-small’s price. Time is the problem, not cost.
So the rewrite is strong, slow, and only fair to measure on real French queries. One practical way to use it: cache the rewrites. Compute them ahead of time for the most common query texts, and store them. In the synthetic log, the 100 most common texts make 51.1% of all searches, and those searches would get the rewrite with no delay. The others use the rules.
Guided practice: test the rules
Section titled “Guided practice: test the rules”-
In
larkfield-search, with the virtual environment active, start Python withpython, and try the expansion:>>> from larkfield_search import synonyms>>> synonyms.expansion("couch")'sofa'>>> synonyms.expansion("modern sofa")'couch'>>> synonyms.expansion("sectional")'sofa'>>> synonyms.expansion("light blue pillow")'lamp'The last line is the planted rule at work: “light” here is a colour, and the rule adds “lamp”. Type
exit()to leave Python. -
Measure all 20 rules together, by kind of query:
Terminal window search compare bm25-norm+spell bm25-norm+spell+syn --by sourcebm25-norm+spell+syn minus bm25-norm+spell on 241 test queries: ndcg@10 +0.005 (95% interval -0.002 to +0.011); better on 13, worse on 7, the same on 221source queries bm25-norm+spell bm25-norm+spell+syn difference low high worsecode 20 1.000 1.000 0.000 0.000 0.000 0planted 3 0.384 0.515 0.132 0.000 0.269 0translated 59 0.301 0.301 -0.000 -0.001 0.000 1typo 40 0.648 0.659 0.011 0.000 0.024 1wands 119 0.701 0.703 0.002 -0.009 0.015 5 -
Write a script that tests one rule alone. Create the file
rule_test.pyin the folderlarkfield-search, with this code:"""Test one synonym rule on the validation and test queries: python rule_test.py "couch, sofa" """import sysimport pandas as pdfrom larkfield_search import compare, data, index, spell, synonymsfrom larkfield_search.runs.understand import norm_scoresrule = sys.argv[1]queries = pd.concat([data.split("validation"), data.split("test")])judged = data.judgments()rows = []for qid, text in zip(queries["query_id"], queries["query"]):text = spell.correct(text)[0]if not synonyms.matches(text, rule):continuescores = norm_scores(text)added = synonyms.expansion(text, only=rule)with_rule = scores + synonyms.EXPANSION_WEIGHT * norm_scores(added) if added else scoresbefore = compare.ndcg([p for p, _ in index.top(scores, 10)], judged[qid])after = compare.ndcg([p for p, _ in index.top(with_rule, 10)], judged[qid])rows.append((qid, text, added, round(before, 3), round(after, 3)))table = pd.DataFrame(rows, columns=["query_id", "query", "added", "without", "with"])print(table.to_string(index=False))better = (table["with"] > table["without"]).sum()worse = (table["with"] < table["without"]).sum()print(f"{len(table)} queries; mean NDCG@10 without {table['without'].mean():.3f}, "f"with {table['with'].mean():.3f}; better {better}, worse {worse}")For each query, the script corrects the spelling, as the run does. If the rule matches, it scores the products twice: without the rule, and with the rule’s added words at half weight. Then it computes NDCG@10 of both top 10s.
-
Run it on the rule “patio, outdoor”:
Terminal window python rule_test.py "patio, outdoor"query_id query added without withW-013 outdoor privacy wall patio 0.443 0.492W-105 outdoor movie screen patio 1.000 1.000W-287 cover set for outdoor furniture patio 0.728 0.611W-367 wooden chair outdoor patio 0.504 0.613W-404 large cushion outdoor chair patio 0.470 0.470T-042 outdoor movie screen patio 1.000 1.000W-090 flared outdoor sofa patio 0.438 0.890W-133 outdoor light fixtures patio 1.000 1.000W-164 modern outdoor furniture patio 0.296 0.353W-325 outdoor waterproof chest patio 0.109 0.046W-388 outdoor sectional dining patio 0.503 0.576W-443 patio design outdoor 0.792 0.371T-055 outdoor light fixtures patio 1.000 1.000T-059 outdoor sectional dining patio 0.503 0.57614 queries; mean NDCG@10 without 0.628, with 0.643; better 6, worse 3Some queries appear twice: a misspelt copy (T-) becomes the same text as its original (W-) after correction. W-090 shows “flared”, the corrector’s mistake from the last lesson.
-
Measure the recorded rewrites, by language:
Terminal window search eval --run llm-rewrite --by languagellm-rewrite on 243 test queries: NDCG@10 0.728 (95% interval 0.690 to 0.764)recall@100 0.617, MRR 0.710, zero results 0.4%, judged in the top 10 88.8%; 0.2 ms a query (p50), 0.4 ms (p95)language queries ndcg@10 low high zero-result rateen 182 0.740 0.695 0.781 0.005fr 61 0.690 0.618 0.759 0.000The time, 0.2 ms, is only the search. The rewrite was recorded, so
search evaldoes not wait for the model. Live, add about 1.2 seconds per query.
Your turn: make the rule one-way
Section titled “Your turn: make the rule one-way”Change one thing: write “patio, outdoor” as a one-way rule. Before you run it, predict: which direction stops the harm to “patio design”, outdoor => patio or patio => outdoor? Which one keeps “wooden chair outdoor” better?
python rule_test.py "outdoor => patio"The output, and what it shows
query_id query added without with W-013 outdoor privacy wall patio 0.443 0.492 W-105 outdoor movie screen patio 1.000 1.000 W-287 cover set for outdoor furniture patio 0.728 0.611 W-367 wooden chair outdoor patio 0.504 0.613 W-404 large cushion outdoor chair patio 0.470 0.470 T-042 outdoor movie screen patio 1.000 1.000 W-090 flared outdoor sofa patio 0.438 0.890 W-133 outdoor light fixtures patio 1.000 1.000 W-164 modern outdoor furniture patio 0.296 0.353 W-325 outdoor waterproof chest patio 0.109 0.046 W-388 outdoor sectional dining patio 0.503 0.576 T-055 outdoor light fixtures patio 1.000 1.000 T-059 outdoor sectional dining patio 0.503 0.57613 queries; mean NDCG@10 without 0.615, with 0.664; better 6, worse 2“patio design” is gone from the list: the rule no longer adds “outdoor” to “patio”. The 6 better queries stay better, and only 2 get worse (“cover set for outdoor furniture” and “outdoor waterproof chest”). The mean gain grows from +0.015 to +0.049.
But remember the trap. You chose this direction by looking at these same queries. Before Larkfield keeps it, check it on queries that you did not look at.
Common mistake
A rule that touches nothing looks harmless
Symptom: Tomás adds a rule and tests it. He forgets the comma:
python rule_test.py "couch sofa"Empty DataFrameColumns: [query_id, query, added, without, with]Index: []0 queries; mean NDCG@10 without nan, with nan; better 0, worse 0“better 0, worse 0”. He reads it as “no harm” and adds the rule to the file.
Cause: without a comma, the rule is one term: the phrase “couch sofa”. No query contains that phrase, so the rule never fires. The mean of zero queries is nan, “not a number”.
Fix: read the number of queries first. Zero queries means the test said nothing, not that the rule is safe. Correct the rule ("couch, sofa") and test again. Four of Larkfield’s real rules also touch no query, such as “tv, television”: test them on the log instead, or wait until queries need them.
Consequence in search work: a test that measures nothing looks like a test that passed. Always report how many queries a change touches, beside how much it changes them.
When to use synonyms and rewrites, and what they cost
Section titled “When to use synonyms and rewrites, and what they cost”Use hand-written rules for the words on which your customers and your catalogue disagree: couch and sofa, French and English product words. A rule is cheap at query time. The whole rule-based search (normalise, spell, synonyms) takes 1.1 ms per query (p50) in the project’s timing.
Do not add a rule because it “makes sense”. “light, lamp” made sense, and it made 4 queries worse. Test each rule alone, count the queries that it touches, and prefer a one-way rule when one word is wider than the other. Treat mined rules as suggestions for a person to check.
Use a language-model rewrite when the gain is large and the delay is acceptable: offline, or for a cache of common queries. On Larkfield’s test queries, it is the strongest change of this module. But its gain on French is too optimistic, and live it takes 1.2 seconds, six times the whole budget.
Knowledge check
Section titled “Knowledge check”Check your understanding
A product scores 2.0 for “sofa” and 6.0 for “couch”. With the rule “couch, sofa” and the added word at half weight, what is its score for the query “sofa”? And for the query “couch”?
Show the answer
For “sofa”: 2.0 + 0.5 × 6.0 = 5.0. For “couch”: 6.0 + 0.5 × 2.0 = 7.0. The person’s own word always counts in full, and the added word counts half.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The rule “patio, outdoor” raises the mean NDCG@10 of its 14 queries from 0.628 to 0.643. Should Larkfield keep it as it is? Use the per-query results.
Show the answer
Not as a two-way rule. It helps “wooden chair outdoor” (0.504 → 0.613) but drops “patio design” from 0.792 to 0.371, because “outdoor” is wider than “patio”. As a one-way rule, outdoor => patio, it keeps the gains and stops that harm (13 queries, 0.615 → 0.664). Then confirm it on queries you did not look at.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar reads: “language-model rewrite, NDCG@10 0.728, French 0.690”, and wants it live tomorrow. Give him the three measured caveats.
Show the answer
First, a live rewrite takes 1.2 s per query (p50), against a 200 ms budget for the whole search. Second, the French result is too easy. Our French queries were machine-translated from English. For 43 of the 60 French test queries, the rewrite is word for word the English original. Third, rewrites exist only for the recorded queries. A cache of rewrites for the most common texts is a safer first step.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A synonym rule adds words to a query (query expansion). A two-way rule (
couch, sofa) adds in both directions; a one-way rule (sectional => sofa) only from left to right. Added words count half. - All 20 rules together: NDCG@10 0.615 → 0.620 on Larkfield’s test queries (+0.005, −0.002 to +0.011). Alone, “couch, sofa” helps 5 queries and hurts 1; the planted “light, lamp” hurts 4 and helps 2.
- “patio, outdoor” helps “wooden chair outdoor” (+0.109) and hurts “patio design” (−0.421). As the one-way rule
outdoor => patio, it stops that harm. - Rules mined from the synthetic log are a query and its shorter form: they show how the log was made.
- The recorded language-model rewrite reaches 0.728 (0.690 to 0.764), French 0.690. It is slow live (1.2 s per query), and too easy on our machine-translated French.
You can now widen a query with other words. But some queries ask for more than words: a price limit, a colour, a kind of product. Next, you detect what the query asks for and turn it into filters.