Detect what the query asks for
- Lesson 14
- 35 min
- Module 4: Understand the query
Where we are
In the last three lessons, we normalised text, fixed spelling and added synonyms, so our queries now match more products. Many queries also ask for a price limit, a colour or a kind of product, and BM25 treats those words like any other word. In this lesson, we detect what the query asks for with rules and a small classifier, and we turn it into filters and boosts.
Lesson 14 of 30 · Search and Ranking: Find the Best MatchDetect what the query asks for
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Read the parts of a query that are not product words: “under 50 dollars”, “grey”, “shower curtain”. You detect them with rules and a small classifier, and turn them into a filter or a boost. Then you measure where that helps and where a wrong guess hurts.
NotebookDo this lesson in a notebook
The notebook writes a price rule in English and French. Then it trains a small classifier that guesses the product class of a query from the product titles. You measure how often its first guess is the class of the query’s exact products. Its setup cells download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The runs of this module up to
bm25-norm+spell+syn, from Synonyms and query rewriting. - Product classes (548 of them, such as “Area Rugs”), from Meet the catalogue and the query log.
- The idea of a classifier that learns from examples helps. The machine learning course teaches it, but this lesson explains what you need.
What you need on your computer (only for the guided practice)
- The project
larkfield-searchwith this module’s files, from the guided practice of Normalise text.python -m pytest -qmust say24 passed.
A shop assistant who hears “under 50 dollars”
Section titled “A shop assistant who hears “under 50 dollars””A customer asks the shop assistant: “Grey outdoor cushions for my couch, under 50 dollars.” She does not look for products with the words “under”, “50” or “dollars” on the label. She hears three different things: a kind of product (outdoor cushions), a colour (grey) and a limit (50 dollars). She walks to the cushions, and she does not even show the ones that cost more than 50 dollars.
A word search cannot do that. Every word of the query is one more word to match. So “under” and “50” are treated exactly like “cushion”.
The problem: some words are not product words
Section titled “The problem: some words are not product words”Here is a query that we wrote to show the problem: “Grey outdor couch cushions under 50 dollars”. It is not in the query set, so it has no judgments: you judge the results by eye. The table follows it through the runs of this module and shows the first result of each.
| Run | First result | Price (dollars) |
|---|---|---|
Larkfield today (current) |
nothing: 0 results | |
Exact matches and rules (bm25-fields+codes) |
4 - light led 9.3 '' under cabinet light bar | 17.99 |
+ normalise (bm25-norm) |
4 - light led 9.3 '' under cabinet light bar | 17.99 |
+ spelling (bm25-norm+spell) |
indoor/outdoor sunbrella chaise cushion | 24.99 |
+ synonyms (bm25-norm+spell+syn) |
indoor/outdoor sunbrella chaise cushion | 24.99 |
The first two runs put an under-cabinet light first, because “under” is in its title. After spelling correction, the second result is “outdoor 50 gallon resin deck box”: the “50” matched. After synonyms, the second result is a patio sofa for 735.99 dollars: the rule “couch, sofa” added “sofa”. Each step of this module helped with words, and none of them understood “under 50 dollars”. The prices are synthetic, made by our code, but the problem is real.
The idea: detect, then filter or boost
Section titled “The idea: detect, then filter or boost”To detect what a query asks for is to find the parts that describe a property of the product, not its name. The project looks for three properties:
- a price limit: “under 50 dollars”, “moins de 50 $”;
- a colour: “grey”, “navy”, “gris”;
- a product class: “Furniture Cushions”, “Area Rugs”.
Then the search uses each property in one of two ways.
- A filter removes every product that does not have the property. The price limit is a filter: a cushion for 60 dollars is not “under 50 dollars”, however well it matches the words.
- A boost moves the products with the property up, and keeps the others. The product class is a boost: the score of a product in the guessed class is multiplied by 1.3.
Why a filter for the price and only a boost for the class? A price limit in the query is almost never wrong: the person wrote the number. A guessed class can be wrong. A wrong filter removes every right product, and the page shows nothing useful. A wrong boost only changes the order, and the right products are still there.
Rules for prices and colours
Section titled “Rules for prices and colours”The price limit is found by a regular expression, the pattern language of the code rule in Exact matches and rules. In plain words, the project’s pattern says: a limit word, maybe a “$”, a number, then maybe “$”, “dollars” or “cad”. The limit words are “under”, “below”, “less than”, “max”, “cheaper than”, “moins de” and “sous”. The pattern finds “under 50 dollars” and “moins de 50 $”. The project then removes these words from the query before the word search, so “under” and “50” no longer match lights and deck boxes.
Colours are found with a list of words: 23 English colour words and 11 French ones, such as “grey”, “navy”, “gris” and “bleu”. The project detects colours, but its run does not filter or boost by colour. You saw why in Synonyms and query rewriting: “light” is a colour in “light blue pillow” and a product in “led light strip”. A colour filter needs more care than this module can give.
People also name a room, as in “red rug for balcony”. The project has no room rule, and the catalogue has no “room” field to filter on. “For balcony” can mean small, outdoor or waterproof. A one-way synonym such as balcony => outdoor is a simple start, tested like any other rule.
A small classifier for the product class
Section titled “A small classifier for the product class”The product class is harder. No list of words can cover 548 classes. So the project learns from the catalogue itself. Every product is an example of its class: its title shows which words its class uses. A classifier is a program that learns from such examples and then guesses the class of new text: here, a query.
The project uses a Naive Bayes classifier, one of the simplest. Here is the idea with a toy example; the counts are made up to show the method.
| Word | In Area Rugs titles (per 1,000 words) | In Table Lamps titles (per 1,000 words) |
|---|---|---|
| rug | 120 | 1 |
| red | 15 | 10 |
| lamp | 1 | 110 |
For the query “red rug”, multiply the shares of its words in each class. Area Rugs: 15/1,000 × 120/1,000 = 0.0018. Table Lamps: 10/1,000 × 1/1,000 = 0.00001. Area Rugs wins by far. “Red” helps a little, and “rug” decides.
The method is called “naive” because it treats every word as independent of the others. It does not know that “shower curtain” is one thing. That is naive, but it is fast, and it often works well for short texts. The project uses a variant called Complement Naive Bayes from scikit-learn, which copes better with classes of very different sizes. It learns from the titles (English and French) and the class names of all 40,066 products, in about 2 seconds.
The classifier always has a first guess, even when it is unsure. So the project keeps a guess only when it is clearly more likely than a random pick among the 548 classes. For “sofa”, the first guess is “Sofa & Console Tables”, a kind of table, closely followed by Sectionals and Sofas. None is clear enough, so the project boosts nothing. Here the caution saves the query.
Larkfield: measure the detector
Section titled “Larkfield: measure the detector”The run bm25-norm+spell+syn+detect adds the price filter and the class boost to bm25-norm+spell+syn. On the test queries, it reaches NDCG@10 0.630 (95% interval 0.584 to 0.670), against 0.620. Each mean is over 241 of the 243 queries. The paired difference is +0.010 (95% interval −0.006 to +0.025). The interval includes 0, so the gain is not clear. And it hides a lot of movement: 67 queries better and 40 worse.
By kind of query, the French queries gain +0.049 (+0.027 to +0.071), a clear gain. The classifier learned from the French titles too, so it probably finds the class of a French query more often than the word search alone. The misspelt queries lose −0.035 (−0.085 to +0.003). The detector runs before the spelling corrector. So the misspelt word reaches the classifier as it is, and it can lead to a wrong class.
A good guess and a bad guess
Section titled “A good guess and a bad guess”The boost works when the guess is right. “palram harmony greenhouses” gets the class Greenhouses, and NDCG@10 goes from 0.358 to 0.801.
The boost hurts when the guess is wrong. “clear shower curtain” gets the class “Shower Curtain Hooks and Accessories”. With the boost, hooks fill the top 5, all graded irrelevant, and NDCG@10 falls from 0.649 to 0.000. The classifier sees “shower” and “curtain”, which are in the titles of 126 of the 146 hooks and accessories. Naive Bayes cannot tell that the person wants the curtain, not the hooks for it.
“farmhouse cabinet” gets the class Kitchen Sinks: 71 of the 129 kitchen sinks have “farmhouse” in their titles. NDCG@10 falls from 0.625 to 0.173.
The price filter is not measured
Section titled “The price filter is not measured”There is a limit to this measurement, and you must say it out loud. None of the 624 queries of the query set asks for a price. WANDS’s real queries do not contain “under 50 dollars”, and we planted none. So the test set cannot measure the price filter at all: the whole gain and loss above come from the class boost.
A test set measures only the queries that it contains. Before you build a feature, check that your test set has queries that need it. For the price filter, Larkfield needs new queries with price limits, and judgments for them. Until then, the price filter is tested only by the project’s unit test and by your eyes.
The whole module: one query through every step
Section titled “The whole module: one query through every step”You have now built every step of this module. The diagram follows the query of the start of this lesson through all of them, with the real values from the project.
Text description of the diagram
A row of boxes joined by arrows. “Typed: Grey outdor couch cushions under 50 dollars. Today’s search: 0 results” leads to “1. Normalise”, then “2. Detect: price limit 50 dollars, colour grey, class none; left: grey outdor couch cushions”, then “3. Correct spelling: outdor → outdoor”, then “4. Expand with synonyms: + sofa · gray · patio (added words count half)”, then “5. Stem and score with BM25”, then a box with a thick border: “6. Filter: price at most 50. First result: texture outdoor bench cushion, 12.99 dollars”.
The first result is now “texture outdoor bench cushion” for 12.99 dollars, and the next ones are outdoor cushions under 50 dollars too. Notice step 4: the synonyms still add “sofa” and “patio”. Without the filter, they brought a patio sofa to the top. The steps work together, and a later step can repair, or hide, the mistake of an earlier one.
The effect of each step, by kind of query
Section titled “The effect of each step, by kind of query”The chart sums up the module on Larkfield’s test queries (mean NDCG@10 of 241 of the 243).
Text description of the chart
Three panels with six rows each. English WANDS queries: Module 3 0.641, normalise 0.709, spelling 0.701, synonyms 0.703, detect 0.711, recorded rewrite 0.721. Misspelt queries: 0.362, 0.416, 0.648, 0.659, 0.624, 0.681. French queries: 0.178, 0.309, 0.301, 0.301, 0.35, 0.69. Each value has its 95% interval as a line.
Read each panel from top to bottom.
- English WANDS queries. Normalising gives the big step (0.641 to 0.709). Spelling takes a little back, because it changes names. Synonyms and detection add little.
- Misspelt queries. Spelling correction gives the big step (0.416 to 0.648). Detection takes some back.
- French queries. The rules lift French from 0.178 to 0.350, and the recorded rewrite to 0.690. That last number is too optimistic, as you saw in the last lesson: our French queries were machine-translated from English.
On all test queries, the rule-based steps take NDCG@10 from 0.506 to 0.620 with normalising, spelling and synonyms, and to 0.630 with detection. That is real progress from Module 3. But French stays weak, and every step made some queries worse. A step that understands meaning, not only words, could help: that is the next module.
Guided practice: detect and measure
Section titled “Guided practice: detect and measure”-
In
larkfield-search, with the virtual environment active, start Python withpython, and try the detector:>>> from larkfield_search.detect import detect>>> detect("grey cushion under 50 dollars"){'price_max': 50.0, 'colours': ['grey'], 'classes': [np.str_('Furniture Cushions')], 'clean': 'grey cushion'}>>> detect("coussin gris moins de 50 $"){'price_max': 50.0, 'colours': ['gris'], 'classes': [np.str_('Accent Pillows')], 'clean': 'coussin gris'}>>> detect("clear shower curtain"){'price_max': None, 'colours': [], 'classes': [np.str_('Shower Curtain Hooks and Accessories')], 'clean': 'clear shower curtain'}The first call trains the classifier on the catalogue and saves it in
outputs/.np.str_is how NumPy prints a text value; read it as plain text.cleanis the query without the price words. Typeexit()to leave Python. -
Compare the detector with the run before it, by kind of query:
Terminal window search compare bm25-norm+spell+syn bm25-norm+spell+syn+detect --by sourcebm25-norm+spell+syn+detect minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.010 (95% interval -0.006 to +0.025); better on 67, worse on 40, the same on 134source queries bm25-norm+spell+syn bm25-norm+spell+syn+detect difference low high worsecode 20 1.000 1.000 0.000 0.000 0.000 0planted 3 0.515 0.515 0.000 0.000 0.000 0translated 59 0.301 0.350 0.049 0.027 0.071 6typo 40 0.659 0.624 -0.035 -0.085 0.003 10wands 119 0.703 0.711 0.008 -0.016 0.031 24 -
Look at the worst query:
Terminal window search query "clear shower curtain" --run bm25-norm+spell+syn+detect -k 3bm25-norm+spell+syn+detect: "clear shower curtain" -> 3 results (query W-246: judged results are marked)1. LK-56534 shower curtain hooks Shower Curtain Hooks and 327.99 [irrelevant]2. LK-53992 sparkle shower curtain hooks Shower Curtain Hooks and 307.99 [irrelevant]3. LK-80781 gilded pineapple shower curtain hooks Shower Curtain Hooks and 107.99 [irrelevant] -
Follow the query of this lesson through the whole module:
Terminal window search query "Grey outdor couch cushions under 50 dollars" --run bm25-norm+spell+syn+detect -k 3bm25-norm+spell+syn+detect: "Grey outdor couch cushions under 50 dollars" -> 3 results1. LK-17722 texture outdoor bench cushion Furniture Cushions 12.992. LK-99990 outdoor lounge chair cushion Furniture Cushions 17.993. LK-87002 scalia indoor/outdoor seat cushion Furniture Cushions 17.99 [out of stock]Run it again with
--run currentand--run bm25-norm+spell+synto see the rows of the table at the start of this lesson. -
Measure the module’s main run, the one that later modules build on:
Terminal window search eval --run bm25-norm+spell+syn --by sourcebm25-norm+spell+syn on 243 test queries: NDCG@10 0.620 (95% interval 0.574 to 0.662)recall@100 0.546, MRR 0.626, zero results 0.0%, judged in the top 10 76.0%; 0.3 ms a query (p50), 1.4 ms (p95)source queries ndcg@10 low high zero-result ratecode 20 1.000 1.000 1.000 0.000planted 3 0.515 0.188 0.812 0.000translated 60 0.301 0.214 0.393 0.000typo 40 0.659 0.569 0.747 0.000wands 120 0.703 0.647 0.753 0.000 -
Run the tests one last time:
Terminal window python -m pytest -q........................ [100%]24 passed in 0.55sYour project now matches the snapshot
end-of-m04. Your time will differ.
Later modules compare their runs with bm25-norm+spell+syn, not with the detector. Its gain is not clear, and a wrong class boost can empty the top of a page, as step 3 shows.
Your turn: how strong should the boost be?
Section titled “Your turn: how strong should the boost be?”The class boost multiplies the score of the guessed class by 1.3. Change one thing: the boost. Before you run them, predict: with a boost of 1 (no boost at all), what NDCG@10 do you get? And with 2?
search eval --run bm25-norm+spell+syn+detect --set class_boost=1search eval --run bm25-norm+spell+syn+detect --set class_boost=2The output, and what it shows
bm25-norm+spell+syn+detect on 243 test queries: NDCG@10 0.620 (95% interval 0.574 to 0.662)recall@100 0.546, MRR 0.626, zero results 0.0%, judged in the top 10 76.0%; 9.8 ms a query (p50), 29.5 ms (p95)bm25-norm+spell+syn+detect on 243 test queries: NDCG@10 0.630 (95% interval 0.586 to 0.673)recall@100 0.547, MRR 0.627, zero results 0.0%, judged in the top 10 78.4%; 10.3 ms a query (p50), 29.0 ms (p95)With a boost of 1, the result is exactly bm25-norm+spell+syn: 0.620. No test query asks for a price, so the price filter changes nothing here. That confirms what the section above said: all of the detector’s effect on the test set is the class boost.
With a boost of 2, the result is 0.630, the same as with 1.3. A stronger boost helps the queries with a right guess more, and hurts the queries with a wrong guess more. On these queries, the two cancel out. Do not tune this number on the test set: if you want to tune it, use the validation queries.
Look at the times too. The detector takes about 10 ms a query here, against 0.3 ms for bm25-norm+spell+syn. The classifier and the boost over 40,066 products run for every query. Your times will differ.
Common mistake
A filter can bring back the empty page
Symptom: a person types “sofa under 5 dollars”. The page is empty:
bm25-norm+spell+syn+detect: "sofa under 5 dollars" -> 0 resultsThis query is one we wrote to show the problem; it is not in the query set.
Cause: the filter removes every product above 5 dollars. The cheapest product in the catalogue costs 4.99 dollars, and only 5 products cost less than 5 dollars. None of them is a sofa. A filter does exactly what it is told, even when the answer is nothing.
Fix: when a filter empties the page, do not show an empty page. Show the products without the filter, with a line such as “No sofas under 5 dollars. Here are the cheapest sofas.” To see what the person would get without the filter, run search query "sofa under 5 dollars" --run bm25-norm+spell+syn.
Consequence in search work: so far, the course has taken the empty pages of the synthetic log from about 31% to 0.3%. A filter can bring them back. Count the empty pages after every new filter, and decide in advance what the page shows instead.
When to detect, and what it costs
Section titled “When to detect, and what it costs”Detect a property when people type it often, when it is easy to recognise, and when your catalogue has a field for it. Prices, sizes and colours often qualify. Check your log and your test set first. In Larkfield’s query set, nobody asked for a price.
Filter only what the person stated exactly; boost what you guess. The classifier trains in about 2 seconds here, but it runs for every query. In search eval on this laptop, the run takes about 10 ms per query, against 0.3 ms without the detector. That fits Amira’s budget of 200 ms, but it is the slowest step of this module’s rules.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does the project use the price limit as a filter but the product class only as a boost? Use “clear shower curtain” in your answer.
Show the answer
The person writes the price limit, so it is almost never wrong, and a product above it does not answer the query. The class is a guess, and guesses can be wrong: for “clear shower curtain”, the classifier guessed “Shower Curtain Hooks and Accessories”. With a boost, the hooks moved to the top (NDCG@10 0.649 → 0.000). With a filter, every curtain would be gone from the page, with no way back.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The detector's run scores 0.630 against 0.620, with a paired interval from −0.006 to +0.025. Tomás says the price filter works well. What is wrong with his claim?
Show the answer
Two things. First, the interval includes 0, so the gain is not clear at all. Second, none of the 624 queries asks for a price, so the test set cannot measure the price filter. With class_boost=1, the run scores exactly 0.620. All of the change comes from the class boost.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Naive Bayes guesses the class of “red rug” from the words of product titles. Which word decides, “red” or “rug”, and why?
Show the answer
“rug”. Naive Bayes multiplies the share of each query word in each class. “rug” is very common in Area Rugs titles and very rare in other classes, so it separates the classes strongly. “red” appears in many classes, so it barely changes the result.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- To detect is to find the parts of a query that describe a property. Rules find a price limit and colours; a Naive Bayes classifier, trained on the product titles, guesses the product class.
- A filter removes products; a boost moves products up. Filter only what the person stated exactly (a price limit); boost what you guess (a class).
- On Larkfield’s test queries, the detector reaches NDCG@10 0.630 (0.584 to 0.670): +0.010 (−0.006 to +0.025), not clear; French +0.049. A wrong guess hurts: “clear shower curtain” fell from 0.649 to 0.000.
- No query in the query set asks for a price, so the price filter is not measured. A test set measures only the queries that it contains.
- Over the whole module, the rules take NDCG@10 from 0.506 to 0.620 (normalise, spelling, synonyms), and the recorded rewrite to 0.728, with its caveats.
You can now read the words of a query in many ways. But every step still matches words, and French stays weak. Next, in Module 5, you search by meaning: a model turns queries and products into numbers that are close when their meanings are close.
Next: Search by meaning