Zero results and business rules
- Lesson 25
- 40 min
- Module 7: The parts of search that people see
Where we are
Filters help people narrow a good list. Some queries still return nothing, and Hana wants sale items first. In this lesson, we add fallbacks for zero results and business rules, and we measure what each rule costs in relevance.
Lesson 25 of 30 · Search and Ranking: Find the Best MatchZero results and business rules
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Never show an empty page without a good reason, and measure what each business rule costs before you accept it.
NotebookMeasure business rules and a fallback in a notebook
The notebook applies “sale first” and a softer sale boost to the recorded hybrid results of the test queries. It measures NDCG@10 and out-of-stock first results. You write a stock rule. Then you build a small word index and find which logged searches need a fallback. Its setup cells download the data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Paired comparisons with an interval (
search compare), from Build a test set that represents your users. - The ranker
ltr, from Learning to rank, and dense search, from Search by meaning.
What you need on your computer
- The project with the files of this module, copied in Autocomplete. This lesson explains
business.pyandruns/rules.py.
A shop assistant who never says only “no”
Section titled “A shop assistant who never says only “no””A customer asks for a “hibou”. The shop assistant does not know the word. A bad assistant says “We don’t have that” and turns away. A good one says: “I don’t know that word. Do you mean something like this?” and shows a few products. She says clearly that she is guessing.
The same shop has a manager who says: “Put the sale items at the front of every shelf.” It sounds like good business. But if the sale rack is half empty, customers walk to the front and find nothing to buy.
This lesson is about both: what to show when nothing matches, and what a rule like the manager’s costs.
Part 1: zero results
Section titled “Part 1: zero results”The problem
Section titled “The problem”In the synthetic log, 30.9% of searches return nothing with Larkfield’s search today, because every word must match. The most common empty searches are “hibou” (216 searches), “ligth bulb” (139), “ampoule” (119) and “canapé” (79). The French queries in our data are machine-translated, and the misspellings are a practice problem we made on purpose. But real shops see the same kinds of failure.
The better runs of Modules 3 to 6 already fix most of this. We ran each search again on every logged search. Today’s search (current) then finds nothing for 30.7%, close to the 30.9% that the log recorded. The share falls to 7.8% (bm25), 2.0% (bm25-norm) and 0.3% (bm25-norm+spell). But an empty page can still happen with any search, for example after a new product name or a strange typo. So the page needs a plan for it.
The idea: a fallback ladder
Section titled “The idea: a fallback ladder”A fallback is what the search does when the normal search finds nothing. Good fallbacks go down a ladder, from strict to loose, and stop at the first step that finds something:
- Every word must match. This is the normal search.
- Relax the query: any word may match. “ligth bulb” has no product with “ligth”, but many with “bulb”.
- Show close matches: search by meaning. “hibou” matches no word, so the dense search of Module 5 finds products that are close in meaning.
At each step down, the page must say what it did: “No product has every word of your search. These products have some of the words.” People then understand why the results look different, and they can try other words.
Text description of the diagram
Three boxes from left to right, joined by arrows labelled “nothing found”. Step 1, every word must be in the product (thick border): 69.2% of the 20,000 logged searches end here; the 80 test queries that end here score NDCG@10 0.588. Step 2, any word, and the page says “These products have some of the words”: 23.4% of searches; 132 test queries, NDCG@10 0.348. Step 3, by meaning, and the page says “These products are close in meaning” (dashed border): 7.4% of searches; 29 test queries, NDCG@10 0.05.
The three shares add up to 100%: no search ends on an empty page. But read the NDCG@10 of each step. The lower the step, the worse the results. Step 3 is reached by 29 test queries, and their results are almost all wrong (0.05). Of those 29, 20 are product codes. The code rule of Exact matches and rules answers them perfectly; meaning cannot.
Larkfield: the fallback in the project
Section titled “Larkfield: the fallback in the project”The run current+fallback is Larkfield’s search today with the ladder. The code is in business.py:
def fallback(query, query_id=None, k=100): """Never an empty page: every word must match; if nothing does, any word; if nothing does, by meaning.""" res = index.fts_search(query, k, "and") if res: return res, "all words" res = index.fts_search(query, k, "or") if res: return res, "any word" return dense.top(dense.scores(query, query_id), k), "by meaning"The function returns the results and the step, so the page can say what it did.
-
Run a misspelt query with today’s search, then with the fallback:
Terminal window search query "ligth bulb" --run current -k 3search query "ligth bulb" --run current+fallback -k 3current: "ligth bulb" -> 0 results (query W-305: judged results are marked)current+fallback: "ligth bulb" -> 3 results (query W-305: judged results are marked)1. LK-75018 4w e12/candelabra led light bulb Light Bulbs 333.99 [exact]2. LK-47333 60 watt equivalent a19 led dimmable light bulb , ( 2700 k ) Light Bulbs 233.99 [exact]3. LK-88046 60 watt equivalent , b13 led , dimmable light bulb , warm wh Light Bulbs 252.99 [partial]Step 2 saves this query: “bulb” alone finds light bulbs.
-
Now a query with no known word:
Terminal window search query "hibou" --run current+fallback -k 3current+fallback: "hibou" -> 3 results (query F-227: judged results are marked)1. LK-84055 fiji shower hooks Shower Curtain Hooks and 163.99 [not judged]2. LK-88452 hoaloha , god of the hawaiian isle tiki wall décor Wall Décor 63.99 [not judged]3. LK-71998 kohler toobi rite-temp valve trim with lever handle Fixture Parts And Compon 71.99 [not judged]“Hibou” means owl. The search by meaning returns shower hooks and a tiki wall decoration. The page is not empty, but it is not useful either.
-
Measure the fallback on the test queries:
Terminal window search compare current current+fallback --by sourcecurrent+fallback minus current on 241 test queries: ndcg@10 +0.197 (95% interval +0.158 to +0.235); better on 105, worse on 0, the same on 136source queries current current+fallback difference low high worsecode 20 0.000 0.000 0.000 0.000 0.000 0planted 3 0.451 0.451 0.000 0.000 0.000 0translated 59 0.012 0.190 0.178 0.111 0.252 0typo 40 0.000 0.384 0.384 0.292 0.475 0wands 119 0.378 0.560 0.182 0.127 0.240 0NDCG@10 rises from 0.195 to 0.392 and no query gets worse. That is what a fallback should do: it only acts when the normal search finds nothing.
Start the page with search serve --run current+fallback and search “hibou”. The note above the results tells the person what happened.

A fallback does not replace a better search. It catches what the search misses. With ltr, no test query has zero results, so the ladder almost never runs.
Part 2: business rules
Section titled “Part 2: business rules”The request
Section titled “The request”Hana, Larkfield’s merchandising lead, asks for a rule: “Always show sale items first.” Sale items earn money now, and the season ends soon. It is a fair request. It is also a change to the ranking, so we measure it like any other change.
A business rule changes the order of the results for a business reason, not for relevance. There are three common kinds:
- A hard rule moves a group as a block: every sale item before every other item, or every out-of-stock product last.
- A soft boost adds a bonus to the score of a group. Its products move up, but they do not jump over much better ones.
- A pinned result puts one chosen product at a fixed place for one query, for example a new collection at position 1 for “sofa”.
The project applies each rule to the top 100 of ltr, so search eval measures it like a run.
One fact about Larkfield’s data matters here. Sale items are out of stock far more often: 39.6% of sale items against 8.2% of the others. This is a practice problem we made on purpose; the stock and sale flags are synthetic. Real shops have it too: a product is often on sale because it is the last of its stock.
A rule before and after
Section titled “A rule before and after”Here is “sale first” on one test query, “clear shower curtain”:
Text description of the diagram
Two lists side by side. Before, with ltr (NDCG@10 0.675): woolley vinyl single shower curtain (exact), siniard single shower curtain (exact, sale, out of stock), woolley vinyl solid color single shower curtain (exact), simmon vinyl single shower curtain liner (exact), vinyl 3 curtains single shower curtain (exact). After, with sale first (NDCG@10 0.353): siniard single shower curtain (exact, sale, out of stock), stainless steel shower curtain hooks (irrelevant, sale, out of stock), adriana shower curtain hooks (irrelevant, sale), sunsetlife shower curtain (partial, sale, out of stock), ruya single shower curtain (partial, sale, out of stock).
The rule did what Hana asked: the top 5 are all sale items. But it pushed four exact curtains out of the top 5. It brought in shower curtain hooks, which the judges graded irrelevant. And four of the five are out of stock. A person who wants a clear shower curtain sees hooks and products they cannot buy.
A soft boost, with every value
Section titled “A soft boost, with every value”The softer rule sale_boost adds a bonus to each sale item. The bonus is 20% of the score range of the list, so it means the same for every query:
- For “clear shower curtain”, the best
ltrscore is 3.099 and the lowest in the top 100 is −1.962. The range is 5.061. - The bonus is 0.2 × 5.061 = 1.012.
- “siniard single shower curtain” (sale): 3.026 + 1.012 = 4.038. It moves from rank 2 to rank 1, above “woolley vinyl single shower curtain” (3.099, no sale).
- “stainless steel shower curtain hooks” (sale): 0.387 + 1.012 = 1.399. It moves from rank 9 to rank 5, above “vinyl 3 curtains single shower curtain” (0.830).
So the boost lifts sale items by a fixed amount. A sale item still cannot pass a product that scores more than 1.012 higher. The three exact curtains between them stay in place.
Measure every rule
Section titled “Measure every rule”Each rule is a run, so we measure it on the 243 test queries. As in every module, the NDCG@10 average uses the 241 queries that have at least one good product:
search eval --run ltr+sale-firstltr+sale-first on 243 test queries: NDCG@10 0.463 (95% interval 0.421 to 0.504)recall@100 0.540, MRR 0.487, zero results 0.0%, judged in the top 10 59.1%; 23.2 ms a query (p50), 41.7 ms (p95)The chart shows all the rules, on two measures: relevance (NDCG@10) and the share of queries whose first result is out of stock.
Text description of the chart
Two panels. Relevance, NDCG@10 with 95% intervals: ltr 0.668 (0.628 to 0.708); sale first 0.463 (0.421 to 0.504); sale boost 0.662 (0.621 to 0.702); stock last 0.66 (0.62 to 0.699). First result out of stock: ltr 11.1%; sale first 37.9%; sale boost 23.5%; stock last 0.0%.
Read the two panels together:
- Sale first costs 0.205 NDCG@10 (paired interval −0.241 to −0.170). It makes 171 test queries worse. And the first result is out of stock for 37.9% of queries, against 11.1% with
ltr. Sale items fill 98.8% of the top 10. - Sale boost costs 0.006 (interval −0.012 to +0.001): not clearly different from no rule. Sale items fill 47.9% of the top 10, against 14.2% with
ltr. But an out-of-stock product comes first for 23.5% of queries. - Stock last costs 0.008 (interval −0.014 to −0.003): small, but clear. It never shows an out-of-stock product first.
A rule can break other rules
Section titled “A rule can break other rules”Look at “sale first” by source:
search compare ltr ltr+sale-first --by sourceltr+sale-first minus ltr on 241 test queries: ndcg@10 -0.205 (95% interval -0.241 to -0.170); better on 35, worse on 171, the same on 35
source queries ltr ltr+sale-first difference low high worse code 20 1.000 0.250 -0.750 -0.900 -0.550 15 planted 3 0.571 0.730 0.159 0.000 0.415 0translated 59 0.474 0.257 -0.216 -0.273 -0.158 44 typo 40 0.615 0.483 -0.132 -0.194 -0.070 28 wands 119 0.729 0.587 -0.142 -0.177 -0.110 84The product codes fall from 1.000 to 0.250. A person who types a product code wants that one product. The code rule put it first. “Sale first” then moved every sale item above it, unless the product was on sale itself. Two rules, each sensible alone, conflict. The last rule wins, and nobody planned that.
Pinned results
Section titled “Pinned results”A pin is the strongest rule: one product at one place for one query. The project has one kind of pin already: the code rule of Exact matches and rules pins the product whose code the person typed. Merchandising pins work the same way, from a list of (query, product, position).
Measure a pin like any rule, but only on its query and on the queries that share its words. And give each pin an end date. A pin outlives its reason more often than you think: the season ends, the product goes out of stock, and the pin stays.
Your turn: a soft stock rule and a fallback
Section titled “Your turn: a soft stock rule and a fallback”Write a rule of your own, measure it, and compare it with the rules above.
- Create a file
larkfield_search/runs/mine.py. Write a runltr+stock-boostthat moves out-of-stock products down by 20% of the score range. Start frombusiness.sale_boostandruns/rules.py. Give your function the@register("ltr+stock-boost", "…")line, as the runs inrules.pyhave: the project loads every file inlarkfield_search/runs/when it starts, sosearch eval --run ltr+stock-boostfinds your run with no other change. - Measure it with
search evalandsearch compare ltr ltr+stock-boost. Then measure how often its first result is out of stock. - For the fallback: run
search eval --run currentandsearch eval --run current+fallback, and compare the zero-result rate and NDCG@10.
A solution, and our results
"""My Module 7 run: a soft stock rule."""
from .. import businessfrom . import registerfrom .rules import _ltr
def stock_boost(results, boost=0.2): """Out-of-stock products move down by `boost` of the score range of the list.""" if not results: return results stock = business._col("in_stock") lo, hi = min(s for _, s in results), max(s for _, s in results) penalty = boost * (hi - lo) return sorted(((p, s - penalty * (not stock[p])) for p, s in results), key=lambda x: (-x[1], x[0]))
@register("ltr+stock-boost", "ltr, out-of-stock products moved down by 20% of the score range", "mine")def ltr_stock_boost(query, query_id=None, k=100): return stock_boost(_ltr(query, query_id))[:k]ltr+stock-boost on 243 test queries: NDCG@10 0.667 (95% interval 0.627 to 0.707)recall@100 0.540, MRR 0.702, zero results 0.0%, judged in the top 10 73.7%; 25.5 ms a query (p50), 57.5 ms (p95)ltr+stock-boost minus ltr on 241 test queries: ndcg@10 -0.001 (95% interval -0.005 to +0.004); better on 42, worse on 56, the same on 143To count the out-of-stock first results, we used this in python:
>>> from larkfield_search import business, data, engine>>> qs = data.split("test")>>> for run in ("ltr", "ltr+stock-last", "ltr+stock-boost"):... ranked, _ = engine.run_queries(run, qs, 100)... lists = [r for r in ranked.values() if r]... first_out = sum(not business._col("in_stock")[r[0]] for r in lists) / len(lists)... print(run, round(first_out, 3))...ltr 0.111ltr+stock-last 0.0ltr+stock-boost 0.016The soft rule costs nothing measurable (−0.001, interval −0.005 to +0.004). The first result is out of stock for only 1.6% of queries. Against ltr+stock-last, it is better by +0.007 (+0.004 to +0.011). An exact product that is out of stock can stay near the top. It does not fall behind every product in stock.
For the fallback, search eval gives zero results 66.7% and NDCG@10 0.195 for current, and 0.0% and 0.392 for current+fallback.
Common mistake
A business rule that quietly breaks product codes
Symptom: after “sale first” goes live, support gets calls: “I typed the product code from your catalogue, and your search shows other products.”
Cause: the sale rule runs after the code rule and moves every sale item above the coded product. The 20 code queries fall from NDCG@10 1.000 to 0.250.
Diagnosis: search compare ltr ltr+sale-first --by source. A slice that drops from 1.000 is a rule conflict, not a ranking problem.
Fix: give rules an order and exceptions. An exact code match comes before any merchandising rule. Better still, use a soft boost, which cannot jump a much better product.
Consequence in AI work: rules added after a model change its output in ways that the model’s own test never sees. Measure the whole pipeline, by slice, after every rule.
Completion check: Hana’s rule
Section titled “Completion check: Hana’s rule”Write Omar and Hana a short answer to “always show sale items first”, with the measurements. Then compare your answer with ours.
Our answer
We measured “sale first” on the test queries (241 of the 243 have an NDCG). It lowers NDCG@10 from 0.668 to 0.463: −0.205, with a 95% interval from −0.241 to −0.170. It makes 171 queries worse and 35 better. It breaks product codes: the right product is no longer first, and those queries fall from 1.000 to 0.250. Sale items are out of stock four to five times more often. So the first result is out of stock for 37.9% of queries, against 11.1% today. People would see more products they cannot buy.
We recommend against the hard rule. We propose a sale boost instead. Sale items rise by 20% of the score range. It costs 0.006 NDCG@10, which is within noise. It shows more than three times as many sale items in the top 10 (47.9% against 14.2%). Combine it with a stock rule, so sale items that are out of stock do not come first. The stock rules alone cost 0.001 to 0.008. Then test the result online, with the add-to-cart rate as the main measure. Test changes online shows how.
Knowledge check
Section titled “Knowledge check”Check your understanding
With current+fallback, no test query has zero results. Tomás says: “So zero results are solved.” What do you answer?
Show the answer
Empty pages are solved, but relevance is not. The 29 test queries that reach the last step, search by meaning, score NDCG@10 0.050: almost every result is wrong. “hibou” shows shower hooks. A fallback must say what it did, and the real fix is a better search, such as spelling correction, the code rule and French titles.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does “sale first” put an out-of-stock product first so often (37.9%), when only 12.9% of all products are out of stock?
Show the answer
Sale items are out of stock much more often: 39.6% of sale items against 8.2% of the others. This is a practice problem we planted on purpose, but real shops see it too. “Sale first” puts a sale item first on almost every query, so the first result inherits the sale items’ stock problem.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The sale boost costs 0.006 NDCG@10, with an interval from −0.012 to +0.001. Is it free?
Show the answer
It is free in relevance, as far as the test can tell, because the interval includes 0. It is not free on the page. An out-of-stock product comes first for 23.5% of queries, against 11.1% without the rule. A rule can cost something that NDCG does not measure. That is why we also count out-of-stock first results.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A fallback ladder goes from strict to loose: every word, then any word, then by meaning. It never shows an empty page, and it must say what it did.
- The fallback lifts Larkfield’s search today from 0.195 to 0.392 NDCG@10 and makes no query worse. Its last step is weak (0.050): a fallback catches failures; it does not replace a better search.
- A business rule re-orders results for a business reason. Measure it like a run: NDCG@10 with its interval, by slice, and the page effects, such as out-of-stock first results.
- “Sale first” costs 0.205 NDCG@10 and breaks product codes. It puts an out-of-stock product first for 37.9% of queries. The reason: sale items are out of stock more often (a planted practice problem). A soft sale boost costs 0.006, within noise.
- Rules conflict. Give them an order, and measure the whole pipeline after each new rule.
Our rules move products that already have a good score. A new product has no clicks, so the ranker never gives it a good score. Next, you see the cold start problem and give new products a place on the page.