Skip to content
Français

Zero results and business rules

  • Lesson 25
  • 40 min
  • Module 7: The parts of search that people see

Where we are

Filters help people narrow a good list. Some queries still return nothing, and Hana wants sale items first. In this lesson, we add fallbacks for zero results and business rules, and we measure what each rule costs in relevance.

What you will be able to do

Never show an empty page without a good reason, and measure what each business rule costs before you accept it.

NotebookMeasure business rules and a fallback in a notebookColab · Kaggle · GitHub

The notebook applies “sale first” and a softer sale boost to the recorded hybrid results of the test queries. It measures NDCG@10 and out-of-stock first results. You write a stock rule. Then you build a small word index and find which logged searches need a fallback. Its setup cells download the data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project with the files of this module, copied in Autocomplete. This lesson explains business.py and runs/rules.py.

A shop assistant who never says only “no”

Section titled “A shop assistant who never says only “no””

A customer asks for a “hibou”. The shop assistant does not know the word. A bad assistant says “We don’t have that” and turns away. A good one says: “I don’t know that word. Do you mean something like this?” and shows a few products. She says clearly that she is guessing.

The same shop has a manager who says: “Put the sale items at the front of every shelf.” It sounds like good business. But if the sale rack is half empty, customers walk to the front and find nothing to buy.

This lesson is about both: what to show when nothing matches, and what a rule like the manager’s costs.

In the synthetic log, 30.9% of searches return nothing with Larkfield’s search today, because every word must match. The most common empty searches are “hibou” (216 searches), “ligth bulb” (139), “ampoule” (119) and “canapé” (79). The French queries in our data are machine-translated, and the misspellings are a practice problem we made on purpose. But real shops see the same kinds of failure.

The better runs of Modules 3 to 6 already fix most of this. We ran each search again on every logged search. Today’s search (current) then finds nothing for 30.7%, close to the 30.9% that the log recorded. The share falls to 7.8% (bm25), 2.0% (bm25-norm) and 0.3% (bm25-norm+spell). But an empty page can still happen with any search, for example after a new product name or a strange typo. So the page needs a plan for it.

A fallback is what the search does when the normal search finds nothing. Good fallbacks go down a ladder, from strict to loose, and stop at the first step that finds something:

  1. Every word must match. This is the normal search.
  2. Relax the query: any word may match. “ligth bulb” has no product with “ligth”, but many with “bulb”.
  3. Show close matches: search by meaning. “hibou” matches no word, so the dense search of Module 5 finds products that are close in meaning.

At each step down, the page must say what it did: “No product has every word of your search. These products have some of the words.” People then understand why the results look different, and they can try other words.

The fallback ladder. Step 1, every word must be in the product: 69.2% of the log's searches end here; on the test queries that end here, NDCG@10 0.588 over 80 queries. Nothing found leads to step 2, any word, with the message "These products have some of the words": 23.4% of searches, NDCG@10 0.348 over 132 queries. Nothing found leads to step 3, by meaning, with the message "These products are close in meaning": 7.4% of searches, NDCG@10 0.05 over 29 queries.
The project's fallback for Larkfield's search today: where each logged search ends, and how good the results are at each step on the judged test queries.
Text description of the diagram

Three boxes from left to right, joined by arrows labelled “nothing found”. Step 1, every word must be in the product (thick border): 69.2% of the 20,000 logged searches end here; the 80 test queries that end here score NDCG@10 0.588. Step 2, any word, and the page says “These products have some of the words”: 23.4% of searches; 132 test queries, NDCG@10 0.348. Step 3, by meaning, and the page says “These products are close in meaning” (dashed border): 7.4% of searches; 29 test queries, NDCG@10 0.05.

The three shares add up to 100%: no search ends on an empty page. But read the NDCG@10 of each step. The lower the step, the worse the results. Step 3 is reached by 29 test queries, and their results are almost all wrong (0.05). Of those 29, 20 are product codes. The code rule of Exact matches and rules answers them perfectly; meaning cannot.

The run current+fallback is Larkfield’s search today with the ladder. The code is in business.py:

def fallback(query, query_id=None, k=100):
"""Never an empty page: every word must match; if nothing does, any word; if nothing does, by meaning."""
res = index.fts_search(query, k, "and")
if res:
return res, "all words"
res = index.fts_search(query, k, "or")
if res:
return res, "any word"
return dense.top(dense.scores(query, query_id), k), "by meaning"

The function returns the results and the step, so the page can say what it did.

  1. Run a misspelt query with today’s search, then with the fallback:

    Terminal window
    search query "ligth bulb" --run current -k 3
    search query "ligth bulb" --run current+fallback -k 3
    current: "ligth bulb" -> 0 results (query W-305: judged results are marked)
    current+fallback: "ligth bulb" -> 3 results (query W-305: judged results are marked)
    1. LK-75018 4w e12/candelabra led light bulb Light Bulbs 333.99 [exact]
    2. LK-47333 60 watt equivalent a19 led dimmable light bulb , ( 2700 k ) Light Bulbs 233.99 [exact]
    3. LK-88046 60 watt equivalent , b13 led , dimmable light bulb , warm wh Light Bulbs 252.99 [partial]

    Step 2 saves this query: “bulb” alone finds light bulbs.

  2. Now a query with no known word:

    Terminal window
    search query "hibou" --run current+fallback -k 3
    current+fallback: "hibou" -> 3 results (query F-227: judged results are marked)
    1. LK-84055 fiji shower hooks Shower Curtain Hooks and 163.99 [not judged]
    2. LK-88452 hoaloha , god of the hawaiian isle tiki wall décor Wall Décor 63.99 [not judged]
    3. LK-71998 kohler toobi rite-temp valve trim with lever handle Fixture Parts And Compon 71.99 [not judged]

    “Hibou” means owl. The search by meaning returns shower hooks and a tiki wall decoration. The page is not empty, but it is not useful either.

  3. Measure the fallback on the test queries:

    Terminal window
    search compare current current+fallback --by source
    current+fallback minus current on 241 test queries: ndcg@10 +0.197 (95% interval +0.158 to +0.235); better on 105, worse on 0, the same on 136
    source queries current current+fallback difference low high worse
    code 20 0.000 0.000 0.000 0.000 0.000 0
    planted 3 0.451 0.451 0.000 0.000 0.000 0
    translated 59 0.012 0.190 0.178 0.111 0.252 0
    typo 40 0.000 0.384 0.384 0.292 0.475 0
    wands 119 0.378 0.560 0.182 0.127 0.240 0

    NDCG@10 rises from 0.195 to 0.392 and no query gets worse. That is what a fallback should do: it only acts when the normal search finds nothing.

Start the page with search serve --run current+fallback and search “hibou”. The note above the results tells the person what happened.

The search page for hibou with the run current+fallback. Above the results, a note with a red line on its left: No product has any word of your search. These products are close in meaning. The results: fiji shower hooks, a tiki wall décor, a valve trim, a corkscrew, a cat figurine.
The fallback says what it did. The first search by meaning also loads the bi-encoder, which is why this one took about 3.6 seconds.

A fallback does not replace a better search. It catches what the search misses. With ltr, no test query has zero results, so the ladder almost never runs.

Hana, Larkfield’s merchandising lead, asks for a rule: “Always show sale items first.” Sale items earn money now, and the season ends soon. It is a fair request. It is also a change to the ranking, so we measure it like any other change.

A business rule changes the order of the results for a business reason, not for relevance. There are three common kinds:

  • A hard rule moves a group as a block: every sale item before every other item, or every out-of-stock product last.
  • A soft boost adds a bonus to the score of a group. Its products move up, but they do not jump over much better ones.
  • A pinned result puts one chosen product at a fixed place for one query, for example a new collection at position 1 for “sofa”.

The project applies each rule to the top 100 of ltr, so search eval measures it like a run.

One fact about Larkfield’s data matters here. Sale items are out of stock far more often: 39.6% of sale items against 8.2% of the others. This is a practice problem we made on purpose; the stock and sale flags are synthetic. Real shops have it too: a product is often on sale because it is the last of its stock.

Here is “sale first” on one test query, “clear shower curtain”:

The top 5 for clear shower curtain before and after sale first. Before, ltr, NDCG@10 0.675: five exact shower curtains; the second, siniard single shower curtain, is on sale and out of stock. After, sale first, NDCG@10 0.353: 1 siniard single shower curtain, exact, sale, out of stock; 2 stainless steel shower curtain hooks, irrelevant, sale, out of stock; 3 adriana shower curtain hooks, irrelevant, sale; 4 sunsetlife shower curtain, partial, sale, out of stock; 5 ruya single shower curtain, partial, sale, out of stock.
One query before and after "sale first". The top 5 goes from five exact curtains to two hooks, two partial matches and one exact curtain; four of the five are out of stock.
Text description of the diagram

Two lists side by side. Before, with ltr (NDCG@10 0.675): woolley vinyl single shower curtain (exact), siniard single shower curtain (exact, sale, out of stock), woolley vinyl solid color single shower curtain (exact), simmon vinyl single shower curtain liner (exact), vinyl 3 curtains single shower curtain (exact). After, with sale first (NDCG@10 0.353): siniard single shower curtain (exact, sale, out of stock), stainless steel shower curtain hooks (irrelevant, sale, out of stock), adriana shower curtain hooks (irrelevant, sale), sunsetlife shower curtain (partial, sale, out of stock), ruya single shower curtain (partial, sale, out of stock).

The rule did what Hana asked: the top 5 are all sale items. But it pushed four exact curtains out of the top 5. It brought in shower curtain hooks, which the judges graded irrelevant. And four of the five are out of stock. A person who wants a clear shower curtain sees hooks and products they cannot buy.

The softer rule sale_boost adds a bonus to each sale item. The bonus is 20% of the score range of the list, so it means the same for every query:

  • For “clear shower curtain”, the best ltr score is 3.099 and the lowest in the top 100 is −1.962. The range is 5.061.
  • The bonus is 0.2 × 5.061 = 1.012.
  • “siniard single shower curtain” (sale): 3.026 + 1.012 = 4.038. It moves from rank 2 to rank 1, above “woolley vinyl single shower curtain” (3.099, no sale).
  • “stainless steel shower curtain hooks” (sale): 0.387 + 1.012 = 1.399. It moves from rank 9 to rank 5, above “vinyl 3 curtains single shower curtain” (0.830).

So the boost lifts sale items by a fixed amount. A sale item still cannot pass a product that scores more than 1.012 higher. The three exact curtains between them stay in place.

Each rule is a run, so we measure it on the 243 test queries. As in every module, the NDCG@10 average uses the 241 queries that have at least one good product:

Terminal window
search eval --run ltr+sale-first
ltr+sale-first on 243 test queries: NDCG@10 0.463 (95% interval 0.421 to 0.504)
recall@100 0.540, MRR 0.487, zero results 0.0%, judged in the top 10 59.1%; 23.2 ms a query (p50), 41.7 ms (p95)

The chart shows all the rules, on two measures: relevance (NDCG@10) and the share of queries whose first result is out of stock.

Each business rule against ltr on the test queries. Sale first loses 0.205 NDCG@10 and puts an out-of-stock product first for 37.9% of queries; stock last costs 0.008 and removes that problem.
Text description of the chart

Two panels. Relevance, NDCG@10 with 95% intervals: ltr 0.668 (0.628 to 0.708); sale first 0.463 (0.421 to 0.504); sale boost 0.662 (0.621 to 0.702); stock last 0.66 (0.62 to 0.699). First result out of stock: ltr 11.1%; sale first 37.9%; sale boost 23.5%; stock last 0.0%.

Read the two panels together:

  • Sale first costs 0.205 NDCG@10 (paired interval −0.241 to −0.170). It makes 171 test queries worse. And the first result is out of stock for 37.9% of queries, against 11.1% with ltr. Sale items fill 98.8% of the top 10.
  • Sale boost costs 0.006 (interval −0.012 to +0.001): not clearly different from no rule. Sale items fill 47.9% of the top 10, against 14.2% with ltr. But an out-of-stock product comes first for 23.5% of queries.
  • Stock last costs 0.008 (interval −0.014 to −0.003): small, but clear. It never shows an out-of-stock product first.

Look at “sale first” by source:

Terminal window
search compare ltr ltr+sale-first --by source
ltr+sale-first minus ltr on 241 test queries: ndcg@10 -0.205 (95% interval -0.241 to -0.170); better on 35, worse on 171, the same on 35
source queries ltr ltr+sale-first difference low high worse
code 20 1.000 0.250 -0.750 -0.900 -0.550 15
planted 3 0.571 0.730 0.159 0.000 0.415 0
translated 59 0.474 0.257 -0.216 -0.273 -0.158 44
typo 40 0.615 0.483 -0.132 -0.194 -0.070 28
wands 119 0.729 0.587 -0.142 -0.177 -0.110 84

The product codes fall from 1.000 to 0.250. A person who types a product code wants that one product. The code rule put it first. “Sale first” then moved every sale item above it, unless the product was on sale itself. Two rules, each sensible alone, conflict. The last rule wins, and nobody planned that.

A pin is the strongest rule: one product at one place for one query. The project has one kind of pin already: the code rule of Exact matches and rules pins the product whose code the person typed. Merchandising pins work the same way, from a list of (query, product, position).

Measure a pin like any rule, but only on its query and on the queries that share its words. And give each pin an end date. A pin outlives its reason more often than you think: the season ends, the product goes out of stock, and the pin stays.

Your turn: a soft stock rule and a fallback

Section titled “Your turn: a soft stock rule and a fallback”

Write a rule of your own, measure it, and compare it with the rules above.

  1. Create a file larkfield_search/runs/mine.py. Write a run ltr+stock-boost that moves out-of-stock products down by 20% of the score range. Start from business.sale_boost and runs/rules.py. Give your function the @register("ltr+stock-boost", "…") line, as the runs in rules.py have: the project loads every file in larkfield_search/runs/ when it starts, so search eval --run ltr+stock-boost finds your run with no other change.
  2. Measure it with search eval and search compare ltr ltr+stock-boost. Then measure how often its first result is out of stock.
  3. For the fallback: run search eval --run current and search eval --run current+fallback, and compare the zero-result rate and NDCG@10.
A solution, and our results
"""My Module 7 run: a soft stock rule."""
from .. import business
from . import register
from .rules import _ltr
def stock_boost(results, boost=0.2):
"""Out-of-stock products move down by `boost` of the score range of the list."""
if not results:
return results
stock = business._col("in_stock")
lo, hi = min(s for _, s in results), max(s for _, s in results)
penalty = boost * (hi - lo)
return sorted(((p, s - penalty * (not stock[p])) for p, s in results), key=lambda x: (-x[1], x[0]))
@register("ltr+stock-boost", "ltr, out-of-stock products moved down by 20% of the score range", "mine")
def ltr_stock_boost(query, query_id=None, k=100):
return stock_boost(_ltr(query, query_id))[:k]
ltr+stock-boost on 243 test queries: NDCG@10 0.667 (95% interval 0.627 to 0.707)
recall@100 0.540, MRR 0.702, zero results 0.0%, judged in the top 10 73.7%; 25.5 ms a query (p50), 57.5 ms (p95)
ltr+stock-boost minus ltr on 241 test queries: ndcg@10 -0.001 (95% interval -0.005 to +0.004); better on 42, worse on 56, the same on 143

To count the out-of-stock first results, we used this in python:

>>> from larkfield_search import business, data, engine
>>> qs = data.split("test")
>>> for run in ("ltr", "ltr+stock-last", "ltr+stock-boost"):
... ranked, _ = engine.run_queries(run, qs, 100)
... lists = [r for r in ranked.values() if r]
... first_out = sum(not business._col("in_stock")[r[0]] for r in lists) / len(lists)
... print(run, round(first_out, 3))
...
ltr 0.111
ltr+stock-last 0.0
ltr+stock-boost 0.016

The soft rule costs nothing measurable (−0.001, interval −0.005 to +0.004). The first result is out of stock for only 1.6% of queries. Against ltr+stock-last, it is better by +0.007 (+0.004 to +0.011). An exact product that is out of stock can stay near the top. It does not fall behind every product in stock.

For the fallback, search eval gives zero results 66.7% and NDCG@10 0.195 for current, and 0.0% and 0.392 for current+fallback.

Common mistake

A business rule that quietly breaks product codes

Symptom: after “sale first” goes live, support gets calls: “I typed the product code from your catalogue, and your search shows other products.”

Cause: the sale rule runs after the code rule and moves every sale item above the coded product. The 20 code queries fall from NDCG@10 1.000 to 0.250.

Diagnosis: search compare ltr ltr+sale-first --by source. A slice that drops from 1.000 is a rule conflict, not a ranking problem.

Fix: give rules an order and exceptions. An exact code match comes before any merchandising rule. Better still, use a soft boost, which cannot jump a much better product.

Consequence in AI work: rules added after a model change its output in ways that the model’s own test never sees. Measure the whole pipeline, by slice, after every rule.

Write Omar and Hana a short answer to “always show sale items first”, with the measurements. Then compare your answer with ours.

Our answer

We measured “sale first” on the test queries (241 of the 243 have an NDCG). It lowers NDCG@10 from 0.668 to 0.463: −0.205, with a 95% interval from −0.241 to −0.170. It makes 171 queries worse and 35 better. It breaks product codes: the right product is no longer first, and those queries fall from 1.000 to 0.250. Sale items are out of stock four to five times more often. So the first result is out of stock for 37.9% of queries, against 11.1% today. People would see more products they cannot buy.

We recommend against the hard rule. We propose a sale boost instead. Sale items rise by 20% of the score range. It costs 0.006 NDCG@10, which is within noise. It shows more than three times as many sale items in the top 10 (47.9% against 14.2%). Combine it with a stock rule, so sale items that are out of stock do not come first. The stock rules alone cost 0.001 to 0.008. Then test the result online, with the add-to-cart rate as the main measure. Test changes online shows how.

Check your understanding

With current+fallback, no test query has zero results. Tomás says: “So zero results are solved.” What do you answer?

Show the answer

Empty pages are solved, but relevance is not. The 29 test queries that reach the last step, search by meaning, score NDCG@10 0.050: almost every result is wrong. “hibou” shows shower hooks. A fallback must say what it did, and the real fix is a better search, such as spelling correction, the code rule and French titles.

Check your understanding

Why does “sale first” put an out-of-stock product first so often (37.9%), when only 12.9% of all products are out of stock?

Show the answer

Sale items are out of stock much more often: 39.6% of sale items against 8.2% of the others. This is a practice problem we planted on purpose, but real shops see it too. “Sale first” puts a sale item first on almost every query, so the first result inherits the sale items’ stock problem.

Check your understanding

The sale boost costs 0.006 NDCG@10, with an interval from −0.012 to +0.001. Is it free?

Show the answer

It is free in relevance, as far as the test can tell, because the interval includes 0. It is not free on the page. An out-of-stock product comes first for 23.5% of queries, against 11.1% without the rule. A rule can cost something that NDCG does not measure. That is why we also count out-of-stock first results.

Recap

  • A fallback ladder goes from strict to loose: every word, then any word, then by meaning. It never shows an empty page, and it must say what it did.
  • The fallback lifts Larkfield’s search today from 0.195 to 0.392 NDCG@10 and makes no query worse. Its last step is weak (0.050): a fallback catches failures; it does not replace a better search.
  • A business rule re-orders results for a business reason. Measure it like a run: NDCG@10 with its interval, by slice, and the page effects, such as out-of-stock first results.
  • “Sale first” costs 0.205 NDCG@10 and breaks product codes. It puts an out-of-stock product first for 37.9% of queries. The reason: sale items are out of stock more often (a planted practice problem). A soft sale boost costs 0.006, within noise.
  • Rules conflict. Give them an order, and measure the whole pipeline after each new rule.

Our rules move products that already have a good score. A new product has no clicks, so the ranker never gives it a good score. Next, you see the cold start problem and give new products a place on the page.

Next: New products and the cold start