Skip to content
Français

A shop where search fails

  • Lesson 1
  • 25 min
  • Module 1: The search problem

Where we are

In the machine learning course, a model gave one answer for each row. A shop's search gives a whole list, and people look only at the top of it. In this lesson, we start with a good shop assistant. Then we see Larkfield's search fail, and what each failure costs the shop.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

See how a shop’s search fails people, in numbers. Then put a price on each failure, so that every later change to the search can be judged by what it is worth.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. You run the same searches yourself in Meet the catalogue and the query log, the last lesson of this module.

Think of a large furniture shop with a good shop assistant. A customer walks in and says: “I need a grey cushion, for outside.” She says it quickly, and she does not know the product names.

The assistant does several things in a few seconds. First, he understands her. “For outside” means a cushion for garden furniture. Then he thinks of the products that could match: seat cushions, bench cushions, a few throw pillows. He puts them in order in his head, best first. He shows her the two or three best ones. If she likes one, she buys it. If not, she asks again in other words, or she leaves.

The customer never sees most of this work. She only sees the result: the first cushion that the assistant shows her. If that cushion is wrong, she thinks the shop does not have what she wants. A good assistant also does not make her wait. An answer after ten minutes is almost as bad as no answer.

A search box must do the same job, for thousands of customers at the same time. This course is about how to do that job well, and how to prove that you did.

Larkfield is the online shop for home and garden products that you may know from other courses. In this course, it sells 40,066 products: sofas, cushions, lamps, rugs, mattresses, garden statues and much more. The products and their descriptions are real. They come from WANDS, a public dataset from Wayfair, an online furniture shop. The last lesson of this module tells you more about the data.

Larkfield’s search box works in a very simple way. It shows a product only if every word of the query is in the product’s title or description. A query is the text that a person types into the search box. The products that the search shows are the results.

Here are four real searches on Larkfield’s search today. You can run each one yourself later in this module.

“grey outdor cushion”. The customer made one small spelling mistake: “outdor”. No product has that word, so the rule “every word must match” fails:

current: "grey outdor cushion" -> 0 results

“canapé”. This is the French word for sofa. Larkfield sells in Canada, and many customers search in French. Most product descriptions are in English, so this search also finds nothing.

“couch”. Here the search finds products, but look at the order:

current: "couch" -> 10 results (query P-002: judged results are marked)
1. LK-98476 waytrim indoor chaise lounge sofa , folding lazy sofa floor Indoor Chaise Lounges 247.99 [out of stock] [not judged]
2. LK-10524 futon sofa bed modern linen upholstered couch , convertible Futons 311.99 [exact]
3. LK-12676 hao 78.8 '' wide microfiber/microsuede reversible sofa & cha Sectionals 60.99 [exact]
4. LK-37600 fernley chaise lounge leisure chair rest sofa chaise lounge Indoor Chaise Lounges 132.99 [not judged]
5. LK-24900 lupo 77.16 '' wide left hand facing sofa & chaise with ottom Sectionals 62.99 [exact]
6. LK-80635 convertible modular sectional sofa u shaped couch with rever Sectionals 72.99 [out of stock] [exact]
7. LK-68088 patio furniture cover , extra large waterproof outdoor furni Furniture Covers 163.99 [not judged]
8. LK-90561 sectional couch with reversible chaise modern l-shape sofa 3 Sectionals 72.99 [exact]
9. LK-30025 carballo decorative cotton lumbar pillow Accent Pillows 66.99 [out of stock] [not judged]
10. LK-50332 laforet decorative cotton lumbar pillow Accent Pillows 67.99 [not judged]

The first result is a folding floor chair, and it is out of stock. A cover for patio furniture is at position 7, and two lumbar pillows are at positions 9 and 10. Real sofas are on the page, but not at the top. The word [exact] means that a person checked this product and marked it as an exact match for the query. You learn how these marks are made in Module 2.

“emma headboard”. Emma is a product name. The search finds one product: a platform bed called “emma-lea”. A person marked it as irrelevant. The real Emma headboards are not on the page at all.

So Larkfield’s search fails in three ways. It shows nothing, it shows the wrong product first, or it shows no right product at all. Prices, stock and sale flags in this course are synthetic: our code made them, so a price can look strange.

Every search, in every shop, goes through the same six steps as the shop assistant’s work. The diagram shows them, and what Larkfield’s search does at each step today.

Six boxes in two rows, joined by arrows: 1 Typed, 2 Understood, 3 Candidates, then down to 4 Ranked, 5 Shown, 6 Clicked or abandoned. Under each step, a dashed box says what Larkfield's search does today: 20,000 searches in eight weeks; no understanding, so outdor stays outdor and canapé stays French; every word must match, so 6,188 searches, 30.9%, find nothing; couch puts a chaise lounge first, out of stock; 38.1% of all clicks are on the first result; a click on 9,144 searches and a basket on 3,652.
The journey of one query. The solid boxes are the steps of any search engine. The dashed boxes are Larkfield's search today, in its synthetic query log.
Text description of the diagram

The diagram has six steps, joined by arrows. The first row goes from left to right: 1, typed; 2, understood; 3, candidates. The second row goes back from right to left: 4, ranked; 5, shown; 6, clicked or abandoned. Under each step, a dashed box shows Larkfield today. Typed: 20,000 searches in eight weeks, in a synthetic log. Understood: Larkfield’s search does nothing here, so “outdor” stays “outdor” and “canapé” stays French. Candidates: every word must match, so 6,188 searches, 30.9%, find nothing. Ranked: “couch” puts a chaise lounge first, and it is out of stock. Shown: 38.1% of all clicks are on the first result. Clicked or abandoned: a click on 9,144 searches, and a basket on 3,652.

Three of these steps get names that you will use in every lesson:

  • Candidates are the products that could match the query. A shop with 40,066 products cannot look at each one carefully for every search. So it first finds a few hundred candidates quickly.
  • Ranking puts the candidates in order, best first. The best product should be at position 1.
  • Relevance says how well a product matches what the person wanted. A good ranking puts the most relevant products at the top.

This course follows the journey. Module 3 improves how the candidates are found, and Module 4 understands the query. Modules 5 and 6 improve the ranking. Module 7 works on the page that people see.

Larkfield keeps a query log: a record of every search, with the time, the query, the number of results and what the person clicked. The course’s log has 20,000 searches from 1 September to 26 October 2025. The log is synthetic: our code made it from the real WANDS queries and a simple model of how people click. It is not a record of real customers. The dataset card explains how it was made.

Now count the empty pages. Of the 20,000 searches, 6,188 returned nothing: 30.9%. Almost one search in three ends on an empty page.

The empty pages are not spread evenly. They come from the rule “every word must match”:

Searches in the log Share with no result
English queries from WANDS 16.1%
French queries (machine-translated) 94.3%
Queries with a spelling mistake (planted) 100%
Product codes, such as LK-91857 (planted) 100%
Shorter queries, typed after a failed search 43.8%

The French queries were translated from the English ones by a language model. The spelling mistakes and the product codes are a practice problem we made on purpose. Real shops see them, so we put them in the log. The last row is people who searched again with fewer words after a failed search.

The most common empty searches show the same causes. “hibou” (French for owl) was searched 216 times and never found anything. Next come “ligth bulb” (139 times), “ampoule” (French for bulb, 119 times) and “canapé” (79 times).

In the log, 2,525 of the 6,188 empty pages (40.8%) were followed by another search. In the other 3,663 (59.2%), the person stopped searching. Of the sessions whose first search found nothing, 5.3% ended with something in the basket. Of the sessions whose first search showed products, 31.2% did.

Read these numbers with care. Our code made the log with a rule: after an empty page, half of the people search again. So the log shows how such a rule plays out, not how real Larkfield customers behave. It does show one true thing: a person who meets an empty page has fewer chances to buy.

Some queries are typed again and again. “bee” was searched 1,157 times, and “rack glass” 606 times. But most queries are rare. The log has 1,903 different query texts, and 876 of them were searched only once.

Search teams split queries into three groups by how often people type them. The head is the few most common queries. The torso comes next. The tail, or long tail, is the many rare queries. For Larkfield, the head is the 40 most common texts, and the torso is the next 120.

The 20,000 searches of the synthetic log, by group. 40 head texts make 37.1% of the searches; 1,743 tail texts make 40.2%.
Text description of the chart

The chart has three bars. The head, the 40 most common query texts, makes 37.1% of all searches. The torso, the next 120 texts, makes 22.7%. The tail, the other 1,743 texts, makes 40.2%. So 40 texts are about as many searches as 1,743 texts together.

This shape matters for every later decision. You can look at the 40 head queries by hand. You cannot look at 1,743 tail queries by hand, and together they are the largest group. So a better search must work for queries that nobody has seen before.

Omar is Larkfield’s product manager. He asks the useful question: “What does each failure cost us?” The course uses the same costs in every lesson. They are assumptions, chosen to be realistic. You can change them for your own shop.

The course assumes that 30% of people leave the shop after an empty results page. It assumes that a basket lost in this way is worth 85 dollars. A person who cannot find a product may also write to support, and one support contact costs about 6 dollars.

Here is the calculation for the eight weeks of the log:

  • Empty pages: 6,188.
  • People who leave: 6,188 × 30% ≈ 1,856.
  • Baskets at risk: 6,188 × 30% × 85 dollars = 157,794 dollars, so about 158,000 dollars.

That is about 19,700 dollars a week. Say “at risk under these assumptions”, never “lost”. Some of these people would not have bought anything. Others search again and find the product. The log does not record support contacts, so the course does not count them. It only gives their price.

People look at the top of a page, and much less further down. In the synthetic log, the product at position 1 got a click in 44.6% of the searches that showed it. The product at position 5 got a click in 9.0%, and position 10 in 4.1%:

How often the product at each position got a click, in the synthetic log. Position 1 alone gets 38.1% of all clicks.
Text description of the chart

The chart has ten bars, one for each position on the page, from 1 to 10. Position 1 has the highest bar: 44.6% of the searches that showed a product there had a click on it. The bars fall quickly: 23.2% at position 2, 14.8% at position 3, 11.2% at position 4 and 9.0% at position 5. Positions 6 to 10 are between 7.0% and 4.1%.

This curve also comes from our code. Its click model says that a person looks at position k with a chance of 1 in k. That is an assumption, and Learn from clicks shows how to measure it in a real log. The lesson for now is simple. A right product at position 5 gets a small part of the attention that position 1 gets. So “the right product is somewhere on the page” is not enough. It must be first.

Amira is the developer who owns Larkfield’s search service. She has a latency budget: every search must answer in under 200 milliseconds. Latency is the time between the question and the answer. The course does not put a dollar value on a slow page. It uses the budget as a hard limit instead.

Larkfield’s search today is fast. It answers a typical query in 1.7 milliseconds (the median, measured on the course’s laptop). Its problem is not speed: it is fast and often wrong. Later methods in this course find better products, but some of them take much more time. Every improvement must fit inside Amira’s 200 milliseconds.

Failure What the person sees Cost in this course
Empty page “No results” 30% leave × 85 dollars per basket; 6 dollars per support contact
Wrong first result A product they did not want, at the top Fewer clicks: position 1 gets 38.1% of all clicks
Slow page A page that waits Not in dollars: a hard limit of 200 milliseconds

Larkfield wants a search that finds the best product first. It must work in English and in French, answer in under 200 milliseconds, and never show an empty page without a good reason. And Larkfield wants to change its search by measurement, not by feel.

Five people work on this in the course:

  • Priya, the data scientist, builds the test set, measures and compares the methods. She is your main guide.
  • Omar, the product manager, asks what each failure costs and what a change is worth.
  • Amira, the developer, owns the search service: the index, the latency budget and the releases.
  • Tomás, the AI engineer, has quick ideas. Some of them are common mistakes, and the course tests them.
  • Hana, the merchandising lead, wants to show sale items and new products. Module 7 deals with her requests.

Before anyone changes the search, Priya needs a way to say “this search is better than that one”. “Couch shows a chaise lounge first” is a story about one query. Larkfield needs a number for hundreds of queries. Module 2 builds that number.

Common mistake

Fix the most common queries by hand

Symptom: Tomás has a quick idea: “Let’s look at the 10 most common queries and choose their results by hand. We pin the right products to the top.” It works for those 10 queries, and the search looks much better in a demo.

Cause: the 10 most common texts make only 20.4% of Larkfield’s searches. 876 texts were searched only once, and the tail of 1,743 texts makes 40.2% of all searches. Tomorrow, new queries arrive that nobody has seen. A pinned list also gets old: when a product goes out of stock, its pin still puts it first.

Fix: improve the search method itself, so that it works for queries that nobody pinned. Measure the change on many queries from the head, the torso and the tail. Pins have a place, for a few important queries. Zero results and business rules shows how to use them with care.

Consequence in AI work: a demo on the most common queries hides the tail. Every search change in this course is measured on a test set with head, torso and tail queries.

Larkfield keeps one shop in view while you learn each method. Real work has other data and other traps, so the course ends with three case studies on real, public data. Each one takes a dataset from the raw files to a recommendation.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

  • Product search for a furniture shopOn the full WANDS dataset: a BM25 baseline, query understanding, hybrid search, a cross-encoder and LambdaMART, compared on NDCG@10 and latency; recommend a design for a 200 ms budget.
  • Shopping queries with hard negativesOn the Amazon Shopping Queries dataset: rank products with exact, substitute, complement and irrelevant labels; measure the cost of showing substitutes; compare English and Spanish slices.
  • Search a help centre or a scientific corpusOn an open retrieval benchmark with a permissive licence: lexical, dense and hybrid retrieval, re-ranking, and when the cheap method is good enough.

Read the list as a map of where the course goes. The case studies use the whole course, so most learners start them after Module 8.

Check your understanding

Larkfield's search shows a product only if every word of the query is in it. Explain why this rule gives an empty page for “grey outdor cushion”, “canapé” and “LK-91857”, but not for “sunflower”.

Show the answer

The rule needs every query word in the title or the description. “outdor” is a spelling mistake, so no product has it, and one missing word is enough to fail. “canapé” is French, and most descriptions are in English. “LK-91857” is a product code, and the code is not in the product’s text. “sunflower” is one common English word that many product titles contain, so the rule finds them. In the synthetic log, 30.9% of all searches end on an empty page for these reasons.

Check your understanding

Omar says: “6,188 empty pages × 30% × 85 dollars: we lost 158,000 dollars.” What is wrong with the sentence, and how should he say it?

Show the answer

The 30% and the 85 dollars are assumptions, not measurements. Also, some people would not have bought anything, and some search again and find the product. So the number is money at risk under these assumptions, not money lost. A better sentence: “Under our assumptions, about 158,000 dollars of baskets were at risk in eight weeks because of empty pages.” The number is useful to compare changes, not as a fact about the past.

Check your understanding

Predict: a new search puts the right product at position 5 for every query, where the old one put it at position 1. It has the same products on the page. Is it as good? Use the click curve in your answer.

Show the answer

No. People look at the top of the page much more than further down. In the synthetic log, position 1 got a click in 44.6% of the searches that showed it, and position 5 in only 9.0%. Position 1 alone gets 38.1% of all clicks. So the same products in a worse order give far fewer clicks and baskets. A search must be judged by its order, not only by what is on the page.

Recap

  • A search goes through six steps: typed, understood, candidates, ranked, shown, then clicked or abandoned. Candidates are the products that could match; ranking puts them in order; relevance says how well a product matches what the person wanted.
  • Larkfield’s search shows a product only if every word matches. In the synthetic log, 30.9% of searches return nothing. That is 16.1% of English queries, 94.3% of French queries, and every misspelt query and product code.
  • It also puts wrong products first: “couch” shows a chaise lounge, out of stock, at position 1.
  • The head (40 texts) makes 37.1% of searches; the tail (1,743 texts) makes 40.2%. Fixing queries by hand cannot cover the tail.
  • Costs (assumptions): after an empty page, 30% of people leave, each with a basket of 85 dollars. That puts about 158,000 dollars at risk in eight weeks. Position 1 gets 38.1% of clicks, and every search must answer in under 200 milliseconds.

You can now say how Larkfield’s search fails and what each failure costs. But “couch”, “emma headboard” and “LK-91857” failed in different ways, because people search for different things. Next, you name the kinds of search, and see why each kind needs a different ranking.

Next: Kinds of search