Kinds of search
- Lesson 2
- 30 min
- Module 1: The search problem
Where we are
In the last lesson, we saw Larkfield's search fail in several ways, and what each failure costs. The failures are different because people search for different things. In this lesson, we name the kinds of search and see why each kind needs a different ranking. We start again in the shop, with four customers.
Lesson 2 of 30 · Search and Ranking: Find the Best MatchKinds of search
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Tell apart the kinds of search that people do, and say what a good first result is for each kind. A search that ranks well for one kind can fail badly for another, and an average over all queries hides it.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s search today and its three failures (an empty page, a wrong first result, no right product), from A shop where search fails.
- The words query, candidates, ranking and relevance, from the same lesson.
What you need on your computer
Nothing. This lesson has no code. You run the searches of this lesson yourself in Meet the catalogue and the query log.
Four customers in the shop
Section titled “Four customers in the shop”Go back to the furniture shop and its good assistant. In one morning, four customers come to him.
The first customer says: “Do you have the Serta Kleinmon mattress?” She knows the exact product. The assistant takes her straight to it. If it is out of stock, he tells her at once. Showing her ten other mattresses first would only waste her time.
The second customer says: “I want a teal chair.” He knows the kind of product and one detail, the colour. The assistant shows him three or four chairs that are teal. A teal rug is the right colour, but it is not a chair, so it is no help.
The third customer says: “I like sunflowers. Do you have anything with sunflowers?” She does not know what she wants yet. The assistant shows her a table with many different things: flowers in a vase, a picture, a plate, a garden ornament. Here, a mix is the best answer.
The fourth customer says: “How do I return a parcel?” He does not want a product at all. He wants an answer. The assistant sends him to the returns desk.
The same assistant gave four very different answers. Each answer was right for its customer. The assistant first understood what kind of help the customer needed.
The problem: one ranking for every query
Section titled “The problem: one ranking for every query”Larkfield’s search does not ask what kind of help people need. It uses the same rule for every query, and the rule fails each kind in a different way. Here are real results from Larkfield’s search today.
“serta kleinmon” finds four products. The first one is a Serta Kleinmon mattress, but it is out of stock, and a person marked it only as a partial match. The exact products are at positions 2, 3 and 4.
current: "serta kleinmon" -> 4 results (query W-074: judged results are marked) 1. LK-83197 serta perfect sleeper 11 '' kleinmon ii plush innerspring ma Innerspring Mattresses 36.99 [out of stock] [partial] 2. LK-77895 serta perfect sleeper 11 '' kleinmon ii plush innerspring ma Innerspring Mattresses 52.99 [exact] 3. LK-72024 serta perfect sleeper kleinmon 11 '' firm innerspring mattre Innerspring Mattresses 33.99 [out of stock] [exact] 4. LK-38374 serta perfect sleeper kleinmon 10 '' firm innerspring mattre Innerspring Mattresses 30.99 [exact]“LK-91857” is a product code: a bunk bed. A customer copied it from an email. The search finds nothing, because the code is not in the product’s text.
“teal chair” is harder to see. Larkfield’s search finds 8 products, and none of them was marked as an exact match. When the course’s first better search (BM25, in the last lesson of this module) tries it, a teal rug comes second:
bm25: "teal chair" -> 3 results (query W-409: judged results are marked) 1. LK-92620 office chair , velvet desk chair with gold metal legs , adju Office Chairs 166.99 [partial] 2. LK-91218 murph teal/gray indoor/outdoor area rug Area Rugs 97.99 [not judged] 3. LK-55458 arteaga ombre flatweave cotton teal rug Area Rugs 98.99 [irrelevant]“sunflower” works well today. All 10 results are exact matches. They are also a good mix: artificial flowers, a framed print, a salad plate, a garden stake and a bird feeder.
So the same search is good for one kind of query and bad for another. If you look only at “sunflower”, you think that it works. If you look only at “LK-91857”, you think that it is broken.
The idea: name the kind of search
Section titled “The idea: name the kind of search”Search teams sort queries by what the person wants. This course uses four kinds:
- In a known-item search, the person wants one product that they already know. They type its name, its brand and model, or its product code. Example: “serta kleinmon”, “LK-91857”.
- In a specific search, the person wants a kind of product with one or two details. Example: “teal chair”, “waterfall faucet”.
- In a broad search, the person wants to explore. Many products are right, and the person does not know which one they want yet. Example: “sunflower”, “meditation”.
- A question asks for an answer, not a product. Example: “how do I return a parcel”.
The name of each kind tells you what a good first result is. The diagram shows one example of each kind from Larkfield’s data.
Text description of the diagram
The diagram has four columns. Each column has the kind of search, an example query, what the person wants and a good first result. Its last box says how many right products a query of this kind typically has. Known item: “serta kleinmon”. The person wants one product that they already know. A good first result is that exact mattress, in stock. A typical known-item query has 1 right product. Specific: “teal chair”. The person wants a kind of product with one or two details. A good first result is a chair that is teal, not a teal rug. A typical specific query has 22 right products. Broad: “sunflower”. The person wants ideas, and many products are right. A good first page is a mix, such as flowers, wall art and a plate. A typical broad query has 78 right products. Question: “how do I return a parcel”. The person wants an answer, not a product. A good first result is the help page about returns. This kind is not in the course’s data; the RAG course teaches it.
A tiny example: how many right answers?
Section titled “A tiny example: how many right answers?”The kind of search changes one number a lot: how many products are right. The WANDS data has people’s judgments for each query. A person looked at the products for the query and marked each one as an exact match, a partial match or irrelevant. So we can count the exact matches for each query, and take the middle value (the median) for each kind:
Text description of the chart
The chart has three bars. A known-item query has a median of 1 exact product. A specific query has a median of 22. A broad query has a median of 78. So a known-item search has one target, and a broad search has many.
Look at the three examples in the same way. “serta kleinmon” has 3 exact products, all of the same kind of mattress. “teal chair” has 37 exact products, in 5 product classes. “sunflower” has 214 exact products, in 68 product classes: from flowers to plates to bird feeders.
The number of right answers changes what “a good ranking” means:
| Kind | Right products | What matters most |
|---|---|---|
| Known item | One, or a few versions | That one product is at position 1. Position 2 is already a failure. |
| Specific | Tens | Every detail holds for each product in the top 10: a chair, and teal. |
| Broad | Many | The first page shows a mix, so the person can choose a direction. |
| Question | An answer | The answer, or the page that holds it. A product is the wrong result. |
Kinds of search at Larkfield
Section titled “Kinds of search at Larkfield”How did Priya get a kind for each query? She did not label 360 queries by hand. A language model labelled each WANDS query as known-item, specific or broad. The course recorded these answers once and replays them, so you get the same labels every time. A model’s labels can be wrong, so treat them as a useful guide, not as the truth. Module 2 shows how to check a model’s labels against people’s labels.
Of the 360 public WANDS queries, 88 are known-item, 230 are specific and 42 are broad. In the synthetic log, broad queries are searched most: 7,372 searches, against 6,640 specific and 2,008 known-item searches. This follows from how our code made the log: short, broad queries are the most common.
Now look at the empty pages again, by kind:
Text description of the chart
The chart has three bars. Known-item searches return nothing in 63.0% of cases. Specific searches return nothing in 31.7%. Broad searches return nothing in 14.6%.
Known-item searches fail most. The reason is the rule “every word must match”. A known-item query has rare, exact words: a brand, a model name or a code. One spelling mistake, or a code that is not in the text, and the page is empty. A broad query is often one common word, such as “sunflower”, and many products contain it.
Priya can also measure the kinds separately on the test queries. You run this command in the last lesson of this module. It measures BM25, a better search that you meet there, by kind. NDCG@10 is a score from 0 to 1 for the top 10 results: higher is better, and Module 2 explains it.
bm25 on 243 test queries: NDCG@10 0.429 (95% interval 0.381 to 0.476)recall@100 0.388, MRR 0.438, zero results 12.8%, judged in the top 10 64.6%; 0.1 ms a query (p50), 0.3 ms (p95)
qtype queries ndcg@10 low high zero-result rate broad 27 0.506 0.366 0.648 0.185known-item 80 0.422 0.338 0.509 0.275 specific 136 0.419 0.360 0.476 0.029Each kind gets its own score. A score for each group of queries is called a slice. The course reports slices beside the overall score in every module, because an average can hide a group that fails.
Questions: search for people and retrieval for a model
Section titled “Questions: search for people and retrieval for a model”The fourth customer asked “how do I return a parcel”. Many shops have a help centre, and people type questions like this into the same search box. A product search cannot answer it: no product is a good result.
Question-style search needs a different system. It searches documents such as help pages, not products. Then it shows the page, or a language model writes a short answer from it. This course does not build it: it works on product search only, and its data has no help pages. The course RAG: Building AI That Uses Your Data teaches this kind of search. Its first lesson, What retrieval-augmented generation is, starts from questions about Larkfield’s policies.
Both systems find text that matches a query, so they share many methods: BM25, embeddings and re-ranking. But they serve different readers, and so “good” means different things:
| Search for people (this course) | Retrieval for a language model (the RAG course) | |
|---|---|---|
| Who reads the results | A person, on a page | A language model, inside its prompt |
| What is found | Products | Passages from documents |
| What matters most | The order: the best product first | That the needed passage is somewhere in the few that are sent |
| How it is measured | Ranking measures with graded judgments; clicks | Recall of the right passages; the answer’s correctness |
| Time | Under 200 milliseconds for the page | The model’s answer takes seconds anyway |
So if a teammate says “we already have a search for our assistant”, ask who reads its results. A search for a model can be good enough with the right passage at position 4. A search for people cannot.
When the kinds help, and what they cost
Section titled “When the kinds help, and what they cost”Use the kinds in two ways. First, measure by kind: always look at the slices, not only the average. Second, choose what to fix. Larkfield’s known-item searches fail most often, so exact-match rules for names and codes are worth a lot. Module 3 builds them.
Do not build a separate search for each kind yet. That needs a reliable way to tell the kind of every new query. The kind labels in this course come from a language model and were recorded once. A wrong guess would send a query to the wrong search. Also, many queries are between two kinds: “emma headboard” is a known item and a specific search at the same time.
Common mistake
Treat a product code as text
Symptom: a customer pastes “LK-91857” from an order email. Larkfield’s search returns nothing. In the synthetic log, every one of the 301 code searches found nothing.
Cause: the search treats the code like any other word. It looks for “LK-91857” in titles and descriptions, and the code is not there. It is a field of the product, not part of its text. Even a better word search cannot help: BM25 also returns nothing for this code.
Fix: recognise a known-item query that is a product code, and look the code up directly. Module 3 adds this rule, and it puts the right product first for every code query of the test set.
Consequence in AI work: a method that improves the average can still miss a whole kind of query. Check each kind as a slice, and handle exact identifiers with exact rules.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name the kind of each query, and say what a good first result is: “arlo bunk bed”, “grey outdoor cushion”, “bohemian”, “can I change my delivery address”.
Show the answer
- “arlo bunk bed”: a known-item search (a product name). A good first result is the Arlo bunk bed itself, in stock.
- “grey outdoor cushion”: a specific search (a kind of product with two details). A good first result is a cushion for outdoor furniture that is grey.
- “bohemian”: a broad search (a style, many right products). A good first page shows a mix of bohemian products, such as rugs, pillows and wall art.
- “can I change my delivery address”: a question. A good first result is the help page about delivery, or an answer. A product is the wrong result, and this course’s product search does not handle it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a new search raises the overall score on the test queries a little. On which kind of query would you check first that it did not get worse, and why?
Show the answer
Check every kind as a slice, and start with the known-item queries. They have only one right product. A small change in order can move it from position 1 to position 2, which is already a failure. They also fail most often today (63.0% empty pages in the log). An overall score can rise because broad or specific queries improve, while known-item queries get worse. Only the slices show it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “Our help-centre assistant already has a search that finds the right passage in its top 5. Let's use the same search for products.” What do you tell him?
Show the answer
The two searches serve different readers. A language model reads all 5 passages, so the right passage can be at position 4. A person reads a product page from the top, and position 1 gets most of the clicks. Product search must also answer in under 200 milliseconds, and it works on products, not documents. The methods may be similar, but the search must be measured again on product queries with a ranking measure, by kind of query.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A known-item search wants one product (“serta kleinmon”, “LK-91857”): it must be at position 1. A specific search wants a kind of product with details (“teal chair”): every result must have the details. A broad search explores (“sunflower”): the first page should show a mix. A question wants an answer, not a product.
- The median number of exact products per WANDS query is 1 for known-item, 22 for specific and 78 for broad queries.
- In the synthetic log, known-item searches return nothing most often: 63.0%, against 31.7% for specific and 14.6% for broad searches.
- Measure each kind as a slice: an average hides a kind that fails.
- Question-style search finds passages for people or for a language model. The RAG course teaches it; this course works on product search only.
You can now say what a good first result is for each kind of search. But so far you have only read results that someone else ran. Next, you set up the project, open the catalogue and the query log yourself, and run Larkfield’s search and BM25 on your own computer.