Build a test set that represents your users
- Lesson 6
- 40 min
- Module 2: Measure relevance before you change anything
Where we are
We can now score a ranked list. A score is useful only if the queries are like the queries people really type. In this lesson, we build a test set from head, torso and tail queries, and we measure how sure each result is.
Lesson 6 of 30 · Search and Ranking: Find the Best MatchBuild a test set that represents your users
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Make the test queries look like the searches people really make, and say how sure each result is. You compare Larkfield’s test set with its query log and compute a confidence interval. Then you learn how many queries you need to see a real difference.
NotebookDo this lesson in a notebook
The notebook compares the bands of the log and the test set, and writes a bootstrap interval by hand. Then it compares two recorded searches on the same queries and asks you to find how many queries you need. Its setup cells download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- NDCG@10, MRR and recall@100, from Ranking measures.
- Head, torso and tail queries in the log, from A shop where search fails.
- Train, validation and test sets help, but are not required. The machine learning course explains them in Train, validation and test sets.
What you need on your computer (only for the guided practice)
- The project
larkfield-searchwith this module’s files, from the guided practice of Relevance judgments.
A critic who tastes only the desserts
Section titled “A critic who tastes only the desserts”A food critic visits a restaurant. She tastes the three most popular desserts, and they are excellent. She writes: “A great restaurant.” But most customers come for the main dishes, and many order dishes that she never tasted. Her review is honest, and it is still wrong for most customers.
A test set is the critic’s plate. If its queries are not like the queries people type, the score describes another shop. And a score from a small plate has a second problem: taste one more dish, and the review could change.
This lesson deals with both problems. First, what goes on the plate. Then, how sure you can be about the score.
The test set and the log
Section titled “The test set and the log”Larkfield’s test split has 243 queries. Its query log has 20,000 searches. Remember that the log is synthetic: our code made it from WANDS queries with a click model. It still lets you check the test set against traffic.
Recall the bands of the log. The head is the 40 most searched texts. The torso is the next 120. The tail is every other text: 1,743 different texts, many searched only once. Compare the share of each band in the log’s searches and in the test queries.
Text description of the chart
Three panels, one per band, each with two rows. Head: 37% of the log’s searches, 7% of the test queries (17 queries). Torso: 23% of searches, 17% of test queries (42). Tail: 40% of searches, 76% of test queries (184).
The test set is not wrong, but it answers a particular question. A mean over test queries gives each query the same weight. So it measures “how good is the search for a typical different query”, and most different queries are in the tail. It does not measure “how good is the search for a typical search”. A head query like “bee” is searched hundreds of times, but it is one row in the test set.
You can answer the second question too. Weight each band’s score by its share of searches. For BM25: 0.371 × 0.638 + 0.227 × 0.623 + 0.402 × 0.368 = 0.526. The plain mean over the test queries is 0.429. Both numbers are right. They answer different questions, so say which one you report.
What goes into Larkfield’s test set
Section titled “What goes into Larkfield’s test set”A good test set is built on purpose. It starts from real queries and adds the queries that matter even when they are rare. Larkfield’s 243 test queries come from five sources:
| Source | Test queries | Why it is there |
|---|---|---|
| English WANDS queries | 120 | Real customer queries with real judgments: the base |
| French versions (machine-translated) | 60 | 12% of the log’s searches are in French, and 94% of them return nothing today |
| Misspellings (planted) | 40 | People mistype. “ligth bulb” is the second most common empty search in the log |
| Product codes (planted) | 20 | Some people paste a product code; each has one exact product |
| Broad queries (planted) | 3 | “sofa”, “couch” and “canapé”, judged by a rule on the product class |
The misspellings, codes and broad queries are practice problems we made on purpose. The French queries were written by a language model from the English ones. Each kind was added because the log shows that it fails today: 30.9% of all searches in the log return nothing. A test set built only from popular queries would not see these failures.
Keep some queries away from your work
Section titled “Keep some queries away from your work”Larkfield’s public queries are in three splits: train (260), validation (121) and test (243). You use the train queries to learn from, the validation queries to choose settings, and the test queries to report the result. The machine learning course explains why in Train, validation and test sets.
There is one more set. The course keeps 80 held-out queries (60 English WANDS queries and 20 French versions) that are not in the practice files at all. Each time you look at the test score and change something, the test set leaks a little into your decisions. After many changes, a good test score can come partly from luck. The held-out queries are used once, at the end, to check that the result is real.
How sure is a number?
Section titled “How sure is a number?”BM25’s NDCG@10 on the test queries is 0.429. Choose other test queries, and the number would be a little different. How different? A confidence interval answers this. It is a range that probably contains the score you would get on many more queries of the same kind. The course uses a 95% interval.
The course computes it with the bootstrap. The idea: you cannot get new queries, so you make many new test sets from the queries you have.
A tiny example: five queries
Section titled “A tiny example: five queries”Here are five test queries and BM25’s NDCG@10 on each:
| Query | NDCG@10 |
|---|---|
| A: “dinosaur” | 1.000 |
| B: “home sweet home sign” | 0.913 |
| C: “large spoon and fork wall decor” | 0.905 |
| D: “beds that have leds” | 0.110 |
| E: “blk 18x18 seat cushions” | 0.485 |
Their mean is (1.000 + 0.913 + 0.905 + 0.110 + 0.485) ÷ 5 = 0.683.
Now draw five queries at random, with replacement. “With replacement” means that after you draw a query, you put it back, so the same query can come twice. Three example draws:
| Draw | Queries drawn | Mean |
|---|---|---|
| 1 | A, A, C, D, E | (1.000 + 1.000 + 0.905 + 0.110 + 0.485) ÷ 5 = 0.700 |
| 2 | B, D, D, E, C | (0.913 + 0.110 + 0.110 + 0.485 + 0.905) ÷ 5 = 0.505 |
| 3 | A, B, C, B, A | (1.000 + 0.913 + 0.905 + 0.913 + 1.000) ÷ 5 = 0.946 |
The means jump from 0.505 to 0.946. With five queries, one bad query like D changes everything. Repeat the draw 2,000 times, sort the 2,000 means, and cut off the lowest 2.5% and the highest 2.5%. The middle 95% is the interval.
Text description of the diagram
A row of boxes. “243 test queries, one NDCG@10 each” leads to “Draw 243 queries at random, with replacement”, then to “Mean of this draw”. An arrow labelled “repeat 2,000 times” goes back to the draw box. From the mean box, an arrow leads to “2,000 means: sort them”, then to a box with a thick border: “Middle 95% = the interval. BM25: 0.381 to 0.476”.
On all the test queries (241 of the 243 have an NDCG), BM25’s interval is 0.381 to 0.476. It is much narrower than the five-query example, because one query has less weight among 243.
Small slices have wide intervals
Section titled “Small slices have wide intervals”The interval depends on the number of queries. Look at BM25’s score in each band:
Text description of the chart
Four rows, each a point with an interval bar. Head (17 queries): 0.638 (0.45 to 0.816). Torso (42): 0.623 (0.525 to 0.715). Tail (184): 0.368 (0.315 to 0.42). All test queries: 0.429 (0.381 to 0.476).
The head’s interval goes from 0.450 to 0.816. With 17 queries, you know little about the head. Before Omar decides anything about the most popular searches, Priya would add more head queries to the test set.
One query in many
Section titled “One query in many”The explorer from the last lesson now also shows how much one query moves an average. The rug-set query is one of 121 validation queries.
Explore
One query in an average of 121
Predict first: Put the irrelevant bath set at rank 1 again. This query falls by more than 0.1. How much does the average over 121 queries fall?
Drag a row, or use its arrow buttons.
- DCG@10
- 9.963
- Ideal DCG@10
- 13.631
- NDCG@10
- 0.731
NDCG@10 is 0.731: the list earns 9.963 of the 13.631 points that the ideal top 10 would earn (DCG@10 ÷ ideal DCG@10). Precision@10: 6 of 10 results are exact. Change in the average over 121 queries: 0.000.
| Rank | Product | Grade | Gain | Discount | Part of DCG |
|---|---|---|---|---|---|
| 1 | ellington circle 3 piece area rug setArea Rugs | 2 exact | 3 | 1.000 | 3.000 |
| 2 | cedric oriental ebony area rug setArea Rugs | 1 partial | 1 | 1.585 | 0.631 |
| 3 | mokvist 3 piece floral border red area rug setArea Rugs | 2 exact | 3 | 2.000 | 1.500 |
| 4 | wilkesboro 3 piece floral border blue area rug setArea Rugs | 2 exact | 3 | 2.322 | 1.292 |
| 5 | braylei simple elegance 3 piece solid bath setBath Rugs & Mats | 0 irrelevant | 0 | 2.585 | 0.000 |
| 6 | fontenay 3 piece braided brown rug setArea Rugs | 2 exact | 3 | 2.807 | 1.069 |
| 7 | kamberk floral teal area rugArea Rugs | 1 partial | 1 | 3.000 | 0.333 |
| 8 | rhodes stripped checker 3 piece hand-knotted beige area rug setArea Rugs | 2 exact | 3 | 3.170 | 0.946 |
| 9 | audrey 3 piece geometric brown/moss area rug setArea Rugs | 2 exact | 3 | 3.322 | 0.903 |
| 10 | shawnna 3 piece bath rug setBath Rugs & Mats | 1 partial | 1 | 3.459 | 0.289 |
Text description of the explorer
The same list of BM25’s ten results for “3 piece rug set with runners”, with grades, gains, discounts and the parts of DCG. The status line also shows the change in the average over 121 queries. Moving the bath set from rank 5 to rank 1 drops this query’s NDCG@10 from 0.731 to 0.615, a change of −0.116. The average over 121 queries moves by about −0.001.
What happens
The query falls by 0.116. The average over 121 queries falls by 0.116 ÷ 121, about 0.001. A large change on one query is almost invisible in the mean. So a change that helps a few queries a lot, or many queries a little, needs many test queries to show up clearly.
Is B really better? Compare on the same queries
Section titled “Is B really better? Compare on the same queries”Now compare two searches. You could compute both intervals and check if they overlap. There is a better way. Both searches run on the same queries, so compare them query by query. For each query, compute B’s score minus A’s score. Then compute the mean of these differences and its bootstrap interval. This is a paired comparison.
Pairing removes the difference between easy and hard queries, because each query is compared with itself. If the interval of the difference does not contain 0, the test set shows a clear difference. The project’s search compare does this.
How many queries do you need?
Section titled “How many queries do you need?”Priya asks a planning question: “If a change really helps, how many test queries do I need to see it?” She tested it on real changes from later modules. She drew smaller test sets at random, 200 times for each size, and counted how often the paired interval was clear.
- A change of +0.077
- A change of +0.037
Text description of the chart
A line chart with the number of test queries (25, 50, 100, 200) on the horizontal axis and the share of draws where the change is clear on the vertical axis. A change of +0.077: 25 queries 63%, 50 queries 88%, 100 queries 100%, 200 queries 100%. A change of +0.037: 25 queries 26%, 50 queries 47%, 100 queries 76%, 200 queries 100%.
The larger change is +0.077, from Normalise text. It is clear in 63% of the draws of 25 queries and in all draws of 100. The smaller change is +0.037, from spelling and synonyms. It is clear in only 26% of the draws of 25 queries. With 25 queries, you would miss this real improvement three times out of four. You need about 200 queries to see it every time.
So decide the size of the test set from the smallest change that matters to you. Small changes need many queries. The evaluation course treats intervals and slices for other AI systems in Uncertainty and slices.
Guided practice: bands, intervals and a paired comparison
Section titled “Guided practice: bands, intervals and a paired comparison”-
Measure BM25 by band:
Terminal window search eval --run bm25 --by bandAfter the first two lines (as in the last lesson):
band queries ndcg@10 low high zero-result ratehead 17 0.638 0.450 0.816 0.176tail 184 0.368 0.315 0.420 0.141torso 42 0.623 0.525 0.715 0.048lowandhighare the 95% interval of each band. -
Compare today’s search and BM25 on the same queries, by band:
Terminal window search compare current bm25 --by bandbm25 minus current on 241 test queries: ndcg@10 +0.234 (95% interval +0.197 to +0.270); better on 138, worse on 9, the same on 94band queries current bm25 difference low high worsehead 16 0.652 0.638 -0.015 -0.036 0.000 2tail 184 0.103 0.368 0.265 0.220 0.312 4torso 41 0.433 0.623 0.190 0.100 0.289 3
Read the first line of step 2. BM25 minus today’s search is +0.234, and the interval (+0.197 to +0.270) is far from 0: a clear difference. The count adds detail: BM25 is better on 138 queries, worse on 9 and the same on 94. The comparison uses 241 queries, because the two queries with no good product have no NDCG.
Then read the bands. On the tail, the difference is large and clear. On the head, BM25 is a little worse (−0.015), and the interval ends at 0.000. Only 16 head queries are in the comparison. The test set cannot say if this small loss is real. Note that the head has 16 queries here and 17 in step 1: “printers” is a head query, and it has no NDCG.
Your turn: compare by source
Section titled “Your turn: compare by source”Change one thing: the slice. Predict which source gains the most from BM25, and which gains nothing. Then run:
search compare current bm25 --by sourceThe output, and what it shows
source queries current bm25 difference low high worse code 20 0.000 0.000 0.000 0.000 0.000 0 planted 3 0.451 0.451 0.000 0.000 0.000 0translated 59 0.012 0.177 0.165 0.103 0.236 0 typo 40 0.000 0.357 0.357 0.267 0.448 0 wands 119 0.378 0.650 0.272 0.215 0.334 9Misspellings gain the most (+0.357): today’s search needs every word, so one wrong word empties the page. Product codes gain nothing: both searches score 0. BM25 does not know that a code is special. All 9 queries where BM25 is worse are English WANDS queries. The three planted broad queries do not change; with only 3 queries, that slice tells you almost nothing.
Common mistake
A winner that was only noise
Symptom: Priya tries 96 settings of field weights on the 121 validation queries and keeps the best one. It is ahead by only 0.006. On the test queries, it loses to the weights that she picked by hand. You meet this case, with all its numbers, in BM25.
Cause: a difference of 0.006 on 121 queries is much smaller than the noise. Among 96 settings, one is always a little ahead by luck. Choosing the best of many settings on the same queries picks up that luck.
Fix: compare the top settings with a paired interval before you trust a winner. Treat a difference inside the noise as a tie, and prefer the simpler setting. Check the final choice once on queries you did not use to choose it.
Consequence in search work: every search team tries many changes. Without intervals, it will ship changes that only looked good. With a fixed test set that it checks again and again, it will also slowly tune the search to that test set. The held-out queries protect against this.
Knowledge check
Section titled “Knowledge check”Check your understanding
Larkfield's head is 37.1% of the searches but only 7% of the test queries. What does the plain mean over test queries measure, and how would you get a number for a typical search?
Show the answer
The plain mean gives each test query the same weight. So it measures the search for a typical different query, and most of those are tail queries. For a typical search, weight each band’s score by its share of searches. For BM25: 0.371 × 0.638 + 0.227 × 0.623 + 0.402 × 0.368 = 0.526, against a plain mean of 0.429. Say which number you report.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Three queries have NDCG@10 of 0.9, 0.3 and 0.6. One bootstrap draw picks the first, the first again, and the third. What is the mean of this draw, and why can a query appear twice?
Show the answer
(0.9 + 0.9 + 0.6) ÷ 3 = 0.8. The bootstrap draws with replacement: after a query is drawn, it goes back, so it can be drawn again. That is how each draw makes a new test set of the same size from the same queries.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A change scores +0.02 NDCG@10 on 30 test queries, with a paired 95% interval of −0.03 to +0.07. Omar asks: “Is it better?” What do you answer, and what would you do next?
Show the answer
The test set cannot say. The interval contains 0, so the change could be better, the same or a little worse. With 30 queries, even a real small change is often not clear. On Larkfield’s data, a +0.037 change was clear in only about a quarter of the draws of 25 queries. Next: run it on more test queries (about 200 for a change this small), and look at the slices that matter.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Check the test set against the log. Larkfield’s head is 37.1% of searches but 7% of test queries; a plain mean speaks mostly for the tail. BM25: 0.429 plain, 0.526 weighted by searches.
- Add the queries that fail today, even when they are rare: French, misspellings, product codes. Keep held-out queries away from your work and use them once.
- A confidence interval says how sure a number is. The bootstrap draws the queries again with replacement 2,000 times and keeps the middle 95% of the means.
- Compare two searches paired, query by query. BM25 minus today: +0.234 (+0.197 to +0.270) on Larkfield’s test queries.
- Small changes need many queries: a +0.037 change was clear in 26% of draws of 25 queries and in all draws of 200.
The test set has one more weak point: the products that nobody judged. Next, you let a language model write the missing labels, and you check it against people.