Skip to content
Français

Build a test set that represents your users

  • Lesson 6
  • 40 min
  • Module 2: Measure relevance before you change anything

Where we are

We can now score a ranked list. A score is useful only if the queries are like the queries people really type. In this lesson, we build a test set from head, torso and tail queries, and we measure how sure each result is.

What you will be able to do

Make the test queries look like the searches people really make, and say how sure each result is. You compare Larkfield’s test set with its query log and compute a confidence interval. Then you learn how many queries you need to see a real difference.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook compares the bands of the log and the test set, and writes a bootstrap interval by hand. Then it compares two recorded searches on the same queries and asks you to find how many queries you need. Its setup cells download the practice data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the guided practice)

  • The project larkfield-search with this module’s files, from the guided practice of Relevance judgments.

A food critic visits a restaurant. She tastes the three most popular desserts, and they are excellent. She writes: “A great restaurant.” But most customers come for the main dishes, and many order dishes that she never tasted. Her review is honest, and it is still wrong for most customers.

A test set is the critic’s plate. If its queries are not like the queries people type, the score describes another shop. And a score from a small plate has a second problem: taste one more dish, and the review could change.

This lesson deals with both problems. First, what goes on the plate. Then, how sure you can be about the score.

Larkfield’s test split has 243 queries. Its query log has 20,000 searches. Remember that the log is synthetic: our code made it from WANDS queries with a click model. It still lets you check the test set against traffic.

Recall the bands of the log. The head is the 40 most searched texts. The torso is the next 120. The tail is every other text: 1,743 different texts, many searched only once. Compare the share of each band in the log’s searches and in the test queries.

The head is more than a third of all searches, but only 7% of the test queries. The tail is the opposite.
Text description of the chart

Three panels, one per band, each with two rows. Head: 37% of the log’s searches, 7% of the test queries (17 queries). Torso: 23% of searches, 17% of test queries (42). Tail: 40% of searches, 76% of test queries (184).

The test set is not wrong, but it answers a particular question. A mean over test queries gives each query the same weight. So it measures “how good is the search for a typical different query”, and most different queries are in the tail. It does not measure “how good is the search for a typical search”. A head query like “bee” is searched hundreds of times, but it is one row in the test set.

You can answer the second question too. Weight each band’s score by its share of searches. For BM25: 0.371 × 0.638 + 0.227 × 0.623 + 0.402 × 0.368 = 0.526. The plain mean over the test queries is 0.429. Both numbers are right. They answer different questions, so say which one you report.

A good test set is built on purpose. It starts from real queries and adds the queries that matter even when they are rare. Larkfield’s 243 test queries come from five sources:

Source Test queries Why it is there
English WANDS queries 120 Real customer queries with real judgments: the base
French versions (machine-translated) 60 12% of the log’s searches are in French, and 94% of them return nothing today
Misspellings (planted) 40 People mistype. “ligth bulb” is the second most common empty search in the log
Product codes (planted) 20 Some people paste a product code; each has one exact product
Broad queries (planted) 3 “sofa”, “couch” and “canapé”, judged by a rule on the product class

The misspellings, codes and broad queries are practice problems we made on purpose. The French queries were written by a language model from the English ones. Each kind was added because the log shows that it fails today: 30.9% of all searches in the log return nothing. A test set built only from popular queries would not see these failures.

Larkfield’s public queries are in three splits: train (260), validation (121) and test (243). You use the train queries to learn from, the validation queries to choose settings, and the test queries to report the result. The machine learning course explains why in Train, validation and test sets.

There is one more set. The course keeps 80 held-out queries (60 English WANDS queries and 20 French versions) that are not in the practice files at all. Each time you look at the test score and change something, the test set leaks a little into your decisions. After many changes, a good test score can come partly from luck. The held-out queries are used once, at the end, to check that the result is real.

BM25’s NDCG@10 on the test queries is 0.429. Choose other test queries, and the number would be a little different. How different? A confidence interval answers this. It is a range that probably contains the score you would get on many more queries of the same kind. The course uses a 95% interval.

The course computes it with the bootstrap. The idea: you cannot get new queries, so you make many new test sets from the queries you have.

Here are five test queries and BM25’s NDCG@10 on each:

Query NDCG@10
A: “dinosaur” 1.000
B: “home sweet home sign” 0.913
C: “large spoon and fork wall decor” 0.905
D: “beds that have leds” 0.110
E: “blk 18x18 seat cushions” 0.485

Their mean is (1.000 + 0.913 + 0.905 + 0.110 + 0.485) ÷ 5 = 0.683.

Now draw five queries at random, with replacement. “With replacement” means that after you draw a query, you put it back, so the same query can come twice. Three example draws:

Draw Queries drawn Mean
1 A, A, C, D, E (1.000 + 1.000 + 0.905 + 0.110 + 0.485) ÷ 5 = 0.700
2 B, D, D, E, C (0.913 + 0.110 + 0.110 + 0.485 + 0.905) ÷ 5 = 0.505
3 A, B, C, B, A (1.000 + 0.913 + 0.905 + 0.913 + 1.000) ÷ 5 = 0.946

The means jump from 0.505 to 0.946. With five queries, one bad query like D changes everything. Repeat the draw 2,000 times, sort the 2,000 means, and cut off the lowest 2.5% and the highest 2.5%. The middle 95% is the interval.

The bootstrap as a loop. 243 test queries with one NDCG@10 each. Draw 243 queries at random, with replacement. Take the mean of this draw. Repeat 2,000 times. Sort the 2,000 means. The middle 95% is the interval: for BM25, 0.381 to 0.476.
The bootstrap: make 2,000 new test sets from the queries you have, and see how much the mean moves.
Text description of the diagram

A row of boxes. “243 test queries, one NDCG@10 each” leads to “Draw 243 queries at random, with replacement”, then to “Mean of this draw”. An arrow labelled “repeat 2,000 times” goes back to the draw box. From the mean box, an arrow leads to “2,000 means: sort them”, then to a box with a thick border: “Middle 95% = the interval. BM25: 0.381 to 0.476”.

On all the test queries (241 of the 243 have an NDCG), BM25’s interval is 0.381 to 0.476. It is much narrower than the five-query example, because one query has less weight among 243.

The interval depends on the number of queries. Look at BM25’s score in each band:

BM25's NDCG@10 per band, with 95% intervals. The head has only 17 test queries, so its interval is very wide.
Text description of the chart

Four rows, each a point with an interval bar. Head (17 queries): 0.638 (0.45 to 0.816). Torso (42): 0.623 (0.525 to 0.715). Tail (184): 0.368 (0.315 to 0.42). All test queries: 0.429 (0.381 to 0.476).

The head’s interval goes from 0.450 to 0.816. With 17 queries, you know little about the head. Before Omar decides anything about the most popular searches, Priya would add more head queries to the test set.

The explorer from the last lesson now also shows how much one query moves an average. The rug-set query is one of 121 validation queries.

Explore

One query in an average of 121

DCG@10
9.963
Ideal DCG@10
13.631
NDCG@10
0.731

NDCG@10 is 0.731: the list earns 9.963 of the 13.631 points that the ideal top 10 would earn (DCG@10 ÷ ideal DCG@10). Precision@10: 6 of 10 results are exact. Change in the average over 121 queries: 0.000.

The query "3 piece rug set with runners": each row adds gain ÷ discount to DCG@10
RankProductGradeGainDiscountPart of DCG
1ellington circle 3 piece area rug setArea Rugs2 exact31.0003.000
2cedric oriental ebony area rug setArea Rugs1 partial11.5850.631
3mokvist 3 piece floral border red area rug setArea Rugs2 exact32.0001.500
4wilkesboro 3 piece floral border blue area rug setArea Rugs2 exact32.3221.292
5braylei simple elegance 3 piece solid bath setBath Rugs & Mats0 irrelevant02.5850.000
6fontenay 3 piece braided brown rug setArea Rugs2 exact32.8071.069
7kamberk floral teal area rugArea Rugs1 partial13.0000.333
8rhodes stripped checker 3 piece hand-knotted beige area rug setArea Rugs2 exact33.1700.946
9audrey 3 piece geometric brown/moss area rug setArea Rugs2 exact33.3220.903
10shawnna 3 piece bath rug setBath Rugs & Mats1 partial13.4590.289
The same rug-set list as in the last lesson. The status line now also shows the change in the average NDCG over the 121 validation queries, when only this query changes.
Text description of the explorer

The same list of BM25’s ten results for “3 piece rug set with runners”, with grades, gains, discounts and the parts of DCG. The status line also shows the change in the average over 121 queries. Moving the bath set from rank 5 to rank 1 drops this query’s NDCG@10 from 0.731 to 0.615, a change of −0.116. The average over 121 queries moves by about −0.001.

What happens

The query falls by 0.116. The average over 121 queries falls by 0.116 ÷ 121, about 0.001. A large change on one query is almost invisible in the mean. So a change that helps a few queries a lot, or many queries a little, needs many test queries to show up clearly.

Is B really better? Compare on the same queries

Section titled “Is B really better? Compare on the same queries”

Now compare two searches. You could compute both intervals and check if they overlap. There is a better way. Both searches run on the same queries, so compare them query by query. For each query, compute B’s score minus A’s score. Then compute the mean of these differences and its bootstrap interval. This is a paired comparison.

Pairing removes the difference between easy and hard queries, because each query is compared with itself. If the interval of the difference does not contain 0, the test set shows a clear difference. The project’s search compare does this.

Priya asks a planning question: “If a change really helps, how many test queries do I need to see it?” She tested it on real changes from later modules. She drew smaller test sets at random, 200 times for each size, and counted how often the paired interval was clear.

  • A change of +0.077
  • A change of +0.037
How often a random test set of each size shows a real change clearly (200 draws per size). A small change needs about 200 queries.
Text description of the chart

A line chart with the number of test queries (25, 50, 100, 200) on the horizontal axis and the share of draws where the change is clear on the vertical axis. A change of +0.077: 25 queries 63%, 50 queries 88%, 100 queries 100%, 200 queries 100%. A change of +0.037: 25 queries 26%, 50 queries 47%, 100 queries 76%, 200 queries 100%.

The larger change is +0.077, from Normalise text. It is clear in 63% of the draws of 25 queries and in all draws of 100. The smaller change is +0.037, from spelling and synonyms. It is clear in only 26% of the draws of 25 queries. With 25 queries, you would miss this real improvement three times out of four. You need about 200 queries to see it every time.

So decide the size of the test set from the smallest change that matters to you. Small changes need many queries. The evaluation course treats intervals and slices for other AI systems in Uncertainty and slices.

Guided practice: bands, intervals and a paired comparison

Section titled “Guided practice: bands, intervals and a paired comparison”
  1. Measure BM25 by band:

    Terminal window
    search eval --run bm25 --by band

    After the first two lines (as in the last lesson):

    band queries ndcg@10 low high zero-result rate
    head 17 0.638 0.450 0.816 0.176
    tail 184 0.368 0.315 0.420 0.141
    torso 42 0.623 0.525 0.715 0.048

    low and high are the 95% interval of each band.

  2. Compare today’s search and BM25 on the same queries, by band:

    Terminal window
    search compare current bm25 --by band
    bm25 minus current on 241 test queries: ndcg@10 +0.234 (95% interval +0.197 to +0.270); better on 138, worse on 9, the same on 94
    band queries current bm25 difference low high worse
    head 16 0.652 0.638 -0.015 -0.036 0.000 2
    tail 184 0.103 0.368 0.265 0.220 0.312 4
    torso 41 0.433 0.623 0.190 0.100 0.289 3

Read the first line of step 2. BM25 minus today’s search is +0.234, and the interval (+0.197 to +0.270) is far from 0: a clear difference. The count adds detail: BM25 is better on 138 queries, worse on 9 and the same on 94. The comparison uses 241 queries, because the two queries with no good product have no NDCG.

Then read the bands. On the tail, the difference is large and clear. On the head, BM25 is a little worse (−0.015), and the interval ends at 0.000. Only 16 head queries are in the comparison. The test set cannot say if this small loss is real. Note that the head has 16 queries here and 17 in step 1: “printers” is a head query, and it has no NDCG.

Change one thing: the slice. Predict which source gains the most from BM25, and which gains nothing. Then run:

Terminal window
search compare current bm25 --by source
The output, and what it shows
source queries current bm25 difference low high worse
code 20 0.000 0.000 0.000 0.000 0.000 0
planted 3 0.451 0.451 0.000 0.000 0.000 0
translated 59 0.012 0.177 0.165 0.103 0.236 0
typo 40 0.000 0.357 0.357 0.267 0.448 0
wands 119 0.378 0.650 0.272 0.215 0.334 9

Misspellings gain the most (+0.357): today’s search needs every word, so one wrong word empties the page. Product codes gain nothing: both searches score 0. BM25 does not know that a code is special. All 9 queries where BM25 is worse are English WANDS queries. The three planted broad queries do not change; with only 3 queries, that slice tells you almost nothing.

Common mistake

A winner that was only noise

Symptom: Priya tries 96 settings of field weights on the 121 validation queries and keeps the best one. It is ahead by only 0.006. On the test queries, it loses to the weights that she picked by hand. You meet this case, with all its numbers, in BM25.

Cause: a difference of 0.006 on 121 queries is much smaller than the noise. Among 96 settings, one is always a little ahead by luck. Choosing the best of many settings on the same queries picks up that luck.

Fix: compare the top settings with a paired interval before you trust a winner. Treat a difference inside the noise as a tie, and prefer the simpler setting. Check the final choice once on queries you did not use to choose it.

Consequence in search work: every search team tries many changes. Without intervals, it will ship changes that only looked good. With a fixed test set that it checks again and again, it will also slowly tune the search to that test set. The held-out queries protect against this.

Check your understanding

Larkfield's head is 37.1% of the searches but only 7% of the test queries. What does the plain mean over test queries measure, and how would you get a number for a typical search?

Show the answer

The plain mean gives each test query the same weight. So it measures the search for a typical different query, and most of those are tail queries. For a typical search, weight each band’s score by its share of searches. For BM25: 0.371 × 0.638 + 0.227 × 0.623 + 0.402 × 0.368 = 0.526, against a plain mean of 0.429. Say which number you report.

Check your understanding

Three queries have NDCG@10 of 0.9, 0.3 and 0.6. One bootstrap draw picks the first, the first again, and the third. What is the mean of this draw, and why can a query appear twice?

Show the answer

(0.9 + 0.9 + 0.6) ÷ 3 = 0.8. The bootstrap draws with replacement: after a query is drawn, it goes back, so it can be drawn again. That is how each draw makes a new test set of the same size from the same queries.

Check your understanding

A change scores +0.02 NDCG@10 on 30 test queries, with a paired 95% interval of −0.03 to +0.07. Omar asks: “Is it better?” What do you answer, and what would you do next?

Show the answer

The test set cannot say. The interval contains 0, so the change could be better, the same or a little worse. With 30 queries, even a real small change is often not clear. On Larkfield’s data, a +0.037 change was clear in only about a quarter of the draws of 25 queries. Next: run it on more test queries (about 200 for a change this small), and look at the slices that matter.

Recap

  • Check the test set against the log. Larkfield’s head is 37.1% of searches but 7% of test queries; a plain mean speaks mostly for the tail. BM25: 0.429 plain, 0.526 weighted by searches.
  • Add the queries that fail today, even when they are rare: French, misspellings, product codes. Keep held-out queries away from your work and use them once.
  • A confidence interval says how sure a number is. The bootstrap draws the queries again with replacement 2,000 times and keeps the middle 95% of the means.
  • Compare two searches paired, query by query. BM25 minus today: +0.234 (+0.197 to +0.270) on Larkfield’s test queries.
  • Small changes need many queries: a +0.037 change was clear in 26% of draws of 25 queries and in all draws of 200.

The test set has one more weak point: the products that nobody judged. Next, you let a language model write the missing labels, and you check it against people.

Next: Judgments from a language model