Skip to content
Français

Test changes online

  • Lesson 28
  • 40 min
  • Module 8: Run search in production

Where we are

Our design is fast, cheap and better offline. Offline tests use judgments, not real people. In this lesson, we test changes online with A/B tests and interleaving, and see why offline and online results can disagree.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find out if people prefer a new ranking, not only the judges. You learn two online tests, A/B testing and interleaving, and run both on a simulated click model. You also see why looking at a test too often finds false winners.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds BM25 and a hybrid design, writes team-draft interleaving and a sign test yourself, and lets a click model play the customers. Then it runs a test of a design against itself, to show false winners. Its setup cells download the practice data, so it needs no local setup and no model download. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project with this module’s files, from Latency and cost budgets.
  • search ltr train run once in this project folder (Module 6).

A restaurant has a new soup recipe. The chef’s friends tasted it and liked it more. Will the customers like it more?

There are two ways to find out. The first: on Monday, half the tables get the old soup and half get the new one. You count the empty bowls in each half. The second: every table gets two small cups, one of each, and says which one it prefers. The second way needs far fewer tables. Each table compares both soups, so one table’s hunger or taste does not matter.

The friends’ test came first, and it was cheap. But friends are not customers.

Every result in this course so far is offline. Offline means: on a fixed test set, with judgments written in advance, and with no customer involved. Offline tests are fast and repeatable. But they can disagree with what people do. There are four common reasons:

  1. Unjudged products. NDCG counts an unjudged product as irrelevant. Dense search showed many unjudged products in its top 10 (judged@10 0.63 on the English and French test queries). Some of them may be good.
  2. The page. People see pictures, prices, stock and reviews. Judges saw only the product and the query.
  3. Position. People look at the top far more than the bottom (Learn from clicks). NDCG has its own discount, which may not match.
  4. The mix of queries. The test set gives each query the same weight. The live shop sees “sofa” hundreds of times and a rare query once.

Here is one disagreement in Larkfield’s data. Offline, hybrid search (hybrid-rrf) and BM25 with query understanding (bm25-norm+spell+syn) are about equal. The difference in NDCG@10 is +0.001, with an interval from −0.022 to +0.024. The course’s click model looks at the same lists in a different way. It expects a click on 60.3% of hybrid searches and on 59.2% of the BM25 searches. Partial products earn clicks too, and NDCG gives them only a small gain.

So you need a test with people. But people make results noisy, and a test with people costs time and risk.

An online test shows a change to real people and measures what they do. Search teams use three steps. Each one proves something different.

Three tests in a row. An offline test on judged test queries with NDCG@10 and an interval, in minutes and with no customers, proves that B ranks judged products better. If B is clearly better, interleaving mixes A and B on one page and asks which side gets more clicks. It proves that people prefer B's results. If people prefer B, an A/B test shows A to half the people and B to the other half. It measures clicks, add-to-cart and reformulation with a fixed plan, and proves the effect on the shop's measures. If the guardrails hold, release B and keep A ready for rollback.
Offline test, then interleaving, then an A/B test. Each step is slower and costs more, and each proves more.
Text description of the diagram

Four boxes from left to right with arrows. “Offline test: judged test queries, NDCG@10 with an interval, minutes, no customers” → (clearly better?) → “Interleaving: one page mixes A and B; which side gets more clicks? a few thousand searches” → (people prefer B?) → “A/B test: half the people see A, half see B; click, add-to-cart, reformulation; more searches, a fixed plan” → (guardrails hold?) → “Release B, keep A ready for rollback”. Under each test, what it proves: offline, “B ranks judged products better” (not yet what people do); interleaving, “people prefer B’s results” (not yet the effect on sales); A/B test, “the effect on the measures the shop cares about” (only for these weeks and these people).

Online, you cannot compute NDCG: nobody judges the results. You count what people do. Each of these is a share of searches:

  • Click-through rate: searches with at least one click.
  • Add-to-cart rate: searches where the person put a product in the cart.
  • Reformulation rate: searches followed by a new, changed search. A high rate means people did not find what they wanted.
  • Zero-result rate: searches with an empty page.

In Larkfield’s synthetic log, 45.7% of searches have a click, 18.3% an add to cart, and 20.7% are reformulated. Our code made these numbers with the click model. They show the scale of the measures, not how real people behave.

In an A/B test, the shop splits people at random into two groups. Group A sees the current ranking; group B sees the new one. After enough searches, you compare a measure between the groups, for example the click-through rate.

The split must be random, so the two groups are alike in everything except the ranking. Then a difference in clicks comes from the ranking. A statistical test, here a two-proportion test, says if the gap is larger than luck would make.

An A/B test measures what the shop cares about: add-to-cart, revenue, support contacts. Its weakness is noise. People differ a lot. One group may get more people who know exactly what they want. Small changes need many searches to show.

Interleaving puts both rankings on one page. Every person sees a mix of A’s and B’s results. Each result remembers which ranking brought it. If people click more of B’s results, B wins that search.

The most common method is team-draft interleaving. A and B are two captains who pick players in turns. In each round, a coin decides who picks first. Each captain adds its best product that is not on the page yet.

A ranks four products P1, P2, P3, P4. B ranks P3, P1, P5, P6. The notebook ran team draft on them with one seed:

Round Coin: first pick Pick Position Team
1 B B’s best: P3 1 B
1 – A’s best not shown: P1 2 A
2 B B’s best not shown: P5 (P3 and P1 are taken) 3 B
2 – A’s best not shown: P2 4 A

The person sees P3, P1, P5, P2. Suppose they click P3 and P5. B gets 2 clicks and A gets 0, so B wins this search. If they click P3 and P1, it is a tie, and the search counts for nobody.

After many searches, you count B’s wins and A’s wins. A sign test asks: if A and B were equally good, how likely is a gap this large? If that chance (the p-value) is below 0.05, we call the result clear.

The same person judges both rankings at the same moment, like the table with two cups of soup. Differences between people cancel out. So interleaving sees a real difference with fewer searches. But it says only which ranking people prefer. It does not measure add-to-cart or revenue. So teams use interleaving to choose, and an A/B test to confirm.

Larkfield has no live customers in this course. So the click model of Learn from clicks plays the customers, with the same assumed chances: examine(k) = 1/k, and a click on an exact, partial or irrelevant product with the chance 0.65, 0.25 or 0.05. A simulation shows how the tests behave when the truth is known. It does not show how real people behave.

search interleave runs both tests at the same time, on 240 test queries (the 3 planted broad queries are left out). Each search picks a random test query. The test is checked every 100 searches, and it stops when the result is clear. Each seed is one complete run with different random choices.

Terminal window
search interleave hybrid-rrf ltr --seeds 5
A = hybrid-rrf, B = ltr; 240 test queries; clicks from the click model (synthetic)
seed 0: interleaving clear after 800 searches (winner B); A/B test clear after 600 searches (winner B)
seed 1: interleaving clear after 100 searches (winner B); A/B test clear after 500 searches (winner B)
seed 2: interleaving clear after 500 searches (winner B); A/B test clear after 100 searches (winner B)
seed 3: interleaving clear after 500 searches (winner B); A/B test clear after 13100 searches (winner B)
seed 4: interleaving clear after 1100 searches (winner B); A/B test clear after 500 searches (winner B)
median searches: interleaving 500.0, A/B 500.0

LambdaMART wins in every seed, as it did offline (+0.047 NDCG@10). The median is 500 searches for both tests. But look at seed 3: the A/B test needed 13,100 searches. And seed 1 says “clear after 100”. Can 100 searches really be enough?

To answer, run a test where you know there is no difference: a design against itself. This is called an A/A test. Any winner it reports is a false alarm.

The course team ran the A/A test with hybrid search against itself, on 100 seeds of 20,000 searches each. They counted the seeds that reported a winner. Three ways to look at the test:

  • interleaving
  • A/B test
Simulated with the click model: hybrid search against itself, 100 seeds of 20,000 searches. A test checked every 100 searches finds a false winner in about half of the seeds. Looked at once (here at 5,000 searches), it is near the 5% that p < 0.05 allows.
Text description of the chart

Six bars in two groups. Interleaving: checked every 100 searches, 47% of the seeds report a false winner; every 1,000 searches, 32%; looked at once at 5,000 searches, 9%. A/B test: every 100 searches, 48%; every 1,000, 23%; once, 3%.

Checked every 100 searches, about half of the A/A tests found a “winner”. The rule “p below 0.05” allows a false alarm 5 times in 100, for one look. Each extra look is a new chance for luck to cross the line. Two hundred looks give luck many chances. This mistake is called peeking.

Now look only once, at a number of searches planned in advance. The false winners fall to between 3% and 9%. With 100 seeds, that is 5% with a few points of noise.

So the “500 searches” above came from peeking. How many searches does a fair test need to see LambdaMART’s real gain? The course team ran hybrid search against LambdaMART on the same 100 seeds, and looked only once, at a fixed number of searches:

  • interleaving
  • A/B test
Simulated with the click model: hybrid-rrf against ltr, 100 seeds. The share of tests that are clear (p < 0.05) when you look only once, after n searches.
Text description of the chart

Two lines. Interleaving finds the gain in 15% of tests after 500 searches, 33% after 1,000, 58% after 2,000, 91% after 5,000, 99% after 10,000 and 100% after 20,000. The A/B test finds it in 16%, 24%, 38%, 72%, 94% and 100%.

Read the chart at 5,000 searches. Interleaving finds the gain in 91% of the tests; the A/B test in 72%. That share is called the power of a test. At 500 searches, both find it only about 15% of the time. So a fair test of LambdaMART needs a few thousand searches, decided before the start. Interleaving reaches high power sooner than the A/B test.

For Larkfield’s traffic, that takes weeks. The synthetic log has 20,000 searches in eight weeks: about 2,500 a week. A test usually gets only part of the traffic.

A good online test has a short written plan:

  1. The question and the measure. “Do people prefer LambdaMART to hybrid search?” Interleaving wins, then the add-to-cart rate in an A/B test.
  2. The size. 5,000 searches for interleaving. Look once.
  3. Guardrails. Measures that must not get worse: the zero-result rate, the p95 latency (budget 200 ms), the share of out-of-stock first results.
  4. The stopping rule. Stop early only if a guardrail breaks. Never stop early because B “looks good”.
  5. Rollback. How to return to A in minutes.

Planning an experiment well is a subject of its own (a planned course on experiments). The MLOps and LLMOps course shows how to roll out a model change in steps and roll it back.

Guided practice: BM25 against the hybrid design

Section titled “Guided practice: BM25 against the hybrid design”
  1. In larkfield-search, with the virtual environment active, check that LambdaMART is trained. If you are not sure, run search ltr train again (a few seconds).

  2. Before you run the test, predict: will BM25 against hybrid search need more or fewer searches than hybrid against LambdaMART? Offline, the gap is +0.192 NDCG@10 (0.429 against 0.621).

  3. Run the test:

    Terminal window
    search interleave bm25 hybrid-rrf --seeds 5
    A = bm25, B = hybrid-rrf; 240 test queries; clicks from the click model (synthetic)
    seed 0: interleaving clear after 100 searches (winner B); A/B test clear after 100 searches (winner B)
    seed 1: interleaving clear after 200 searches (winner B); A/B test clear after 600 searches (winner B)
    seed 2: interleaving clear after 300 searches (winner B); A/B test clear after 800 searches (winner B)
    seed 3: interleaving clear after 100 searches (winner B); A/B test clear after 300 searches (winner B)
    seed 4: interleaving clear after 300 searches (winner B); A/B test clear after 100 searches (winner B)
    median searches: interleaving 200.0, A/B 300.0

    It takes about 2 seconds. The seeds are fixed, so your output is the same.

  4. Compare with hybrid against LambdaMART (median 500 and 500). A large difference shows sooner. These medians still come from peeking, so treat them as the earliest moment, not as a plan.

  5. Run the pair that is equal offline:

    Terminal window
    search interleave bm25-norm+spell+syn hybrid-rrf --seeds 5

    Interleaving is clear in 3 of 5 seeds, after 11,300, 2,100 and 900 searches. The A/B test is clear in 3 of 5, after 7,300, 4,400 and 3,000 searches. The other seeds print more than 20000. The click model gives hybrid search about 1 point more clicks, so a small real difference exists in clicks. It is too small to find reliably, and peeking makes the early “winners” untrustworthy.

Run hybrid search against itself on 20 seeds:

Terminal window
search interleave hybrid-rrf hybrid-rrf --seeds 20

Before you look at the output, write your prediction: in how many of the 20 seeds will a test report a winner? Then count the seeds that do not say more than 20000, for interleaving and for the A/B test.

Answer

The seeds are fixed, so you get the same output as the course team. Interleaving reported a winner in 7 of 20 seeds, and the A/B test in 11 of 20. Some “won” after only 100 or 300 searches. Some “winners” are A and others are B, as luck makes them. Each one is false, because both sides show the same ranking. If you look every 100 searches for 20,000 searches, you take 200 looks, and luck crosses the line often.

The notebook runs 20 seeds of 5,000 searches. Checked every 100 searches, 4 of 20 tests found a false winner. Looked at once, 0 of 20 did.

Common mistake

Stopping the test when B first looks better

Symptom: Tomás starts an A/B test of LambdaMART on Monday. On Tuesday, after 500 searches, the dashboard shows p = 0.03 for B. He stops the test and tells Omar: “LambdaMART won.”

Cause: he looked at the test many times and stopped at the first good look. In the simulated A/A test, this way of looking finds a winner in about half of the tests, where no difference exists. And seed 3 above shows the opposite risk: the same real gain can take 13,100 searches to show.

Fix: decide the number of searches before the test, from a power estimate like the chart above, and look once at the end. Stop early only for a broken guardrail. If you must look often, use a method made for it (a sequential test), not the plain test checked again and again.

Consequence in search work: a peeked test is not evidence. Here the conclusion happened to be right, because LambdaMART is really better. With a small or no real difference, the same habit ships changes that do nothing, or harm.

Test online when an offline result is clear and the change matters to people. Use interleaving first: it needs fewer searches to choose between two rankings. Then confirm with an A/B test on the measures the shop cares about: add-to-cart and reformulation.

Do not test online:

  • when the offline test already shows a clear loss. Do not show a worse ranking to customers;
  • when the change breaks the latency budget. Fix that first;
  • when traffic is too small to reach the planned size in a reasonable time. Then rely on offline tests and careful release steps.

An online test costs three things. Time: weeks of traffic for small gains. Engineering: logging which ranking showed which product. Risk: some people see the worse ranking. The click model of this lesson is a practice tool. Use it to plan the size of a test, not as a replacement for people.

Check your understanding

Team draft: A ranks P1, P2, P3; B ranks P2, P4, P1. The coin says A picks first in round 1, and B picks first in round 2. Which four products are on the page, and from which team?

Show the answer

Round 1: A picks P1 (position 1, team A); B picks its best not shown, P2 (position 2, team B). Round 2: B picks first, its best not shown: P4 (position 3, team B); A picks its best not shown: P3 (position 4, team A). The page is P1 (A), P2 (B), P4 (B), P3 (A).

Check your understanding

An A/B test is checked every 100 searches for 20,000 searches. Why can it find a winner where none exists, although each check uses p < 0.05?

Show the answer

The 5% false-alarm rate is for one look. Checking 200 times gives luck 200 chances to cross the line. In the simulated A/A test, about half of the tests found a false winner this way. Decide the size before the test and look once.

Check your understanding

Interleaving found LambdaMART better in 91% of fair tests of 5,000 searches; the A/B test in 72%. Why is interleaving more sensitive, and what does it not tell you?

Show the answer

Each person sees both rankings on one page and compares them at the same moment, so differences between people cancel out. But interleaving only says which ranking people prefer. It does not measure the add-to-cart rate or revenue of each ranking alone: that needs an A/B test.

Recap

  • Offline tests use judgments; people can disagree with them (unjudged products, the page, position, the mix of queries). Test in steps: offline, then interleaving, then an A/B test.
  • An A/B test shows each ranking to a random half of people and compares click-through, add-to-cart and reformulation. Interleaving mixes both rankings on one page and counts which one gets more clicks.
  • With the course’s click model (simulated), tests of 5,000 searches looked at once find LambdaMART’s gain over hybrid search: 91% of interleaving tests and 72% of A/B tests.
  • Peeking finds false winners: checked every 100 searches, about half of the A/A tests “won”. Write a plan first: the measure, the size, guardrails, the stopping rule and rollback.

You now know how to show that a change is better for people. Next, the catalogue itself changes every day. You keep the index fresh, and switch versions without stopping search.

Next: Keep the index fresh