Skip to content
Français

Latency and cost budgets

  • Lesson 27
  • 35 min
  • Module 8: Run search in production

Where we are

Our search page is complete and measured offline. Before it serves people, it must be fast and affordable. In this lesson, we measure the time of each stage, cache popular queries, and compute the cost per thousand queries.

What you will be able to do

Make search fast enough and cheap enough before it serves people. You measure the time of each stage on your own computer and turn milliseconds into a cost per 1,000 queries. You also see how much a cache of popular queries saves.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook times BM25 and the dense search on 50 test queries. It turns milliseconds into money with the course’s example price, and measures what a cache of the 100 most common queries would answer. Its setup cells download the practice data, so it needs no local setup and no model download. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project larkfield-search at the end of Module 7, with its virtual environment active, and search check passing.
  • The two models of Modules 5 and 6 (about 800 MB in all). If they are not on your computer yet, search bench downloads them the first time.

A restaurant promises a meal in 15 minutes. The chef knows each step: 1 minute to take the order, 8 minutes on the grill, 2 minutes to plate it. If a new dish needs 30 minutes in the oven, it cannot be on the lunch menu, however good it is.

The kitchen also prepares some things before lunch. The soup of the day is ready in a pot. Half the customers order it, and they get it at once.

The chef watches two things. Time: does each meal arrive inside the promise? And money: how many cooks, for how many meals? Search works in the same way.

The problem: a better design that is too slow

Section titled “The problem: a better design that is too slow”

In Module 6, the cross-encoder re-ranked the top 50 candidates. On Larkfield’s test queries, NDCG@10 went from 0.621 to 0.635. That was not a clear gain, but suppose it were. Is it ready for people?

Remember Amira’s latency budget from A shop where search fails: 200 ms, the longest time one search may take, from the query to the ranked list. It is an assumption of this course, a common target for a search page that feels instant.

The cross-encoder on 50 candidates took 1,121 ms per search (the median) on the course team’s laptop with 1 thread. A thread is one line of work on one CPU core. That is more than five times the budget. People would wait more than a second after every search. So the best design offline can be the wrong design for people.

A latency budget turns “fast enough” into a number that you can measure.

The idea: measure each stage, then spend the budget

Section titled “The idea: measure each stage, then spend the budget”

A search is a chain of stages. The whole time is roughly the sum of the stage times. So you measure each stage alone, and you see where the time goes.

Search times vary. A short query is fast; a long one is slower. So one number is not enough. We use two percentiles:

  • p50 (the median): half of the searches are faster than this.
  • p95: 95 of every 100 searches are faster than this. The other 5 are the slow ones.

Here is a tiny constructed example. Ten searches took these times, sorted: 4, 5, 5, 6, 6, 7, 7, 8, 9 and 40 ms.

Measure How to read it Value
p50 the middle of the sorted list: halfway between the 5th (6) and the 6th (7) 6.5 ms
mean the sum divided by 10 9.7 ms
p95 near the top of the list about 40 ms

One slow search pulls the mean up to 9.7 ms. Without it, the mean would be 6.3 ms. The p50 hardly moves. The p95 shows the slow search. A budget is for the slow searches too, so check p95 as well as p50.

The course team measured each stage of one search on 40 English test queries, on their laptop (Apple M4), with 1 thread:

One search against a 200 ms budget. First a cache check: the 100 most common query texts answer 51% of searches with a stored page. Otherwise: spelling 0.01 ms, then lexical BM25 1.1 ms and query encoding 5.5 ms followed by dense search 2.1 ms, fusion 0.05 ms, then the LambdaMART step, features and model, 35 ms measured with its own candidate search. The whole design is measured at 36 ms (p50). A dotted path: a cross-encoder on 50 candidates instead takes 977 ms, over the budget.
The stages of one search, p50 on 1 thread, measured on the course team's laptop. The cache answers half of the searches before any stage runs.
Text description of the diagram

A flow from left to right. “A search, budget 200 ms” goes to a question: “In the cache?” (the 100 most common texts answer 51% of searches). Yes: “Stored page, no search needed”. No: “Spelling, 0.01 ms”, then two branches: “Lexical BM25, 1.1 ms” and “Encode the query, 5.5 ms” followed by “Dense search, 2.1 ms”. Both go to “Fusion, 0.05 ms”, then “LambdaMART step, features and model, 35 ms, measured with its own candidate search”, then “The page: whole design measured, 36 ms (p50)”. A dotted arrow from fusion: “instead: cross-encoder on 50, 977 ms”, to a red box “Over the budget”.

Measured on the course team's laptop, 40 English test queries, 1 thread. The LambdaMART step includes its own candidate search, so the bars do not add up exactly; the whole design was measured at 36.1 ms.
Text description of the chart

Six bars. Spelling 0.01 ms, lexical BM25 1.1 ms, encoding the query with the bi-encoder 5.5 ms, the dense search 2.1 ms, fusion 0.05 ms, and the LambdaMART step 35.3 ms.

Read the chart from left to right:

  • Spelling and fusion cost almost nothing: hundredths of a millisecond.
  • BM25 reads an index, not every product, so it takes about 1 ms.
  • Encoding the query is the slowest part of hybrid search: the bi-encoder is a small neural network, and it runs once per search. The dense search itself multiplies one vector by a matrix of 40,066 rows in 2.1 ms.
  • The LambdaMART step computes 19 features for 100 candidates and runs the model. It takes most of the time, and it still fits easily.

The cross-encoder is not on the chart, because it would not fit. In this measurement of the stages, it took 977 ms on 50 candidates with 1 thread, and 710 ms with 4 threads. It is the only stage that breaks the budget. The whole search with it took 1,121 ms in the design table below. Each measurement was a separate run, so the numbers differ a little.

More threads do not save the cross-encoder

Section titled “More threads do not save the cross-encoder”

A model can split its work over several threads. Cross-encoder re-ranking showed that its time grows with the number of re-ranked candidates, and that only 10 candidates fit the budget. Here is the same explorer. This time, compare only the 1-thread and 4-thread times.

Explore

The cross-encoder against the 200 ms budget

NDCG@10
0.635
Time per search (p50)
981 ms
Budget
200 ms

Re-ranking 50 candidates takes 981 ms per search on 1 thread: 781 ms over the budget. NDCG@10 is 0.635 (0.589 to 0.679), +0.014 against no re-ranking. In its top 50, stage one has 44% of the exact products: the cross-encoder can only order what stage one found. For comparison, LambdaMART reaches 0.668 in 36 ms.

  1. 40,066products in the catalogue
  2. Stage one: hybrid search100 candidates9.2 ms
  3. Stage two: cross-encoderre-orders the top 50972 ms
  4. The pagetop 10 shown
Two-stage ranking on Larkfield’s 243 test queries (1 thread). Rows marked “over” miss the budget.
Re-rankedExact products in stage one’s top nNDCG@10 (95% interval)Time per search, p50, 1 thread budget 200 ms
0 (stage one only)none0.621 (0.576–0.663)9.2 ms · fits
1027%0.631 (0.587–0.672)194 ms · fits
2033%0.641 (0.595–0.684)400 ms · over
5044%0.635 (0.589–0.679)981 ms · over
10054%0.631 (0.584–0.674)1,904 ms · over
LambdaMART insteadnone0.66836 ms · fits
Hybrid search finds 100 candidates; the cross-encoder re-orders the top n. Live time per search (p50) on the course team's laptop, on 1 or 4 threads, and NDCG@10 on Larkfield's test queries (mean of 241 of the 243).
Text description of the explorer

The explorer starts at 50 re-ranked candidates on 1 thread. Buttons choose 0, 10, 20, 50 or 100 candidates and 1 or 4 threads. The table gives the time per search on 1 and 4 threads: re-ranked 0, 9.2 ms and 10.3 ms; 10, 194 ms and 155 ms; 20, 400 ms and 295 ms; 50, 981 ms and 732 ms; 100, 1,904 ms and 1,396 ms. The budget line is at 200 ms. LambdaMART takes 36 ms on 1 thread and 34 ms on 4 threads, with NDCG@10 0.668.

What the explorer shows

No. Four threads cut the time from 981 ms to 732 ms: a quarter less, not four times less. And four threads keep four CPU cores busy for that one search. So the server can serve fewer searches at the same time, as the cost section below shows.

A cloud server is paid by the hour, busy or not. This course uses an example price: US$0.17 an hour for a server with 4 virtual CPUs (vCPUs) and 16 GB. It is an assumption, not a quote. Prices change: check your provider’s page.

Section titled “A worked example: the cost of one LambdaMART search”

One search keeps one vCPU busy for its time. So the cost of a search is its CPU time multiplied by the price of one vCPU-second.

Step Calculation Value
Price of one vCPU for one hour 0.17 ÷ 4 US$0.0425
Price of one vCPU-second 0.0425 ÷ 3,600 US$0.0000118
CPU time of one LambdaMART search 36 ms × 1 thread 0.036 vCPU-seconds
CPU time of 1,000 searches 0.036 × 1,000 36 vCPU-seconds
Cost of 1,000 searches 36 × 0.0000118 about US$0.0004

So 1,000 LambdaMART searches cost less than a tenth of a cent. A thread count multiplies the CPU time: 750 ms on 4 threads is 3 vCPU-seconds for one search.

The course team measured each design live on 40 English test queries. The cost uses the price above, and p50 times. The language-model rewrite of Module 4 is a call to a hosted model, so its cost is the provider’s price per call.

Design p50 / p95 ms US$ per 1,000 queries Fits 200 ms?
Larkfield today (current) 1.7 / 16.2 0.00002 yes
BM25 with query understanding (bm25-norm+spell+syn) 1.1 / 2.0 0.00001 yes
Hybrid search (hybrid-rrf) 9.2 / 15.1 0.0001 yes
LambdaMART (ltr) 36.1 / 54.1 0.0004 yes
Cross-encoder on 50 (rerank-ce), 1 thread 1,121 / 1,445 0.013 no
Cross-encoder on 50, 4 threads 753 / 1,119 0.035 no
Language-model rewrite (network call) 1,195 / 1,581 0.04 (the provider’s price) no

Two things stand out.

  1. Cost is not the problem here. Time is. Even the most expensive design costs a few cents per 1,000 searches. But three designs break the budget: at p95, by a factor of five or more.
  2. The best offline design is also fast. LambdaMART has the highest NDCG@10 of Module 6 (0.668) and uses a fifth of the budget.

The real cost: servers for the busiest hour

Section titled “The real cost: servers for the busiest hour”

Cost per query hides one thing. You pay for servers by the hour, and you need enough of them for the busiest moment. One server has 4 vCPUs. If each LambdaMART search keeps one vCPU busy for 36 ms, one vCPU can serve about 1,000 ÷ 36 ≈ 27 searches a second. The server, with 4 vCPUs, can serve about 110 searches a second when it is fully busy.

The cross-encoder on 4 threads keeps all 4 vCPUs busy for about 750 ms. The same server serves about 1.3 searches a second. For the same traffic, it needs about 80 times more servers. Real servers also need spare room, so plan for less than full use. The Scaling APIs and AI workloads course measures capacity in detail.

A cache keeps the results of a recent query, so the next person with the same query gets them without a search. Module 1 showed that a few query texts make a large part of Larkfield’s log. That is the head of the log. A cache works well on the head.

Measured on the synthetic query log (20,000 searches): the share of searches whose text is among the n most common texts. A real log has a head and a long tail too, but its numbers differ.
Text description of the chart

Five bars. A cache of the 10 most common query texts answers 20.4% of the searches; 40 texts, 37.1%; 100 texts, 51.1%; 500 texts, 84.5%; 1,000 texts, 95.4%.

The 100 most common texts answer half of the searches. Larkfield’s log is synthetic, and our code made short, broad queries common. In a real shop, check this curve on your own log before you build a cache.

Cache the ranked list of product IDs, not the whole page. Price, stock and the sale flag change during the day. Read them fresh when you build the page.

Every cached answer gets old. Cache invalidation means deleting old answers when something changes. Delete the cache when you switch to a new index or a new ranking model, as Keep the index fresh shows. Also give each answer a short life, for example an hour, so new products appear. The Scaling APIs and AI workloads course covers cache keys and invalidation in depth.

A cache does not make the slow tail fast. Half of Larkfield’s searches still miss the cache. They need a design that fits the budget on its own.

The indexes must fit in memory, or each search waits for the disk. The course team measured:

  • The lexical indexes (SQLite FTS5 and BM25 for every field and analyzer): 102.5 MB.
  • The product embeddings file (40,066 × 384 numbers, float16): 31.3 MB on disk (30.8 MB for the numbers alone in memory). Float32 would be twice the size. On 200 queries, float16 kept 99.3% of the float32 top 10.

Both fit easily in a 16 GB server. The dense search reads every row of the embedding matrix, so its time grows with the catalogue. At 40,066 products it takes 2.1 ms. A catalogue ten times larger would need roughly ten times longer: still fast. For millions of products, teams use an approximate index such as HNSW (a graph of near neighbours, in libraries like FAISS). It reads only a small part of the vectors and may miss a few true neighbours. Larkfield does not need one at this size. The same ideas in a search server shows one.

Guided practice: measure your own computer

Section titled “Guided practice: measure your own computer”

This module adds three Python files and one test file to the project. You copy all of them now, with the README, from the snapshot end-of-m08. This lesson uses bench.py. The next lesson uses interleave.py.

  1. In larkfield-search, with the virtual environment active, copy the files:

    Terminal window
    S=../nextia-learning-labs/C38/snapshots/end-of-m08
    for f in larkfield_search/bench.py larkfield_search/interleave.py larkfield_search/commands/online.py tests/test_online.py README.md; do
    cp "$S/$f" "$f"
    done

    No new package is needed.

  2. Run the tests:

    Terminal window
    python -m pytest -q
    ....................................... [100%]
    39 passed in 3.91s

    Three tests are new. One checks the cost formula: 100 ms on one thread at US$0.17 an hour for 4 vCPUs.

  3. Before you run the next step, write down your prediction: on your computer, how long will the LambdaMART step take?

  4. Measure every stage on 30 test queries, with the models on 4 threads (the default):

    Terminal window
    search bench --queries 30

    The first run loads both models, so it takes about 30 seconds. PyTorch may print one warning line about “Redirects” first. You can ignore it. On the course team’s laptop, the output was:

    30 test queries, models on 4 thread(s); milliseconds per query:
    stage p50 p95
    understand 0.0 1.9
    lexical 0.6 1.0
    encode 5.8 6.8
    dense search 1.2 1.3
    fusion 0.1 0.1
    rerank 50 726.5 868.7
    ltr features and model 32.7 59.6
    design p50 ms p95 ms US$ per 1,000
    hybrid 7.7 20.5 0.00036
    hybrid + rerank 50 734.2 876.4 0.03467
    ltr (hybrid inside) 38.5 65.4 0.00182
    index (FTS5 and BM25): 81.2 MB
    product embeddings: 31.3 MB

    Your numbers will differ: they depend on your computer and on what else runs on it.

  5. Run it again on 1 thread, and compare the cost column:

    Terminal window
    search bench --queries 30 --threads 1

    On the course team’s laptop, hybrid + rerank 50 took 1,036.8 ms (p50) and cost US$0.01224 per 1,000. The cost is lower on 1 thread, because it uses one vCPU instead of four. The time is longer.

Look at the last two lines. Your index is smaller than 102.5 MB (here 81.2 MB; yours can differ). The project builds an index for a field only when a run first needs it. You have not yet run every design in this folder.

Larkfield’s real server may be slower than the course team’s laptop. Suppose every stage takes three times as long there. Use the stage times of this lesson (p50, 1 thread) and answer without running anything:

  1. Does LambdaMART still fit the 200 ms budget? Check its p95 too.
  2. Omar wants the language-model rewrite only for queries that are not in the cache. How many searches of the synthetic log would still wait for it, with a cache of the 100 most common texts?
Answers
  1. Yes. 36.1 ms × 3 ≈ 108 ms at p50, and 54.1 ms × 3 ≈ 162 ms at p95. Both fit inside 200 ms, but the p95 leaves little room for the network and the page.
  2. About half: 48.9%. The 100 most common texts answer 51.1% of the searches. The other 48.9% would wait about 1.2 seconds (p50) for the rewrite. A better plan: rewrite the head queries once, store the rewrites, and keep the tail on the fast path.

Common mistake

Measuring on a warm laptop and promising it to production

Symptom: Tomás runs search bench --queries 10 once, sees the cross-encoder on 10 candidates at 155 ms on 4 threads, and tells Omar: “It fits the budget.”

Cause: one short run on a quiet laptop measures the best case. The p50 hides the slow searches. In the course team’s own measurement, 10 candidates on 4 threads had a p50 of 155 ms but a p95 of 196 ms. On 1 thread, the p95 was 525 ms. A busy server also runs many searches at once, so they share the CPU and each one gets slower.

Fix: measure p95 as well as p50, on at least 30 queries, on hardware like the server. Leave room for the network and the page: a stage that alone takes 155 of 200 ms does not fit.

Consequence in search work: a budget is a promise about the slow searches, under real traffic. Measure it there, or keep a large margin.

Measure the time of each stage before you choose a design, not after. A design that wins offline but breaks the budget is not a candidate. Measure again on the server, under load, before release.

The cost of this work is small. search bench takes under a minute. The notebook takes about 20 minutes. What it saves is large: Larkfield did not build a cross-encoder service that would have needed many more servers and still been too slow.

Use a cache when the head of your log is large. Do not use it as a reason to keep a slow design: the tail still meets the slow path.

Check your understanding

Ten searches took 3, 3, 4, 4, 5, 5, 5, 6, 6 and 90 ms. What is the p50, and which number should Amira check against her budget?

Show the answer

The p50 is 5 ms (between the 5th and 6th sorted values, 5 and 5). Amira should also check the p95, which is near 90 ms here: the budget is a promise for the slow searches too.

Check your understanding

With US$0.17 an hour for 4 vCPUs, what do 1,000 searches of 9 ms on 1 thread cost?

Show the answer

9 ms × 1,000 = 9 vCPU-seconds. One vCPU-second costs 0.17 ÷ 4 ÷ 3,600 ≈ US$0.0000118. So 9 × 0.0000118 ≈ US$0.0001. That is the hybrid search’s cost in this lesson’s table.

Check your understanding

A cache of the 100 most common texts answers 51.1% of Larkfield's searches. Why must the uncached design still fit the budget?

Show the answer

Because the other 48.9% of searches miss the cache and run the full search. Most of them are tail queries, typed by few people each. A cache makes the average faster, but it does not help a person who types a rare query.

Recap

  • A latency budget is the longest time one search may take: 200 ms for Larkfield (an assumption). Check p95 as well as p50.
  • On the course team’s laptop, every stage except the cross-encoder takes a few milliseconds. LambdaMART takes 36 ms (p50); the cross-encoder on 50 candidates takes 1,121 ms on 1 thread and 753 ms on 4.
  • Cost per 1,000 queries = CPU-seconds × the price of a vCPU-second. At US$0.17 an hour for 4 vCPUs (an assumption), every design costs a few cents or less. Time is the limit, and servers for the busiest hour are the real cost.
  • A cache of the 100 most common texts answers 51.1% of the synthetic log. Cache product IDs, read price and stock fresh, and clear the cache when the index or the model changes.

You now have a design that is fast and cheap. But offline NDCG uses judgments, not people. Next, you test a change with people’s clicks: interleaving and A/B tests.

Next: Test changes online