Latency and cost budgets
- Lesson 27
- 35 min
- Module 8: Run search in production
Where we are
Our search page is complete and measured offline. Before it serves people, it must be fast and affordable. In this lesson, we measure the time of each stage, cache popular queries, and compute the cost per thousand queries.
Lesson 27 of 30 · Search and Ranking: Find the Best MatchLatency and cost budgets
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Make search fast enough and cheap enough before it serves people. You measure the time of each stage on your own computer and turn milliseconds into a cost per 1,000 queries. You also see how much a cache of popular queries saves.
NotebookDo this lesson in a notebook
The notebook times BM25 and the dense search on 50 test queries. It turns milliseconds into money with the course’s example price, and measures what a cache of the 100 most common queries would answer. Its setup cells download the practice data, so it needs no local setup and no model download. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Two-stage ranking, from Two-stage ranking.
- How the cross-encoder’s time grows with the number of candidates it re-ranks, from Cross-encoder re-ranking.
- LambdaMART (
ltr), from Learning to rank. - The search page (
search serve), from Autocomplete.
What you need on your computer
- The project
larkfield-searchat the end of Module 7, with its virtual environment active, andsearch checkpassing. - The two models of Modules 5 and 6 (about 800 MB in all). If they are not on your computer yet,
search benchdownloads them the first time.
A restaurant kitchen at lunchtime
Section titled “A restaurant kitchen at lunchtime”A restaurant promises a meal in 15 minutes. The chef knows each step: 1 minute to take the order, 8 minutes on the grill, 2 minutes to plate it. If a new dish needs 30 minutes in the oven, it cannot be on the lunch menu, however good it is.
The kitchen also prepares some things before lunch. The soup of the day is ready in a pot. Half the customers order it, and they get it at once.
The chef watches two things. Time: does each meal arrive inside the promise? And money: how many cooks, for how many meals? Search works in the same way.
The problem: a better design that is too slow
Section titled “The problem: a better design that is too slow”In Module 6, the cross-encoder re-ranked the top 50 candidates. On Larkfield’s test queries, NDCG@10 went from 0.621 to 0.635. That was not a clear gain, but suppose it were. Is it ready for people?
Remember Amira’s latency budget from A shop where search fails: 200 ms, the longest time one search may take, from the query to the ranked list. It is an assumption of this course, a common target for a search page that feels instant.
The cross-encoder on 50 candidates took 1,121 ms per search (the median) on the course team’s laptop with 1 thread. A thread is one line of work on one CPU core. That is more than five times the budget. People would wait more than a second after every search. So the best design offline can be the wrong design for people.
A latency budget turns “fast enough” into a number that you can measure.
The idea: measure each stage, then spend the budget
Section titled “The idea: measure each stage, then spend the budget”A search is a chain of stages. The whole time is roughly the sum of the stage times. So you measure each stage alone, and you see where the time goes.
Two numbers for time: p50 and p95
Section titled “Two numbers for time: p50 and p95”Search times vary. A short query is fast; a long one is slower. So one number is not enough. We use two percentiles:
- p50 (the median): half of the searches are faster than this.
- p95: 95 of every 100 searches are faster than this. The other 5 are the slow ones.
Here is a tiny constructed example. Ten searches took these times, sorted: 4, 5, 5, 6, 6, 7, 7, 8, 9 and 40 ms.
| Measure | How to read it | Value |
|---|---|---|
| p50 | the middle of the sorted list: halfway between the 5th (6) and the 6th (7) | 6.5 ms |
| mean | the sum divided by 10 | 9.7 ms |
| p95 | near the top of the list | about 40 ms |
One slow search pulls the mean up to 9.7 ms. Without it, the mean would be 6.3 ms. The p50 hardly moves. The p95 shows the slow search. A budget is for the slow searches too, so check p95 as well as p50.
Where the time goes in one search
Section titled “Where the time goes in one search”The course team measured each stage of one search on 40 English test queries, on their laptop (Apple M4), with 1 thread:
Text description of the diagram
A flow from left to right. “A search, budget 200 ms” goes to a question: “In the cache?” (the 100 most common texts answer 51% of searches). Yes: “Stored page, no search needed”. No: “Spelling, 0.01 ms”, then two branches: “Lexical BM25, 1.1 ms” and “Encode the query, 5.5 ms” followed by “Dense search, 2.1 ms”. Both go to “Fusion, 0.05 ms”, then “LambdaMART step, features and model, 35 ms, measured with its own candidate search”, then “The page: whole design measured, 36 ms (p50)”. A dotted arrow from fusion: “instead: cross-encoder on 50, 977 ms”, to a red box “Over the budget”.
Text description of the chart
Six bars. Spelling 0.01 ms, lexical BM25 1.1 ms, encoding the query with the bi-encoder 5.5 ms, the dense search 2.1 ms, fusion 0.05 ms, and the LambdaMART step 35.3 ms.
Read the chart from left to right:
- Spelling and fusion cost almost nothing: hundredths of a millisecond.
- BM25 reads an index, not every product, so it takes about 1 ms.
- Encoding the query is the slowest part of hybrid search: the bi-encoder is a small neural network, and it runs once per search. The dense search itself multiplies one vector by a matrix of 40,066 rows in 2.1 ms.
- The LambdaMART step computes 19 features for 100 candidates and runs the model. It takes most of the time, and it still fits easily.
The cross-encoder is not on the chart, because it would not fit. In this measurement of the stages, it took 977 ms on 50 candidates with 1 thread, and 710 ms with 4 threads. It is the only stage that breaks the budget. The whole search with it took 1,121 ms in the design table below. Each measurement was a separate run, so the numbers differ a little.
More threads do not save the cross-encoder
Section titled “More threads do not save the cross-encoder”A model can split its work over several threads. Cross-encoder re-ranking showed that its time grows with the number of re-ranked candidates, and that only 10 candidates fit the budget. Here is the same explorer. This time, compare only the 1-thread and 4-thread times.
Explore
The cross-encoder against the 200 ms budget
Predict first: Switch from 1 thread to 4. Does the cross-encoder on 50 candidates fit the 200 ms budget then?
- NDCG@10
- 0.635
- Time per search (p50)
- 981 ms
- Budget
- 200 ms
Re-ranking 50 candidates takes 981 ms per search on 1 thread: 781 ms over the budget. NDCG@10 is 0.635 (0.589 to 0.679), +0.014 against no re-ranking. In its top 50, stage one has 44% of the exact products: the cross-encoder can only order what stage one found. For comparison, LambdaMART reaches 0.668 in 36 ms.
- 40,066products in the catalogue
- Stage one: hybrid search100 candidates9.2 ms
- Stage two: cross-encoderre-orders the top 50972 ms
- The pagetop 10 shown
| Re-ranked | Exact products in stage one’s top n | NDCG@10 (95% interval) | Time per search, p50, 1 thread budget 200 ms |
|---|---|---|---|
| 0 (stage one only) | none | 0.621 (0.576–0.663) | 9.2 ms · fits |
| 10 | 27% | 0.631 (0.587–0.672) | 194 ms · fits |
| 20 | 33% | 0.641 (0.595–0.684) | 400 ms · over |
| 50 | 44% | 0.635 (0.589–0.679) | 981 ms · over |
| 100 | 54% | 0.631 (0.584–0.674) | 1,904 ms · over |
| LambdaMART instead | none | 0.668 | 36 ms · fits |
Text description of the explorer
The explorer starts at 50 re-ranked candidates on 1 thread. Buttons choose 0, 10, 20, 50 or 100 candidates and 1 or 4 threads. The table gives the time per search on 1 and 4 threads: re-ranked 0, 9.2 ms and 10.3 ms; 10, 194 ms and 155 ms; 20, 400 ms and 295 ms; 50, 981 ms and 732 ms; 100, 1,904 ms and 1,396 ms. The budget line is at 200 ms. LambdaMART takes 36 ms on 1 thread and 34 ms on 4 threads, with NDCG@10 0.668.
What the explorer shows
No. Four threads cut the time from 981 ms to 732 ms: a quarter less, not four times less. And four threads keep four CPU cores busy for that one search. So the server can serve fewer searches at the same time, as the cost section below shows.
Turn milliseconds into money
Section titled “Turn milliseconds into money”A cloud server is paid by the hour, busy or not. This course uses an example price: US$0.17 an hour for a server with 4 virtual CPUs (vCPUs) and 16 GB. It is an assumption, not a quote. Prices change: check your provider’s page.
A worked example: the cost of one LambdaMART search
Section titled “A worked example: the cost of one LambdaMART search”One search keeps one vCPU busy for its time. So the cost of a search is its CPU time multiplied by the price of one vCPU-second.
| Step | Calculation | Value |
|---|---|---|
| Price of one vCPU for one hour | 0.17 ÷ 4 | US$0.0425 |
| Price of one vCPU-second | 0.0425 ÷ 3,600 | US$0.0000118 |
| CPU time of one LambdaMART search | 36 ms × 1 thread | 0.036 vCPU-seconds |
| CPU time of 1,000 searches | 0.036 × 1,000 | 36 vCPU-seconds |
| Cost of 1,000 searches | 36 × 0.0000118 | about US$0.0004 |
So 1,000 LambdaMART searches cost less than a tenth of a cent. A thread count multiplies the CPU time: 750 ms on 4 threads is 3 vCPU-seconds for one search.
Every design, side by side
Section titled “Every design, side by side”The course team measured each design live on 40 English test queries. The cost uses the price above, and p50 times. The language-model rewrite of Module 4 is a call to a hosted model, so its cost is the provider’s price per call.
| Design | p50 / p95 ms | US$ per 1,000 queries | Fits 200 ms? |
|---|---|---|---|
Larkfield today (current) |
1.7 / 16.2 | 0.00002 | yes |
BM25 with query understanding (bm25-norm+spell+syn) |
1.1 / 2.0 | 0.00001 | yes |
Hybrid search (hybrid-rrf) |
9.2 / 15.1 | 0.0001 | yes |
LambdaMART (ltr) |
36.1 / 54.1 | 0.0004 | yes |
Cross-encoder on 50 (rerank-ce), 1 thread |
1,121 / 1,445 | 0.013 | no |
| Cross-encoder on 50, 4 threads | 753 / 1,119 | 0.035 | no |
| Language-model rewrite (network call) | 1,195 / 1,581 | 0.04 (the provider’s price) | no |
Two things stand out.
- Cost is not the problem here. Time is. Even the most expensive design costs a few cents per 1,000 searches. But three designs break the budget: at p95, by a factor of five or more.
- The best offline design is also fast. LambdaMART has the highest NDCG@10 of Module 6 (0.668) and uses a fifth of the budget.
The real cost: servers for the busiest hour
Section titled “The real cost: servers for the busiest hour”Cost per query hides one thing. You pay for servers by the hour, and you need enough of them for the busiest moment. One server has 4 vCPUs. If each LambdaMART search keeps one vCPU busy for 36 ms, one vCPU can serve about 1,000 ÷ 36 ≈ 27 searches a second. The server, with 4 vCPUs, can serve about 110 searches a second when it is fully busy.
The cross-encoder on 4 threads keeps all 4 vCPUs busy for about 750 ms. The same server serves about 1.3 searches a second. For the same traffic, it needs about 80 times more servers. Real servers also need spare room, so plan for less than full use. The Scaling APIs and AI workloads course measures capacity in detail.
Cache the head of the query log
Section titled “Cache the head of the query log”A cache keeps the results of a recent query, so the next person with the same query gets them without a search. Module 1 showed that a few query texts make a large part of Larkfield’s log. That is the head of the log. A cache works well on the head.
Text description of the chart
Five bars. A cache of the 10 most common query texts answers 20.4% of the searches; 40 texts, 37.1%; 100 texts, 51.1%; 500 texts, 84.5%; 1,000 texts, 95.4%.
The 100 most common texts answer half of the searches. Larkfield’s log is synthetic, and our code made short, broad queries common. In a real shop, check this curve on your own log before you build a cache.
What to cache, and when to throw it away
Section titled “What to cache, and when to throw it away”Cache the ranked list of product IDs, not the whole page. Price, stock and the sale flag change during the day. Read them fresh when you build the page.
Every cached answer gets old. Cache invalidation means deleting old answers when something changes. Delete the cache when you switch to a new index or a new ranking model, as Keep the index fresh shows. Also give each answer a short life, for example an hour, so new products appear. The Scaling APIs and AI workloads course covers cache keys and invalidation in depth.
A cache does not make the slow tail fast. Half of Larkfield’s searches still miss the cache. They need a design that fits the budget on its own.
Index size
Section titled “Index size”The indexes must fit in memory, or each search waits for the disk. The course team measured:
- The lexical indexes (SQLite FTS5 and BM25 for every field and analyzer): 102.5 MB.
- The product embeddings file (40,066 × 384 numbers, float16): 31.3 MB on disk (30.8 MB for the numbers alone in memory). Float32 would be twice the size. On 200 queries, float16 kept 99.3% of the float32 top 10.
Both fit easily in a 16 GB server. The dense search reads every row of the embedding matrix, so its time grows with the catalogue. At 40,066 products it takes 2.1 ms. A catalogue ten times larger would need roughly ten times longer: still fast. For millions of products, teams use an approximate index such as HNSW (a graph of near neighbours, in libraries like FAISS). It reads only a small part of the vectors and may miss a few true neighbours. Larkfield does not need one at this size. The same ideas in a search server shows one.
Guided practice: measure your own computer
Section titled “Guided practice: measure your own computer”This module adds three Python files and one test file to the project. You copy all of them now, with the README, from the snapshot end-of-m08. This lesson uses bench.py. The next lesson uses interleave.py.
-
In
larkfield-search, with the virtual environment active, copy the files:Terminal window S=../nextia-learning-labs/C38/snapshots/end-of-m08for f in larkfield_search/bench.py larkfield_search/interleave.py larkfield_search/commands/online.py tests/test_online.py README.md; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C38\snapshots\end-of-m08"foreach ($f in 'larkfield_search\bench.py', 'larkfield_search\interleave.py', 'larkfield_search\commands\online.py', 'tests\test_online.py', 'README.md') { Copy-Item "$S\$f" $f }No new package is needed.
-
Run the tests:
Terminal window python -m pytest -q....................................... [100%]39 passed in 3.91sThree tests are new. One checks the cost formula: 100 ms on one thread at US$0.17 an hour for 4 vCPUs.
-
Before you run the next step, write down your prediction: on your computer, how long will the LambdaMART step take?
-
Measure every stage on 30 test queries, with the models on 4 threads (the default):
Terminal window search bench --queries 30The first run loads both models, so it takes about 30 seconds. PyTorch may print one warning line about “Redirects” first. You can ignore it. On the course team’s laptop, the output was:
30 test queries, models on 4 thread(s); milliseconds per query:stage p50 p95understand 0.0 1.9lexical 0.6 1.0encode 5.8 6.8dense search 1.2 1.3fusion 0.1 0.1rerank 50 726.5 868.7ltr features and model 32.7 59.6design p50 ms p95 ms US$ per 1,000hybrid 7.7 20.5 0.00036hybrid + rerank 50 734.2 876.4 0.03467ltr (hybrid inside) 38.5 65.4 0.00182index (FTS5 and BM25): 81.2 MBproduct embeddings: 31.3 MBYour numbers will differ: they depend on your computer and on what else runs on it.
-
Run it again on 1 thread, and compare the cost column:
Terminal window search bench --queries 30 --threads 1On the course team’s laptop,
hybrid + rerank 50took 1,036.8 ms (p50) and cost US$0.01224 per 1,000. The cost is lower on 1 thread, because it uses one vCPU instead of four. The time is longer.
Look at the last two lines. Your index is smaller than 102.5 MB (here 81.2 MB; yours can differ). The project builds an index for a field only when a run first needs it. You have not yet run every design in this folder.
Your turn: a budget for a slower server
Section titled “Your turn: a budget for a slower server”Larkfield’s real server may be slower than the course team’s laptop. Suppose every stage takes three times as long there. Use the stage times of this lesson (p50, 1 thread) and answer without running anything:
- Does LambdaMART still fit the 200 ms budget? Check its p95 too.
- Omar wants the language-model rewrite only for queries that are not in the cache. How many searches of the synthetic log would still wait for it, with a cache of the 100 most common texts?
Answers
- Yes. 36.1 ms × 3 ≈ 108 ms at p50, and 54.1 ms × 3 ≈ 162 ms at p95. Both fit inside 200 ms, but the p95 leaves little room for the network and the page.
- About half: 48.9%. The 100 most common texts answer 51.1% of the searches. The other 48.9% would wait about 1.2 seconds (p50) for the rewrite. A better plan: rewrite the head queries once, store the rewrites, and keep the tail on the fast path.
Common mistake
Measuring on a warm laptop and promising it to production
Symptom: Tomás runs search bench --queries 10 once, sees the cross-encoder on 10 candidates at 155 ms on 4 threads, and tells Omar: “It fits the budget.”
Cause: one short run on a quiet laptop measures the best case. The p50 hides the slow searches. In the course team’s own measurement, 10 candidates on 4 threads had a p50 of 155 ms but a p95 of 196 ms. On 1 thread, the p95 was 525 ms. A busy server also runs many searches at once, so they share the CPU and each one gets slower.
Fix: measure p95 as well as p50, on at least 30 queries, on hardware like the server. Leave room for the network and the page: a stage that alone takes 155 of 200 ms does not fit.
Consequence in search work: a budget is a promise about the slow searches, under real traffic. Measure it there, or keep a large margin.
When to measure, and what it costs
Section titled “When to measure, and what it costs”Measure the time of each stage before you choose a design, not after. A design that wins offline but breaks the budget is not a candidate. Measure again on the server, under load, before release.
The cost of this work is small. search bench takes under a minute. The notebook takes about 20 minutes. What it saves is large: Larkfield did not build a cross-encoder service that would have needed many more servers and still been too slow.
Use a cache when the head of your log is large. Do not use it as a reason to keep a slow design: the tail still meets the slow path.
Knowledge check
Section titled “Knowledge check”Check your understanding
Ten searches took 3, 3, 4, 4, 5, 5, 5, 6, 6 and 90 ms. What is the p50, and which number should Amira check against her budget?
Show the answer
The p50 is 5 ms (between the 5th and 6th sorted values, 5 and 5). Amira should also check the p95, which is near 90 ms here: the budget is a promise for the slow searches too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With US$0.17 an hour for 4 vCPUs, what do 1,000 searches of 9 ms on 1 thread cost?
Show the answer
9 ms × 1,000 = 9 vCPU-seconds. One vCPU-second costs 0.17 ÷ 4 ÷ 3,600 ≈ US$0.0000118. So 9 × 0.0000118 ≈ US$0.0001. That is the hybrid search’s cost in this lesson’s table.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A cache of the 100 most common texts answers 51.1% of Larkfield's searches. Why must the uncached design still fit the budget?
Show the answer
Because the other 48.9% of searches miss the cache and run the full search. Most of them are tail queries, typed by few people each. A cache makes the average faster, but it does not help a person who types a rare query.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A latency budget is the longest time one search may take: 200 ms for Larkfield (an assumption). Check p95 as well as p50.
- On the course team’s laptop, every stage except the cross-encoder takes a few milliseconds. LambdaMART takes 36 ms (p50); the cross-encoder on 50 candidates takes 1,121 ms on 1 thread and 753 ms on 4.
- Cost per 1,000 queries = CPU-seconds × the price of a vCPU-second. At US$0.17 an hour for 4 vCPUs (an assumption), every design costs a few cents or less. Time is the limit, and servers for the busiest hour are the real cost.
- A cache of the 100 most common texts answers 51.1% of the synthetic log. Cache product IDs, read price and stock fresh, and clear the cache when the index or the model changes.
You now have a design that is fast and cheap. But offline NDCG uses judgments, not people. Next, you test a change with people’s clicks: interleaving and A/B tests.
Next: Test changes online