Keep the index fresh
- Lesson 29
- 30 min
- Module 8: Run search in production
Where we are
We know how to test a change. The catalogue also changes every day: new, changed and removed products. In this lesson, we keep the index fresh, reindex without downtime, and version synonyms and models.
Lesson 29 of 30 · Search and Ranking: Find the Best MatchKeep the index fresh
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Keep search right while the catalogue changes. You find what a stale index misses and choose the right update for each kind of change. Then you switch to a new index version without stopping search, with a quick way back.
NotebookDo this lesson in a notebook
The notebook builds yesterday’s index without this week’s products and measures what it misses. Then it rebuilds the index, updates stock, hides a removed product, and switches index versions with a pointer file. Its setup cells download the practice data, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- How BM25 scores a product, and what IDF is, from BM25.
- Synonym rules, from Synonyms and query rewriting.
- Why a new product has no clicks, from New products and the cold start.
What you need on your computer
- The project with this module’s files, from Latency and cost budgets, and about 400 MB of free disk space for a second index version.
A printed catalogue in a busy shop
Section titled “A printed catalogue in a busy shop”A garden shop prints a catalogue of everything on its shelves. Every day, new plants arrive, prices change, and some products sell out. The printed catalogue does not know.
A customer asks for a new kind of rose. The assistant looks in the catalogue, does not find it, and says “we do not have it”. The rose is on the shelf behind her.
The shop cannot reprint the catalogue every minute. So it does three things. It writes today’s prices on a small board by the door. It crosses out sold-out products by hand. And every night, it prints a new catalogue, checks it, and puts it on the counter before opening. Yesterday’s copy stays in a drawer, in case the new one has a mistake.
The problem: a product that is not in the index cannot be found
Section titled “The problem: a product that is not in the index cannot be found”A search index is a copy of the catalogue, built at one moment. Search reads the index, not the catalogue. So anything that changed after the build is invisible or wrong.
Larkfield adds new products every week. The “added” dates are synthetic: our code made them.
Text description of the chart
Eight bars, one per week of the log: 272, 254, 275, 278, 305, 248, 325 and 275 products. About 39 new products a day.
About 39 products a day. The notebook builds “yesterday’s index” without the last week’s products, and measures what it misses:
| Measure | Value |
|---|---|
| Products in yesterday’s index | 39,791 |
| Products added this week, not in it | 275 |
| English validation and test queries with an exact product from this week | 54 |
| New products in yesterday’s top 10 for “home sweet home sign” | none |
A person who types “home sweet home sign” cannot find this week’s signs, however good they are. And price and stock are worse: they change many times a day. An index built last night may show a product as in stock that sold out an hour ago.
The idea: match the update to the change
Section titled “The idea: match the update to the change”Not every change needs the same work. Sort the changes by how often they happen.
Text description of the diagram
Three rows, each with an arrow to its kind of update. “Every few minutes: price, stock, sale flag” → “Update the fields in place, no new index”. “Every day: new, changed and removed products” → “Add or delete documents, or rebuild the index”. “By a release: analyzer and synonyms.csv, spelling dictionary, bi-encoder revision and embeddings, LambdaMART model” → “A new index version, tested offline, then swapped”.
Fields that change every minute
Section titled “Fields that change every minute”Keep price, stock and the sale flag out of the text index. Store them in a small table, keyed by product ID, and update the table when the warehouse changes. When you build the page, read the ranked list of IDs from search, then read price and stock from the table.
This is why Latency and cost budgets said: cache the ranked IDs, not the page. In the notebook, one line marks a product as sold out, and the next search shows it at once. No rebuild is needed.
The ranking can use stock too: the rule “stock last” of Zero results and business rules reads the table at query time.
New and changed products
Section titled “New and changed products”A new product needs three things before search can find it:
- its words in the lexical index;
- its embedding, made by the same bi-encoder as the others;
- its features for LambdaMART (it has no clicks yet, as New products and the cold start showed).
A changed title is a delete and an add of the same product.
There are two ways to apply them. An incremental update adds or replaces only the changed products in the live index. Search servers do this all day. A full rebuild builds the whole index again from the catalogue.
At Larkfield’s size, a full rebuild is simple and fast:
- The lexical index of 40,066 products: 1.3 to 1.9 seconds (the notebook and
search index). - Embeddings for the new products: the bi-encoder encoded 257.8 products a second on 4 threads. This week’s 275 products take about 1 second. All 42,066 products (with the 2,000 held back for the final assignment) took 163 seconds.
So Larkfield can rebuild every night, or every hour. Incremental updates matter for catalogues of millions of products, or for changes that must show within seconds.
One thing changes for everyone: IDF
Section titled “One thing changes for everyone: IDF”Remember IDF from BM25. A rare word gets a high IDF. When new products arrive, word counts change, so the IDF of some words changes. The BM25 scores of old products change a little too. A rebuild applies these changes everywhere at once. An incremental update in a search server applies them as it goes.
Removed products
Section titled “Removed products”A removed product must leave the page at once, not after tonight’s rebuild. Until the rebuild, hide it at query time. Keep a list of removed IDs, and drop them from every ranked list. Such a mark, “removed, but still in the index”, is called a tombstone. The next rebuild leaves the product out, and the tombstone can go.
Reindex without downtime
Section titled “Reindex without downtime”Never build the new index over the one that serves people. During the build, the old index would be half replaced. Searches would fail or return half the catalogue.
Instead, build each version beside the live one, check it, then switch in one step. Teams call this a blue-green switch: two copies, one live (blue), one new (green).
Text description of the diagram
On the left, a box “Serving now” with “Pointer: current” pointing to “Index version 41, yesterday’s catalogue”. Then “Build version 42 in a new folder, today’s catalogue” → a question “Checks pass? counts, a known query, NDCG@10 on validation”. Yes → “Move the pointer to version 42, one atomic step” → “Keep version 41 for a quick rollback”. No → a red box “Keep version 41, fix, build again”.
The pointer and the atomic switch
Section titled “The pointer and the atomic switch”Search does not open a folder by a fixed name. It reads a pointer first: a small file, or a link, that names the version in use. To switch, you change the pointer.
The change must be atomic: it happens in one step, so no search ever sees half of it. On a computer, renaming a file over another file is atomic. In Python, os.replace does this. The notebook writes the new name into current.tmp, then renames it over current.
Checks before the switch
Section titled “Checks before the switch”A switch is only as safe as its checks. Before you move the pointer, check at least:
- Counts. The new version has the number of products you expect. The notebook refuses a version with one product too few.
- A known query. One query returns its known first result.
search checkdoes this with “garden hose”. - Quality. NDCG@10 on the validation queries is close to the last version’s. A sudden fall means something broke.
Rollback
Section titled “Rollback”Keep the old version on disk until the new one has served for a while. If people report a problem, move the pointer back. Rollback then takes a second, not a rebuild. The CI/CD, testing and safe releases course covers rollback plans for any release.
After a switch, clear the cache of ranked lists. Its answers came from the old version.
Version synonyms and models with the index
Section titled “Version synonyms and models with the index”The index is not the only part of a search release. The results also depend on:
- the analyzer and
synonyms.csv(Module 4); - the spelling dictionary (built from the catalogue and the log);
- the bi-encoder’s revision and the product embeddings it made (Module 5);
- the LambdaMART model (Module 6).
These parts must match each other. Two examples:
- Embeddings and the query encoder. Product embeddings made by one revision of the bi-encoder, and queries encoded by another revision, live in different spaces. Their cosine scores mean nothing. The course pins the model revision (
835ad140…) for this reason. - The model and its features. LambdaMART learned from features of one index. If the index changes its analyzer, the BM25 features change too, and the model may rank worse.
So treat all of these parts as one release: build them together, give the set one version number, test it offline, then switch. Write the versions in a small file, a manifest, next to the index. The notebook’s manifest records the number of products and the BM25 settings. A real one also names the model revisions and the synonym file’s version. The MLOps and LLMOps course calls this artifact lineage: knowing which data and code made each part.
Larkfield: a new version of the project’s outputs
Section titled “Larkfield: a new version of the project’s outputs”The project keeps everything it builds in one folder, outputs/: the indexes, the spelling dictionary, the class classifier and the LambdaMART model. The variable LARKFIELD_OUTPUTS tells the project to use another folder. So you can build a complete new version beside the live one.
-
In
larkfield-search, with the virtual environment active, build the index into a new folder:Terminal window LARKFIELD_OUTPUTS=outputs-next search indexTerminal window $env:LARKFIELD_OUTPUTS = "outputs-next"; search indexIndexed 40,066 products in 1.9 s: SQLite FTS5 (title, description) and BM25 (title and description). Index size 54.0 MB. -
Check the new version, still in its own folder:
Terminal window LARKFIELD_OUTPUTS=outputs-next search checkData: 15 files, checksums right (data)Index: foundQuery "garden hose": 10 results; first: LK-74728 anand self watering vinyl window box planterBM25 on the validation queries: NDCG@10 0.432 (expected 0.432)Setup check passed. -
Check the design you serve. The first run builds the parts that LambdaMART needs (the other indexes, the spelling dictionary, the class classifier and the model):
Terminal window LARKFIELD_OUTPUTS=outputs-next search eval --run ltrltr on 243 test queries: NDCG@10 0.668 (95% interval 0.628 to 0.708)recall@100 0.540, MRR 0.698, zero results 0.0%, judged in the top 10 73.9%; 18.7 ms a query (p50), 63.9 ms (p95)From an empty folder, steps 1 and 3 took 25 seconds on the course team’s laptop. The folder holds 342 MB.
-
Switch: keep the old version, and put the new one in its place.
Terminal window mv outputs outputs-previousmv outputs-next outputssearch checkTerminal window Remove-Item Env:LARKFIELD_OUTPUTSRename-Item outputs outputs-previousRename-Item outputs-next outputssearch checkThe last line says
Setup check passed.To roll back, rename the folders the other way.
The PowerShell commands were not run by the course team. The two renames in step 4 are two steps, not one. A search that starts between them finds no outputs/ folder. That is acceptable on your laptop. A live service uses a pointer, as in the notebook, or a search server’s alias, as the next lesson shows.
Guided practice: the notebook
Section titled “Guided practice: the notebook”Open the notebook Keep the index fresh (the link is at the top of this lesson).
- Run the setup cells. The last one prints
Ready: 4 files in larkfield-search. - Run section 1. It prints
Yesterday's index: 39,791 productsandNot in it: 275 products added this week. The build time depends on your computer. - Run section 2. It finds 54 queries with an exact new product.
- In section 3, set
productsto the whole catalogue (cat), and run the check. It printsCheck passed: found is 7. - Run sections 4 and 5. Watch the sold-out product leave the “in stock now” list, and the removed product leave the results.
- In section 7, complete
switch, and run the check:Check passed: serving is v42.
Why do only 7 of the 54 queries show an exact new product after the rebuild? The notebook uses plain BM25, and an exact new product competes with many older exact products. Being in the index makes a product findable. It does not put it in the top 10. That is the cold-start problem of New products and the cold start.
Your turn: the module task, a release note
Section titled “Your turn: the module task, a release note”Write a one-page release note for one ranking change: LambdaMART (ltr) in place of hybrid search (hybrid-rrf). Use only numbers from this course. It needs five parts:
- The change and why. What changes for people, in two sentences.
- Offline result. NDCG@10 with its interval, overall and for the slices that change most. Name any slice that gets worse.
- Online plan. Interleaving or an A/B test, the measure, the size, decided in advance, and the stopping rule.
- Guardrails. What must not get worse, with numbers: latency, zero results, out-of-stock first results.
- Release and rollback. The index and model versions, how you switch, and how you switch back.
A worked example (compare yours with it)
1. The change. Search orders the top 100 hybrid candidates with LambdaMART, a model trained on 260 judged training queries. People should see the right product higher, mostly for French and broad queries.
2. Offline result (243 test queries). NDCG@10 0.668 (0.628 to 0.708) against 0.621. Paired difference +0.047 (+0.023 to +0.073): clear. French +0.159, broad +0.130. English WANDS queries +0.010: not clear. recall@100 does not change (0.540): the same candidates.
3. Online plan. Team-draft interleaving, LambdaMART against hybrid search, 5,000 searches, looked at once (simulated power 91%). Then an A/B test on the add-to-cart rate, with its size from the same kind of power estimate. No early stop except a broken guardrail.
4. Guardrails. p95 latency under 200 ms (measured 54 ms on 1 thread). Zero-result rate stays 0% on the test queries. The share of out-of-stock first results stays near 11.1%.
5. Release and rollback. Version 42 bundles the index, the spelling dictionary, the bi-encoder revision 835ad140… and the LambdaMART model, with a manifest. It is built beside version 41 and checked with search check and search eval --run ltr (0.668). Then the pointer moves and the cache is cleared. Rollback: move the pointer back to version 41; it stays on disk for two weeks.
Common mistake
Editing the synonym file in the live index
Symptom: Hana asks for “light” to also find “lamp”. Tomás adds the rule light, lamp to the live synonyms.csv. Results change at once, for every query with “light”. Two days later, people say that some searches with “light” or “lamp” got worse. Nobody knows which rules were live before.
Cause: the synonym file is part of the release, but it changed outside one. It skipped the offline test, and the old file was overwritten, so there is no rollback. This rule is a practice problem we made on purpose. On Larkfield’s validation and test queries, it touches 10 queries. NDCG@10 goes from 0.784 to 0.778: 2 queries better, 4 worse (Synonyms and query rewriting).
Fix: keep synonyms.csv under version control, with the index version it belongs to. Test a new rule offline with search compare on the queries it touches. Release it in a new version, and switch with the pointer.
Consequence in search work: every part that changes the ranking is code, even a text file. Version it, test it, and keep the old version to switch back.
When to rebuild, when to update, and what it costs
Section titled “When to rebuild, when to update, and what it costs”- Rebuild fully when the catalogue is small enough to rebuild in minutes: Larkfield’s lexical index takes under two seconds. A rebuild is simple, and it applies IDF changes everywhere.
- Update incrementally when the catalogue is large, or a change must show within seconds. A search server does this, at the cost of running the server.
- Update fields in place for price, stock and the sale flag, always. Never rebuild for them.
- Release a new version for synonyms, analyzers, models and embeddings. Test it offline first.
Versions cost disk space: one complete version of the project’s outputs is 342 MB. Keep one or two old versions, not all of them.
Knowledge check
Section titled “Knowledge check”Check your understanding
A product sold out ten minutes ago. Should Larkfield rebuild the index? What should it do instead?
Show the answer
No. Stock is a field that changes often, so it lives outside the text index, in a small table keyed by product ID. Update the table; the page reads it for every search. The rule “stock last” can use the same table at query time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why must Larkfield not build the new index in the same folder that serves people?
Show the answer
During the build, the folder holds half of the old index and half of the new one. Searches would fail or miss products. Build the new version beside the old one, check it, then switch a pointer in one atomic step.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Amira updates the bi-encoder to a new revision, and re-encodes the queries with it, but keeps last month's product embeddings. What goes wrong?
Show the answer
The queries and the products are now in different vector spaces, so their cosine scores mean nothing, and dense search ranks badly. The model revision and the embeddings are one part of a release: change them together, test offline, then switch.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A product that is not in the index cannot be found. Yesterday’s index misses this week’s 275 products; 54 test and validation queries have an exact one.
- Match the update to the change. Price and stock go in a side table. New and changed products need a rebuild or incremental updates. A tombstone hides a removed product at once.
- Reindex without downtime. Build beside the live index, and check counts, a known query and NDCG@10. Switch a pointer atomically, keep the old version for rollback, and clear the cache.
- Synonyms, the spelling dictionary, the model revision, the embeddings and the LambdaMART model are one release, with a manifest. Version them with the index.
You built every part of search in Python. Next, you see the same ideas in a search server, OpenSearch, and what a server adds and what it costs to run.