Skip to content
Français

Keep the index fresh

  • Lesson 29
  • 30 min
  • Module 8: Run search in production

Where we are

We know how to test a change. The catalogue also changes every day: new, changed and removed products. In this lesson, we keep the index fresh, reindex without downtime, and version synonyms and models.

What you will be able to do

Keep search right while the catalogue changes. You find what a stale index misses and choose the right update for each kind of change. Then you switch to a new index version without stopping search, with a quick way back.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds yesterday’s index without this week’s products and measures what it misses. Then it rebuilds the index, updates stock, hides a removed product, and switches index versions with a pointer file. Its setup cells download the practice data, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • The project with this module’s files, from Latency and cost budgets, and about 400 MB of free disk space for a second index version.

A garden shop prints a catalogue of everything on its shelves. Every day, new plants arrive, prices change, and some products sell out. The printed catalogue does not know.

A customer asks for a new kind of rose. The assistant looks in the catalogue, does not find it, and says “we do not have it”. The rose is on the shelf behind her.

The shop cannot reprint the catalogue every minute. So it does three things. It writes today’s prices on a small board by the door. It crosses out sold-out products by hand. And every night, it prints a new catalogue, checks it, and puts it on the counter before opening. Yesterday’s copy stays in a drawer, in case the new one has a mistake.

The problem: a product that is not in the index cannot be found

Section titled “The problem: a product that is not in the index cannot be found”

A search index is a copy of the catalogue, built at one moment. Search reads the index, not the catalogue. So anything that changed after the build is invisible or wrong.

Larkfield adds new products every week. The “added” dates are synthetic: our code made them.

Products added to Larkfield's catalogue in each of the eight weeks of the query log (synthetic dates). The last week is the one that yesterday's index misses in the notebook.
Text description of the chart

Eight bars, one per week of the log: 272, 254, 275, 278, 305, 248, 325 and 275 products. About 39 new products a day.

About 39 products a day. The notebook builds “yesterday’s index” without the last week’s products, and measures what it misses:

Measure Value
Products in yesterday’s index 39,791
Products added this week, not in it 275
English validation and test queries with an exact product from this week 54
New products in yesterday’s top 10 for “home sweet home sign” none

A person who types “home sweet home sign” cannot find this week’s signs, however good they are. And price and stock are worse: they change many times a day. An index built last night may show a product as in stock that sold out an hour ago.

Not every change needs the same work. Sort the changes by how often they happen.

Three groups of changes. Price, stock and the sale flag change every few minutes: update these fields in place, with no new index. New products, changed titles and descriptions, and removed products change every day: add or delete documents, or rebuild the index. Some parts change only with a release: the analyzer and synonyms.csv, the spelling dictionary, the bi-encoder revision with its embeddings, and the LambdaMART model. For them, build a new index version, test it offline, then swap.
Three speeds of change, three kinds of update.
Text description of the diagram

Three rows, each with an arrow to its kind of update. “Every few minutes: price, stock, sale flag” → “Update the fields in place, no new index”. “Every day: new, changed and removed products” → “Add or delete documents, or rebuild the index”. “By a release: analyzer and synonyms.csv, spelling dictionary, bi-encoder revision and embeddings, LambdaMART model” → “A new index version, tested offline, then swapped”.

Keep price, stock and the sale flag out of the text index. Store them in a small table, keyed by product ID, and update the table when the warehouse changes. When you build the page, read the ranked list of IDs from search, then read price and stock from the table.

This is why Latency and cost budgets said: cache the ranked IDs, not the page. In the notebook, one line marks a product as sold out, and the next search shows it at once. No rebuild is needed.

The ranking can use stock too: the rule “stock last” of Zero results and business rules reads the table at query time.

A new product needs three things before search can find it:

  1. its words in the lexical index;
  2. its embedding, made by the same bi-encoder as the others;
  3. its features for LambdaMART (it has no clicks yet, as New products and the cold start showed).

A changed title is a delete and an add of the same product.

There are two ways to apply them. An incremental update adds or replaces only the changed products in the live index. Search servers do this all day. A full rebuild builds the whole index again from the catalogue.

At Larkfield’s size, a full rebuild is simple and fast:

  • The lexical index of 40,066 products: 1.3 to 1.9 seconds (the notebook and search index).
  • Embeddings for the new products: the bi-encoder encoded 257.8 products a second on 4 threads. This week’s 275 products take about 1 second. All 42,066 products (with the 2,000 held back for the final assignment) took 163 seconds.

So Larkfield can rebuild every night, or every hour. Incremental updates matter for catalogues of millions of products, or for changes that must show within seconds.

Remember IDF from BM25. A rare word gets a high IDF. When new products arrive, word counts change, so the IDF of some words changes. The BM25 scores of old products change a little too. A rebuild applies these changes everywhere at once. An incremental update in a search server applies them as it goes.

A removed product must leave the page at once, not after tonight’s rebuild. Until the rebuild, hide it at query time. Keep a list of removed IDs, and drop them from every ranked list. Such a mark, “removed, but still in the index”, is called a tombstone. The next rebuild leaves the product out, and the tombstone can go.

Never build the new index over the one that serves people. During the build, the old index would be half replaced. Searches would fail or return half the catalogue.

Instead, build each version beside the live one, check it, then switch in one step. Teams call this a blue-green switch: two copies, one live (blue), one new (green).

Serving now: a pointer named current points to index version 41, yesterday's catalogue. Version 42 is built in a new folder from today's catalogue. Checks: product counts, a known query, NDCG@10 on validation. If they pass, the pointer moves to version 42 in one atomic step, and version 41 is kept for a quick rollback. If they fail, version 41 stays live, and the team fixes the problem and builds again.
Build beside, check, switch in one step, keep the old version.
Text description of the diagram

On the left, a box “Serving now” with “Pointer: current” pointing to “Index version 41, yesterday’s catalogue”. Then “Build version 42 in a new folder, today’s catalogue” → a question “Checks pass? counts, a known query, NDCG@10 on validation”. Yes → “Move the pointer to version 42, one atomic step” → “Keep version 41 for a quick rollback”. No → a red box “Keep version 41, fix, build again”.

Search does not open a folder by a fixed name. It reads a pointer first: a small file, or a link, that names the version in use. To switch, you change the pointer.

The change must be atomic: it happens in one step, so no search ever sees half of it. On a computer, renaming a file over another file is atomic. In Python, os.replace does this. The notebook writes the new name into current.tmp, then renames it over current.

A switch is only as safe as its checks. Before you move the pointer, check at least:

  • Counts. The new version has the number of products you expect. The notebook refuses a version with one product too few.
  • A known query. One query returns its known first result. search check does this with “garden hose”.
  • Quality. NDCG@10 on the validation queries is close to the last version’s. A sudden fall means something broke.

Keep the old version on disk until the new one has served for a while. If people report a problem, move the pointer back. Rollback then takes a second, not a rebuild. The CI/CD, testing and safe releases course covers rollback plans for any release.

After a switch, clear the cache of ranked lists. Its answers came from the old version.

Version synonyms and models with the index

Section titled “Version synonyms and models with the index”

The index is not the only part of a search release. The results also depend on:

  • the analyzer and synonyms.csv (Module 4);
  • the spelling dictionary (built from the catalogue and the log);
  • the bi-encoder’s revision and the product embeddings it made (Module 5);
  • the LambdaMART model (Module 6).

These parts must match each other. Two examples:

  1. Embeddings and the query encoder. Product embeddings made by one revision of the bi-encoder, and queries encoded by another revision, live in different spaces. Their cosine scores mean nothing. The course pins the model revision (835ad140…) for this reason.
  2. The model and its features. LambdaMART learned from features of one index. If the index changes its analyzer, the BM25 features change too, and the model may rank worse.

So treat all of these parts as one release: build them together, give the set one version number, test it offline, then switch. Write the versions in a small file, a manifest, next to the index. The notebook’s manifest records the number of products and the BM25 settings. A real one also names the model revisions and the synonym file’s version. The MLOps and LLMOps course calls this artifact lineage: knowing which data and code made each part.

Larkfield: a new version of the project’s outputs

Section titled “Larkfield: a new version of the project’s outputs”

The project keeps everything it builds in one folder, outputs/: the indexes, the spelling dictionary, the class classifier and the LambdaMART model. The variable LARKFIELD_OUTPUTS tells the project to use another folder. So you can build a complete new version beside the live one.

  1. In larkfield-search, with the virtual environment active, build the index into a new folder:

    Terminal window
    LARKFIELD_OUTPUTS=outputs-next search index
    Indexed 40,066 products in 1.9 s: SQLite FTS5 (title, description) and BM25 (title and description). Index size 54.0 MB.
  2. Check the new version, still in its own folder:

    Terminal window
    LARKFIELD_OUTPUTS=outputs-next search check
    Data: 15 files, checksums right (data)
    Index: found
    Query "garden hose": 10 results; first: LK-74728 anand self watering vinyl window box planter
    BM25 on the validation queries: NDCG@10 0.432 (expected 0.432)
    Setup check passed.
  3. Check the design you serve. The first run builds the parts that LambdaMART needs (the other indexes, the spelling dictionary, the class classifier and the model):

    Terminal window
    LARKFIELD_OUTPUTS=outputs-next search eval --run ltr
    ltr on 243 test queries: NDCG@10 0.668 (95% interval 0.628 to 0.708)
    recall@100 0.540, MRR 0.698, zero results 0.0%, judged in the top 10 73.9%; 18.7 ms a query (p50), 63.9 ms (p95)

    From an empty folder, steps 1 and 3 took 25 seconds on the course team’s laptop. The folder holds 342 MB.

  4. Switch: keep the old version, and put the new one in its place.

    Terminal window
    mv outputs outputs-previous
    mv outputs-next outputs
    search check

    The last line says Setup check passed. To roll back, rename the folders the other way.

The PowerShell commands were not run by the course team. The two renames in step 4 are two steps, not one. A search that starts between them finds no outputs/ folder. That is acceptable on your laptop. A live service uses a pointer, as in the notebook, or a search server’s alias, as the next lesson shows.

Open the notebook Keep the index fresh (the link is at the top of this lesson).

  1. Run the setup cells. The last one prints Ready: 4 files in larkfield-search.
  2. Run section 1. It prints Yesterday's index: 39,791 products and Not in it: 275 products added this week. The build time depends on your computer.
  3. Run section 2. It finds 54 queries with an exact new product.
  4. In section 3, set products to the whole catalogue (cat), and run the check. It prints Check passed: found is 7.
  5. Run sections 4 and 5. Watch the sold-out product leave the “in stock now” list, and the removed product leave the results.
  6. In section 7, complete switch, and run the check: Check passed: serving is v42.

Why do only 7 of the 54 queries show an exact new product after the rebuild? The notebook uses plain BM25, and an exact new product competes with many older exact products. Being in the index makes a product findable. It does not put it in the top 10. That is the cold-start problem of New products and the cold start.

Your turn: the module task, a release note

Section titled “Your turn: the module task, a release note”

Write a one-page release note for one ranking change: LambdaMART (ltr) in place of hybrid search (hybrid-rrf). Use only numbers from this course. It needs five parts:

  1. The change and why. What changes for people, in two sentences.
  2. Offline result. NDCG@10 with its interval, overall and for the slices that change most. Name any slice that gets worse.
  3. Online plan. Interleaving or an A/B test, the measure, the size, decided in advance, and the stopping rule.
  4. Guardrails. What must not get worse, with numbers: latency, zero results, out-of-stock first results.
  5. Release and rollback. The index and model versions, how you switch, and how you switch back.
A worked example (compare yours with it)

1. The change. Search orders the top 100 hybrid candidates with LambdaMART, a model trained on 260 judged training queries. People should see the right product higher, mostly for French and broad queries.

2. Offline result (243 test queries). NDCG@10 0.668 (0.628 to 0.708) against 0.621. Paired difference +0.047 (+0.023 to +0.073): clear. French +0.159, broad +0.130. English WANDS queries +0.010: not clear. recall@100 does not change (0.540): the same candidates.

3. Online plan. Team-draft interleaving, LambdaMART against hybrid search, 5,000 searches, looked at once (simulated power 91%). Then an A/B test on the add-to-cart rate, with its size from the same kind of power estimate. No early stop except a broken guardrail.

4. Guardrails. p95 latency under 200 ms (measured 54 ms on 1 thread). Zero-result rate stays 0% on the test queries. The share of out-of-stock first results stays near 11.1%.

5. Release and rollback. Version 42 bundles the index, the spelling dictionary, the bi-encoder revision 835ad140… and the LambdaMART model, with a manifest. It is built beside version 41 and checked with search check and search eval --run ltr (0.668). Then the pointer moves and the cache is cleared. Rollback: move the pointer back to version 41; it stays on disk for two weeks.

Common mistake

Editing the synonym file in the live index

Symptom: Hana asks for “light” to also find “lamp”. Tomás adds the rule light, lamp to the live synonyms.csv. Results change at once, for every query with “light”. Two days later, people say that some searches with “light” or “lamp” got worse. Nobody knows which rules were live before.

Cause: the synonym file is part of the release, but it changed outside one. It skipped the offline test, and the old file was overwritten, so there is no rollback. This rule is a practice problem we made on purpose. On Larkfield’s validation and test queries, it touches 10 queries. NDCG@10 goes from 0.784 to 0.778: 2 queries better, 4 worse (Synonyms and query rewriting).

Fix: keep synonyms.csv under version control, with the index version it belongs to. Test a new rule offline with search compare on the queries it touches. Release it in a new version, and switch with the pointer.

Consequence in search work: every part that changes the ranking is code, even a text file. Version it, test it, and keep the old version to switch back.

When to rebuild, when to update, and what it costs

Section titled “When to rebuild, when to update, and what it costs”
  • Rebuild fully when the catalogue is small enough to rebuild in minutes: Larkfield’s lexical index takes under two seconds. A rebuild is simple, and it applies IDF changes everywhere.
  • Update incrementally when the catalogue is large, or a change must show within seconds. A search server does this, at the cost of running the server.
  • Update fields in place for price, stock and the sale flag, always. Never rebuild for them.
  • Release a new version for synonyms, analyzers, models and embeddings. Test it offline first.

Versions cost disk space: one complete version of the project’s outputs is 342 MB. Keep one or two old versions, not all of them.

Check your understanding

A product sold out ten minutes ago. Should Larkfield rebuild the index? What should it do instead?

Show the answer

No. Stock is a field that changes often, so it lives outside the text index, in a small table keyed by product ID. Update the table; the page reads it for every search. The rule “stock last” can use the same table at query time.

Check your understanding

Why must Larkfield not build the new index in the same folder that serves people?

Show the answer

During the build, the folder holds half of the old index and half of the new one. Searches would fail or miss products. Build the new version beside the old one, check it, then switch a pointer in one atomic step.

Check your understanding

Amira updates the bi-encoder to a new revision, and re-encodes the queries with it, but keeps last month's product embeddings. What goes wrong?

Show the answer

The queries and the products are now in different vector spaces, so their cosine scores mean nothing, and dense search ranks badly. The model revision and the embeddings are one part of a release: change them together, test offline, then switch.

Recap

  • A product that is not in the index cannot be found. Yesterday’s index misses this week’s 275 products; 54 test and validation queries have an exact one.
  • Match the update to the change. Price and stock go in a side table. New and changed products need a rebuild or incremental updates. A tombstone hides a removed product at once.
  • Reindex without downtime. Build beside the live index, and check counts, a known query and NDCG@10. Switch a pointer atomically, keep the old version for rollback, and clear the cache.
  • Synonyms, the spelling dictionary, the model revision, the embeddings and the LambdaMART model are one release, with a manifest. Version them with the index.

You built every part of search in Python. Next, you see the same ideas in a search server, OpenSearch, and what a server adds and what it costs to run.

Next: The same ideas in a search server