Operational handoff

What you will be able to do

Write down everything that another person needs to run the assistant, rebuild its index and get the same results. That is the versions that an answer depends on, its cost and time, and one command that rebuilds it.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Why the embedding model and its revision belong to the index, from Embeddings for search.
• How the mock finds a recorded answer, from Answer and cite.
• audit and ingest --rebuild, from Updates and deletions.

What you need on your computer

• The project after Permissions and isolation: python -m pytest shows 90 passed, 1 deselected.

No model call and no account. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A recipe card for the next cook

A restaurant’s best dish is in the head cook’s memory. She goes on holiday, and the dish tastes different the next week. The new cook used another flour, a hotter oven and a little less salt. Nobody wrote down that it mattered.

So kitchens keep a recipe card: the exact ingredients with their brands, the amounts, the oven temperature, the time, and what one plate costs. With the card, a new cook can make the same dish. When the dish changes, the card shows why.

The problem: the same project, a different assistant

Amira goes on holiday, and Tomás takes over the assistant. He knows Python well. But the assistant’s answers depend on more than its code, and small changes break things without a clear error:

• An embedding model swapped for another one of the same size. In Embeddings for search, searching the e5 index with all-MiniLM-L6-v2 question vectors gave hit@5 0.068, with no error message.

• A prompt file with one extra space. Tomás adds a space between two words of answer_v1.md, as a tidy-up. Then ask Q22 answers:

  Output

  No answer: No recording for this request (key 6c964385674edd7d). The mock replays only the questions that were recorded for the course, with the same prompt and the same model.

  With a live model, nothing fails. But every measure that Priya made in Module 5 was made with the old prompt, so none of them is proof for the new one.

• A document edited in place. Grace fixes a sentence, as in Updates and deletions. Two chunk IDs change, and old answers cite IDs that no longer exist.

None of these is in the code. Each one is a version of something that the code uses.

The idea: a handoff record

A handoff record is the recipe card of the assistant. It lists everything that an answer depends on, with its exact version. It also says what the assistant costs, how long it takes, and how to rebuild it. Some of it the code can print, because the index and the project already store it. The rest goes into the README, next to the code.

An answer of the policy assistant depends on these parts:

• the documents, and so the chunks and their IDs;
• the chunker;
• the embedding model, with its revision;
• the reranker, with its revision;
• the prompts, byte for byte;
• the answer model, with its date;
• the limits: the context budget and the answer limit;
• the packages: Python and the libraries that run the models.

A tiny worked example: one fingerprint per part

A version must be exact, and short enough to compare by eye. The project uses the same trick as for chunk IDs: a short SHA-256 fingerprint. For a prompt file, it hashes the file’s bytes and keeps 8 characters:

Python session, in ~/projects/policy-assistant

import hashlib
hashlib.sha256(open("policy_assistant/prompts/answer_v1.md", "rb").read()).hexdigest()[:8]

Output

'7b7c7836'

With Tomás’s extra space, the same line gives 290968ed. The difference is one invisible character, and the fingerprint shows it at once. For the documents, the project hashes the list of every file with its content hash, so that one fingerprint stands for all 36 documents: ec1d4db6e641. The models have their own fingerprints already: the Hugging Face revision is the hash of the model’s files on a given day.

Larkfield’s handoff record

The new command info prints the part of the record that the project stores:

python -m policy_assistant info

Output

index: index.sqlite | last sync 2026-10-09T06:28:30+00:00
documents: 36 (fingerprint ec1d4db6e641) | chunks: 185 | vectors: 185 x 384
chunker: structure
embedding model: intfloat/multilingual-e5-small (614241f622f53c4eeff9890bdc4f31cfecc418b3)
reranker: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 (1427fd652930e4ba29e8149678df786c240d8825) | candidates: 20
prompts: answer_v1 7b7c7836, answer_v2 c4e40fe8, closed_book 9131b52f, rewrite 71abe061
context budget: 1500 estimated tokens | answer limit: 2000 tokens
answers: provider mock | model chat-small
python 3.12.13 | sentence-transformers 6.1.0 | torch 2.14.1 | pypdf 6.19.0 | openai 3.26.1
rebuild: python -m policy_assistant ingest --rebuild --chunker structure --embedder intfloat/multilingual-e5-small

Your last sync time is different; every other line should be the same. vectors: 185 x 384 is a quick health check: one vector for each chunk, with e5’s 384 numbers. The diagram maps each part to its line, and says what a change to it means:

What one answer depends on, where info shows it, and what a change means. The project refuses the changes that would give nonsense; the others need a new measurement.

Text description of the diagram

A table with seven rows and three columns: the part, its value in info, and what happens if it changes. Documents: 36, fingerprint ec1d4db6e641; chunk text and IDs change, and recorded answers replay with a note, so measure again. Chunker: structure; ingest refuses to mix chunkers, so rebuild, then measure again. Embedding model: multilingual-e5-small, revision 614241f622f5 and more; ingest and search refuse another model. Mixing two models gave hit@5 0.068. Reranker: mmarco-mMiniLMv2-L12-H384-v1, revision 1427fd652930 and more; the order of the top 5 changes, so measure again. Prompts: answer_v1 7b7c7836 and answer_v2 c4e40fe8; a changed file no longer matches the recorded requests, so record again. Answer model: the mock with chat-small, recorded with gpt-6-luna-2026-09-22. Answers, cost and time change, so measure again and note the date. Budget and limits: 1500 context tokens and 2000 answer tokens; more or fewer passages fit, so measure again.

What it costs, and how long it takes

Omar’s questions are always the same: what does it cost, and how long does an agent wait? The answers come from the course’s recorded runs, at the prices when we recorded them. Prices change, so check the current price list. Per million input and output tokens, chat-small costs 0.10 and 0.50 US$, and chat-strong 2 and 10.

Building the index costs nothing with the local e5 model. With the hosted embed-small instead, one full embedding run cost US$0.0005. It was 25,362 tokens for 443 texts: the chunks of all three chunkers, the questions and their rewrites. One question costs US$0.00017 with chat-small, and US$0.0031 with chat-strong. Most of the time goes to the model, and most of the rest to the reranker:

Where the time of one question goes: embedding the question with e5, BM25, the vector search, reranking 20 candidates (local, 2 CPU threads), and the answer from chat-small (small) or chat-strong (strong), recorded, median.

Text description of the chart

A column chart in milliseconds, medians. Embedding the question with e5: 5.2. BM25: 0.26. Vector search: 0.09. Reranking 20 candidates: 322. The answer from chat-small (small): 1460. The answer from chat-strong (strong): 2460.

The search steps are too short to see, except the reranker: 322 ms. With chat-small, an agent waits about 1.8 seconds. For one question in twenty, the model alone takes 3.72 seconds or more (its 95th percentile). For a whole day, the choice of model matters more than anything else:

The cost of a day of 45 questions: chat-small and chat-strong with retrieval (top 5, about 900 input tokens), and chat-strong with the whole collection in every prompt (about 17,850 tokens). Prices change: check the current price list.

Text description of the chart

A column chart of US dollars for 45 questions a day. chat-small with retrieval: 0.0077. chat-strong with retrieval: 0.14. chat-strong with the whole collection in the prompt: 1.7.

The README of the snapshot end-of-m06 holds these numbers, in a section Handoff record. It also holds the setup, the models with their revisions and licences, and the environment variables. It says what invalidates the recorded answers, and when and where the project was last tested. Prices change, so the README says to check the price list, and gives the date.

One command that rebuilds it

The last line of info is the rebuild command. Run it, and check that the rebuilt index is the same: the same fingerprint, the same counts, and the same measures.

Output of the rebuild command

36 added, 0 updated, 0 unchanged, 0 removed | chunks: +185 -0
  added: 36 files
index index.sqlite: 36 documents, 185 chunks | chunker structure | embeddings intfloat/multilingual-e5-small

Then info shows ec1d4db6e641 again, and eval shows hit@5 0.898 | recall@5 0.856 | precision@5 0.231 | MRR 0.831, as in Module 5. A rebuild that gives other numbers means that something in the record is not what it says.

Common mistake

A tidy-up that breaks every recorded answer

Symptom: after a small edit to answer_v1.md, ask Q22 prints No recording for this request. With a live model, the answers look fine, so nobody notices for weeks.

Cause: the prompt is part of every request, and so part of the key that finds a recording. A real cache or a recorded test set has the same key. One extra space makes a new prompt.

Diagnose: run info and compare the prompt fingerprints with the handoff record: answer_v1 290968ed instead of 7b7c7836.

Fix: never edit a prompt in place. Copy it to a new version (answer_v3.md), measure the new version on the question set, and record its answers. Keep the old one for the old measurements. This is how answer_v2 was made in Permissions and isolation.

When a handoff record is worth it, and what it costs

Write one for every assistant that another person will run, or that you will run again in a month. It costs one command and one README section, and it must be updated when a part changes. A record that is out of date is worse than none, because people trust it. So let the code print what it can (info). Keep the README for what only people know: prices, measures and their dates, and known limits.

This record is a start, not operations. Running the assistant for real users needs more: scheduled re-ingestion, monitoring of answers and costs, and alerts. It also needs safe releases of a new prompt or model, and a way back when a release goes wrong. The course MLOps and LLMOps: Operating AI in Production teaches that.

Guided practice: the handoff record

1. Copy the full README from the snapshot:

   macOS / Linux

   cp ../nextia-learning-labs/C09/snapshots/end-of-m06/README.md .

   Windows

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m06\README.md .

   Your project is now the same as the snapshot end-of-m06, the end of the course’s project.

2. Open README.md, and read the sections Rebuild and update and Handoff record.

3. Print the record:

   python -m policy_assistant info

4. Rebuild the index with the command from the last line of the output, then run info again. The fingerprint is still ec1d4db6e641.

5. Measure the rebuilt index. You see hit@5 0.898 on the second line:

   python -m policy_assistant eval

6. In the README, add a line under Tested with your date, your operating system and your Python version.

Module practice: replace a policy version and remove a document

Finance publishes version 2 of the gift-card policy, in force from today, 9 October 2026: gift cards now go from 20 to 750 dollars. The same day, Grace removes the holiday returns notice of 2025. It ended on 31 January 2026, and no return from that time can still be open. Do both, then prove that the old text no longer appears.

1. Copy corpus/documents/gift-cards.md to corpus/documents/gift-cards.v2.md.

2. In gift-cards.v2.md, change three things: version: 1 to version: 2, effective_from: 2025-06-01 to effective_from: 2026-10-09, and “from 25 to 500 dollars” to “from 20 to 750 dollars”.

3. In gift-cards.md, add the line effective_to: 2026-10-08 under effective_from. Version 1 now ends the day before version 2 starts.

4. Delete corpus/documents/holiday-returns-2025.md.

5. Run ingest. Predict the report before you read it.

6. Search “What amounts do gift cards come in?” with --method bm25 --k 2, once without --as-of and once with --as-of 2026-10-08.

7. Search “Holiday returns extension 2025” with --method bm25 --k 3 --as-of none.

Show the expected results

Step 5: 1 added, 1 updated, 34 unchanged, 1 removed | chunks: +6 -6. The new file is added (3 chunks). Version 1 is read again because its metadata changed (3 out, 3 in). The notice is removed (3 out). chunks gift-cards lists 6 chunks from documents/gift-cards.md, documents/gift-cards.v2.md. Version 1 keeps its IDs: the new line is metadata, not text.

Step 6: today, the first two results are [21240db11429] gift-cards v2 and [68d2e5bc69bc] gift-cards v2, “from 20 to 750 dollars”. On 2026-10-08, they are [f27bb50ce8b7] and [8c4bc4299042], version 1, “from 25 to 500 dollars”. Version 1 is retired by its date, not deleted: a question about a gift card bought on 8 October still finds it.

Step 7: even with no date filter, the notice is gone. The results are Document types, Returns for trade customers and Returns and points.

Module check

The completion check of this module: confirm that obsolete and removed passages no longer appear in answers. Search is half of it, and you did it above. Now the answers: run audit while your two changes are still in place.

python -m policy_assistant audit

Output (the first 9 of 15 lines)

Q25 chat-small rerank answer_v1: [bbd5032e5107] removed from the index
Q25 chat-small bm25 answer_v1: [a5a937704e3f] removed from the index
Q25 chat-small bm25 answer_v1: [bbd5032e5107] removed from the index
Q25 chat-strong rerank answer_v1: [a5a937704e3f] removed from the index
Q25 chat-strong rerank answer_v1: [bbd5032e5107] removed from the index
Q25 chat-strong bm25 answer_v1: [a5a937704e3f] removed from the index
Q25 chat-strong bm25 answer_v1: [bbd5032e5107] removed from the index
Q63 chat-small rerank answer_v1: [52ad3a57c1d1] gift-cards v1 is not in force on 2026-10-09
Q63 chat-strong rerank answer_v1: [52ad3a57c1d1] gift-cards v1 is not in force on 2026-10-09

The last line is 314 recorded answers | 15 citations to passages that are removed, not in force or not allowed. The other six lines are recorded extras of Q25 and Q26 that also cite the notice. These are the stored answers that still carry the old text. Q25 cites the deleted notice. Q63, the French question about returning a gift card, cites version 1’s Lost cards and returns, which is no longer in force today. Ask it:

python -m policy_assistant ask Q63

The search now gives version 2’s passage ba3e515c5d04. The mock replays the recorded answer, and both of its claims show CHECK: not_in_context:52ad3a57c1d1. The obsolete passage did not reach the context; only the old stored answer still names it. In a real assistant, the 15 citations that audit lists mark the cached answers to drop. A live model then answers again from the passages in force.

Then undo the practice, so that your project matches the course again: delete gift-cards.v2.md, copy gift-cards.md and holiday-returns-2025.md back from ../nextia-learning-labs/C09/corpus/documents/, and run ingest. audit must end with 0 citations.

Check your understanding

Q25 asks about a lamp delivered on 1 December 2025. After the practice, what should a live assistant answer, and what must Grace change in the question set?

Show the answer

The holiday notice is gone, so the passages in force for 2025-12-01 are version 3 of the returns policy: 30 days. A live assistant should answer from it. The lamp was delivered on 1 December, so the 30 days ended on 31 December. The return on 15 January is late. Grace removed the evidence on purpose. So Q25’s reference answer and relevant passages are now wrong. Grace must update Q25 or remove it, and record the change in the dataset card, before anyone compares measures again.

Check your understanding

Tomás wants to switch the embedding model to a newer, larger one. Using the handoff record, list what he must do, in order.

Show the answer

1. Record the new model’s name and revision. 2. Rebuild the index with ingest --rebuild --embedder NAME, because vectors of two models cannot be mixed. 3. Run info: the vectors line shows the new number of dimensions. 4. Run eval and compare with hit@5 0.898 and MRR 0.831 on the same 67 questions. 5. Measure the answers with a live model, because the passages change and the recorded answers no longer match. 6. Note the new time and cost of ingestion and search. 7. Update the README’s handoff record, with the date.

Check your understanding

Predict: Grace changes no file, but Tomás upgrades sentence-transformers to a newer version. Which lines of info change, and why should he still run eval?

Show the answer

Only the package line changes. The model revision is the same, so the stored vectors are still valid. But a new library version can compute the same model’s vectors a little differently: tokenization, pooling or numeric precision. Near-ties in the ranking can then change. eval shows whether the measures are still hit@5 0.898 and MRR 0.831. If they moved, rebuild the index and record why.

Recap

• A handoff record lists everything that an answer depends on, with exact versions: documents, chunker, embedding model, reranker, prompts, answer model, limits and packages. info prints the part that the project stores, with short fingerprints.
• One extra space in a prompt made a new prompt: answer_v1 290968ed instead of 7b7c7836, and no recording matched. Make a new prompt version instead of editing one.
• Costs,: a full embedding run with embed-small US$0.0005; one question US$0.00017 with chat-small, US$0.0031 with chat-strong. The model takes most of the time, then the reranker (322 ms).
• The rebuild command gives the same fingerprint and the same measures. If it does not, the record is wrong.
• Module check: after a new version and a deletion, search no longer returns the old text, and audit lists every stored answer that still cites it.

Your project now matches the snapshot end-of-m06: the course’s project is complete. The case studies apply the same method to real open data, starting with public guidance.

Next: Answer questions from public guidance