Updates and deletions

What you will be able to do

Keep the index equal to the collection while documents change. Read only the files that changed, let dates retire old versions, and refuse duplicates. Make a deleted document disappear from search and from answers.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Stable chunk IDs, effective dates and access labels, from Preserve metadata.
• The date filter, from Lexical, dense and hybrid search.
• Recorded answers and their citation checks, from Answer and cite.

What you need on your computer

• The project as it is at the end of Module 5 (the snapshot end-of-m05). python -m pytest shows 77 passed, 1 deselected.
• An index built by python -m policy_assistant ingest, and a terminal in ~/projects/policy-assistant with .venv active.

No model call and no account: answers come from the recordings. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A library that keeps its catalogue right

A library gets new books every week. The librarian does not catalogue the whole library again each time. She compares the delivery with the catalogue. A new book gets a card, and a new edition replaces the old card. An unchanged book needs nothing.

Taking a book away is harder than it looks. The book leaves the shelf. But its card is still in the catalogue, it is still on the “new arrivals” list, and a reader’s notes still cite it. Until every copy of the reference is gone, people keep asking for a book that the library no longer has.

The problem: a deleted document that search still finds

Module 5’s ingest rebuilds the whole index every time. It deletes every row, then parses, chunks and embeds all 36 documents again. That is simple and always correct. On the course’s Mac it took 2.09 s, with the embedding model already loaded. A collection of 36,000 documents would take about a thousand times longer, for every small change.

So Mei writes the obvious faster version. For each file in the folder, it skips a file that the index already has, and it reads a new or changed one. It is fast. Then the People team moves the staff discount rules to its own system, and asks Grace to remove staff-discount-and-perks.md from the help desk’s collection. Mei deletes the file and runs her sync. This is a constructed version of the code, run on a copy of the collection (reference/c09/m06/naive_incremental.py):

Output (a constructed sync that only adds and updates)

NaiveStore: after deleting staff-discount-and-perks.md: 0 added, 0 updated, 35 unchanged, 0 removed | chunks: +0 -0
  chunks in the index: 185; keyword search for 'What is the staff discount?': ['[f02ee0910bbf] staff-discount-and-perks', '[c13d27947ed1] staff-discount-and-perks']

Nothing failed, and the report looks calm: 35 files unchanged. But the sync looks only at the files that exist, so it never notices a file that is gone. The index still has all 185 chunks, and the search still returns the deleted document first. This is one of the misconceptions that this course warns about: leaving deleted content searchable. The file is gone, so people believe the text is gone. It is not.

The idea: compare fingerprints, and delete everywhere

The fix has two parts.

The first part is how to know what changed. When ingest reads a file, it stores the file’s content hash: the SHA-256 fingerprint of its bytes. It is the same hash function that makes the chunk IDs. The next time, it computes the hash of each file again and compares the two lists, as the librarian compares the delivery with the catalogue. Every file falls into one of four cases:

• New (no row for this file): parse, chunk, embed and add it.
• Changed (a different hash): delete the old rows first, then add the file again.
• Unchanged (the same hash): skip it. Nothing is parsed or embedded.
• Gone (a row, but no file): delete it.

This is incremental ingestion: each run does only the work that the changes need. The fourth case is the one Mei forgot.

The second part is what “delete” means. One document lives in many places: its row in documents, its chunks, their keyword rows and their vectors. Any process that is running may also keep copies in memory. Answers that cite it may also be stored somewhere. A deletion is complete only when every one of these places has forgotten the document.

A tiny worked example: one changed file

Take the gift-card policy. First, compare the hash of the file with the hash that the index stored at the last ingest:

Python session, in ~/projects/policy-assistant

from pathlib import Path
from policy_assistant.store import Store, file_hash
file_hash(Path("corpus/documents/gift-cards.md"))[:12]
[d["content_hash"][:12] for d in Store("index.sqlite").documents() if d["doc_id"] == "gift-cards"]

Output

'dbfb9792f883'
['dbfb9792f883']

The two are equal, so ingest skips this file. Now Grace fixes one sentence in Using a gift card: “your account” becomes “your Larkfield account”. The new hash is 7b83eafaf8b6. One changed word is enough to change the whole fingerprint, so ingest sees the change:

Output of python -m policy_assistant ingest

0 added, 1 updated, 35 unchanged, 0 removed | chunks: +3 -3
  updated: documents/gift-cards.md
index index.sqlite: 36 documents, 185 chunks | chunker structure | embeddings intfloat/multilingual-e5-small

It deleted the 3 old chunks, with their keyword rows and vectors, and added 3 new ones. Compare the chunk IDs before and after:

Chunk	Before	After	Why
#0 Buying a gift card	8c4bc4299042, characters 12–125	8c4bc4299042, characters 12–125	Same version, same position: same ID
#1 Using a gift card	f27bb50ce8b7, 127–230	76f2d13b502f, 127–240	The sentence is 10 characters longer
#2 Lost cards and returns	52ad3a57c1d1, 232–424	6c36ce3a91a6, 242–434	Its position moved by 10 characters

The chunk IDs come from the document, the version and the position, so the changed section and the one after it got new IDs. An old answer that cites f27bb50ce8b7 now cites a chunk that does not exist, and the citation check says so. That is correct: the cited text really changed.

The whole life of a document, from the first ingest to the last check, looks like this:

The life of a document in the index. A date retires a version; only a deletion removes it, and only a check proves that it is gone.

Text description of the diagram

Six steps in two groups. The first group is “In the index”. Step 1, Published: a new file in corpus/documents. Step 2, Ingested: its hash is recorded, and its chunks, keyword rows and vectors are added. Step 3, In force: from its effective_from date, search returns it. A dashed arrow goes from step 3 back to step 2: a correction gives the file a new hash, so it is ingested again. When the next version takes effect, the document moves to the second group, “On the way out”. Step 4, Retired by date: after its effective_to date it is still in the index, but the date filter leaves it out. Step 5, Deleted: the file is removed, and ingest removes every row made from it. Step 6, Checked: no answer, cache or running process still uses it.

Larkfield’s index: retire by date, delete by decision

Retiring a version needs no deletion

On 1 November 2026, version 4 of the returns policy replaces version 3. Nobody has to touch the index on that day. Version 3 has effective_to: 2026-10-31, so the date filter of Module 3 retires it at midnight:

python -m policy_assistant search "Can a customer return an unused item?" --method bm25 --k 2 --as-of 2026-10-31

Output

1.  11.6078  [7839ded2d52b] returns-policy v3 | Return window | public
   Return window You can return an unused item within 30 days of delivery for a full refund of the ite…
2.  10.9722  [c236034e24d2] help-centre-faq v7 | How do I return an item? | public
   How do I return an item? Go to Orders in your account, choose the item and select Return an item. O…
bm25, top 2, as of 2026-10-31, audience staff

With --as-of 2026-11-01, the first result is [963dae4229eb] returns-policy v4 | Return window (“within 45 days”), and the help article stays second. Version 3 must stay in the index. How Larkfield policies are managed says that the old version stays in the archive. It also says that a return follows the version in force on the delivery date. In December, an agent will still ask about an order delivered on 20 October, and only version 3 answers that.

So the rule is simple. A new version gets its own file, a new version number and its effective_from, and the old version gets an effective_to. Delete a version only when no open question can need it any more.

Deleting a document

The staff discount document is different: the People team wants it gone from the collection. With the project’s new ingest, the deletion reaches the index:

Output of python -m policy_assistant ingest, after the file was deleted

0 added, 0 updated, 35 unchanged, 1 removed | chunks: +0 -2
  removed: documents/staff-discount-and-perks.md
index index.sqlite: 35 documents, 183 chunks | chunker structure | embeddings intfloat/multilingual-e5-small

Keyword search now returns Using a discount code first, and chunks staff-discount-and-perks answers No chunks for staff-discount-and-perks. Now ask the question that the recordings know:

python -m policy_assistant ask Q51

Output

Question (date 2026-10-09): What is the staff discount?
Passages: 094a8535171c, 1b1c047750bf, d753fb02bc8c, b16f983ce250, a559fb986971 (568 tokens, estimated)
Note: Recorded with passages c13d27947ed1, f02ee0910bbf, 094a8535171c, 1b1c047750bf, d753fb02bc8c; yours: 094a8535171c, 1b1c047750bf, d753fb02bc8c, b16f983ce250, a559fb986971. The recorded answer is replayed; its citations are checked against your passages.
Answer (answerable): The staff discount is 20% off all products except gift cards, on up to $2,000 of purchases per calendar year. It cannot be combined with other promotions or discount codes.
  - The discount is 20% off all products except gift cards, and applies to at most $2,000 of purchases per calendar year. [c13d27947ed1] CHECK: missing_id:c13d27947ed1
  - The staff discount cannot be combined with other promotions or discount codes. [c13d27947ed1] CHECK: missing_id:c13d27947ed1
gpt-6-luna-2026-09-22 | 951 tokens in, 127 out | 1.416 s

The search no longer finds the document, but the answer still says “20% off”. The mock replays the answer that chat-small (gpt-6-luna on Azure) gave earlier, when the document existed. In this project, that is how the mock works. In a real assistant, the same thing happens with an answer cache: a store of earlier answers, kept to save time and money. A cache that finds answers by the question’s text alone serves deleted text to the next person who asks. The citation check catches it here (missing_id), but only because every claim cites its source.

So the new command audit looks through every recorded answer. It lists each citation of a passage that the index no longer serves for that question. The passage is removed, or not in force on the question’s date:

Output of python -m policy_assistant audit

Q51 chat-small rerank answer_v1: [c13d27947ed1] removed from the index
Q51 chat-small bm25 answer_v1: [c13d27947ed1] removed from the index
Q51 chat-strong rerank answer_v1: [c13d27947ed1] removed from the index
Q51 chat-strong rerank answer_v1: [f02ee0910bbf] removed from the index
Q51 chat-strong bm25 answer_v1: [c13d27947ed1] removed from the index
Q51 chat-strong bm25 answer_v1: [f02ee0910bbf] removed from the index
268 recorded answers | 6 citations to passages that are removed, not in force or not allowed

Those four answers to Q51 are the ones to drop from any answer cache. One more place remembers the document: Grace’s inventory.csv. inventory still lists staff-discount-and-perks, because nobody edited the list. The diagram collects every place, with what each sync did:

Where a deleted document can still be found. Orange: the text is still there. Every value comes from a real run.

Text description of the diagram

A table with six rows for the deleted file staff-discount-and-perks.md. Its two columns are a sync that only adds and updates, and the project’s ingest. Row 1, the folder corpus/documents: the file is gone in both columns. Row 2, the index tables documents and chunks: the first sync removed nothing and still has 185 chunks. It never looks for files that are gone. The project’s ingest reports 1 removed and chunks minus 2, so 183 chunks. Row 3, the keyword index and the vectors on disk: with the first sync, “What is the staff discount?” still finds f02ee0910bbf and c13d27947ed1 first. With the project, the rows are deleted with the chunks. The first result is promotions-and-discount-codes. Row 4, the memory of a process that keeps running: with Module 5’s code, vector search still returned c13d27947ed1 from the vectors in memory. The project checks the index’s version before every search, and reads the memory again. Row 5, the recorded answers, which work like an answer cache: Q51 still replays “20% off all products” in both columns. audit finds 6 citations to removed chunks, and those answers must be dropped. Row 6, Grace’s inventory.csv: it still lists the document in both columns; update it by hand.

Duplicates: one document version, one file

A duplicate is the same content stored twice. In this project it comes in three forms.

• The same file twice. Somebody saves gift-cards-copy.md next to gift-cards.md. Both say that they are gift-cards version 1. ingest refuses, and the index stays as it was: Nothing changed in the index. documents/gift-cards-copy.md and documents/gift-cards.md are both gift-cards version 1. Keep one file per document version; a new version needs a new version number. Without the check, both copies would compete in every top 5.
• A renamed file. Mei renames gift-cards.md to gift-cards-2025.md. ingest deletes the files that are gone first, then adds the new name: 1 added, 0 updated, 35 unchanged, 1 removed | chunks: +3 -3. The chunk IDs stay the same, because they come from the document, the version and the position, not from the file name.
• Near-duplicate paragraphs in different documents. “How to start a return” appears in four documents. Each is a real source with its own owner, so the index keeps all four. Assemble context removes the repeats when the passages go into one prompt. Two versions of one document are never merged: their dates differ, and the answer must say which one it used.

Common mistake

The server that still knows the deleted text

Symptom: the assistant runs as a service: one Python process that answers questions all day. In another terminal, Mei deletes the staff discount file and runs ingest. The service still answers “What is the staff discount?” from the deleted chunks. A real run with Module 5’s code (reference/c09/m06/running_process.py):

Output

end-of-m05: before: dense [c13d27947ed1] staff-discount-and-perks, [f02ee0910bbf] staff-discount-and-perks
another process: 35 documents, 183 chunks (rebuilt from scratch)
end-of-m05: after:  dense [c13d27947ed1] staff-discount-and-perks, [f02ee0910bbf] staff-discount-and-perks
end-of-m05: after:  bm25  [ea0bff365b2c] promotions-and-discount-codes, [f78646d93435] policy-governance

Cause: for speed, the service loads all the vectors and chunks into memory once. Keyword search asks SQLite every time, so it sees the deletion. Vector search uses the copy in memory, which still has the deleted vectors. The memory is a cache, and nobody told it that the index changed.

Fix: check the index’s version before each search, and read the memory again when it changed. SQLite counts the changes that other connections make (PRAGMA data_version). The new store.py combines that count with a count of its own writes in Store.generation(), and search.py uses it. With the snapshot end-of-m06, the same run returns two privacy-policy chunks after the deletion. The test test_a_running_assistant_sees_the_deletion keeps it so.

For a real assistant: list every cache before you trust a deletion. Examples are vectors in memory, answer caches, search results kept by a web server, and copies on other machines. Each one needs a rule for when it is read again.

When incremental ingestion is worth it, and what it costs

On the course’s Mac, a second ingest with nothing changed took 2 ms, and one changed document took 28.4 ms. A full rebuild took 2.09 s. On 36 documents, both are fast. The gap matters when the collection grows, or when embeddings come from a paid service: every rebuild pays again for every chunk.

The cost is more cases to get right. Renames, duplicates, deletions and running processes each needed code and a test here. A change of chunker or embedding model cannot be incremental at all, because vectors of two models cannot be mixed. So ingest refuses it, and ingest --rebuild starts a new index.

Use incremental ingestion when documents change often and the collection is large. Keep a full rebuild for small collections, and after a change of chunker or model. Also use it as a regular check: a rebuilt index must give the same chunk IDs and measures.

Guided practice: add incremental ingestion

1. Copy the new index code, search code, command line and store tests from the snapshot:

   macOS / Linux

   cp ../nextia-learning-labs/C09/snapshots/end-of-m06/policy_assistant/store.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m06/policy_assistant/search.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m06/policy_assistant/__main__.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m06/tests/test_store.py tests/

   Windows

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m06\policy_assistant\store.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m06\policy_assistant\search.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m06\policy_assistant\__main__.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m06\tests\test_store.py tests\

   The new __main__.py also has --prompt, audit and info. The next two lessons use them.

2. Open policy_assistant/store.py, and find sync. Find the four cases: gone, unchanged, changed and new.

3. Run the tests. You see 83 passed, 1 deselected:

   python -m pytest

4. Update the index. Your index from Module 5 already has a hash for every file:

   python -m policy_assistant ingest

   Output

   0 added, 0 updated, 36 unchanged, 0 removed | chunks: +0 -0
   index index.sqlite: 36 documents, 185 chunks | chunker structure | embeddings intfloat/multilingual-e5-small

5. In VS Code, open corpus/documents/gift-cards.md. Change “in your account” to “in your Larkfield account”, and save.

6. Run ingest, then chunks gift-cards. You see 1 updated | chunks: +3 -3 and the new IDs 76f2d13b502f and 6c36ce3a91a6, as in the worked example.

7. Put the original file back, and run ingest again:

   macOS / Linux

   cp ../nextia-learning-labs/C09/corpus/documents/gift-cards.md corpus/documents/
   python -m policy_assistant ingest

   Windows

   Copy-Item ..\nextia-learning-labs\C09\corpus\documents\gift-cards.md corpus\documents\
   python -m policy_assistant ingest

   chunks gift-cards shows f27bb50ce8b7 and 52ad3a57c1d1 again: the same text gives the same IDs.

8. Delete the staff discount document, and update the index:

   macOS / Linux

   rm corpus/documents/staff-discount-and-perks.md
   python -m policy_assistant ingest

   Windows

   Remove-Item corpus\documents\staff-discount-and-perks.md
   python -m policy_assistant ingest

   You see 1 removed | chunks: +0 -2.

9. Search for it with keywords:

   python -m policy_assistant search "What is the staff discount?" --method bm25 --k 3

   The first result is [ea0bff365b2c] promotions-and-discount-codes v2 | Using a discount code. No result comes from the deleted document.

10. Run ask Q51, then audit. You see the replayed “20% off” with missing_id, and 6 citations to passages that are removed.

11. Put the document back, and check that everything is as before:

    macOS / Linux

    cp ../nextia-learning-labs/C09/corpus/documents/staff-discount-and-perks.md corpus/documents/
    python -m policy_assistant ingest
    python -m policy_assistant audit

    Windows

    Copy-Item ..\nextia-learning-labs\C09\corpus\documents\staff-discount-and-perks.md corpus\documents\
    python -m policy_assistant ingest
    python -m policy_assistant audit

    Output

    1 added, 0 updated, 35 unchanged, 0 removed | chunks: +2 -0
      added: documents/staff-discount-and-perks.md
    index index.sqlite: 36 documents, 185 chunks | chunker structure | embeddings intfloat/multilingual-e5-small
    268 recorded answers | 0 citations to passages that are removed, not in force or not allowed

If ingest prints Nothing changed in the index, one document version is in corpus/documents twice. Delete the copy, and run ingest again. If ingest reports a different chunker or embedding model, run python -m policy_assistant ingest --rebuild.

Your turn: rename corpus/documents/gift-cards.md to gift-cards-2025.md, and predict the report of ingest before you run it. Then rename it back.

Show the answer

1 added, 0 updated, 35 unchanged, 1 removed | chunks: +3 -3, with added: documents/gift-cards-2025.md and removed: documents/gift-cards.md. The chunk IDs do not change, because the file name is not part of them. Renaming it back gives the same report with the two names swapped.

See it end to endAnswer from versioned regulationsCase study

The regulations case study answers from versions that change over time, and shows what happens without the date filter.

Check your understanding

Mei's first sync skipped unchanged files and added new ones. Why did a deleted file stay searchable, and why did its report not show a problem?

Show the answer

The sync looped over the files in the folder. A deleted file is not in the folder, so the loop never reached it, and its rows stayed in every table. The report counted only the files it saw: 35 unchanged, 0 removed. To find deletions, compare the other way too: every document in the index whose file is gone must be removed.

Check your understanding

Predict: on 2 November 2026, should Grace delete returns-policy.v3.md from the collection? What would go wrong?

Show the answer

No. The date filter already retires version 3 for questions dated from 1 November. But returns follow the version in force on the delivery date. An agent will still ask about orders delivered in October, such as Q22 (delivered on 20 October). Without version 3, the search would find only version 4, and the assistant would say 45 days instead of 30. Delete a version only when no open question can need it.

Check your understanding

After the deletion, ask Q51 still printed '20% off'. Which part of the system was wrong, and what is the fix in a real assistant?

Show the answer

The index was right: the search returned no staff discount passage. The stored answer was the problem: the mock replays a recording found by the question, like an answer cache keyed by the question’s text. In a real assistant, remove the cached answers that cite a removed chunk, as audit lists them, or include the passage IDs in the cache key. The citation check (missing_id) is a last safety net, not the fix.

Recap

• Incremental ingestion compares each file’s content hash with the stored one. New and changed files are read again, and unchanged files are skipped. Documents whose file is gone are removed. A second ingest took 2 ms instead of 2.09 s.
• A deletion must reach every copy: the tables, the keyword index, the vectors, the memory of running processes, answer caches and the inventory. A sync that only adds and updates left the deleted staff discount first in the search.
• Retiring a version is the date filter’s job: version 3 of the returns policy stays in the index for deliveries before 1 November.
• One document version comes from one file: copies are refused, renames are moved, and near-duplicates in different documents are left for context assembly.
• audit lists recorded answers that cite removed or obsolete passages. The citation check catches them too, but only as the last guard.

Your project now has the incremental store.py. Next, the access labels: who may see which passage.

Next: Permissions and isolation