Preserve metadata

What you will be able to do

Give every chunk an ID that never changes for the same passage. Give it the facts that lead back to its source: document, version, dates, access label, section, page and characters. Then build the project’s index on disk, and trace any chunk to the exact place it came from.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The three chunkers and the warranty policy’s chunks, from Chunk deliberately.
• Block positions in the parsed text, from Parse and normalize.
• SQLite, from SQL and Data Preparation for AI. The index is one SQLite file.

What you need on your computer

• The project with parse.py, chunk.py and metadata.py, from Chunk deliberately: Guided practice. python -m pytest shows 24 passed.
• A terminal in ~/projects/policy-assistant, with .venv active.

Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A call number on the spine

In a library, every book has a call number on its spine, such as “641.5 LAR”. The number belongs to the book. When the library buys new books or gives old ones away, “641.5 LAR” still finds the same book. Nobody uses “the 132nd book on the third shelf”: one new book, and the 132nd book is a different one.

The catalogue card says more: the title, the edition, the year, the shelf, and whether the book may leave the building. A reader with only the call number can find the book, check the edition, and know whether they may take it home.

The problem: a citation that points to the wrong passage

Answers in this course cite chunks by their ID. Grace’s agents will read an answer, click the ID, and check the passage. So the ID must lead to the same passage tomorrow, after the collection changes.

The simplest ID is a counter: chunk 0, chunk 1, chunk 2, in the order that ingestion reads the files. Amira tested it. She numbered the 185 structure-aware chunks, then left out one document, account-and-security.md, as if Grace had retired it. Then she numbered the chunks again.

One document leaves the collection. The counter 132 now means another passage; the hash ID 7839ded2d52b still means the return window. Computed with the project's code; the retired document is a constructed case.

Text description of the diagram

Two small tables. Before: 36 documents and 185 chunks. Counter 132 is the chunk with hash ID 7839ded2d52b, the returns policy version 3, section Return window. Counter 133 is beb2688295be, section Condition of returned items. An old answer cites [132] and 7839ded2d52b, and both mean the 30-day return window. After one document leaves: 179 chunks. Counter 132 is now b7711c885517, the section Exchanges, marked in orange. The return window is now counter 126, and its hash ID is still 7839ded2d52b. So [132] in the old answer now points to the wrong passage, while the hash ID still points to the right one.

All 179 remaining chunks changed their counter. Not one kept it. An old answer that cited “[132]” for the 30-day return window now points to the section Exchanges. Nothing fails, and no error appears. The citation is just wrong. The hash IDs of all 179 chunks stayed the same.

There is a second trap. Version 3 and version 4 of the returns policy have a section Exchanges with exactly the same text. If the ID came from the text alone, both versions would get the same ID, and an answer could not say which version it used. For a return, that matters: the delivery date decides which of the two versions applies.

The idea: a stable ID and the chunk’s metadata

Metadata is data about data: the facts about a chunk that are not its text. A stable chunk ID is an ID that the same chunk gets on every ingestion, and that no other chunk gets.

The project builds the ID from four values: the document ID, the version, and the start and end of the chunk in the parsed text. It joins them with |, computes SHA-256 of that text, and keeps the first 12 characters. SHA-256 is a hash function: it turns any text into a fingerprint of 64 hexadecimal characters. The same text always gives the same fingerprint, and any change gives a different one.

Each of the four values has a job.

• The document ID says which document. Another document can never move it, so leaving one out changes no other ID.
• The version gives a new version new IDs, even when its text is the same. An answer never cites version 4 with an ID of version 3.
• The start and the end say which part of the document. Two chunks of one document never share them.

The chunk’s number in its document (position, #3) is stored too, for reading order, but it is not part of the ID.

A tiny worked example: one ID, by hand

Take chunk #3 of the warranty policy, the structure-aware chunk of What is not covered: characters 1517 to 1967 of version 5. Compute its ID twice: with the project’s function, and by hand with Python’s hashlib.

Python session, in ~/projects/policy-assistant

import hashlib
from policy_assistant.metadata import chunk_id
chunk_id("warranty-policy", "5", 1517, 1967)
hashlib.sha256("warranty-policy|5|1517|1967".encode()).hexdigest()

'a4dc9ec49818'
'a4dc9ec498189eea06b752305cc84105636c67ea6767e348552973c7a6bc0cea'

The first 12 characters of the fingerprint are the ID. Twelve hexadecimal characters allow about 281 million million different IDs, so two of Larkfield’s 185 chunks sharing one by chance is very unlikely. Run it again tomorrow, on another computer: you get a4dc9ec49818 again. Change the version to "6", and you get a different ID.

The ID finds the chunk. The rest of the record says where it comes from, and who may see it:

One real chunk and where each value of its metadata comes from. Every value is in the index; the chunk's text is stored with it.

Text description of the diagram

The record of chunk a4dc9ec49818, structure-aware chunk #3 of the warranty policy. Each field has an arrow to its source. chunk_id a4dc9ec49818: the first 12 characters of SHA-256 of “warranty-policy|5|1517|1967”. doc_id and version, warranty-policy and 5: the PDF’s metadata (its Keywords field) and the inventory. effective_from 2026-01-01, and no effective_to: the PDF’s metadata. access public: the PDF’s metadata; a missing label becomes staff. source documents/warranty-policy.pdf: the file that ingestion read. sections What is not covered: the heading block above the text. pages 1: the page where each block starts. start and end, 1517 and 1967: the characters in the parsed text; the parsed text from 1517 to 1967 is the chunk’s source text.

Two fields need a word more. The access label says who may see a chunk: public (customers and agents) or staff (agents only). A document with no label gets staff. If someone forgets the label, the passage stays hidden from customers; it is never shown by mistake. The effective dates say when a version applies. Version 3 of the returns policy is in force until 2026-10-31, and version 4 from 2026-11-01. Module 3 filters on both, before the top 5 is taken.

The index on disk

So far the chunks lived only in a Python session. The project’s store.py keeps them in one SQLite file, index.sqlite, with four tables:

• documents: one row per document version, with its metadata, its file and the SHA-256 of the file’s bytes;
• chunks: one row per chunk, with the record above and its text;
• keyword_index: the chunks’ text in SQLite’s full-text index, for BM25;
• meta: how the index was built, for now only the chunker.

The command ingest reads every file in corpus/documents/, parses it, chunks it and writes it. For now, it deletes everything and builds the index again each time. That is simple and always correct, and it takes well under a second for 36 documents. Module 6 makes it update only what changed. A document without a document ID or a version stops ingest with an error: a chunk without them could not be cited.

Larkfield’s index: trace a chunk to its source

After ingestion, the command chunks lists the chunks of one document with their metadata:

python -m policy_assistant chunks warranty-policy

144a40851685 v5 #0 chars 27-369 pages 1 | 60 words | What the warranty covers
d9a5bd8ea531 v5 #1 chars 371-1453 pages 1 | 200 words | Warranty periods by product category
b47ef9ac1993 v5 #2 chars 1455-1515 pages 1 | 18 words | Warranty periods by product category
a4dc9ec49818 v5 #3 chars 1517-1967 pages 1 | 80 words | What is not covered
3f7023270716 v5 #4 chars 1969-2276 pages 2 | 64 words | How to make a claim
ae23e5ca9b13 v5 #5 chars 2278-2409 pages 2 | 27 words | Proof of purchase
66cd6ab97679 v5 #6 chars 2411-2579 pages 2 | 30 words | Customers in Quebec
86d27b0a52ac v5 #7 chars 2581-2744 pages 2 | 31 words | Safety problems
8 chunks (structure) from documents/warranty-policy.pdf

These are the IDs of the explorer in Chunk deliberately. To trace one chunk, follow its record back. Take d9a5bd8ea531, the table chunk. Its source is documents/warranty-policy.pdf, version 5. Parse that file, and characters 371 to 1453 of the text are the heading and the whole table. They are in the section Warranty periods by product category, on page 1 of the PDF. Grace can open the printed policy on page 1 and find the same table. That is a trace: from an ID to a file, a version, a section, a page and the exact characters.

Common mistake

A page number that is only half right

Symptom: Grace traces chunk a4dc9ec49818 to the printed policy. The index says pages 1. Page 1 has the first four items of the list What is not covered, but the chunk has seven. The last three, from “Damage from the wrong fuel, voltage or chemicals.”, are on page 2. You saw them at the top of page 2 in the raw text of Parse and normalize.

Cause: the parser joins the items of a list into one block, and a block has one page: the page where it starts. A chunk takes the pages of its blocks. This list crosses a page boundary, so the chunk reports only its first page. In the chunk explorer, the dashed page line falls inside this chunk.

Fix: know what each field means, and trust the most exact one. In this project, pages means “the pages where the chunk’s blocks start”. The characters 1517 to 1967 are exact, and they always lead to the full text. When you cite a PDF for a person, give the section and say “page 1, continued on page 2” if the text goes on. Optional stretch work: give Block a last page as well, and let make_chunk use both. This changes no text and no chunk ID.

For a real assistant: every metadata field has a definition, and you should write it down. A page number that means “where it starts” is useful. If someone reads it as “where it all is”, it sends them to the wrong page.

When metadata is worth it, and what it costs

Metadata costs little to store: a few short fields per chunk. It costs discipline. Every source must bring a document ID, a version, its dates and an access label, and somebody must keep them right. Grace’s inventory and the metadata inside each file do this at Larkfield.

Stable IDs have one more cost: when a document changes, its chunks get new IDs. That is on purpose. An old answer still cites the old version’s chunk, and it says so. Module 6 removes old chunks from the index when a version is retired.

Keep all of it whenever answers must be checked, filtered by date or limited by access, which is every assistant over company documents. A search box for your own notes needs less.

Guided practice: build the index

1. Copy the index, the full command line and two tests from the snapshot. On macOS and Linux:

   cp ../nextia-learning-labs/C09/snapshots/end-of-m02/policy_assistant/store.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m02/policy_assistant/__main__.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m02/tests/test_metadata.py tests/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m02/tests/test_store.py tests/

   On Windows, in PowerShell:

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\policy_assistant\store.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\policy_assistant\__main__.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\tests\test_metadata.py tests\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\tests\test_store.py tests\

   The new __main__.py replaces yours. It keeps your parse command, adds ingest and chunks, and makes search and eval use the index.

2. Open policy_assistant/metadata.py. Find chunk_id and the Chunk record, and match its fields to the picture above.

3. Run the tests. You see 31 passed:

   python -m pytest

   test_metadata.py checks that a new version gets new IDs, and that the IDs do not depend on the other documents.

4. Build the index:

   python -m policy_assistant ingest

   36 documents, 185 chunks (rebuilt from scratch)
   index index.sqlite: 36 documents, 185 chunks | chunker structure

5. List the chunks of the warranty policy, as above:

   python -m policy_assistant chunks warranty-policy

6. Measure the search again, now over all 36 documents:

   python -m policy_assistant eval

   BM25 over structure chunks, top 5: hit@5 0.797 (59 questions with relevant passages)
     answerable       18 questions, hit@5 0.778
     contradiction     4 questions, hit@5 1.000
     exact-code       11 questions, hit@5 1.000
     injection-bait    3 questions, hit@5 0.333
     staff-only        7 questions, hit@5 1.000
     table             8 questions, hit@5 0.625
     version           8 questions, hit@5 0.625
   missed: Q05, Q12, Q13, Q26, Q28, Q29, Q32, Q34, Q37, Q56, Q57, Q63

   The baseline was 0.695. The table questions rose from 0.125 to 0.625: the PDF table is now in the index.

7. Ask the question that failed in Module 1:

   python -m policy_assistant search "Which pump kit fits the PW-2200?"

   The first result is now the PW-2200’s own parts table, from the PDF:

   1.  19.6787  [f80f7e98f870] product-care-pw2200 v2 | Parts and part numbers | public
      Parts and part numbers Part | Part number | Notes Pump kit | PK-2200-07 | fits the PW-2200 only Noz…

If step 5 prints No index at …, step 4 did not run in the project folder. Go to ~/projects/policy-assistant, and run step 4 again. index.sqlite is made again on every ingest, so you can delete it at any time.

Module practice: inspect the chunks of a policy with a table

Do this with the warranty policy first, then on your own with the delivery policy. Both have several headings and a table; the delivery policy is an HTML page.

1. Rebuild the index with fixed chunks, and list the warranty policy’s chunks:

   python -m policy_assistant ingest --chunker fixed
   python -m policy_assistant chunks warranty-policy

   521ff4295be4 v5 #0 chars 0-613 pages 1 | 100 words | What the warranty covers / Warranty periods by product category
   7ac8fe213b88 v5 #1 chars 614-1159 pages 1 | 100 words | Warranty periods by product category
   957c935ee2fa v5 #2 chars 1160-1708 pages 1 | 100 words | Warranty periods by product category / What is not covered
   174d797eb4bd v5 #3 chars 1709-2252 pages 1,2 | 100 words | What is not covered / How to make a claim
   0131e6fb8ee6 v5 #4 chars 2253-2744 pages 2 | 84 words | How to make a claim / Proof of purchase / Customers in Quebec / Safety problems
   5 chunks (fixed) from documents/warranty-policy.pdf

   Write down, for each chunk, its sections and pages. Which chunks hold part of the table (characters 409 to 1453)?

2. Run eval. Then rebuild with --chunker overlap, and run eval again. You get hit@5 0.864 and 0.881, as in Chunk deliberately.

3. Rebuild with the default chunker, so that the next module starts from it:

   python -m policy_assistant ingest

4. On your own: run python -m policy_assistant parse corpus/documents/delivery-policy.html to find its table. Then compare chunks delivery-policy with structure-aware chunks and with fixed chunks. For each chunk, write its source location. Which method keeps the delivery table whole?

Show the answers

Warranty policy, fixed chunks: #0, #1 and #2 hold parts of the table (409 to 1453); #0 starts before it and #2 ends after it. #3 is the only chunk on two pages. Every fixed chunk except #1 mixes two or more sections.

Delivery policy: its table is the block at characters 480 to 1193, in Delivery times and fees. The structure-aware chunk 617a6fe666fc (characters 274 to 1245, 167 words) holds the whole section with the whole table. The fixed chunks a577c4f431c0 (0 to 606), 30cff6982b67 (607 to 1177) and 886ece394acc (1178 to 1746) cut it into three. HTML has no pages, so every chunk says pages -; its location is the file, the version (4), the section and the characters.

Module check

The completion check of this module: trace every chunk back to a specific source location. For one chunk, you did it by hand above. For all of them, ask the code. For each chunk in the index, it parses the source file and checks that the chunk’s characters are in the chunk’s text.

Python session, in ~/projects/policy-assistant

from policy_assistant.store import Store
from policy_assistant.parse import parse
from policy_assistant.documents import CORPUS
store = Store("index.sqlite")
chunks = store.chunks()
bad = [c.chunk_id for c in chunks if parse(CORPUS / c.source).text[c.start:c.end] not in c.text]
print(len(chunks), "chunks,", len(bad), "not traceable")

185 chunks, 0 not traceable

Every chunk leads back to a file, a version and the exact characters of its source, and from there to its section and page. Then answer these:

See it end to endAnswer from versioned regulationsCase study

The regulations case study keeps the effective dates of every version, so that a question with a date finds the version in force.

Check your understanding

Why is a chunk's ID built from the document ID, the version and its character positions, and not from a counter or from its text alone?

Show the answer

A counter depends on every chunk before it. When one document left Larkfield’s collection, all 179 remaining chunks got a new counter, and the old citation [132] pointed to Exchanges instead of the return window. The text alone is not enough either. Version 3 and version 4 of the returns policy have the same Exchanges text. They must have different IDs so that an answer says which version it used. Document ID, version and position identify exactly one passage, and they do not change when other documents change.

Check your understanding

Trace chunk 3f7023270716: which file, version, section, page and characters? Use only the output of `chunks warranty-policy`.

Show the answer

3f7023270716 v5 #4 chars 1969-2276 pages 2 | 64 words | How to make a claim. The file is documents/warranty-policy.pdf (the last line of the output), version 5. The section is How to make a claim, on page 2. The source text is characters 1969 to 2276 of the parsed document. In the parse output of the PDF, these are the heading How to make a claim and its three paragraphs.

Check your understanding

A new document arrives with no access label. What does the project do, and why is that the safe choice?

Show the answer

make_chunk gives its chunks the label staff. A missing label then hides the passage from the public assistant, and a person notices and adds the right label. The other choice, public, would show a staff-only procedure to customers by mistake, and nobody would notice until it happened.

Check your understanding

Predict: Grace publishes version 6 of the warranty policy with one changed sentence in Safety problems. What happens to the IDs of the warranty chunks, and to old answers that cited version 5?

Show the answer

Every chunk of version 6 gets a new ID, because the version is part of the ID, even for sections with the same text. The chunks of version 5 keep their IDs, so old answers still point to the version-5 passages they used, and say so. Retiring version 5 from the index, so that it is no longer searched, is a step of its own: Module 6 does it.

Recap

• Metadata is the data about a chunk: document ID, version, effective dates, access label, language, type, source file, sections, pages and characters.
• A stable chunk ID is the first 12 characters of SHA-256 of “document ID | version | start | end”. The same passage gets the same ID on every ingestion; a new version gets new IDs; other documents cannot move it. Counter IDs all shifted when one document left.
• A missing access label becomes staff, so a mistake hides a passage instead of showing it.
• ingest builds index.sqlite (documents, chunks, keyword index, meta) from all 36 documents: 185 chunks. hit@5 rose from 0.695 to 0.797, and the PW-2200 question finds its own parts table first.
• Every chunk traces back to a file, a version, a section, a page and the exact characters. In this project, pages means the pages where the chunk’s blocks start.

Your project now matches the snapshot end-of-m02. In Module 3, search learns the meaning of words, not only the words, and filters by date and access label.

Next: Embeddings for search