Chunk deliberately

Lesson 5
50 min
Module 2: Ingest documents reproducibly

Lesson 5 of 18 · RAG: Building AI That Uses Your DataChunk deliberately

Course overview

1. Define the knowledge problem

2. Ingest documents reproducibly

3. Build and compare retrieval

4. Construct grounded answers

5. Evaluate and debug the pipeline

6. Maintain and protect the knowledge base

7. Case studies

What you will be able to do

Cut every parsed document into the pieces that search returns. Choose the cut on purpose, so that a table stays whole, every piece says where it comes from, and you know what your choice did to the search.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

Blocks, sections and character positions, from Parse and normalize.
hit@5, the share of questions with a relevant passage in the top 5, from A keyword-search baseline.

What you need on your computer

The project with parse.py and the parse command, from Parse and normalize: Guided practice. python -m pytest shows 20 passed.
A terminal in ~/projects/policy-assistant, with .venv active.

Tested with Python 3.12.13 on macOS (Apple silicon). The numbers in this lesson are a snapshot of that date. Tools and models change fast; the way to compare them lasts.

Index cards from a cookbook

You copy a big cookbook onto index cards, so that you can find a recipe fast. There are two ways to do it.

The fast way: write exactly 100 words on every card, then start the next card. It is quick, and every card is the same size. But a list of ingredients now starts on one card and ends on the next. And a card that starts with “bake for 40 minutes” does not say which cake.

The careful way: one card per recipe, with the name of the recipe at the top. The cards have different sizes, and you must find where each recipe starts. But every card stands alone: you can take it and use it alone.

The problem: search returns pieces, not documents

Search does not hand the assistant whole documents. It returns the top 5 pieces, and the model reads only those. A whole document is too big and too broad. The warranty policy has seven sections, and a question about sprinklers needs one row of one table. The baseline of Module 1 used the sections of the Markdown files as its pieces. Now all 36 documents are parsed, and the project needs one rule for all of them.

The simplest rule is the fast way: 100 words per piece. Here it is on the warranty policy. Look at where the second piece ends and the third begins:

…nd metal sheds, greenhouses | 5 years | glass and polycarbonate panels: 1 year
Watering | hoses, hose reels, sprinklers,

timers | 1 year | none
Small kitchen appliances | kettles, t…

The row for watering products is cut in two. One piece says “sprinklers” but not how long; the next says “1 year” but not for what. Grace’s question Q34 is “How long is the warranty on a garden sprinkler?”. No single piece holds the answer.

The idea: chunks, cut in one of three ways

A chunk is a piece of a document that the index stores, and that search returns as one result. Cutting documents into chunks is chunking. The passages of Module 1 were chunks of one kind: one Markdown section each. From now on, the passages that search returns are chunks. The project’s chunk.py has three ways to do it.

Fixed size: windows of 100 words, one after the other. The fast way of the index cards. It cuts wherever the 100th word falls: in a sentence, a list or a table row.
Overlapping: windows of 100 words, but each one starts 75 words after the last one, so 25 words repeat. A sentence that is cut at the end of one chunk is whole in the next one.
Structure-aware: one chunk per section, from its heading to the next heading. A section of more than 200 words is split between two blocks, never inside a table. The document’s title goes in front of every chunk, and the heading too when a chunk continues a section. The careful way of the index cards.

To the chunker, a “word” is any piece of text between two spaces. So the | between two table cells counts as a word too.

A tiny worked example: one sentence, three cuts

Take the first section of the warranty policy, What the warranty covers. Its last sentence is “A repaired or replacement item keeps the remaining period of the original warranty; it does not start a new one.” It sits at characters 257 to 369 of the parsed text. The table shows what each method does with it. The windows here are smaller, 50 words, so that the cut falls inside the sentence.

Method (size)	Chunk	Characters	What happens to the sentence
Fixed (50 words)	#0	0–317	ends with “…keeps the remaining period of”
Fixed (50 words)	#1	318–613	starts with “the original warranty; it does not start a new one”
Overlapping (50, 12 repeated)	#1	244–542	starts with “the remedy. A repaired or…”: the sentence is whole
Structure-aware	#0	27–369	the whole section, with “Larkfield warranty policy” in front

The fixed chunks cut the sentence in half. Neither half says that a repair does not start a new warranty. The overlap repeats the end of chunk #0 at the start of chunk #1, so chunk #1 holds the whole sentence. The price is 12 words stored twice. The structure-aware chunk holds the whole section, because it never cuts inside a section that fits.

Explore: the warranty policy, cut three ways

The explorer below shows the warranty policy as the parser reads it, with every chunk drawn over it. Each colour is one chunk; the colours alternate, and words in two chunks are orange. A mark with a number shows where each chunk starts, and ✂ means that it starts inside a sentence, a list or a table row. The dashed line is the page break of the PDF. The table under the document lists each chunk with its ID, its characters, its pages and its sections. Every chunk comes from the project’s own chunk.py.

Explore

Cut the warranty policy into chunks

Chunks: 5
Words per chunk: 84–100
The table: cut, 3 chunks
Headings: 4 of 5 chunks without

Fixed-size chunks (100 words): 5 chunks for the 484 words of the document. The table is cut into 3 chunks (#0, #1, #2), so some rows are cut away from their column names. 4 of 5 chunks start in the middle of a sentence, a list or a table row, and 4 hold parts of two or more sections.

one chunk (colours alternate)
the next chunk
in two chunks (repeated)
in no chunk

documents/warranty-policy.pdf · warranty-policy, version 5

#0Larkfield warranty policy

What the warranty covers

Our warranty covers defects in materials and workmanship under normal household use. The warranty period starts on the delivery date.

We repair the item, replace it, or refund it. We choose the remedy.

A repaired or replacement item keeps the remaining period of the original warranty; it does not start a new one.

Warranty periods by product category

Product category	Examples	Warranty period	Exceptions
Hand garden tools	spades, rakes, pruners, shears	2 years	none
Garden furniture	chairs, tables, loungers, benches	2 years	cushions and ✂ #1fabric covers: 1 year
Corded power tools	hedge trimmers such as the HT-550, corded lawn mowers	3 years	blades: not covered
Battery power tools	the Larkfield Volt 20 V range	3 years	batteries and chargers: 1 year
Pressure washers	PW-2200, PW-2400	2 years	pump: 3 years; nozzles and hoses: 1 year
BBQs and grills	the Ember GR-310	2 years	stainless-steel burners: 5 years
Sheds and greenhouses	wooden and metal sheds, greenhouses	5 years	glass and polycarbonate panels: 1 year
Watering	hoses, hose reels, sprinklers, ✂ #2timers	1 year	none
Small kitchen appliances	kettles, toasters, blenders	1 year	none
Lighting	indoor lamps, outdoor and solar lights	1 year	none
Textiles	parasols, hammocks, outdoor rugs	1 year	none
Larkfield Pro range	any product marked Pro	5 years	covers trade use

If a product fits two categories, the longer period applies.

What is not covered

Normal wear.
Misuse, accidents, and repairs or changes made by someone other than Larkfield.
Rust from leaving tools outside.
Frost damage, for example a pressure ✂ #3washer stored with water inside below 0 °C.
Page 2Damage from the wrong fuel, voltage or chemicals.
Commercial or trade use, except for the Larkfield Pro range.
Consumable parts, such as blades, nozzles, filters and cutting line, unless the table lists them.

How to make a claim

Contact the help desk with your order number and a photo or short video of the fault.

For power tools and pressure washers, also give the serial number from the label on the product.

Do not send the product back until we ask you to. We send a free return label ✂ #4if we need the product.

Proof of purchase

Your order number is your proof of purchase. For a gift, the gift receipt or the giver's order number is enough.

Customers in Quebec

Customers in Quebec also have the legal warranty under Quebec law, in addition to this warranty. See the French document Garantie légale au Québec.

Safety problems

If a product smokes, sparks, overheats, smells of burning or leaks gas, stop using it and contact us at once. Safety notices win over this policy.

The 5 chunks: ID, characters in the parsed text, PDF pages, words, sections
#	Chunk ID	Characters	Pages	Words	Sections
#0	`521ff4295be4`	0–613	1	100	What the warranty covers / Warranty periods by product category
#1	`7ac8fe213b88`	614–1159	1	100	Warranty periods by product category
#2	`957c935ee2fa`	1160–1708	1	100	Warranty periods by product category / What is not covered
#3	`174d797eb4bd`	1709–2252	1, 2	100	What is not covered / How to make a claim
#4	`0131e6fb8ee6`	2253–2744	2	84	How to make a claim / Proof of purchase / Customers in Quebec / Safety problems

Larkfield's warranty policy (the PDF, as parse.py reads it), cut by the project's three chunkers at three sizes. Every chunk, ID and position is real output of chunk.py.

Text description of the explorer

The explorer shows the warranty policy as the parser reads it: the title, seven headings, the paragraphs, the list What is not covered, and the table of 12 products with 4 columns. A dashed line marks the start of page 2, inside the list. Two controls: the method (fixed size, overlapping, structure-aware) and the size (fixed and overlapping: 50, 100 or 200 words; structure-aware: at most 100, 200 or 400 words). The readout shows the number of chunks, the words per chunk, what happens to the table and how many chunks start without a heading.

Fixed size, 100 words (the start): 5 chunks of 84 to 100 words. The table is cut into 3 chunks (#0, #1 and #2), and 4 of the 5 chunks start inside a block, without a heading. Chunk #1 ends in the watering row after “sprinklers,” and chunk #2 starts with “timers | 1 year | none”. At 50 words: 10 chunks, the table cut into 5. At 200 words: 3 chunks, the table cut into 2.

Overlapping, 100 words with 25 repeated: 7 chunks; the table is in 4 chunks, and the index stores 634 words for a document of 484. At 50 words: 13 chunks, the table in 6. At 200 words: 3 chunks, the table in 2.

Structure-aware, at most 200 words: 8 chunks of 18 to 200 words, one per section, plus one: the table section is longer than 200 words, so its last sentence, “If a product fits two categories, the longer period applies.”, becomes chunk #2, with the title and the heading in front. The table is whole in chunk #1, with its heading. At most 100 words: 9 chunks; the heading of the table section becomes a chunk of 8 words of its own, and the table stays whole (200 words) because a table is never split. At most 400 words: 7 chunks, one per section; the table and its last sentence are one chunk of 210 words.

With fixed chunks of 100 words, the table is cut into three chunks, and the cuts fall inside rows. With structure-aware chunks, the table is one chunk, with “Larkfield warranty policy” and its heading at the top. Try the sizes too. Fixed chunks cut the table at every size. Structure-aware chunks keep it whole at every size, even at 100 words, because the chunker never splits a table.

The same comparison as a still picture, with the real chunk IDs and positions:

Three columns. In the middle, the warranty policy as parsed, from top to bottom: the title, What the warranty covers, Warranty periods by product category, the table, What is not covered, the start of page 2 inside that list, How to make a claim, Proof of purchase, Customers in Quebec, Safety problems. On the left, 5 fixed chunks of 100 words with their IDs and characters: #0 0 to 613, #1 614 to 1159, #2 1160 to 1708, #3 1709 to 2252 on pages 1 and 2, #4 2253 to 2744. Two dashed orange lines show where fixed chunks start inside the table. On the right, 8 structure-aware chunks, one per section: #0 27 to 369, #1 371 to 1453 with the whole table, #2 1455 to 1515, #3 1517 to 1967, #4 1969 to 2276, #5 2278 to 2409, #6 2411 to 2579, #7 2581 to 2744. — The warranty policy mapped to its chunks: fixed size on the left, structure-aware on the right. Two fixed chunks start inside the table; structure-aware chunks follow the headings.

Why does the heading in front matter so much? Larkfield has two care guides that are almost the same: one for the PW-2200 and one for the PW-2400. Both have a section Specifications. A chunk that holds only that section’s text does not say which pressure washer it describes. With the title in front, the chunk starts “PW-2200 electric pressure washer: care and parts”. Now search and the model can tell the two apart.

Larkfield’s collection: does the choice change the search?

Amira cut all 36 documents in the three ways and ran Grace’s 59 questions with the keyword search of Module 1. With the default sizes, the collection gives 185 structure-aware chunks (a median of 34 words), 80 fixed chunks and 88 overlapping chunks (100 words each). Then she measured hit@5:

Chunks	hit@5	Words in the top 5 (median)
Structure-aware	0.797	197
Fixed, 100 words	0.864	432
Overlapping, 100 words	0.881	455

Tomás reads the hit@5 column. “Fixed chunks win. Take them, and forget the headings.” Priya reads the last column. A top 5 of fixed chunks holds a median of 432 words; a top 5 of structure-aware chunks holds 197. That is more than twice the text. Of course more text finds more evidence. The comparison is not fair.

A fair test gives every method the same amount of text. Take the results in rank order, and stop before the next chunk would pass the limit. Then measure hit with 100, 200 and 400 words of context. Amira did that:

Structure-aware
Fixed, 100 words
Overlapping, 100 words

Keyword search (BM25) over the 36 parsed documents, no filters. With the same amount of text, the three chunkers find the evidence for the same share of questions. The fixed line lies under the structure-aware line.

Text description of the chart

A line chart with three lines. The x axis is the context size: 100, 200 and 400 words taken from the top of the results. The y axis is hit: the share of the 59 questions with at least one relevant chunk in that context. Structure-aware chunks: 0.661, 0.746, 0.864. Fixed chunks of 100 words: 0.661, 0.746, 0.864, exactly the same, so the two lines lie on top of each other. Overlapping chunks: 0.678, 0.746, 0.831.

At an equal context size, the three chunkers tie. Structure-aware and fixed chunks give exactly the same hit at 100, 200 and 400 words. Overlapping chunks are one question higher at 100 words and two lower at 400. With 59 questions, one or two questions is not a difference. The course’s recorded runs with vector search, the method of Module 3, are close too. They measure recall: the share of all relevant passages that the context holds, not only the first one (Retrieval measures defines it). With 100, 200 and 400 words, recall is 0.681, 0.794 and 0.870 for structure-aware chunks. It is 0.590, 0.751 and 0.870 for fixed chunks, and 0.667, 0.768 and 0.862 for overlapping chunks. At 400 words, the three are equal.

Why the measure did not move here

Three reasons, and each one teaches something.

First, Larkfield’s documents are short: 6,520 words in 36 documents, about 180 words each. A window of 100 words often holds two or three whole short sections, so it rarely cuts the one sentence a question needs.

Second, hit@5 is a coarse measure. It counts a chunk as relevant when it covers any part of the right section. For Q34, the sprinkler question, the fixed chunker’s top result is chunk #0 of the warranty policy, 521ff4295be4. It covers the start of the table section, so the measure counts a hit. But it holds only the first row of the table and part of the second: no sprinklers, and no “1 year”. The answer is in two other chunks, cut in half. The measure cannot see that.

Third, the comparison at equal k hid the size. That is the general lesson: compare chunkers at an equal context size, not at an equal number of results.

So what does chunking change, if not this number? Three things, which matter from Module 4 on, when the model reads the chunks and cites them.

Tables stay whole. The model reads “Watering | hoses, hose reels, sprinklers, timers | 1 year | none” as one row, with the column names above it.
Headings travel with the text. Every chunk says which document and which section it comes from: the PW-2200, not “Specifications”.
Citations are precise. A structure-aware chunk is one section, so a citation points to Warranty periods by product category, page 1. A fixed chunk can cover four sections and two pages.

The course keeps structure-aware chunks of at most 200 words as its default. On a collection of long documents, the numbers can come out differently; measure on your own questions, at an equal context size.

When to use each method, and what it costs

Fixed size fits text with no structure, such as chat logs or transcripts. Every chunk has the same size, so a context budget is easy to plan. It costs you the structure: cuts in sentences, rows and lists, and chunks that do not say where they come from.
Overlapping protects a sentence that falls on a cut. It costs repeated text: the collection’s 88 overlapping chunks store 7,820 words for 6,520, and two results can show the same sentence.
Structure-aware needs a parser that finds the headings, which you now have. Its chunks have uneven sizes (from 18 to 200 words on the warranty policy), and some are very short. In return, each chunk is a unit that a person can check.

Common mistake

One long text, cut every 100 words

Symptom: Tomás’s first prototype joined the text of all 36 documents into one long text, and cut it every 100 words. “Chunking is chunking,” he said. Amira checked the 66 windows. 33 of them hold text from two documents. 12 of them mix a public document and a staff-only one. The first of these ends the public Contact and hours page and starts a staff procedure:

… in French is available Monday to Friday, 9:00 to 17:00 Eastern Time. || Courier claims for damaged or lost parcels Purpose We file claims with the courier …

Cause: the cuts ignore documents, sections and access labels. Such a window has no single document, version or access label. If it is labelled public, a public assistant can show staff text. If it is labelled staff, a public question loses the public half.

Fix: chunk each document on its own, and give every chunk the metadata of its document. The project’s three chunkers all do this, even the fixed one: a window never crosses into the next document. The script is reference/c09/m02/one_long_text.py; you do not need to run it.

For a real assistant: structure and permissions are part of the data, not decoration. A chunk that forgets its document cannot be cited, filtered by date or protected by an access label. The next lesson gives every chunk this metadata.

Guided practice: add the chunkers

chunk.py uses metadata.py, which holds the Chunk record and its ID. You copy both now; the next lesson explains metadata.py.

Copy the two files and the test from the snapshot. On macOS and Linux:

cp ../nextia-learning-labs/C09/snapshots/end-of-m02/policy_assistant/chunk.py policy_assistant/
cp ../nextia-learning-labs/C09/snapshots/end-of-m02/policy_assistant/metadata.py policy_assistant/
cp ../nextia-learning-labs/C09/snapshots/end-of-m02/tests/test_chunk.py tests/

On Windows, in PowerShell:

Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\policy_assistant\chunk.py policy_assistant\
Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\policy_assistant\metadata.py policy_assistant\
Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\tests\test_chunk.py tests\

Open policy_assistant/chunk.py. Find windows, fixed, overlapping and structure, and the three numbers at the top: SIZE = 100, OVERLAP = 25 and MAX_WORDS = 200.
Run the tests. You see 24 passed:
Terminal window
```
python -m pytest
```
One of the new tests checks that the fixed chunker cuts the table, and another that the structure-aware chunker keeps it whole.

Start Python with python, and cut the warranty policy in two ways:

from policy_assistant.parse import parse
from policy_assistant.chunk import fixed, overlapping, structure
doc = parse("corpus/documents/warranty-policy.pdf")
source = "documents/warranty-policy.pdf"
for c in fixed(doc, source):
    print(c.position, c.start, c.end, len(c.text.split()), c.sections)

for c in structure(doc, source):
    print(c.position, c.start, c.end, len(c.text.split()), c.sections)

Press Enter on the empty line after each loop. The output:

0 0 613 100 ('What the warranty covers', 'Warranty periods by product category')
1 614 1159 100 ('Warranty periods by product category',)
2 1160 1708 100 ('Warranty periods by product category', 'What is not covered')
3 1709 2252 100 ('What is not covered', 'How to make a claim')
4 2253 2744 84 ('How to make a claim', 'Proof of purchase', 'Customers in Quebec', 'Safety problems')

0 27 369 60 ('What the warranty covers',)
1 371 1453 200 ('Warranty periods by product category',)
2 1455 1515 18 ('Warranty periods by product category',)
3 1517 1967 80 ('What is not covered',)
4 1969 2276 64 ('How to make a claim',)
5 2278 2409 27 ('Proof of purchase',)
6 2411 2579 30 ('Customers in Quebec',)
7 2581 2744 31 ('Safety problems',)

Look at the cut in the table, and at the start of the structure-aware table chunk:

print(fixed(doc, source)[1].text[-120:])
print(structure(doc, source)[1].text[:160])

nd metal sheds, greenhouses | 5 years | glass and polycarbonate panels: 1 year
Watering | hoses, hose reels, sprinklers,
Larkfield warranty policy
Warranty periods by product category

Product category | Examples | Warranty period | Exceptions
Hand garden tools | spades, rakes, pr

Keep Python open for the next task.

Compare the two lists with the explorer at “Fixed size, 100 words” and “Structure-aware, at most 200 words”. They are the same chunks.

Your turn: overlapping chunks

Predict before you run: how many overlapping chunks does the warranty policy give, with 100 words and 25 repeated? Where does chunk 1 start: before or after character 613, where fixed chunk 0 ends? Then run:

for c in overlapping(doc, source):
    print(c.position, c.start, c.end, len(c.text.split()))

Show the answer

0 0 613 100
1 473 1006 100
2 878 1424 100
3 1297 1846 100
4 1709 2252 100
5 2129 2695 100
6 2537 2744 34

Seven chunks, two more than the fixed chunker. Chunk 1 starts at character 473, before 613. Its first 25 words are the last 25 words of chunk 0, “garden tools | spades, rakes, pruners, shears | 2 years | none Garden furniture …”. The table is still cut, now into four chunks, and some of its rows are stored twice. Overlap protects a sentence at a cut; it does not keep a table together. Leave Python with exit().

Knowledge check

Check your understanding

In your own words: what is a chunk, and why does the way you cut chunks matter even when the search measure does not change?

Show the answer

A chunk is a piece of a document that the index stores and that search returns as one result. The cut decides what the model reads later, and what a citation points to. A structure-aware chunk keeps a table row whole, carries its document title and heading, and covers one section, so a citation is precise. A fixed chunk can cut a row in two and mix four sections. hit@5 counts any chunk that touches the right section, so it cannot see these differences.

Check your understanding

Predict: chunker A gives hit@5 0.88 and its top 5 holds about 450 words. Chunker B gives hit@5 0.80 and its top 5 holds about 200 words. Is A better? What would you measure before you decide?

Show the answer

Not necessarily. A’s top 5 holds more than twice as much text, and more text finds more evidence. Measure both at an equal context size: take results in rank order up to 100, 200 and 400 words, and compare hit (or recall) at each size. On Larkfield’s collection, this is what happened. Fixed chunks had the higher hit@5 (0.864 against 0.797), and at an equal context size the two tied.

Check your understanding

Tomás wants to join all the documents into one text and cut it every 100 words, because it is simpler. Give two problems, one of them about access.

Show the answer

First, windows cross documents. In Larkfield’s collection, 33 of 66 windows hold text from two documents, so a chunk has no single document, version or section to cite. Second, 12 windows mix public and staff-only text. Whatever access label such a window gets, it is wrong for half of it. A public assistant can show staff text, or lose public text. Chunk each document on its own, and copy its metadata to every chunk.

Recap

A chunk is a piece of a document that the index stores and that search returns. Chunking can cut by size (fixed), by size with repeated words (overlapping), or by the document’s structure (one section per chunk, tables never split, title and heading in front).
On the warranty policy, fixed chunks of 100 words cut the table into 3 chunks and the watering row in two. Structure-aware chunks keep the table whole in one chunk with its heading.
At top 5, fixed chunks scored higher (hit@5 0.864 against 0.797), but their top 5 holds twice the text. At an equal context size, the three chunkers tie on this collection.
The measure did not move because the documents are short and hit@5 only checks the section. Chunking still changes what the model reads and what a citation points to.
Chunk each document on its own, never one long text: a chunk must keep one document, version and access label.

Next, every chunk gets a stable ID and its metadata, and the project gets an index on disk.

Next: Preserve metadata