Sections and hierarchy

What you will be able to do

Turn a policy with two columns, a running header and footnotes into blocks that keep their place in the document. Each title, heading, paragraph and list item keeps its section path, and each footnote is joined to its mark. A search or a model then gets a paragraph together with what it belongs to.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The text layer, its lines with boxes and font sizes, and the reading order of two columns, from Larkfield’s policy, in reading order in Layout and meaning.
• The normalised representation (a Page with Lines), from The idea: three paths, one record in Text and OCR paths.
• Optional: how a chunk keeps its section in its metadata, from Preserve metadata in RAG: Building AI That Uses Your Data. That lesson uses the same policy.

What you need on your computer

• Your project ~/projects/doc-extract at the end of Module 2: with the virtual environment active, python -m pytest -q prints 63 passed. If your project is broken, copy the snapshot end-of-m02 as the snapshots README says.
• The course’s documents: python -m doc_extract.generate --check prints All 65 files match the course's checksums.
• The labs repository in ~/projects/nextia-learning-labs. Get its newest files with git -C ~/projects/nextia-learning-labs pull.

Every document in this course is made by the course’s generator. No real customer or shop is in it. Tested with Python 3.12.13 and pdfplumber 0.11.10 on macOS (Apple silicon). The Windows and Linux commands are not tested.

Get the lesson’s file

Each lesson of this module adds its own files from the snapshot end-of-m03. This lesson adds sections.py and its tests.

1. In the project folder, with the virtual environment active, copy the two files:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m03
   cp $S/doc_extract/sections.py doc_extract/
   cp $S/tests/test_sections.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m03"
   Copy-Item "$S\doc_extract\sections.py" doc_extract\
   Copy-Item "$S\tests\test_sections.py" tests\

2. Run the tests:

   python -m pytest -q

   The last line is 68 passed: the 63 tests of Module 2 and 5 new ones.

A manual read aloud on the phone

Imagine that a friend reads you a garden-tool manual over the phone. The friend reads every line from top to bottom. So you hear “Page 1” and “Larkfield, Returns policy, version 4” in the middle of a sentence, each time a new page starts. You hear a small note, “Business days and weekends both count”, but not which sentence it explains. And you never hear the headings as headings: “Return window” sounds like the start of a sentence. At the end, you have every word, but you do not know which rule belongs to which topic.

A good reader does three more things. They skip what repeats on every page. They say “Under Return window: …” before a paragraph. And when they reach a small number after a word, they read its note there.

The problem: flat text loses what a line belongs to

D054 is Larkfield’s returns policy: two pages, two columns on page 1, a running header and footer, and two footnotes. It is the same text that the RAG course searches. In Layout and meaning, you fixed the reading order: the left column first, then the right one. So every paragraph is whole. But what does the text look like when you join the lines in that order? Here are two places from the real output:

chemicals. | ¹ Business days and weekends both count. | ² Prices in Canadian dollars, tax included. | Larkfield · Made-up document for a course. | Page 1 | How to start a return
item and place a new order. | Larkfield · Returns policy · version 4 | Page 2 | What changed in this version | • Return window: 45 days instead of 30 days, | for orders delivered on or after 1 November

(The | shows where one line ends.) Three things went wrong, although each line is read perfectly:

1. The page furniture is inside the text. “Page 1”, “Page 2”, the running header and the footer sit between two rules of the policy. A chunk for a search can now start with “Page 2 What changed”.
2. The footnotes are far from their marks. “² Prices in Canadian dollars, tax included” is at the bottom of the left column. Its mark is after “12.95 dollars”, in the right column. A chunk with “You pay return shipping: 12.95 dollars²” does not say that the tax is included. An assistant that answers “12.95 dollars plus tax” is wrong.
3. The headings are plain lines. “How to start a return” looks like any other line. A chunk that holds “We email a return label within 1 business day” does not say which section it is from.

The RAG course met this problem with its own parser. A paragraph without its context is easy to find, and easy to misread.

The idea: blocks with a section path

Structure recovery turns the lines of a document into blocks, the units that a person sees. A block is a title, a heading, a paragraph, a list item or a footnote. Each block keeps four things:

• its kind;
• its text, with the lines of the paragraph joined;
• its section path: the headings above it, from the top, such as Returns policy > Return window. It works like the path of a file in folders: the path says where the block lives;
• its page and its box, so you can still show it on the page.

Two more ideas complete it. Page furniture is the text that repeats on every page and is not content: the running header, the footer and the page number. Printers call it “furniture” because it stays in place on every page, like the furniture of a room. Structure recovery drops it. And a footnote link joins each footnote to the block that holds its mark, so that the two travel together.

How can a program see that a line is a heading? It cannot read the meaning. But a digital PDF knows how each character is drawn: its font, its size and its weight. The project’s text layer keeps the size, and whether the font is bold. A person also sees a heading because it is bigger and bolder. So the rules use the same clues.

A tiny example: ten lines, sorted by hand

Here are ten real lines of D054’s first page, with the clues that the text layer gives. top is the distance from the top of the page in points (the page is 792 points high).

 8.0 pt  bold=False  top=   34  Larkfield · Returns policy · version 4
 8.0 pt  bold=False  top=   34  Page 1
20.0 pt  bold=True   top=   69  Returns policy
13.0 pt  bold=True   top=  125  Who this policy is for
11.0 pt  bold=False  top=  148  This policy applies to orders delivered in
11.0 pt  bold=False  top=  284  You can return an unused item within 45 days of
11.0 pt  bold=False  top=  300  delivery¹ for a full refund of the item price.
11.0 pt  bold=False  top=  544  • Live plants and seeds.
 7.5 pt  bold=False  top=  705  ¹ Business days and weekends both count.
 8.0 pt  bold=False  top=  746  Larkfield · Made-up document for a course.

First, find the body size. The middle value of all sizes on both pages, the median, is 11 pt: that is the normal text. Now apply the rules of sections.py to each line:

Clue	Rule	Lines it catches
In the top or bottom 8% of the page (above 63 pt or below 729 pt), and the same text on most pages, numbers ignored	furniture: drop	the header, “Page 1”, the footer
More than 5 pt above the body size (over 16 pt)	title; the path starts again	“Returns policy” (20 pt)
Bold, and more than 1 pt above the body size	heading; the path becomes title > heading	“Who this policy is for” (13 pt)
Starts with “•”	a new list item	“• Live plants and seeds.”
More than 1.5 pt smaller than the body, in the bottom 20% of the page, starts with a mark such as ¹	footnote	“¹ Business days…” (7.5 pt)
Anything else	part of a paragraph	the other 11 pt lines

Two lines of a paragraph join when the gap between them is small. Take “You can return an unused item within 45 days of” and “delivery¹ for a full refund of the item price.”. Their tops are 16 points apart: one paragraph. A bigger gap, or a jump to the other column, starts a new block. The heading “Return window” is above them, so their path is Returns policy > Return window. Last, the footnote “¹ Business days…” starts with ¹, and the paragraph holds ¹: the two are linked.

Notice why “Page 1” counts as furniture: “Page 2” is at the same place on the next page. With the digits ignored, both become “page #”, so they repeat.

What sections.parse finds on D054, page 1: every block by kind, in reading order. The page furniture is dropped, and each footnote is joined to the block that holds its mark.

Text description of the diagram

The picture shows the first page of the returns policy with a box around each block that the project finds. Grey, crossed-out boxes are the page furniture: the running header, “Page 1” and the footer. They are dropped. The title has a dark box. The headings have blue boxes, the paragraphs green ones and the five list items orange ones. Numbers give the reading order. The left column comes first (1 to 17), then the two footnotes at its bottom (18 and 19), then the right column (20 to 34). Two purple circles mark the footnote signs: ¹ after “delivery” in the Return window section, and ² after “12.95 dollars” in the Return shipping costs section. The footnotes at the bottom carry the same signs, and the project joins each one to its paragraph.

Recover the structure of Larkfield’s policy

1. Run the structure recovery on the returns policy:

   python -m doc_extract.sections D054

   returns-policy-v4.pdf: 2 pages, 37 blocks
   page furniture dropped: ['Larkfield · Made-up document for a course.', 'Larkfield · Returns policy · version 4', 'Page 1', 'Page 2']
     p1 # Returns policy
     p1 ## Who this policy is for
     p1    This policy applies to orders delivered in Canada.
     p1    The version that applies is the one in force on the date the order was...
     p1 ## Return window
     p1    You can return an unused item within 45 days of delivery¹ for a full r...  [notes: ¹]
     p1    The 45 days start on the delivery date shown in your account.
   ...
     p1 ## Items that cannot be returned
     p1    - Live plants and seeds.
     p1    - Products cut to length, such as hose or fabric sold by the metre.
   ...
     p1    footnote: ¹ Business days and weekends both count.  (section: Returns policy > Return window)
     p1    footnote: ² Prices in Canadian dollars, tax included.  (section: Returns policy > Return shipping costs)
     p1 ## How to start a return
   ...
     p1 ## Return shipping costs
     p1    You pay return shipping: 12.95 dollars² is taken from the refund.  [notes: ²]
   ...
     p2 ## What changed in this version
     p2    - Return window: 45 days instead of 30 days, for orders delivered on or ...
     p2    - New: a free return label for Larkfield Rewards members.
   paragraphs and list items found word for word: 25 of 25

   The dots ... mark lines left out here; your terminal shows all 37 blocks. # is the title, ## a heading, and - a list item. The screen cuts long blocks at 70 characters; the blocks keep their whole text.

2. Read the result. All 4 lines of furniture are gone. The policy has 1 title and 9 headings. All 25 paragraphs and list items match the text that the generator wrote, word for word. The two footnotes are linked. Each one now carries the section of its mark, Return window or Return shipping costs. Both sit at the bottom of the left column.

3. Open doc_extract/sections.py and find parse(). The rules of the tiny example are the if tests in its loop, in the same order: footnote, title, heading, then list item or paragraph.

The same blocks, drawn as a tree, show what the flat text lost:

The returns policy as a tree of sections, from the real output. Every paragraph has a path to the title, and each footnote hangs from the block with its mark.

Text description of the diagram

The title “Returns policy” is the root. Below it are 9 headings: 8 on page 1 and “What changed in this version” on page 2. Each heading shows how many paragraphs or list items it holds. “Return window” is opened to show its two paragraphs. A dashed line joins the first paragraph, which holds the mark ¹, to the footnote “Business days and weekends both count”. A second dashed line joins “Return shipping costs” to the footnote “Prices in Canadian dollars, tax included”, because the mark ² is in its first paragraph.

Sam asks: “Why keep the path in every block? The heading is already above it.” Mei answers: “Above it on the page, yes. But a search returns one block, alone. With its path, the block ‘The 45 days start on the delivery date’ still says that it is about the return window. In the RAG course, that path went into the metadata of each chunk, and the answers could cite the section.”

Grace adds what her team needs: when the assistant quotes 12.95 dollars, the footnote must come too. Her agents get questions about tax on the return fee every week.

Document boundaries: two documents in one file

A file is not always one document. D058, a warranty claim, holds a digital claim form on page 1 and a scanned receipt on page 2. The customer filled in the form, added the receipt and sent one PDF. Treat the file as one document, and the receipt’s total can end up in the claim’s text. Or the claim’s date ends up in the receipt’s fields.

So the second job of sections.py is to find where one document ends and the next begins. Its rule is simple. A new part starts when the way to read the page changes (text layer or picture). A new part also starts at a page that begins with a title in capital letters.

python -m doc_extract.sections D058 --parts

warranty-claim-wc-2026-0418.pdf: pages 1-1: text layer (a digital form or letter)
warranty-claim-wc-2026-0418.pdf: pages 2-2: picture (a scan: read it with OCR)

Each part then goes its own way: the form to the text layer, the receipt to OCR and the receipt contract. You met the routes of D058 in Document types. This rule finds only the edges that the routes and titles show. Two scanned receipts in one PDF have the same route. So the rule would need a better clue, for example a new “LARKFIELD” at the top of a page. The course’s set has no such file, so the rule was not tested on one.

When to recover structure, and what it costs

Recover structure for long documents that people read by section: policies, contracts, manuals, reports. A search or an assistant cuts these documents into pieces later. The cost is small: sections D054 takes about 0.1 seconds for two pages, with no model.

Do not use it for receipts. A receipt has no sections that matter. Its structure is the fields of the contract, which the rules of Text and OCR paths fill.

Know the limits of these rules:

• They need a text layer. The font size and weight come from the PDF. On a scanned policy, OCR gives only the height of each box (Line.size), and nothing says “bold”. The rules could use the height, but the course has no scanned policy, so this was not tested.
• They know one style of document. “Bold and bigger” means a heading in Larkfield’s policies. Another company may use numbered headings (“3.2 Refunds”), or capitals in the body size. Check the rules on a few pages of each new source before you trust them.
• A table is not paragraphs. D055, the Quebec warranty note, has a small table on page 2, and the paragraph rules read it as lines. You recover tables in the next lesson.

Common mistake

A header that appears only once is not dropped

Symptom: On D055, Garantie légale au Québec, the first block is the running header, before the title:

garantie-legale-quebec.pdf: 2 pages, 26 blocks
page furniture dropped: ['Page 1']
  p1    Larkfield · Garantie légale au Québec
  p1 # Garantie légale au Québec

Further down, the footer “Larkfield · Made-up document for a course.” also stays as a paragraph.

Cause: The furniture rule asks for the same text at the edge of most pages, and at least two. D055’s page 2 is a table, stored turned (/Rotate 90), and it has no header or footer. So the header appears once, and the rule keeps it. “Page 1” is dropped only because of its own pattern: “Page” and a number.

Diagnose: Look at the page furniture dropped line, and at the first and last blocks of each page. A short line at the top or bottom of a page that is not a heading is often furniture.

Fix: Do not lower the rule to “appears once”: then every line near the top of a one-page letter would go, the title too. Add what you know about the source instead. Larkfield prints “Larkfield ·” at the start of every running header and footer. A rule for that source, with a test, is safer than a looser general rule.

Consequence in AI work: furniture in the text costs little in one document. But in a search over hundreds of policies, the same header in every chunk makes the chunks look alike, and the search ranks them worse.

Your turn: the right column

Change one condition: take two blocks of the right column instead of the left one. Predict, for each block, its kind, its section path, its page and its notes:

• “Items over 30 kg: we arrange a pickup instead of a label. The pickup fee is 49 dollars, taken from the refund.”
• “You pay return shipping: 12.95 dollars² is taken from the refund.”

Then check it. Start python in the project folder and type:

from doc_extract import documents, sections, textlayer
blocks, _ = sections.parse(textlayer.read(documents.get("D054").path))
for b in blocks:
    if "12.95" in b.text or "pickup fee is 49" in b.text:
        print(b.kind, "|", " > ".join(b.path), "|", b.page, [round(v) for v in b.box], "|", b.notes)

Show the answer

paragraph | Returns policy > How to start a return | 1 [318, 247, 555, 289] | []
paragraph | Returns policy > Return shipping costs | 1 [318, 329, 548, 356] | ['² Prices in Canadian dollars, tax included.']

Both are paragraphs on page 1, and their boxes start at x = 318 points: the right column. The pickup fee is under “How to start a return”, although “Return shipping costs” is printed just below it. That heading comes after the paragraph, so it does not change the paragraph’s path. The return fee carries its footnote: a search that returns this block also returns “tax included”. The box of the first paragraph is three lines high (247 to 289 points); the second one is two lines high.

Knowledge check

Check your understanding

Explain: the returns policy is read in the right column order, and every paragraph is whole. Why is that not enough for a search or an assistant? Name the three things that the flat text loses.

Show the answer

The flat text keeps every word, but not what each line belongs to. First, the page furniture (“Page 1”, the running header, the footer) sits between the rules, so chunks can start or end with it. Second, the footnotes are far from their marks: the note “tax included” is not next to “12.95 dollars”. Third, the headings are plain lines, so a paragraph does not say which section it is from. Blocks with a kind, a section path and linked footnotes fix all three, and each block keeps its page and box.

Check your understanding

Predict: Larkfield publishes a one-page notice with a running header at the top. Will the project drop the header? What would you do?

Show the answer

No. The furniture rule needs the same text at the edge of at least two pages. On a one-page document nothing repeats, and only a “Page 1” line goes, by its own pattern. The header stays as the first block, as on D055. Do not make the general rule looser, or real titles near the top would go too. Add a rule for the source, such as “a line at the top that starts with ‘Larkfield ·’”, and a test with that notice.

Check your understanding

Practical check: run python -m doc_extract.sections D055. How many headings does the French note have, which section is its footnote linked to, and what happens to the table on page 2?

Show the answer

The note has a title, “Garantie légale au Québec”, and 5 headings. Four are on page 1: “Deux garanties différentes”, “Après la fin de la garantie Larkfield”, “Garantie prolongée” and “Comment faire une demande”. Page 2 has “Durées de garantie Larkfield (exemples)”. The footnote “¹ Loi sur la protection du consommateur, art. 37 et 38.” is linked to Garantie légale au Québec > Deux garanties différentes, where its mark is. All 7 paragraphs are found word for word. The table on page 2 comes out as short paragraphs, column by column. First “Code Produit”, then each code with its product name, then “Garantie Larkfield” and the four durations. So “PW-2200 Nettoyeur haute pression” is no longer next to its “2 ans”. Paragraph rules cannot read a table: the next lesson recovers tables.

Recap

• Text in the right reading order still loses what each line belongs to. Page furniture sits inside the text, footnotes are far from their marks, and headings are plain lines.
• Structure recovery turns lines into blocks: title, heading, paragraph, list item, footnote. Each block keeps its section path, page and box. The rules use the clues that a person uses: size, weight, position and a mark such as ¹.
• On D054, the project drops 4 lines of page furniture and finds 1 title and 9 headings. It links both footnotes to their sections and finds all 25 paragraphs and list items word for word, in about 0.1 seconds.
• A file can hold more than one document. D058 splits into a digital form and a scanned receipt, where the route changes.
• The rules know Larkfield’s style and need a text layer. A header that appears only once stays, and a table is not paragraphs.

Next, you recover a table that continues on a second page, and read the same price list as a spreadsheet.

Next: Tables and spreadsheets