Parse and normalize

Lesson 4
50 min
Module 2: Ingest documents reproducibly

Lesson 4 of 18 · RAG: Building AI That Uses Your DataParse and normalize

Course overview

1. Define the knowledge problem

2. Ingest documents reproducibly

3. Build and compare retrieval

4. Construct grounded answers

5. Evaluate and debug the pipeline

6. Maintain and protect the knowledge base

7. Case studies

What you will be able to do

Turn Larkfield’s PDF and HTML documents into clean text that search can use, and keep the page and the section of every part. Then the four documents that the Module 1 baseline could not read join the other 32.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

The keyword baseline and its hit@5 of 0.695, from A keyword-search baseline. It searched only the 32 Markdown documents.
The inventory of the collection: 32 Markdown, 2 HTML and 2 PDF documents, from Corpus and questions.

What you need on your computer

The project policy-assistant at the end of Module 1, in ~/projects/policy-assistant, with .venv active. python -m pytest shows 11 passed.
The labs repository in ~/projects/nextia-learning-labs, from Corpus and questions.
If your project is missing or broken, copy the snapshot end-of-m01. Use a snapshot has the steps.

Tested with Python 3.12.13 and pypdf 6.19.0 on macOS (Apple silicon). The Windows and Linux commands are not tested.

Reading a policy over the phone

A colleague phones Grace and asks about the warranty. Grace has the printed policy in front of her, and she reads it aloud. She does not read the line at the top of every page, or “Page 1 of 2”. At the table, she reads across one row: “Pressure washers: 2 years, the pump 3 years.” She does not read the first column from top to bottom, then the second. When a word breaks at the end of a line, “greenhou-” and “ses”, she says “greenhouses”. And when the colleague asks “Where does it say that?”, she answers “Page 1, under Warranty periods.”

A person does all of this without thinking. A program does none of it unless you tell it how.

The problem: four documents the search cannot read

The baseline of Module 1 searched the sections of the 32 Markdown documents. Four documents were left out. The returns policy version 4 and the delivery policy are HTML pages. The warranty policy and the PW-2200 care guide are PDF files. The cost was visible in the baseline. For the table questions, its hit@5 was 0.125: the warranty table holds the answers, and the warranty table is in the PDF. “Which pump kit fits the PW-2200?” returned the PW-2400 parts table first, because the PW-2200 guide is a PDF.

So why not take the text out of the files and search it? Python can do that in three lines. Amira tried it first with pypdf, a common library for PDF files. Here is what it gives for page 2 of the warranty policy. (You run the same lines in the guided practice, after you install pypdf.)

from pypdf import PdfReader
reader = PdfReader("corpus/formats/warranty-policy.pdf")
for line in reader.pages[1].extract_text().splitlines()[:10]:
    print(repr(line))

repr shows each line with its quotes, so you can see characters that do not print. The output:

'Larkfield  |  Larkfield warranty policy  |  Version 5, effective from 2026-01-01'
'Printed copies are not controlled. Check the help centre for the current version.'
'Page 2 of 2'
'\x7f'
'Damage from the wrong fuel, voltage or chemicals.'
'\x7f'
'Commercial or trade use, except for the Larkfield Pro range.'
'\x7f'
'Consumable parts, such as blades, nozzles, filters and cutting line, unless the table lists them.'
'How to make a claim'

Three problems are already on screen. The first three lines are the page header and footer, and they are on every page. '\x7f' is an invisible control character: pypdf returns the bullet of the list as this character. And the list What is not covered starts on page 1 and continues here, so the header now sits in the middle of the list. That is a page boundary: the place where one page ends and the next begins.

The table is worse. Here are lines 11 to 26 of page 1:

lines = reader.pages[0].extract_text().splitlines()
for line in lines[10:26]:
    print(repr(line))

'Warranty periods by product category'
' Product category'
'Examples'
'Warranty period'
'Exceptions'
'Hand garden tools'
'spades, rakes, pruners,'
'shears'
'2 years'
'none'
'Garden furniture'
'chairs, tables, loungers,'
'benches'
'2 years'
'cushions and fabric covers: 1'
'year'

Every cell is a line of its own, and some cells are two lines. The rows are gone. “2 years” appears twice, and only the order of the lines says which product it belongs to. Further down, lines[43:47] shows a word cut in two: 'Sheds and greenhou-' and 'ses'.

The HTML page has the opposite problem: too much text. Remove every tag, and you get everything that the browser shows around the policy:

import re
html = open("corpus/formats/warranty-policy.html", encoding="utf-8").read()
text = " ".join(re.sub(r"<[^>]+>", " ", html).split())
print(text[:330])
print(text[-250:])

Larkfield warranty policy | Larkfield Help Centre body { font-family: sans-serif; } .cookie { background: #eee; } window.dataLayer = window.dataLayer || []; dataLayer.push({"page": "warranty-policy"}); We use cookies to improve your visit. Accept all Settings Larkfield Shop Garden Home Help centre My account Help centre &rsaquo;
ety notices win over this policy. Related articles Contact us and opening hours Help centre: frequently asked questions Was this article helpful? Yes No &copy; 2026 Larkfield. All prices in Canadian dollars. Privacy &middot; Terms &middot; Contact us

Style rules, a script, the cookie banner, the menu, links to other articles and the footer: none of it is policy. Text like this is called boilerplate: the same text on every page of a site or a document.

The idea: parse into blocks, and keep the way back

Parsing means reading a file in its own format and turning it into a structure that your program understands. Normalizing means making the same content look the same, whatever file it came from. It gives the same characters, spaces and table form, and it removes what is not content. Together, they do what Grace did on the phone.

One thing must not get lost on the way: where each piece came from. This is called provenance: the file, the version, the section and the page of a piece of text. Grace’s “page 1, under Warranty periods” is provenance. Without it, the assistant cannot show its evidence, and nobody can check an answer against the real document.

The project’s parser gives every document the same form, whatever its format. A block is one part of the document: the title, a heading, a text (a paragraph or a list) or a table. Each block knows its section, its page in a PDF, and its exact position in the clean text.

Three files, three parsers, one result. The Markdown, HTML and PDF versions of the warranty policy give the same metadata, the same 20 blocks and the same text.

Text description of the diagram

Three source files: the warranty policy in Markdown (front matter, headings that start with ##, a table written with | characters), in HTML (meta tags, a cookie banner, a menu, the article and a footer) and in PDF (2 pages, a header and a footer on each page, a table drawn with lines). Each goes to its own parser. parse_markdown reads the front matter and removes the ** bold marks. parse_html keeps only the <article> and reads the metadata from the <meta> tags. parse_pdf drops the repeated lines, rebuilds the table rows and joins the cut words. All three give one ParsedDocument: the metadata (document ID, version, dates, access label, language), then 20 blocks, each a title, heading, text or table with its section, page and character positions. The text is the same 2,744 characters for all three files.

The folder corpus/formats/ exists for this reason: it holds four documents, each in all three formats. If the parser is right, the three files of a document give exactly the same text. A test in the project checks this for all four.

A tiny worked example: how the PDF parser decides

A PDF file does not store paragraphs, rows or headings. It stores instructions like “draw this text at this place, in this font”. pypdf can give each line with its position: the page, x and y. x is the distance from the left edge, and y the distance from the bottom, in points. A point is 1/72 of an inch, and Larkfield’s pages are 792 points high. pypdf also gives the font and its size.

With these numbers, five rules do the work. Read the picture from top to bottom: each group of real lines from the warranty policy, and the rule that handles it.

What pypdf reads from the warranty policy, and the five rules of parse.py. Every number is a real position from the file.

Text description of the diagram

Five groups of real lines, each with its page, x, y and font size, and the rule that parse.py applies to it. Group 1, the header and the footer: the header at y 742 and “Page 1 of 2” at y 40 on page 1, and the same lines on page 2. The rule: the same text in the same place on every page is a header or a footer; digits are ignored, so “Page 1 of 2” and “Page 2 of 2” both read “Page # of #”. They are dropped. Group 2: the title in size 17 at y 702, and “Warranty periods by product category” in size 13 bold. The rule: size 17 is the title, size 13 bold is a heading, and the heading names the section of the blocks below it. Group 3, a table row: “Hand garden tools” at x 76, “spades, rakes, pruners,” at x 186.6, y 518, “shears” at x 186.6, y 506, “2 years” at x 322.8 and “none” at x 407.9, all size 9.5. The rule: size 9.5 is a table cell; the same x is the same column; a line 12 points lower is the same cell, and a line more than 15 points lower starts a new row. The result: “Hand garden tools | spades, rakes, pruners, shears | 2 years | none”. Group 4: “Sheds and greenhou-” at y 350 and “ses” at y 338. The rule: a letter and a hyphen at the end of a line, and a lowercase letter next, make one word: “Sheds and greenhouses”. Group 5: the character U+007F at x 74 on page 2, and “Damage from the wrong fuel, …” at x 86. The rule: pypdf returns this font’s bullet as U+007F, so it becomes a list dash.

Take the table row, the hardest one. “spades, rakes, pruners,” is at y 518 and “shears” at y 506, in the same column. The difference is 518 − 506 = 12 points, one line of small text: the same cell. “Garden furniture” is at y 488, 30 points below “Hand garden tools”. That is more than 15, so a new row starts. The cells of one row are joined with |, the same form that the Markdown table gives.

The hyphen rule needs care, because product codes have hyphens too. “greenhou-” ends with a letter and a hyphen, and “ses” starts with a lowercase letter: one word. But a line that ends with “PK-” followed by “2200-07” stays apart, because a digit follows. A test in the project checks both cases.

One more rule handles a PDF with only one page. Nothing repeats, so rule 1 finds no header. The parser then also drops every line in the top and bottom 60 points of the page. That is where Larkfield’s template puts the header and the footer.

These rules fit Larkfield’s PDF files because all of them come from one template. They are not a general PDF reader.

Larkfield’s warranty policy, parsed

The project’s command parse prints the blocks that the parser finds. For the PDF of the warranty policy:

python -m policy_assistant parse corpus/formats/warranty-policy.pdf

pdf | doc_id=warranty-policy, title=Larkfield warranty policy, version=5, effective_from=2026-01-01, owner=Support (Grace), access=public, language=en, doc_type=policy
[    0-25   ] p1 title                                | Larkfield warranty policy
[   27-51   ] p1 heading What the warranty covers     | What the warranty covers
[   53-186  ] p1 text    What the warranty covers     | Our warranty covers defects in materials and workmanship under normal…
[  188-255  ] p1 text    What the warranty covers     | We repair the item, replace it, or refund it. We choose the remedy.
[  257-369  ] p1 text    What the warranty covers     | A repaired or replacement item keeps the remaining period of the orig…
[  371-407  ] p1 heading Warranty periods by product… | Warranty periods by product category
[  409-1453 ] p1 table   Warranty periods by product… | Product category | Examples | Warranty period | Exceptions Hand garde…
[ 1455-1515 ] p1 text    Warranty periods by product… | If a product fits two categories, the longer period applies.
[ 1517-1536 ] p1 heading What is not covered          | What is not covered
[ 1538-1967 ] p1 text    What is not covered          | - Normal wear. - Misuse, accidents, and repairs or changes made by so…
[ 1969-1988 ] p2 heading How to make a claim          | How to make a claim
[ 1990-2075 ] p2 text    How to make a claim          | Contact the help desk with your order number and a photo or short vid…
[ 2077-2173 ] p2 text    How to make a claim          | For power tools and pressure washers, also give the serial number fro…
[ 2175-2276 ] p2 text    How to make a claim          | Do not send the product back until we ask you to. We send a free retu…
[ 2278-2295 ] p2 heading Proof of purchase            | Proof of purchase
[ 2297-2409 ] p2 text    Proof of purchase            | Your order number is your proof of purchase. For a gift, the gift rec…
[ 2411-2430 ] p2 heading Customers in Quebec          | Customers in Quebec
[ 2432-2579 ] p2 text    Customers in Quebec          | Customers in Quebec also have the legal warranty under Quebec law, in…
[ 2581-2596 ] p2 heading Safety problems              | Safety problems
[ 2598-2744 ] p2 text    Safety problems              | If a product smokes, sparks, overheats, smells of burning or leaks ga…

The first line is the metadata, read from the PDF’s Keywords field. Then there is one line per block: its characters in the clean text, its page, its kind, its section and the start of its text. The header and the footer are gone. The table is one block, characters 409 to 1453, on page 1, in the section Warranty periods by product category. The list What is not covered is one block again, with dashes instead of U+007F.

Run the same command for warranty-policy.html and warranty-policy.md. You get the same 20 lines, with the same positions, but no page numbers: HTML and Markdown have no pages. For HTML, the metadata comes from the <meta> tags. The parser reads only the <article>, so the banner, the menu and the footer never reach the text.

Clean, but keep the way back

The parser also cleans every piece of text in the same way. The function clean() does three small things. It puts accented letters into one standard Unicode form, so “é” is always one character and not “e” plus an accent; the French documents need this. It turns non-breaking spaces into normal spaces. And it makes a run of spaces one space. A search for “légale” then finds “légale” in every document.

Cleaning changes the text, so the parser keeps the way back. Each block has start and end: its position in the clean text of the document, doc.text. For every block, doc.text[start:end] is exactly its text, and a test checks this. In the next two lessons, a chunk keeps the same kind of address, and that address leads back to the section and the page.

When to parse like this, and what it costs

Write rules like these when your documents come from a few known templates, as Larkfield’s do. Test them on the real files, and keep the raw problems in the tests. The project’s test_parse.py checks that the raw PDF text has the repeated header, the cut word and the U+007F bullet. It also checks that the clean text has none of them.

Do not use them for every PDF in the world. A scanned PDF holds pictures of pages and no text at all. It needs OCR (optical character recognition), which reads the letters from the picture. Pages with two columns, or tables inside tables, need a tool that understands the layout. Whatever you use, check its output on your own documents, as Amira did above.

The cost is code that you maintain. A new template from the printer can break a rule, and nobody notices unless a test fails. The parser itself is fast: it reads all 36 documents in less than a second. The cheapest fix is often not code at all: ask the owner for the source file. Grace’s team writes most policies in Markdown, and Markdown keeps the headings and the tables.

Guided practice: add the parser to the project

The parser is long (about 340 lines), so you copy it from the snapshot and read it, rather than type it.

In the terminal, go to ~/projects, and get the newest labs files:
Terminal window
```
cd ~/projects
git -C nextia-learning-labs pull
```
If you downloaded the labs as a ZIP file, download it again instead.

Go to the project folder, and copy the parser, its test and the new requirements from the snapshot end-of-m02. On macOS and Linux:

cd policy-assistant
cp ../nextia-learning-labs/C09/snapshots/end-of-m02/policy_assistant/parse.py policy_assistant/
cp ../nextia-learning-labs/C09/snapshots/end-of-m02/tests/test_parse.py tests/
cp ../nextia-learning-labs/C09/snapshots/end-of-m02/requirements.txt .

On Windows, in PowerShell:

cd policy-assistant
Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\policy_assistant\parse.py policy_assistant\
Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\tests\test_parse.py tests\
Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m02\requirements.txt .

Install the new package. requirements.txt now has pypdf==6.19.0:
Terminal window
```
python -m pip install -r requirements.txt
```
The last line is Successfully installed pypdf-6.19.0.
Start Python with python, and type the three blocks of The problem. Compare your output with the lesson. Leave Python with exit().
Open policy_assistant/parse.py in VS Code. Find parse_pdf and rebuild_table, and match them to the five rules of the picture above.
Open policy_assistant/__main__.py. Add this line below import sys:
policy_assistant/__main__.py
```
from pathlib import Path
```

Add this function below cmd_questions:

def cmd_parse(args) -> None:
    from .parse import parse

    doc = parse(Path(args.file))
    print(f"{doc.format} | " + ", ".join(f"{k}={v}" for k, v in doc.meta.items()))
    for b in doc.blocks:
        where = f"p{b.page}" if b.page else "  "
        print(f"[{b.start:>5}-{b.end:<5}] {where} {b.kind:7} {short(b.section, 28):28} | {short(b.text, 70)}")

In main(), add the parse command below the questions line. Then add "parse": cmd_parse to the dictionary on the last line. The end of main() becomes:

    s = sub.add_parser("questions"); s.add_argument("--kind")
    s = sub.add_parser("parse"); s.add_argument("file")
    s = sub.add_parser("search"); s.add_argument("query"); s.add_argument("--k", type=int, default=5)
    s = sub.add_parser("eval"); s.add_argument("--k", type=int, default=5)
    args = p.parse_args(argv)
    {"inventory": cmd_inventory, "questions": cmd_questions, "parse": cmd_parse, "search": cmd_search, "eval": cmd_eval}[args.command](args)

Run the tests:
Terminal window
```
python -m pytest
```
You see 20 passed: the 11 tests of Module 1 and the 9 new parser tests.

Parse the PDF, then the HTML and the Markdown versions:

python -m policy_assistant parse corpus/formats/warranty-policy.pdf
python -m policy_assistant parse corpus/formats/warranty-policy.html
python -m policy_assistant parse corpus/formats/warranty-policy.md

Check that the three outputs have the same 20 blocks, and that only the PDF has pages.

If step 4 or step 9 shows ModuleNotFoundError: No module named 'pypdf', your virtual environment was not active in step 3. Activate it (source .venv/bin/activate, or .venv\Scripts\Activate.ps1 on Windows), and do step 3 again.

Your turn: a PDF with one page

The care guide of the PW-2200 is a PDF with one page: corpus/formats/product-care-pw2200.pdf. Its raw text starts with the same kind of header and footer, and “Page 1 of 1”. Before you run parse on it, predict two things. Which rule removes the header and the footer, now that nothing repeats? And in which block is the pump kit PK-2200-07?

Show the answer

Rule 1 finds nothing, because a one-page PDF has no second page to compare with. The margin rule removes the header (at y 742), the footer and “Page 1 of 1” (both at y 40). They are in the top and bottom 60 points. The output starts with the title, and the pump kit is in the parts table:

pdf | doc_id=product-care-pw2200, title=PW-2200 electric pressure washer: care and parts, version=2, effective_from=2025-05-01, owner=Product team, access=public, language=en, doc_type=guide
[    0-48   ] p1 title                                | PW-2200 electric pressure washer: care and parts
[   50-64   ] p1 heading Specifications               | Specifications
…
[ 1000-1022 ] p1 heading Parts and part numbers       | Parts and part numbers
[ 1024-1255 ] p1 table   Parts and part numbers       | Part | Part number | Notes Pump kit | PK-2200-07 | fits the PW-2200 o…
[ 1257-1265 ] p1 heading Warranty                     | Warranty
[ 1267-1362 ] p1 text    Warranty                     | See the Larkfield warranty policy for the warranty periods of pressur…

This is the table that the Module 1 baseline could not see. Once this guide is in the index, the PW-2200 question can find its own parts table.

Common mistake

The footer that answers a question

Symptom: Amira tried the quick way first. She put the raw text of every PDF page and every HTML page in a keyword index, with the 32 Markdown documents. She asked “Which version of the policy is current?”. Two of the top 5 results were whole pages:

  1.   8.777  policy-governance.md | Versions and effective dates
  2.   6.622  returns-policy.v3.md | Who this policy is for
  3.   6.281  warranty-policy.pdf page 2 (whole page)
  4.   4.986  returns-policy.v4.html (whole page)
  5.   4.517  safety-notice-kt180.md | Which kettles are affected

The text of page 2 of the warranty policy says nothing about versions. The page came third because of its header and its footer: “Version 5, effective from 2026-01-01” and “Check the help centre for the current version.”

Cause: the header and the footer were part of the searched text. Keyword search counts every word, so the footer’s “current version” matched the question better than most real sections.

Fix: parse before you search, and remove the boilerplate. With the parsed documents, the same search gives the governance policy’s Versions and effective dates first. Then come the returns policies’ Who this policy is for and What changed in this version. No result is there because of a footer. The script is reference/c09/m02/naive_extraction.py in the course repository; you do not need to run it.

For a real assistant: boilerplate is not harmless noise. It is text that the search can match, and that the assistant can quote. Look at what your parser keeps before you build anything on it.

Knowledge check

Check your understanding

In your own words: what is provenance, and why must the parser keep it while it cleans the text?

Show the answer

Provenance is where a piece of text came from: the file, the version, the section and, for a PDF, the page. The assistant must show its evidence, and a person must be able to check it in the real document. Cleaning changes the text: it drops headers, joins cut words and rebuilds rows. So the parser keeps each block’s section, page and position in the clean text. Without them, a correct answer cannot be checked.

Check your understanding

Predict: a PDF line 'Pressure washers' is at x 76, y 300, and the next table line in the same column is at x 76, y 288. Are they one cell or two rows? And if the second one were at y 270?

Show the answer

300 − 288 = 12 points: one line of small text, so it is the same cell, and the two lines are joined. 300 − 270 = 30 points is more than 15, so the second line starts a new row. This is the rule of rebuild_table in parse.py.

Check your understanding

Tomás says: 'pypdf already gives us the text. We can index it as it is and save a week.' Give two concrete problems with the warranty policy that his plan would leave in the index.

Show the answer

Any two of these. The header and the footer are in the text of every page, so a question about versions matches a footer (the failure case). The table is one cell per line, so a row such as “Pressure washers | … | 2 years | pump: 3 years” is lost, and “2 years” is no longer tied to its product. “greenhou-” and “ses” are two words, so a search for “greenhouses” misses that row. The bullets are the invisible character U+007F. And a list that crosses a page boundary is cut by the header of the next page.

Recap

Parsing reads each format into one structure. Normalizing makes the same content look the same and removes boilerplate: headers, footers, banners and menus.
The raw text of Larkfield’s PDF has a header and a footer on every page, a table with one cell per line, a word cut in two and bullets read as U+007F. The raw HTML has a cookie banner, a menu and a footer.
parse.py uses each line’s page, position and font size. Repeated lines are dropped, and the size gives the title and the headings. The positions rebuild the table rows, and cut words are joined.
Every block keeps its provenance: its section, its page, and its position in the clean text. The Markdown, HTML and PDF versions of a document give the same text.
Rules like these fit a few known templates. Scanned or complex PDFs need other tools, and every parser needs tests on the real files.

Next, you cut the parsed documents into the pieces that search returns: chunks.

Next: Chunk deliberately