A keyword-search baseline

What you will be able to do

Build the simplest search that can work, a keyword search, and give it a score on Grace’s questions. Every later change to the assistant must beat that score, or it is not worth its cost.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Larkfield’s collection, the inventory and Grace’s questions, from Corpus and questions.
• SQLite and Python’s sqlite3 module, from Set up a practice database in SQL and Data Preparation for AI.

What you need on your computer

• The project ~/projects/policy-assistant from the last lesson, with its .venv active. python -m pytest gives 4 passed.
• If you do not have it, copy the snapshot C09/snapshots/start of the labs, as in Set up the project.
• Nothing new to install: SQLite’s full-text search is part of Python.

No account and no payment. Nothing in this lesson calls a model.

Tested with Python 3.12.13 (SQLite 3.53.4 inside it) on macOS. The Windows and Linux steps are not tested.

The index at the back of a book

You want to know how long a kettle is covered. You do not read the whole shop manual. You turn to the index at the back, find “kettle”, and it lists pages 12 and 48. If “kettle” had pages 3, 5, 6, 8 and 11 to 90, the index would not help much: the word is everywhere. A word that appears on two pages tells you a lot. A word that appears on every page tells you nothing.

A keyword search works the same way. It looks up the words of the question, finds the passages that contain them, and puts the most useful passages first.

The problem: which section comes first?

In this module, the search unit is one section of a Markdown document: the text under one ## heading. The 32 Markdown documents have 154 sections. The 2 HTML and 2 PDF documents are not searched yet; reading them is Module 2.

Take Grace’s question “Which pump kit fits the PW-2200?” Count the sections that contain each of its words:

Word	which	pump	kit	fits	the	pw	2200
Sections (of 154)	3	4	3	1	106	8	2

Try the two obvious searches. “Sections with all the words” finds nothing: no section has all seven. “Sections with any of the words” finds 108 sections, because “the” is in 106 of them. Neither is useful. The search must rank the sections: put a few good ones first and the rest after. And a match on “fits” or “2200” must count for much more than a match on “the”.

The idea: count often, count rare words more

Two simple ideas give a good ranking.

• Term frequency. A section that uses a word of the question more often is more likely to be about it. The extra credit shrinks, though: a word five times is not five times as good as once.
• Rarity. A word that few sections contain is worth more. “Fits” (1 section) points at one place; “the” (106 sections) points everywhere. This weight is called the inverse document frequency: the fewer documents have the word, the higher it is.

A third, smaller idea: a short section with the word is more focused than a long one. So it gets a little more credit.

BM25 combines the three. “BM” means “best matching”. 25 is the number of the formula in a series of experiments by Stephen Robertson and his colleagues, in the 1990s. It is still the standard ranking of keyword search. Searching by the exact words of the question is called keyword search, or lexical search (“lexical” means “about words”).

A tiny worked example

Here are eight short sections, made up for this example. The question is “kettle refund”.

Section	Text	Words
A	Kettle notice: stop using the kettle.	6
B	Kettle care: descale it every month.	6
C	Refund timing: we send it in 2 days.	8
D	Gift cards: a gift card has no refund.	8
E	Price drops: ask for a refund of the difference.	9
F	Delivery: parcels arrive in 3 to 5 days.	8
G	Returns: return unused items within 30 days.	7
H	Rewards: points expire after 18 months.	6

Step 1, rarity. “kettle” is in 2 of the 8 sections, and “refund” in 3. So “kettle” is rarer, and its weight is about twice as high: 0.96 against 0.45. F, G and H contain neither word, so the search never scores them.

Step 2, term frequency. A says “kettle” twice, and B once. Each section gets a credit for the count of each word: 1 time gives about 1, 2 times gives about 1.4, not 2.

Step 3, multiply and add. For each word of the question, the score adds rarity × credit:

Section	“kettle” count	“refund” count	Score
A	2	0	0.96 × 1.45 = 1.38
B	1	0	0.96 × 1.08 = 1.03
C	0	1	0.45 × 0.96 = 0.43
D	0	1	0.45 × 0.96 = 0.43
E	0	1	0.45 × 0.91 = 0.41

• contains the rare word kettle
• contains only refund

BM25 scores of the five sections that contain a word of the query, computed by SQLite. F, G and H contain no query word and get no score.

Text description of the chart

A column chart with five columns.

• A has kettle twice and scores 1.38.
• B has kettle once and scores 1.03.
• C has refund once and scores 0.43.
• D has refund once and scores 0.43.
• E has refund once in a longer section and scores 0.41.

Read the chart from left to right. A beats B because it says “kettle” twice, but by 1.34 times, not 2 times. B beats C, D and E, because one rare word is worth more than one common word. And E is just below C and D: same count, but E is one word longer. These are the three ideas of BM25, in real numbers.

The formula, for the curious

For a question with words $w$, the score of a section is:

$$\text{score} = \sum_{w} \text{rarity}(w) \times \frac{f \times (k_1 + 1)}{f + k_1 \times \left(1 - b + b \times \frac{\text{length}}{\text{average length}}\right)}$$

$$\text{rarity}(w) = \ln \frac{N - n + 0.5}{n + 0.5}$$

Here $f$ is how often the word is in the section, $N$ is the number of sections (8), and $n$ is the number of sections with the word. SQLite uses $k_1 = 1.2$ and $b = 0.75$. For “kettle”, the rarity is ln(6.5 ÷ 2.5) = 0.96.

Section A has 6 words, and the average is 7.25. Its credit:

$$\frac{2 \times 2.2}{2 + 1.2 \times (0.25 + 0.75 \times 6 \div 7.25)} = 1.45$$

So A scores 0.96 × 1.45 = 1.38.

A word in more than half of the sections gets a negative rarity. SQLite makes it almost 0 instead. That is why “the” counts for almost nothing.

Where the search runs: SQLite full-text search

In the SQL course, Mei kept Larkfield’s tickets in SQLite: a database in one file, used through Python’s sqlite3 module. SQLite also has FTS5, a full-text search table, and Python’s sqlite3 includes it, so there is nothing to install.

An FTS5 table keeps an inverted index: for each word, the list of rows that contain it. It works like the index at the back of a book. A search reads only the lists of the question’s words, then ranks those rows with bm25().

The inverted index of the worked example. The search reads two short lists, not eight sections.

Text description of the diagram

The query “kettle refund” splits into its two words. The word “kettle” is in 2 of 8 sections, with rarity 0.96. It is 2 times in section A (Kettle notice) and 1 time in section B (Kettle care). The word “refund” is in 3 of 8 sections, with rarity 0.45. It is 1 time each in C (Refund timing), D (Gift cards) and E (Price drops). Sections F, G and H contain no word of the query, so the search never scores them.

FTS5 first cuts text into words with a tokenizer. The project’s tokenizer, unicode61 remove_diacritics 2, makes every word lowercase and removes accents, so “prolongée” matches “prolongee”. It also cuts “PW-2200” into “pw” and “2200”. The question is cut the same way, so a product code still matches. It does not reduce words to a root: “refunds” and “refund” are two different words.

Build the search

You add four things to the project: a function that splits each document into sections, the search, a scoring module and two commands. At the end, your project is the same as the snapshot end-of-m01.

Step 1: one passage per section

Open policy_assistant/documents.py. Add import re after import csv. Then add this at the end of the file:

policy_assistant/documents.py (add at the end)

@dataclass(frozen=True)
class Section:
    """One section of a Markdown document: its heading and its text (the Module 1 search unit)."""
    passage_id: str   # doc_id@vVERSION#heading-slug, for example returns-policy@v3#return-window
    doc_id: str
    version: str
    section: str
    text: str

    @property
    def sections(self) -> tuple:   # the same attribute as a chunk's, so evaluate.py can score both
        return (self.section,)


def slug(text: str) -> str:
    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")


def sections(info: DocInfo) -> list[Section]:
    """Split a Markdown document at its '## ' headings. Each section starts with the document title."""
    meta, body = read_markdown(info.path)
    out, heading, lines = [], None, []

    def close():
        if heading is not None:
            text = f"{info.title}\n## {heading}\n" + "\n".join(lines).strip()
            out.append(Section(f"{info.doc_id}@v{info.version}#{slug(heading)}", info.doc_id, info.version, heading, text))

    for line in body.splitlines():
        if line.startswith("## "):
            close()
            heading, lines = line[3:].strip(), []
        elif heading is not None:
            lines.append(line)
    close()
    return out

Each section keeps its document ID, its version and its heading. Its ID joins all three, for example returns-policy@v3#return-window, so a result always says where it came from. The document title goes first in the text: “Returns policy” then helps the search too. Check it:

python -c "from policy_assistant.documents import load_inventory, sections; print(sum(len(sections(d)) for d in load_inventory() if d.format == 'md'))"

Output

154

Step 2: the keyword index

Make the file policy_assistant/lexical.py:

policy_assistant/lexical.py

"""Keyword search with SQLite full-text search (FTS5) and BM25.

FTS5 keeps an inverted index: for every word, the items that contain it. BM25 scores an item higher
when it contains the query's words often (term frequency, with diminishing returns), when those words
are rare in the collection (inverse document frequency), and when the item is short.
SQLite's bm25() returns lower numbers for better matches; `search` turns them round, so that a
higher score is better, as with every other search method in this project.

The tokenizer `unicode61 remove_diacritics 2` lowercases words and removes accents (é -> e), so a
French question matches French text written with or without accents. It splits "PW-2200" into "pw" and
"2200": the query is split the same way, so exact product codes still match.
"""

import re
import sqlite3
from dataclasses import dataclass
from typing import Callable, Iterable


@dataclass(frozen=True)
class Hit:
    id: str
    score: float
    rank: int  # 1 = best


def words(text: str) -> list[str]:
    """The query's words, lowercased, without duplicates, in order."""
    return list(dict.fromkeys(w.lower() for w in re.findall(r"\w+", text)))


def fts_query(text: str) -> str:
    """'Which pump kit fits the PW-2200?' -> '"which" OR "pump" OR ... OR "pw" OR "2200"'.

    Every word is quoted, so FTS5 never reads a word as an operator (AND, OR, NOT, NEAR, -).
    OR means that an item needs only some of the words; BM25 ranks the items with more of them,
    and with rarer ones, higher.
    """
    return " OR ".join(f'"{w}"' for w in words(text))


class KeywordIndex:
    """An FTS5 table of (item ID, text) inside a SQLite database."""

    TABLE = "keyword_index"

    def __init__(self, conn: sqlite3.Connection):
        self.conn = conn
        conn.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS {self.TABLE} USING fts5("
                     "item_id UNINDEXED, text, tokenize = 'unicode61 remove_diacritics 2')")

    def add(self, items: Iterable[tuple[str, str]]) -> None:
        self.conn.executemany(f"INSERT INTO {self.TABLE} (item_id, text) VALUES (?, ?)", list(items))

    def remove(self, ids: Iterable[str]) -> None:
        self.conn.executemany(f"DELETE FROM {self.TABLE} WHERE item_id = ?", [(i,) for i in ids])

    def search(self, query: str, k: int = 5, allowed: Callable[[str], bool] | None = None) -> list[Hit]:
        """The k best items for the query. `allowed(item_id)` removes items before the top k is taken."""
        q = fts_query(query)
        if not q:
            return []
        rows = self.conn.execute(
            f"SELECT item_id, bm25({self.TABLE}) FROM {self.TABLE} WHERE {self.TABLE} MATCH ? "
            f"ORDER BY bm25({self.TABLE}), item_id", (q,)).fetchall()
        hits = [(item_id, -score) for item_id, score in rows if allowed is None or allowed(item_id)]
        return [Hit(item_id, round(score, 4), rank) for rank, (item_id, score) in enumerate(hits[:k], 1)]

Part	What it does
CREATE VIRTUAL TABLE … USING fts5(…)	Makes the full-text table. item_id UNINDEXED stores the ID without searching it.
fts_query()	Quotes every word and joins them with OR, so that a word such as “not” is never read as an operator
MATCH ?	Finds the rows with at least one of the words, through the inverted index
ORDER BY bm25(…)	Ranks them. SQLite’s bm25() gives lower numbers to better rows, so the code changes the sign.
item_id in ORDER BY	Breaks ties the same way every time, so results do not change between runs
allowed	Removes rows before the top k is taken. Module 3 uses it for access labels.

Check it with the worked example. Start Python with python, and type:

Python

import sqlite3
from policy_assistant.lexical import KeywordIndex, fts_query
index = KeywordIndex(sqlite3.connect(":memory:"))
index.add([
    ("A", "Kettle notice: stop using the kettle."),
    ("B", "Kettle care: descale it every month."),
    ("C", "Refund timing: we send it in 2 days."),
    ("D", "Gift cards: a gift card has no refund."),
    ("E", "Price drops: ask for a refund of the difference."),
    ("F", "Delivery: parcels arrive in 3 to 5 days."),
    ("G", "Returns: return unused items within 30 days."),
    ("H", "Rewards: points expire after 18 months."),
])
fts_query("kettle refund")
for hit in index.search("kettle refund", k=8):
    print(hit)

Output

'"kettle" OR "refund"'
Hit(id='A', score=1.3808, rank=1)
Hit(id='B', score=1.028, rank=2)
Hit(id='C', score=0.4336, rank=3)
Hit(id='D', score=0.4336, rank=4)
Hit(id='E', score=0.4114, rank=5)

These are the scores of the worked example. ":memory:" makes a database that lives only in memory, for this test. Type exit() to leave Python.

Step 3: score the search

The search needs a score of its own. The idea is the simplest one possible: for each answerable question, did the search put a relevant passage near the top?

The search returns its best results first. The top k are the first k results; Larkfield uses k = 5, because five short passages fit easily in a prompt. hit@k is 1 for a question when at least one of its relevant passages is in the top k, and 0 when none is. Its mean over all the answerable questions is the share of questions for which the evidence reached the model.

A worked example with four real questions and the search of this lesson:

Question	Rank of the first relevant section	hit@5
Q01, double charge	1	1
Q22, return on 1 December	2	1
Q05, Rewards points after two years	8	0
Q31, BBQ burners	not found (the answer is in the PDF)	0
Q14, student discount	no relevant passage: not scored	–

hit@5 = (1 + 1 + 0 + 0) ÷ 4 = 0.5. Q14 has no relevant passage, so it cannot count. hit@k is the simplest form of retrieval recall: how much of the needed evidence the search found. Module 5 adds finer measures, built on The precision–recall trade-off of the machine learning course.

Make the file policy_assistant/evaluate.py:

policy_assistant/evaluate.py

"""Measure retrieval against Grace's question set: the first answerability check.

A question lists its relevant passages (document, version, section). A search result is relevant when
it comes from one of them (same document and version, and it covers that section).

hit@k: 1 if at least one of the top k results is relevant, else 0. Its mean over the questions is the
share of answerable questions for which the evidence reached the top k: the number to beat.
Unanswerable questions have no relevant passage and are left out.
"""

import json
import statistics
from dataclasses import dataclass
from pathlib import Path

PROJECT = Path(__file__).resolve().parent.parent
QUESTIONS = PROJECT / "questions" / "questions.jsonl"


@dataclass(frozen=True)
class Question:
    id: str
    question: str
    language: str
    kind: str
    as_of: str
    public_ok: bool
    abstain: bool
    reference_answer: str
    keys: list
    must_not: list
    relevant: list            # [{"doc_id", "version", "section", "access"}]
    relevant_chunks: list
    wrong_version: list
    tags: list
    notes: str


def load_questions(path: Path = QUESTIONS) -> dict[str, Question]:
    qs = [Question(**json.loads(line)) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
    return {q.id: q for q in qs}


def covers(result, passage: dict) -> bool:
    return result.doc_id == passage["doc_id"] and result.version == passage["version"] and passage["section"] in result.sections


def retrieval_scores(q: Question, ranked: list, k: int) -> dict:
    if not q.relevant:
        return {"id": q.id, "kind": q.kind, "scored": False}
    first = next((i for i, r in enumerate(ranked) if any(covers(r, p) for p in q.relevant)), None)
    return {"id": q.id, "kind": q.kind, "scored": True, "hit": 1.0 if first is not None and first < k else 0.0,
            "first_rank": None if first is None else first + 1}


def summarise(rows: list[dict]) -> dict:
    scored = [r for r in rows if r["scored"]]
    return {"questions": len(scored), "hit": round(statistics.fmean(r["hit"] for r in scored), 3) if scored else None}


def by_kind(rows: list[dict]) -> dict:
    return {kind: summarise([r for r in rows if r["kind"] == kind]) for kind in sorted({r["kind"] for r in rows})}

covers() is the match rule: a result is relevant when its document, version and section are those of a relevant passage. The version matters: version 4 of the returns policy is not relevant to a question about an October delivery.

Step 4: the commands search and eval

Replace the whole of policy_assistant/__main__.py with this version. It keeps inventory and questions, and adds search and eval:

policy_assistant/__main__.py

"""The command line of the policy assistant.

    python -m policy_assistant inventory                    the documents, their versions, owners and access labels
    python -m policy_assistant questions [--kind KIND]      Grace's question set
    python -m policy_assistant search "QUESTION" [--k 5]    keyword search (BM25) over the sections of the Markdown documents
    python -m policy_assistant eval [--k 5]                 hit@k on the question set: the baseline to beat

Module 1 searches the 32 Markdown documents, one section at a time. The 4 HTML and PDF documents are
not searched yet: parsing them is Module 2.
"""

import argparse
import sqlite3
import sys

from .documents import load_inventory, sections
from .lexical import KeywordIndex


def short(text: str, n: int = 90) -> str:
    text = " ".join(text.split())
    return text if len(text) <= n else text[: n - 1] + "…"


def cmd_inventory(args) -> None:
    docs = load_inventory()
    print(f"{'document':34} {'v':>2} {'from':10} {'to':10} {'access':6} {'lang':4} {'type':9} {'format':6} owner")
    for d in docs:
        print(f"{d.doc_id:34} {d.version:>2} {d.effective_from:10} {d.effective_to or '-':10} {d.access:6} "
              f"{d.language:4} {d.doc_type:9} {d.format:6} {d.owner}")
    print(f"{len(docs)} documents: {sum(d.access == 'staff' for d in docs)} staff-only, "
          f"{sum(d.language == 'fr' for d in docs)} in French, {sum(d.format != 'md' for d in docs)} not Markdown")


def cmd_questions(args) -> None:
    from .evaluate import load_questions

    qs = [q for q in load_questions().values() if not args.kind or q.kind == args.kind]
    for q in qs:
        print(f"{q.id} {q.kind:15} {q.language} {q.as_of}  {short(q.question, 80)}")
    print(f"{len(qs)} questions")


def section_index():
    """An in-memory keyword index of every section of the Markdown documents."""
    all_sections = [s for d in load_inventory() if d.format == "md" for s in sections(d)]
    index = KeywordIndex(sqlite3.connect(":memory:"))
    index.add((s.passage_id, s.text) for s in all_sections)
    return index, {s.passage_id: s for s in all_sections}


def cmd_search(args) -> None:
    index, by_id = section_index()
    for hit in index.search(args.query, args.k):
        s = by_id[hit.id]
        print(f"{hit.rank}. {hit.score:>8.4f}  {hit.id}")
        print(f"   {short(s.text.split(chr(10), 2)[-1], 100)}")
    print(f"BM25 over {len(by_id)} sections of the Markdown documents, top {args.k}")


def cmd_eval(args) -> None:
    from .evaluate import by_kind, load_questions, retrieval_scores, summarise

    index, by_id = section_index()
    rows = []
    for q in load_questions().values():
        ranked = [by_id[h.id] for h in index.search(q.question, args.k)]
        rows.append(retrieval_scores(q, ranked, args.k))
    s = summarise(rows)
    print(f"BM25 over sections, top {args.k}: hit@{args.k} {s['hit']:.3f} ({s['questions']} questions with relevant passages)")
    for kind, k in by_kind(rows).items():
        if k["questions"]:
            print(f"  {kind:15} {k['questions']:>3} questions, hit@{args.k} {k['hit']:.3f}")
    print(f"missed: {', '.join(r['id'] for r in rows if r['scored'] and not r['hit'])}")


def main(argv=None) -> None:
    p = argparse.ArgumentParser(prog="python -m policy_assistant", description="Larkfield's policy assistant")
    sub = p.add_subparsers(dest="command", required=True)
    sub.add_parser("inventory")
    s = sub.add_parser("questions"); s.add_argument("--kind")
    s = sub.add_parser("search"); s.add_argument("query"); s.add_argument("--k", type=int, default=5)
    s = sub.add_parser("eval"); s.add_argument("--k", type=int, default=5)
    args = p.parse_args(argv)
    {"inventory": cmd_inventory, "questions": cmd_questions, "search": cmd_search, "eval": cmd_eval}[args.command](args)


if __name__ == "__main__":
    sys.exit(main())

The index lives in memory and is built again for each command. With 154 short sections, that takes a moment. Module 2 keeps the index in a file.

Step 5: the tests

Two test files keep what you built. Make tests/test_lexical.py:

tests/test_lexical.py

import sqlite3

from policy_assistant.lexical import KeywordIndex, fts_query, words


def make(items):
    index = KeywordIndex(sqlite3.connect(":memory:"))
    index.add(items)
    return index


def test_query_quotes_every_word_and_splits_codes():
    assert words("Which pump kit fits the PW-2200?") == ["which", "pump", "kit", "fits", "the", "pw", "2200"]
    assert fts_query("NOT a code") == '"not" OR "a" OR "code"'   # NOT is a word here, not an operator


def test_bm25_prefers_the_rare_word():
    index = make([("a", "the kettle and the toaster"), ("b", "the kettle"), ("c", "the toaster and the lamp")])
    hits = index.search("kettle lamp", k=3)
    # "lamp" occurs once in the collection, "kettle" twice: the item with the rare word wins.
    assert hits[0].id == "c"
    assert [h.rank for h in hits] == [1, 2, 3]
    assert hits[0].score > hits[1].score


def test_exact_code_and_accents():
    index = make([("pw", "Pump kit | PK-2200-07 | fits the PW-2200 only"), ("fr", "Larkfield ne vend pas de garantie prolongée.")])
    assert index.search("PK-2200-07", k=1)[0].id == "pw"
    assert index.search("garantie prolongee", k=1)[0].id == "fr"   # accents removed by the tokenizer


def test_allowed_is_applied_before_the_top_k():
    index = make([("staff", "goodwill credit 25 dollars"), ("public", "store credit never expires")])
    hits = index.search("credit", k=1, allowed=lambda i: i == "public")
    assert [h.id for h in hits] == ["public"]


def test_remove():
    index = make([("x", "deleted text"), ("y", "kept text")])
    index.remove(["x"])
    assert [h.id for h in index.search("text", k=5)] == ["y"]

And tests/test_baseline.py, which fixes the number of this lesson:

tests/test_baseline.py

"""Module 1's keyword baseline: BM25 over the sections of the Markdown documents. The number to beat."""

import sqlite3

from policy_assistant.documents import load_inventory, sections
from policy_assistant.evaluate import load_questions, retrieval_scores, summarise
from policy_assistant.lexical import KeywordIndex


def test_sections_of_a_document():
    info = next(d for d in load_inventory() if d.doc_id == "returns-policy" and d.version == "3")
    secs = sections(info)
    assert [s.section for s in secs][:2] == ["Who this policy is for", "Return window"]
    assert secs[1].passage_id == "returns-policy@v3#return-window"
    assert secs[1].text.startswith("Returns policy\n## Return window\n")


def test_baseline_hit_at_5():
    all_sections = [s for d in load_inventory() if d.format == "md" for s in sections(d)]
    index = KeywordIndex(sqlite3.connect(":memory:"))
    index.add((s.passage_id, s.text) for s in all_sections)
    by_id = {s.passage_id: s for s in all_sections}
    rows = [retrieval_scores(q, [by_id[h.id] for h in index.search(q.question, 5)], 5) for q in load_questions().values()]
    assert len(all_sections) == 154
    assert summarise(rows)["hit"] == 0.695      # 41 of 59 questions with relevant passages

Run all the tests:

python -m pytest

Output (end)

tests/test_baseline.py ..                                                [ 18%]
tests/test_first_run.py ....                                             [ 54%]
tests/test_lexical.py .....                                              [100%]

============================== 11 passed in 0.06s ==============================

If you see 11 passed, your project is the same as the snapshot end-of-m01.

Search Larkfield’s documents

Ask Grace’s exact-code question:

python -m policy_assistant search "Which pump kit fits the PW-2200?"

Output

1.  20.3759  product-care-pw2400@v1#parts-and-part-numbers
   | Part | Part number | Notes | | --- | --- | --- | | Pump kit | PK-2400-03 | fits the PW-2400 only …
2.  18.9867  warranty-claims-procedure@v3#pressure-washer-pumps
   - If the pump of a PW-2200 fails within 3 years, send the pump kit PK-2200-07 instead of a new pres…
3.   6.2832  product-care-pw2400@v1#winter-storage
   Drain all water from the pump and hoses before temperatures fall below 0 °C. Store it indoors, abov…
4.   5.8360  product-care-pw2400@v1#using-it-safely
   Never run the pressure washer for more than 2 minutes with the trigger released: the pump overheats…
5.   3.9398  refund-approval-procedure@v2#when-this-procedure-applies
   Refunds for normal returns are approved automatically after inspection. Use this procedure for manu…
BM25 over 154 sections of the Markdown documents, top 5

The first result is the parts table of the wrong pressure washer: the PW-2400. Its table has “pump”, “kit”, “fits”, “pw” and also “2200”, because one nozzle set fits both models. The guide of the PW-2200 is a PDF file, so this search does not have it. The second result does name PK-2200-07, but it is a staff-only procedure, which a customer must never see. The search did its job with the words that it had; the right document was missing. That is a missing source, and no ranking formula can fix it.

The first score: the baseline

Now score the search on all of Grace’s questions:

python -m policy_assistant eval

Output

BM25 over sections, top 5: hit@5 0.695 (59 questions with relevant passages)
  answerable       18 questions, hit@5 0.722
  contradiction     4 questions, hit@5 1.000
  exact-code       11 questions, hit@5 1.000
  injection-bait    3 questions, hit@5 0.333
  staff-only        7 questions, hit@5 1.000
  table             8 questions, hit@5 0.125
  version           8 questions, hit@5 0.500
missed: Q05, Q11, Q12, Q13, Q23, Q26, Q27, Q29, Q31, Q32, Q33, Q34, Q35, Q36, Q37, Q56, Q57, Q63

For 41 of the 59 answerable questions, a relevant section was in the top 5: hit@5 = 0.695. This is the baseline: the score of the simplest reasonable method, which every later change must beat. A new method that does not beat 0.695 on these questions is not worth its cost.

The keyword baseline by kind of question: BM25 over the 154 sections of the 32 Markdown documents, top 5, no filters. The dashed line is the score over all 59 questions.

Text description of the chart

Seven rows, on a scale from 0 to 1. A dashed line marks 0.695, the score over all 59 questions.

• Answerable: 13 of 18, 0.722.
• Contradiction: 4 of 4, 1.000.
• Exact code: 11 of 11, 1.000.
• Injection bait: 1 of 3, 0.333.
• Staff only: 7 of 7, 1.000.
• Table: 1 of 8, 0.125.
• Version: 4 of 8, 0.500.

Keyword search is perfect on exact codes and staff questions: their words are distinctive. It is weak on tables and versions. Why? Look at the 18 misses one by one, and the reason is almost always the same:

• 14 misses need an HTML or PDF file: the warranty table (PDF), the delivery table (HTML) or version 4 of the returns policy (HTML). These are missing sources, as with the PW-2200.
• 2 misses use other words: Q05 asks about a customer who “has not bought anything” for two years. The policy speaks of 18 months without buying. Q12 says “died” and “delivered”, and the policy says “dies” and “delivery”. For this search, those are different words. The right sections are at ranks 8 and 6.
• 2 misses cross languages: Q13 asks in English, and the only answer is in a French document. Q63 asks in French, and the answer is in English. Keyword search matches words, not meanings.

And look at a hit that is not quite right. For Q22, the first result is the expired holiday notice, and the right section is second. The search has no idea of dates yet; Module 3 adds a date filter.

When to use keyword search

Use it as the first search of any document assistant, and keep it. It is exact on product codes, part numbers and names, and it is easy to explain. On this computer, one query over the course’s final index takes about 0.26 ms. It needs no model, no download and no key.

Do not rely on it alone when people use other words than the documents, or another language. “Garden sprinkler” will not find a table row that says “Watering”. Module 3 adds search by meaning for that, and compares both on the same questions.

What it costs: almost nothing in time or money. The real cost is the misses that you do not see: a search always returns five results, even when none of them is relevant.

Common mistake

A vector database before a baseline

Symptom: Tomás says: “Keyword search is from the 1990s. Let us start with a vector database, it understands meaning.” Two weeks later, the assistant runs on a new database service. Omar asks, “Is it better than before?” Nobody can say, because there is no “before”.

Cause: a new method was built before the simplest method was measured. Without a baseline, there is nothing to compare with. The team also does not know where the misses come from. Here, 14 of the 18 misses are documents that no search reads yet. A vector database would miss them too.

Fix:

1. Build the simplest search first, and measure it on fixed questions: hit@5 0.695.
2. Read the misses, and find their cause before you choose a fix.
3. Change one thing at a time, and measure it the same way.

What it means for a real AI service: a baseline makes every later decision cheap to check. Module 3 measures search by meaning against this baseline, and keyword search still wins on some questions there.

Your turn: a larger top k

Predict first: if the assistant took the top 10 sections instead of the top 5, which of the 18 misses would become hits? Then run:

python -m policy_assistant eval --k 10

Show the answer

Output

BM25 over sections, top 10: hit@10 0.729 (59 questions with relevant passages)
  answerable       18 questions, hit@10 0.833
  contradiction     4 questions, hit@10 1.000
  exact-code       11 questions, hit@10 1.000
  injection-bait    3 questions, hit@10 0.333
  staff-only        7 questions, hit@10 1.000
  table             8 questions, hit@10 0.125
  version           8 questions, hit@10 0.500
missed: Q11, Q13, Q23, Q26, Q27, Q29, Q31, Q32, Q33, Q34, Q35, Q36, Q37, Q56, Q57, Q63

Only Q05 and Q12 become hits: their sections were at ranks 8 and 6. The 14 missing sources and the 2 language misses stay. A larger k cannot find a document that the search does not have, and twice the passages make every prompt longer. The fix for most misses is to read the HTML and PDF files, in Module 2.

Module practice

Worked example: Grace’s inventory and questions

Grace’s two files are the model for your practice. Each document is one row of corpus/inventory.csv:

corpus/inventory.csv (the header and one row)

file,doc_id,title,version,effective_from,effective_to,owner,access,language,doc_type,format
documents/returns-policy.v3.md,returns-policy,Returns policy,3,2025-03-01,2026-10-31,Support (Grace),public,en,policy,md

Each question names its kind, its date, its answer and where the answer is. For three of her questions, Grace wrote this, and then checked each one with search:

Question	Kind	Relevant passage	Answer	Found by search?
“A customer was charged twice for the same order. How quickly do we refund the extra charge?”	answerable	refunds-and-payments v3, Double charges	Within 3 business days after Larkfield confirms the double charge.	Yes, rank 1
“Which pump kit fits the PW-2200?”	exact code	product-care-pw2200 v2 (PDF), Parts and part numbers	PK-2200-07	Only the staff procedure; the guide is a PDF
“Do we offer a student discount?”	unanswerable	none	The documents do not say.	Five results, none relevant

The last row is the important one. The search returns five sections for any question, so a list of results proves nothing. Only a person who knows the documents can say that none of them answers.

Your task: a corpus inventory and fifteen questions

Choose a small collection that you know: 5 to 10 documents from your work, a club, a public help centre, or Larkfield’s own collection.

1. Make an inventory, my_inventory.csv, with the columns of Grace’s file. Give each document an owner, a version, an effective date and an access label. If you do not know a value, write unknown, not a guess.
2. Write fifteen representative questions in my_questions.md. Write them in the words that real users use, not in the words of the documents.
3. Mix the kinds. Write at least:
   • 3 questions that the documents cannot answer,
   • 2 that depend on a date or a version,
   • 2 with a code or a name,
   • 1 with the answer in a table.
4. For each question, write the answer, and the document and section that hold it. For an unanswerable question, write “none”.
5. If you used Larkfield’s collection, run python -m policy_assistant search "…" for each question. Note whether the relevant section is in the top 5.

Show a model answer (Larkfield, five of the fifteen)

Question	Kind	Relevant passage	Answer	Top 5?
“How long do we keep a customer’s order history?”	answerable	privacy-policy v3, How long we keep it	Orders and invoices: 7 years, as tax law requires.	Yes, rank 1
“Can a customer return an opened bag of soil?”	answerable	returns-policy v3, Items that cannot be returned	No: opened bags of soil, fertiliser or garden chemicals cannot be returned.	Yes, rank 1
“An order arrived on 2 November 2026. Does a Rewards member pay for the return label?”	version	returns-policy v4 (HTML), Return shipping costs	No: from version 4, Rewards members get a free return label.	No: version 4 is an HTML file
“Do we match the price of another shop?”	answerable	refunds-and-payments v3, Price adjustments; help-centre-faq v7, Do you match other shops’ prices?	No. If Larkfield lowers its own price within 14 days after delivery, the customer can ask for the difference.	Yes, ranks 1 and 2
“Do we offer gift wrapping?”	unanswerable	none	The documents do not say.	Five results (the kettle offer, gift cards, payments, Rewards), none relevant

A good set has questions that the search gets wrong for a known reason, and questions with no answer at all. A set that the search gets 100% right is too easy to teach you anything.

Module check

Check your understanding

Retrieval: BM25 ranks sections with term frequency and rarity. What does each one reward, and why does the word 'the' count for almost nothing?

Show the answer

Term frequency rewards a section that uses a word of the question more often. The returns shrink: twice is worth about 1.4 times once, not 2 times. Rarity (inverse document frequency) rewards words that few sections contain, because they point at a few places. “The” is in 106 of the 154 sections, more than half, so its rarity is about 0 and it adds almost nothing to any score. Shorter sections also get a little more credit.

Check your understanding

Predict: you add the PW-2200 guide (now a PDF) as a Markdown file to the search. What happens to the results for 'Which pump kit fits the PW-2200?', and to hit@5 for exact-code questions?

Show the answer

Its section Parts and part numbers has the words “pump”, “kit”, “fits”, “pw” and “2200”. In a test, it ranked first (19.24), above the PW-2400 table (17.21). A customer-facing answer could then come from a public passage. hit@5 for exact-code questions cannot rise: it is already 1.000, because the staff procedure counted as relevant. This shows that hit@5 does not see everything: a hit can still be the wrong passage for the audience. Module 2 reads the PDF, and Module 3 adds the access filter.

Check your understanding

Completion check: which of these questions can Larkfield's documents not answer? For each answerable one, name the document. (1) Can customers pick up an online order in a Larkfield store? (2) Does Larkfield sell extended warranties? (3) What is the weight limit of the AquaFlow HR-50 cart? (4) How long is the pump of a PW-2200 covered? (5) Can a customer return a gift card? (6) Do we offer a student discount?

Show the answer

The documents cannot answer (1) store pickup, (3) the HR-50 weight limit and (6) a student discount: no document states them. Search still returns five sections for each. For (1), they are about store credit and installation. For (3), they come from the AquaFlow supplier sheet. For (6), they are about the staff discount and discount codes. None of them answers the question. The others are answerable. (2) No: the only statement is in the French document garantie-legale-quebec, section Garantie prolongée (“Nous ne vendons pas de garantie prolongée”), so an English keyword search misses it. (4) 3 years: the warranty table in warranty-policy (PDF), and the staff procedure warranty-claims-procedure. (5) No: the returns policy, version 3, lists gift cards among the items that cannot be returned; the gift-cards policy says the same.

Recap

In this module, you defined the knowledge problem before solving it:

• RAG retrieves passages, puts them in the prompt and asks for cited answers, or “the documents do not say”. Alone, the models mostly declined and sometimes answered wrongly; with five passages, chat-small answered 51 of 59 correctly.
• The corpus and its inventory say which documents count, with owners, versions, effective dates and access labels. Grace’s test set has 67 questions, 8 of them unanswerable.
• Keyword search (lexical search) with BM25 ranks by term frequency and rarity. SQLite’s FTS5 keeps an inverted index, inside Python.
• hit@5 is 1 when a relevant passage is in the top k = 5. The baseline: hit@5 0.695 (41 of 59), over the 154 sections of the 32 Markdown documents.
• 14 of the 18 misses are missing sources: the HTML and PDF files. A larger k or a new search method cannot fix that.

Next, you read the HTML and PDF documents, and see what text extraction loses on the way.

Next: Parse and normalize