Answer questions from public guidance

What you will be able to do

Build an assistant that answers questions about rights at work from real GOV.UK guidance pages. Every claim cites a passage, and every source shows its address and the date we fetched it. When the guidance does not answer, the assistant says so. You measure the search and the answers on a fixed question set, and you compare the cost of a small and a strong model.

Case studyPortfolio piece

Dataset

GOV.UK guidance on rights at work: 137 pages, fetched on 2026-10-09

Licence

Open Government Licence v3.0

Time

About 4 hours

Runs in

VS Code, Python 3.12+

Skills used

• Licences of public data
• A fixed snapshot with checksums
• Parsing with provenance
• Stable chunk IDs
• BM25, e5 and reranking
• Cited answers and checks
• Abstention
• recall@k and answer measures
• Cost and latency

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is a project in the shape of the course’s policy-assistant. It already has the snapshot, the questions and the recorded answers. It also has the parts that you built in the course: search, reranking, the context, the answer schema, the citation checks and the mock provider. You write the three files that are new for public guidance: the parser, the chunker, and the code that shows the sources. The finished project is the result, for when you are stuck or want to compare. You need no account: the mock replays the answers that real models gave.

Before you startWhat to know and to have readyAbout 240 minutes

What you need to know

This case study applies the whole course to new documents. The lessons it uses most:

• Blocks that keep the way back to the source, from Parse and normalize.
• Stable chunk IDs, from Preserve metadata.
• BM25, vectors, fusion and reranking, from Lexical, dense and hybrid search and Reranking and query changes.
• Claims with passage IDs and the citation checks, from Answer and cite, and abstention, from Handle difficult evidence.
• recall@k and the answer measures, from Retrieval measures and Answer measures.

What you need on your computer

• Python 3.12 or newer, VS Code and a terminal, as in the course.
• About 2 GB of free disk space: 0.9 GB for the packages and 1 GB for the two local models. If you did the course, the models are already in your Hugging Face cache.

Tested on macOS (Apple silicon) with Python 3.12.13. The versions are the course’s: sentence-transformers 6.1.0, torch 2.14.1, pydantic 2.14.0, pytest 9.1.1. The Windows commands follow the course’s commands, but they are not tested.

The situation

Joy Mensah runs the sessions of a small advice centre in an English town. Volunteers help people with a problem at work: holiday refused, sick pay stopped, a redundancy letter. Most answers are in the GOV.UK guidance, spread over hundreds of pages, and a volunteer with a queue cannot read a guide of eight parts for each question.

Ravi Patel, a volunteer developer, offers to build an assistant that finds the passages and gives a short answer with its pages. Joy agrees on three conditions. Every answer must show where it comes from and how old the page is. A volunteer must be able to open the page and read it with the person. When the guidance does not answer, the assistant must say so. A confident wrong answer about a dismissal can cost someone their claim. And the assistant must never look like a government service or a lawyer.

Joy, Ravi and the centre are invented. The pages are real, and so are the questions people ask about them.

What you will build

You will build guidance-assistant, a small Python project. The picture shows one question from start to end.

The design of guidance-assistant. The search and the answer come from the course; the snapshot, the addresses and the dates on every source are new.

Text description of the diagram

1. A snapshot of 137 GOV.UK pages, fetched once on 2026-10-09 and checked with SHA-256, is parsed and cut into 504 chunks. Each chunk keeps its ID, its page address with an anchor, and its dates.
2. An adviser’s question goes to the course’s search: BM25 and e5 vectors, fused with RRF, then reranked. The top 5 chunks are kept.
3. The context gives each passage with its page, section, fetch date and address.
4. The model answers in claims, and each claim cites passage IDs.
5. Code checks every citation: the ID exists, it was in the context, its numbers and words are in the passage.
6. If the pages answer the question, the adviser sees an answer card with numbered sources, their addresses and dates. A notice says that this is guidance, not legal advice. If the pages do not answer, the card says so and points to Acas or an adviser.

Three files are new. The parser reads GOV.UK’s HTML, the chunker gives each chunk a web address, and sources.py shows the sources.

The data

The pages come from GOV.UK, the UK government’s website. We chose one topic that an advice centre meets every day: rights at work in England, Scotland and Wales (holidays, sick pay, family leave, notice, redundancy and dismissal, working hours, contracts, the minimum wage). The snapshot holds 38 guides with 137 pages, about 36,000 words.

The licence, and what it asks of us

Most GOV.UK content is Crown copyright under the Open Government Licence v3.0 (OGL). Anyone may copy, adapt and share it, also commercially. The licence asks for one thing: name the source. GOV.UK gives no special wording for these pages, so we use the licence’s default text: “Contains public sector information licensed under the Open Government Licence v3.0.” The project prints it under every answer.

The licence does not cover personal data, logos, crests or third-party rights; our snapshot holds only the body text, with none of these. The licence also gives no right “to use the Information in a way that suggests any official status”. GOV.UK’s own terms add that it does not publish advice: “You should get professional or specialist advice before doing anything on the basis of the content.” Both sentences become rules of the project, in Responsible use.

We also looked at Canada.ca’s guidance pages. Their terms allow only non-commercial reproduction without permission, so they fail the course’s licence rule. The Open Government Licence – Canada covers the datasets on open.canada.ca, not the web pages.

A snapshot, fetched once

A snapshot is a copy of the pages as they were at one moment, kept with a fingerprint of every file. Think of a dated photocopy of a notice board: the board will change, and the copy will not. Without it, the questions, measures and recorded answers would stop matching as soon as GOV.UK changed a sentence.

We did not copy the web pages. GOV.UK has an official Content API, which returns each page as JSON with its body as HTML and its dates. The script scripts/fetch_snapshot.py asked for 47 candidate guides once, one request every 1.2 seconds, after reading GOV.UK’s robots.txt (it allows the API). We kept 38 of them. guidance/pages.jsonl has one line per page, with:

Field	Example
page_id, url	statutory-sick-pay/eligibility, https://www.gov.uk/statutory-sick-pay/eligibility
guide_title, part_title	Statutory Sick Pay (SSP), Eligibility
public_updated_at	2015-03-09: GOV.UK’s date of the last major change
updated_at	2026-10-05: the date of the last change of any size
retrieved_at	2026-10-09T06:21:10Z: when we fetched it
api_sha256, body_sha256	fingerprints of the API response and of the body
body_html	the page body, exactly as the API returned it

First checks

We read the snapshot before writing any search. Four findings shape the project:

• The “last updated” date can mislead. public_updated_at is before 2020 for 26 of the 38 guides (the oldest 2014-11-17), yet the sick pay guide gives the April 2026 rate, £123.25. updated_at (2026-09-24 to 2026-10-05) is closer to the truth, but neither says which sentence changed. So the assistant shows the fetch date and updated_at, and says to check the page.
• Some pages hold their own history. The minimum wage page has four tables: the current rates and three tables of earlier rates. Some rules carry a date on the page: neonatal care is for babies born on or after 6 April 2025. A paternity notice rule is for babies due before 25 July 2026.
• Northern Ireland is outside. Several pages say “There are different rules if you live in Northern Ireland”, but do not give those rules.
• A repeated line. Nine pages carry the line “This guide is also available in Welsh (Cymraeg).” It says nothing about any question, and it makes those pages look alike to a search.

The questions

Before any search, Joy and Ravi fix what a good answer is, as in Corpus and questions. The file questions/questions.jsonl has 44 questions in six kinds. Each has a reference answer, the relevant passages (page and section), and keys: regular expressions that a right answer must match, as in the course.

Kind	How many	Example	A right answer
Fact	12	For how long can someone get Statutory Sick Pay?	Up to 28 weeks
Condition	14	Am I entitled to statutory redundancy pay?	Normally yes, if you are an employee with 2 years’ service
Table	3	What is the minimum hourly wage for an 18-year-old?	£10.85 (from April 2026)
Dated	5	What was the minimum wage for a 22-year-old in June 2023?	£10.18
Calculation	2	I work 4 days a week. How many days of statutory holiday do I get?	22.4 days (4 × 5.6)
Not covered	8	What is the rate of Statutory Sick Pay in Northern Ireland?	“The guidance does not say”

The condition questions matter most for an advice centre. In employment law, the answer to “Can I…?” is often “yes, if…”. An answer that drops the condition is wrong even when its numbers are right. Their keys check for the condition, for example “2 years”. The not covered questions are traps. Some ask about Northern Ireland, one asks for personal advice, and one invites a number from the wrong rule.

The author of this case study wrote every question, answer and key by reading the pages; no model wrote or labelled a question. A script checks that each passage exists and that each key matches its reference answer. After reading the first recorded answers, we changed two checks, and we say so in DATASET.md. The key of Q34 now also accepts “not met”, because a right answer said “the birth-date condition is not met”. And a must_not pattern of Q41 no longer catches “whether you should accept”, an abstention that it had scored as advice. Then we froze the set, before any comparison. Answer measures explains why a test set must be frozen before a final comparison.

Set up the project

The steps use VS Code and its terminal. Every command runs in the project folder.

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder guidance-assistant.

2. In VS Code, choose File > Open Folder, and open guidance-assistant. Then open a terminal: Terminal > New Terminal.

3. Make and activate a virtual environment:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1

   Linux

   python3 -m venv .venv
   source .venv/bin/activate

   The prompt starts with (.venv). If VS Code asks whether to use the new environment for this folder, choose Yes.

4. Install the course’s versions. On Linux, install the CPU-only PyTorch first: python -m pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu.

   python -m pip install -r requirements.txt

5. Run the tests, and check the snapshot:

   python -m pytest
   python scripts/check_snapshot.py

Output (end of pytest, then check_snapshot.py)

========================= 5 passed, 9 skipped in 0.46s =========================
OK  pages.jsonl  SHA-256 b48f97342a68…
OK  sources.csv  SHA-256 499806c88713…
137 pages from 38 GOV.UK guides, fetched 2026-10-09

What this tells us. The packages work, and the snapshot is the one we tested: every load compares the files with guidance/SHA256SUMS first, and a test changes £123.25 to £132.25 in a copy and checks that the load stops. The 9 skipped tests are for the three files that you write. Each one runs as soon as its file exists.

Look at the pages and the questions that the guidance does not cover:

python -m guidance_assistant pages
python -m guidance_assistant questions --kind not_covered

Output (the last lines of each)

 1 page   2026-10-05  Payslips: employee rights
 1 page   2026-10-05  Pay and work rights helpline and complaints
137 pages from 38 GOV.UK guides | fetched 2026-10-09 | checksums OK
...
Q43 not_covered  What is the rate of Statutory Sick Pay in Northern Ireland?
Q44 not_covered  How much compensation will I get if I win an unfair dismissal claim?
8 questions

Part 1: read the pages, and keep the way back

A GOV.UK body is HTML: headings with an id, paragraphs, lists, a few tables, and notice boxes. The search needs clean text. The adviser needs the way back: which page, which section, and the address of that section on GOV.UK. This is the idea of Parse and normalize, on a new format: parse into blocks, and let each block remember its section, its heading’s id (the anchor, the part of an address after #) and its position in the clean text.

Python’s standard library has an HTML parser, so the project needs no new package. Create guidance_assistant/parse.py:

guidance_assistant/parse.py

"""Parse a GOV.UK page body (HTML) into clean blocks that remember where they came from.

A block is a heading, a paragraph, a list item, a table or a notice. Each block keeps:
- section: the heading it is under ("" before the first heading),
- anchor:  that heading's id, so a citation can link to https://www.gov.uk/<page>#<anchor>,
- start, end: its character positions in the page's clean text.

What the parser removes, and why:
- tags (links keep their words, abbreviations their short form: "SSP");
- the line "This guide (or page) is also available in Welsh (Cymraeg)": it repeats on many
  pages, says nothing about the question, and makes those pages look alike to a search.
Tables become one block with one line per row ("cell | cell | cell"): a table is never split.
"""

import html
import re
from dataclasses import dataclass, field
from html.parser import HTMLParser

DROP = re.compile(r"^This (guide|page) is also available in Welsh", re.I)
BLOCK_TAGS = {"p": "paragraph", "li": "item", "h2": "heading", "h3": "heading", "h4": "heading"}


@dataclass(frozen=True)
class Block:
    kind: str        # heading, paragraph, item, table, notice
    text: str
    section: str
    anchor: str
    start: int = 0
    end: int = 0


@dataclass
class ParsedPage:
    page_id: str
    title: str
    blocks: list = field(default_factory=list)
    text: str = ""
    dropped: list = field(default_factory=list)   # texts removed as boilerplate


def clean(text: str) -> str:
    return re.sub(r"\s+", " ", html.unescape(text)).strip()


class _Reader(HTMLParser):
    """Collects (kind, text, heading id) in reading order."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.out: list[tuple[str, str, str]] = []
        self.buffer: list[str] | None = None
        self.kind = ""
        self.heading_id = ""
        self.notice_depth = 0          # inside <div role="note"> (info and help notices)
        self.table: list[list[str]] | None = None
        self.cell: list[str] | None = None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "div" and a.get("role") == "note":
            self.notice_depth += 1
        if tag == "table":
            self.table = []
        elif self.table is not None:
            if tag == "tr":
                self.table.append([])
            elif tag in ("td", "th"):
                self.cell = []
            return
        if tag in BLOCK_TAGS and self.buffer is None:
            self.buffer, self.kind = [], BLOCK_TAGS[tag]
            if self.kind == "heading":
                self.heading_id = a.get("id", "")
        elif tag == "li" and self.buffer is not None:
            self.buffer.append(" ")    # a nested list item continues its parent item
        if tag == "br" and self.buffer is not None:
            self.buffer.append(" ")

    def handle_endtag(self, tag):
        if self.table is not None:
            if tag in ("td", "th") and self.cell is not None:
                self.table[-1].append(clean("".join(self.cell)))
                self.cell = None
            elif tag == "table":
                rows = [" | ".join(r) for r in self.table if any(r)]
                self.out.append(("table", "\n".join(rows), ""))
                self.table = None
            return
        if tag in BLOCK_TAGS and self.buffer is not None and BLOCK_TAGS[tag] == self.kind:
            text = clean("".join(self.buffer))
            if text:
                kind = "notice" if self.notice_depth and self.kind == "paragraph" else self.kind
                self.out.append((kind, text, self.heading_id if kind == "heading" else ""))
            self.buffer = None
        if tag == "div" and self.notice_depth:
            self.notice_depth -= 1

    def handle_data(self, data):
        if self.cell is not None:
            self.cell.append(data)
        elif self.buffer is not None:
            self.buffer.append(data)


def parse_page(page) -> ParsedPage:
    """page: a snapshot.Page. Returns its blocks and clean text, with positions."""
    reader = _Reader()
    reader.feed(page.body_html)
    reader.close()
    out = ParsedPage(page.page_id, page.title)
    section, anchor, parts, pos = "", "", [], 0
    for kind, text, heading_id in reader.out:
        if DROP.search(text):
            out.dropped.append(text)
            continue
        if kind == "heading":
            section, anchor = text, heading_id
        shown = f"- {text}" if kind == "item" else text
        start = pos
        parts.append(shown)
        pos += len(shown)
        out.blocks.append(Block(kind, shown, section, anchor, start, pos))
        parts.append("\n")
        pos += 1
    out.text = "".join(parts)
    return out

Three choices matter. A table becomes one block, so no chunk can cut it. The Welsh line is removed, and dropped says so: cleaning that you can see is cleaning that you can check. And start and end let every block be found again.

Run the tests, then parse the minimum wage page:

python -m pytest
python -m guidance_assistant parse national-minimum-wage-rates

Output (the end of pytest, then part of the blocks)

========================= 8 passed, 8 skipped in 0.05s =========================
National Minimum Wage and National Living Wage rates | https://www.gov.uk/national-minimum-wage-rates
[    0-90   ] paragraph                            | The hourly rate for the minimum wage depends on your age an…
[  275-288  ] heading   Current rates              | Current rates
[  289-328  ] paragraph Current rates              | The rates change on 1 April every year.
[  329-417  ] table     Current rates              | | 21 and over | 18 to 20 | Under 18 | Apprentice April 2026…
...
[ 1095-1139 ] heading   Rates between 1 April 202… | Rates between 1 April 2024 and 31 March 2026
[ 1221-1384 ] table     Rates between 1 April 202… | | 21 and over | 18 to 20 | Under 18 | Apprentice April 2025…
...
[ 1887-2077 ] table     Rates before 1 April 2021  | | 25 and over | 21 to 24 | 18 to 20 | Under 18 | Apprentice…
dropped: This page is also available in Welsh (Cymraeg).
33 blocks, 2373 characters

What this tells us. The page has four tables, and each one is a single block under its own heading. The current table and the old tables have the same columns and similar numbers: only the heading says which one is current. Keep this in mind for Part 3.

Part 2: chunks that know their address

Cut the blocks into chunks as in Chunk deliberately. Each section is one chunk of at most 200 words, never cut inside a table, with the page title in front. Two things are new. The chunk ID hashes the page, its updated_at and the positions, as in Preserve metadata: a page that GOV.UK changes gets new IDs, so an old citation cannot point to new text. And each chunk carries its web address with the anchor and its three dates, so a citation can link to the exact section.

Create guidance_assistant/chunk.py:

guidance_assistant/chunk.py

"""Structure-aware chunks with stable IDs, as in the course's policy assistant.

One chunk per section (from a heading to the next heading; the text before the first heading is a
section too). A section longer than MAX_WORDS is split between blocks, never inside a table.
Each chunk starts with the page title (and the section heading when a part does not start with it),
so a chunk keeps its context: "Exceptions" alone does not say which guide.

The chunk ID is the first 12 hexadecimal characters of SHA-256 over "page_id|updated_at|start|end":
the same text of the same page gives the same ID every time, a changed page (a new updated_at)
gives new IDs, and adding a page changes no other ID.
"""

import hashlib
from dataclasses import asdict, dataclass

from .parse import ParsedPage

MAX_WORDS = 200


def chunk_id(page_id: str, updated_at: str, start: int, end: int) -> str:
    return hashlib.sha256(f"{page_id}|{updated_at}|{start}|{end}".encode("utf-8")).hexdigest()[:12]


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    page_id: str
    title: str              # the page title: "Guide: Part"
    section: str            # the heading ("" = the top of the page)
    anchor: str             # the heading's id on GOV.UK
    position: int
    start: int
    end: int
    text: str               # what is indexed and shown to the model
    url: str                # the page, with #anchor when the section has one
    public_updated_at: str
    updated_at: str
    retrieved_at: str

    def to_dict(self) -> dict:
        return asdict(self)


def structure(parsed: ParsedPage, page, max_words: int = MAX_WORDS) -> list[Chunk]:
    sections: list[list] = []
    for b in parsed.blocks:
        if b.kind == "heading" or not sections:
            sections.append([])
        sections[-1].append(b)
    chunks: list[Chunk] = []
    for blocks in sections:
        part: list = []
        for b in blocks:
            words = sum(len(x.text.split()) for x in part)
            if part and words + len(b.text.split()) > max_words:
                chunks.append(_make(parsed, page, part, len(chunks)))
                part = []
            part.append(b)
        if part:
            chunks.append(_make(parsed, page, part, len(chunks)))
    return chunks


def _make(parsed: ParsedPage, page, blocks: list, position: int) -> Chunk:
    start, end = blocks[0].start, blocks[-1].end
    section, anchor = blocks[0].section, blocks[0].anchor
    prefix = page.title if blocks[0].kind == "heading" or not section else f"{page.title}\n{section}"
    return Chunk(
        chunk_id=chunk_id(page.page_id, page.updated_at, start, end), page_id=page.page_id, title=page.title,
        section=section, anchor=anchor, position=position, start=start, end=end,
        text=f"{prefix}\n{parsed.text[start:end]}", url=page.url + (f"#{anchor}" if anchor else ""),
        public_updated_at=page.public_updated_at[:10], updated_at=page.updated_at[:10],
        retrieved_at=page.retrieved_at[:10])

Run the tests, build the index, and look at the chunks of one page:

python -m pytest
python -m guidance_assistant ingest
python -m guidance_assistant chunks statutory-sick-pay/eligibility

Output

======================== 18 passed, 1 skipped in 0.14s =========================
index index.sqlite: 504 chunks from 137 pages | embeddings intfloat/multilingual-e5-small | 31.5 s
0a1832934dc9 #0 chars 0-221 |  48 words | https://www.gov.uk/statutory-sick-pay/eligibility
daa76305e45b #1 chars 222-449 |  49 words | https://www.gov.uk/statutory-sick-pay/eligibility#telling-your-employer
da9155fa4f2f #2 chars 450-580 |  28 words | https://www.gov.uk/statutory-sick-pay/eligibility#exceptions
261d90e47a19 #3 chars 581-887 |  63 words | https://www.gov.uk/statutory-sick-pay/eligibility#linked-periods-of-sickness
a09f0d23000a #4 chars 888-1270 |  72 words | https://www.gov.uk/statutory-sick-pay/eligibility#fit-notes-and-asking-for-proof
e668f89c2ed3 #5 chars 1271-1566 |  56 words | https://www.gov.uk/statutory-sick-pay/eligibility#other-proof-of-sickness
1229b76afc00 #6 chars 1567-2114 | 110 words | https://www.gov.uk/statutory-sick-pay/eligibility#if-youre-not-eligible-or-your-ssp-ends
7 chunks

What this tells us. Each section is one chunk, and each chunk has an address that opens GOV.UK at that section. The median chunk has 72 words; the longest has 211, a single block that cannot be split. ingest took 31.5 seconds on our Mac with 2 threads, most of it loading the e5 model and embedding the 504 chunks. The one skipped test is for sources.py.

Part 3: search, measured

The index holds the course’s four search methods, unchanged. Measure them on the 36 questions that have relevant passages (the 8 not covered questions have none):

python -m guidance_assistant eval --method bm25
python -m guidance_assistant eval --method dense
python -m guidance_assistant eval --method hybrid
python -m guidance_assistant eval --method rerank

Output

bm25, top 5: hit@5 0.778 | recall@5 0.701 | precision@5 0.244 | MRR 0.549 (36 questions with relevant passages)
dense, top 5: hit@5 0.972 | recall@5 0.861 | precision@5 0.300 | MRR 0.867 (36 questions with relevant passages)
hybrid, top 5: hit@5 0.889 | recall@5 0.829 | precision@5 0.289 | MRR 0.785 (36 questions with relevant passages)
rerank, top 5: hit@5 0.944 | recall@5 0.859 | precision@5 0.294 | MRR 0.872 (36 questions with relevant passages)

The chart adds recall at other values of k, as in Retrieval measures.

• bm25 (dashed)
• dense
• hybrid
• rerank

recall@k of the four search methods on the 36 questions with relevant passages (snapshot of 2026-10-09, structure chunks, e5, the course's reranker).

Text description of the chart

Four lines show recall at k = 1, 3, 5 and 10. BM25: 0.301, 0.595, 0.701, 0.843. Dense: 0.535, 0.792, 0.861, 0.956. Hybrid: 0.484, 0.755, 0.829, 0.914. Rerank: 0.604, 0.729, 0.859, 0.951. BM25 is lowest at every k. Rerank is highest at k = 1; dense and rerank are close at 5 and 10.

What this tells us. The pattern of Larkfield’s collection holds on real pages. Keywords miss many questions, because people’s words are not the page’s words. Vectors find more, and hybrid with RRF is again below dense alone, for the reason in Lexical, dense and hybrid search. The reranker puts the right passage first most often (recall@1 0.604, MRR 0.872), at a cost: about 340 ms per question on our Mac, against 5.5 ms for dense search. With 36 questions, one question is 0.028, so dense and rerank are not clearly different. We keep rerank, the course’s default. Every method misses one of the three table questions, and that miss teaches the most.

One page, four tables

A real miss (Q27, rerank). The page holds its own history, and the reranker cannot tell the current table from the old ones.

Text description of the diagram

The question asks for the minimum hourly wage of an 18-year-old. It meets one page with four tables of the same shape. The reranker puts the table of rates before 1 April 2021 at rank 3 and the table for 2024 to 2026 at rank 4. The current table, with £10.85, comes 6th, just out of the top 5. The table for 2021 to 2024 comes 7th. So the model gets two old tables and no current one. chat-small abstained: “The guidance does not say what the minimum hourly wage is for an 18-year-old on the fetched date.”

At Larkfield, an old policy was a separate document with effective dates, and a filter kept it out. Here, one page holds its own history: the current rates and three earlier tables, under headings with dates. To a reranker, “rates for 18 to 20” in 2020 and in 2026 read alike. BM25 does better on this question (the current table at rank 2), and dense search has it at rank 6, like the reranker.

Two things kept the miss safe. The table’s heading reached the model with each passage (“section: Rates before 1 April 2021”), and rule 3 of the prompt forbids guessing; so chat-small did not quote an old rate as today’s rate. It abstained. To fix the retrieval itself, give each history section the dates in its heading as effective dates, and filter by date as in Lexical, dense and hybrid search. That is a good next step for your version.

Part 4: a baseline, the model alone

Before you trust the pages to help, measure what the model knows without them. This is the open-book exam of What retrieval-augmented generation is, with the book taken away. The project can ask chat-small each question with no passages:

python -m guidance_assistant ask Q33 --closed-book

Output

Question: My baby is due on 1 July 2026 and I joined my employer 10 weeks ago. Do I need to give 15 weeks' notice for paternity leave?
The model alone, without the guidance:
No—you must notify your employer **by the end of the 15th week before the expected week of childbirth**; it is not 15 weeks’ notice counted from when you joined. For a due date of 1 July 2026, that deadline was around 21 March 2026, so if you missed it, tell your employer as soon as possible. Since paternity leave became a day-one right for babies due from 6 April 2026, your short service should not prevent leave, though statutory paternity pay has separate eligibility rules.

gpt-6-luna-2026-09-22 | 75 tokens in, 1152 out | 7.874 s

This answer is confident and wrong. The page has a rule for exactly this case. For a baby due before 25 July 2026, someone with less than 26 weeks’ service does not need to give 15 weeks’ notice. The model says the deadline “was around 21 March 2026” and tells the person to hurry.

Over the whole set, the model alone did better than you might expect: by the keys, 31 of the 36 answerable questions right. We read the 5 misses: three say the right thing in other words, one leaves out the main limit (Q15), and one is wrong (Q33). So accuracy is not why Joy needs the pages. These are:

• No source and no date. Not one answer can be checked by opening a page.
• It answers what it cannot know from GOV.UK. Of the 8 not covered questions, it answered 7 from its own knowledge. For Northern Ireland it said “£123.25 per week for 2026/27”. For the settlement agreement it gave personal advice: “Don’t accept or sign it until…”. Some of this may be true; none of it comes from the guidance, and nothing shows that.
• Wrong and right look the same. The Q33 answer has the same confident tone as the 31 right ones.

As the course found in What retrieval-augmented generation is, the pages bring evidence and dates, not only fewer inventions.

Part 5: answers with their sources

The answer pipeline is the course’s ask(). What is missing is what a person sees. The model gives only claims and passage IDs. The application writes the sources, by looking up each cited ID in the index. So a source line is always a real page with its real address and dates, even when the model is wrong. The notice and the licence’s attribution are fixed text, under every answer.

Create guidance_assistant/sources.py:

guidance_assistant/sources.py

"""Show an answer the way a person should see it: every claim numbered, every source with its address
and its dates, and what the assistant is not.

The application writes the sources and the notice, not the model: the model gives only claims and
passage IDs, and this code looks up each ID in the index. So a source line is always a real page,
with the date we fetched it, even when the model is wrong.
"""

from .snapshot import ATTRIBUTION

NOTICE = ("This is what GOV.UK guidance says, not legal advice. This assistant is not a government service. "
          "Pages fetched on {fetched}: check the page for changes.")


def sources_for(answer, store) -> list:
    """The cited chunks, in the order of first citation, without repeats."""
    chunks = [store.chunk(cid) for cid in answer.cited()]
    return [c for c in chunks if c is not None]


def render(answer, checks: list, store) -> str:
    if not answer.answerable:
        lines = [f"Answer: {answer.answer}", "", "The pages in the snapshot do not answer this question.",
                 "Ask the Acas helpline or an adviser."]
        return "\n".join(lines + ["", NOTICE.format(fetched=_fetched(store)), ATTRIBUTION])
    sources = sources_for(answer, store)
    number = {c.chunk_id: i for i, c in enumerate(sources, 1)}
    lines = [f"Answer: {answer.answer}", ""]
    for claim, check in zip(answer.claims, checks):
        refs = "".join(f"[{number[i]}]" for i in claim.chunk_ids if i in number) or "[no source]"
        status = "" if check.ok else "  (check: " + "; ".join(check.problems) + ")"
        lines.append(f"- {claim.text} {refs}{status}")
    lines += ["", "Sources:"]
    for i, c in enumerate(sources, 1):
        section = f', section "{c.section}"' if c.section else ""
        lines.append(f"[{i}] {c.title}{section}")
        lines.append(f"    {c.url}")
        lines.append(f"    fetched {c.retrieved_at}; GOV.UK last updated {c.updated_at}")
    return "\n".join(lines + ["", NOTICE.format(fetched=_fetched(store)), ATTRIBUTION])


def _fetched(store) -> str:
    dates = sorted({c.retrieved_at for c in store.chunks()})
    return dates[-1] if dates else "?"

Run the tests, then ask a recorded question:

python -m pytest
python -m guidance_assistant ask Q14

Output (recorded answer, chat-small)

============================== 20 passed in 0.19s ==============================
Question: I have worked for my employer for 18 months and was just dismissed. Can I claim unfair dismissal?
Passages: 59652cdb4e72, f4443fade2e6, f44be5705d22, 407a19ade544, 023fd2596ef3 (1100 tokens, estimated)

Answer: The guidance says you must be classed as an employee and normally have 2 years’ service; with 18 months, you would not meet that usual qualifying period. In Northern Ireland, the usual period is 1 year, so 18 months meets the service-length requirement; there is no qualifying period if you were dismissed since 25 June 2013 because of your political opinions or affiliation. Any unfair dismissal claim must be made to a tribunal within 6 months of dismissal.

- You must be classed as an employee and normally have 2 years’ service to qualify, so 18 months does not meet the usual qualifying period. [1]
- In Northern Ireland the usual qualifying period is 1 year, and there is no qualifying period for a dismissal since 25 June 2013 because of political opinions or affiliation. [1]
- An unfair dismissal claim must be made to a tribunal within 6 months of dismissal. [1]

Sources:
[1] Dismissal: your rights: What to do if you're dismissed, section "Qualifying period to claim unfair dismissal"
    https://www.gov.uk/dismissal/what-to-do-if-youre-dismissed#qualifying-period-to-claim-unfair-dismissal
    fetched 2026-10-09; GOV.UK last updated 2026-10-05

This is what GOV.UK guidance says, not legal advice. This assistant is not a government service. Pages fetched on 2026-10-09: check the page for changes.
Contains public sector information licensed under the Open Government Licence v3.0.

gpt-6-luna-2026-09-22 | 1413 tokens in, 570 out | 4.761 s

What this tells us. The answer keeps the conditions: employee status, “normally” 2 years, the exception, the 6-month limit. It says what the guidance says, not “you cannot claim”, as rule 5 of the prompt asks. Every claim passed the checks, and the adviser can open the section with the person.

When a check flags a right answer

python -m guidance_assistant ask Q35

Output (part)

- Almost all people classed as workers are entitled to 5.6 weeks’ paid holiday a year. [1]
- At 4 working days a week, 5.6 weeks is 22.4 working days. [1][2]  (check: number_not_found:22.4)

The answer is right: 4 × 5.6 = 22.4. But 22.4 is not written on the page, so the number check flags it. This is the limit that Answer and cite taught: code can check that a number is in the passage, not that a calculation is right. Over the 44 answers, the checks flagged 4 of 70 claims, and all 4 are right. Two are calculations (Q35, Q36), and two use other words than the passage (Q25, Q34). No answer cited an ID that did not exist or was not in its passages.

When the pages do not answer

python -m guidance_assistant ask Q43

Output (part)

Answer: The guidance does not say what rate of Statutory Sick Pay applies in Northern Ireland.

The pages in the snapshot do not answer this question.
Ask the Acas helpline or an adviser.

The sick pay passages, with £123.25, were in the context, but no passage says that they apply to Northern Ireland. This is the abstention of Handle difficult evidence; the model alone gave a number.

Common mistake

The source line says one thing, the page says another

Symptom. An adviser opens a cited address, and the section says something different from the claim; or the address is a GOV.UK page that is not in the snapshot at all.

Cause. The model wrote the source lines (the prompt asked it to “include the URL”). A model can write a real-looking address for a page that it never saw.

Fix. Let the model give only passage IDs. Look up every ID in the index, and reject an ID that is not in the context. Write the title, the address and the dates from the index, as sources.py does. The page you show is then always the page the passage came from.

Part 6: a fair comparison

All runs below use the same 44 questions, the same snapshot, prompt answer_v1 and the top 5. The mock replays the answers recorded on Azure. First, the course’s main question: does better retrieval help the answers?

python -m guidance_assistant eval --answers --method rerank
python -m guidance_assistant eval --answers --method bm25

Output

chat-small, rerank, top 5, answer_v1: 44 questions
  answerable, correct:          34/36
  answerable, abstained:        2/36
  not covered, abstained:       8/8
  shown but wrong:              0  (none)
  claims flagged by the checks: 4 of 70
  tokens in / out (median):     1235 / 206
  latency median / p95:         2.26 / 5.04 s (recorded)
  cost per question (median):   US$0.00023
chat-small, bm25, top 5, answer_v1: 44 questions
  answerable, correct:          28/36
  answerable, abstained:        8/36
  not covered, abstained:       7/8
  shown but wrong:              1  (Q40)
  claims flagged by the checks: 3 of 59
  tokens in / out (median):     1260 / 182
  latency median / p95:         2.12 / 4.02 s (recorded)
  cost per question (median):   US$0.00022

The measure that matters most for Joy is “shown but wrong”: an answer that is not an abstention and not right. An adviser would read it as the guidance. With rerank it is zero. Better retrieval gave 6 more right answers (28 → 34) at the same model and the same cost. With BM25, the model abstained on 8 answerable questions: 7 because the right passage was not found, and once (Q32) although it was. The one BM25 answer shown but wrong, Q40, is mild. Asked what an employer’s own maternity scheme pays, it gave the rule that a scheme cannot pay less than the statutory amount. It should have said that the guidance does not say.

With rerank, both misses are retrieval misses: the relevant passage was not in the top 5, and the model abstained. Q27 is the four tables of Part 3; for Q19 (maternity pay after 3 months in a job), the eligibility section was not found. This is the stage analysis of Find where it fails: no wrong answer came from the model.

Then the model. chat-strong was recorded on a sample: the 22 questions with an even number, which include every kind. Compare it with chat-small on the same 22:

python -m guidance_assistant eval --answers --ids Q02,Q04,Q06,Q08,Q10,Q12,Q14,Q16,Q18,Q20,Q22,Q24,Q26,Q28,Q30,Q32,Q34,Q36,Q38,Q40,Q42,Q44
python -m guidance_assistant eval --answers --ids Q02,Q04,Q06,Q08,Q10,Q12,Q14,Q16,Q18,Q20,Q22,Q24,Q26,Q28,Q30,Q32,Q34,Q36,Q38,Q40,Q42,Q44 --model chat-strong

On the 22 even questions	chat-small	chat-strong
Answerable, right	18/18	18/18
Not covered, abstained	4/4	4/4
Shown but wrong	0	0
Claims (flagged)	37 (2)	51 (3)
Latency, median / p95 (recorded)	2.35 / 4.28 s	3.25 / 10.37 s
Cost per question (median)	US$0.00023	US$0.00417
Cost per 1,000 questions	about US$0.25	about US$4.65

What this tells us. On this sample, the strong model is not more correct. It writes more claims (51 against 37), is slower, and costs about 18 times more per question. For a few hundred questions a month, both cost very little; with no difference in the measures, chat-small is the choice. 22 questions cannot show a small difference, so record the versions and measure again later, as in Operational handoff.

With the reranker (about 0.34 seconds on a CPU) and the model call (about 2.3 seconds), an adviser waits about 3 seconds.

The final design, end to end

The final design: rerank top 5, chat-small, prompt answer_v1, sources written by the application. Here is one question from start to end, a dated question that the model alone got wrong:

python -m guidance_assistant ask Q33

Output (recorded answer, chat-small)

Question: My baby is due on 1 July 2026 and I joined my employer 10 weeks ago. Do I need to give 15 weeks' notice for paternity leave?
Passages: 96d822f2d09f, dbdb4b6565f7, 1d938f602c0b, e30d70b6e089, 6f79065eed0a (1075 tokens, estimated)

Answer: Your baby’s due date is before 25 July 2026. You do not need to give 15 weeks’ notice if you’ll have been employed for less than 26 weeks up to any day in the qualifying week, which is the 15th week before the baby is due; you’ll still need to give 28 days’ notice of when you want leave to start and whether you want 1 or 2 weeks.

- A baby due on 1 July 2026 is due before 25 July 2026. [1]
- For a baby due before 25 July 2026, 15 weeks’ notice is not required if the employee will have been employed for less than 26 weeks up to any day in the qualifying week. The qualifying week is the 15th week before the baby is due. [1]
- The employee must still give 28 days’ notice of when they want leave to start and whether they want 1 or 2 weeks. [1]

Sources:
[1] Paternity pay and leave: How to claim, section "If your baby is due before 25 July 2026"
    https://www.gov.uk/paternity-pay-leave/how-to-claim#if-your-baby-is-due-before-25-july-2026
    fetched 2026-10-09; GOV.UK last updated 2026-10-05

This is what GOV.UK guidance says, not legal advice. This assistant is not a government service. Pages fetched on 2026-10-09: check the page for changes.
Contains public sector information licensed under the Open Government Licence v3.0.

gpt-6-luna-2026-09-22 | 1496 tokens in, 649 out | 5.048 s

The search found the one section written for this case. The answer applies the question’s date to the page’s date, keeps the condition, and cites the section.

To ask a question of your own, you need a live model. The mock knows only the recorded questions, and says so: “No recording for this request”. Copy .env.example to .env and choose a free local model with Ollama, or your own key, as in the course. Your answers will differ from the recorded ones; the checks and the sources work the same way.

Insights and limits

• The pages bring evidence and dates more than accuracy. Model alone: 31 of 36 by the keys, no sources, 7 of 8 uncovered questions answered anyway. With the pages: 34 of 36, 8 of 8 abstentions, none shown but wrong.
• Retrieval mattered more than the model: +6 right answers from BM25 to rerank; none from chat-small to chat-strong, at 18 times the cost.
• Both misses were retrieval misses, and both were safe abstentions.
• Pages hold their own history, and GOV.UK’s major-change date was years old on pages with 2026 rates.
• Limits. 44 questions written by one person; 36 with passages, so one question moves a measure by about 0.03. One snapshot of one topic. Answers recorded once; another run of the same model can differ. The automatic checks are a net with holes, so we read every answer.

Responsible use

• Not legal advice. GOV.UK itself says that it does not publish advice. The assistant says what the guidance says, not what a person should do (rule 5 of the prompt). The notice says so under every answer. A person with a real dispute needs an adviser, Acas or a solicitor, and deadlines matter (the pages give 6-month limits for several tribunal claims).
• Not an official service. The licence gives no right to suggest official status, and GOV.UK asks permission before anyone says it endorses them. Do not use GOV.UK’s name, crown or design in the assistant’s name or screens. The notice says: “This assistant is not a government service.”
• Show the source and the date of every page. The guidance changes every April at least. Every answer shows the fetch date and GOV.UK’s last update, and tells the reader to check the page. Refresh the snapshot, and run the tests and the evaluation before you replace the old one.
• Say where it does not apply. The pages are for England, Scotland and Wales. Questions from Northern Ireland need nidirect and the Labour Relations Agency; the assistant abstains, and an adviser should know why.
• Personal data. Advisers will type real stories: names, employers, health. Keep questions out of logs, and do not send them to a hosted model without consent and a contract that covers it. Only the model call leaves the computer.
• Attribution. Keep the licence’s attribution line wherever you show or publish text from the pages, including your repository.

Make it a portfolio piece

This project shows what employers look for in RAG work. It has a licensed data source, provenance, measured retrieval, checked citations, abstention, and an honest comparison with cost.

What to put in the repository:

guidance-assistant/
  README.md               what it is, what it is not, how to run and check it, the results
  DATASET.md              source, licence, attribution, fetch date, checksums, limits
  LICENSE                 MIT, for the code
  guidance/               the snapshot (OGL v3.0, with the attribution line in README and DATASET.md)
  guidance_assistant/     the code, including parse.py, chunk.py and sources.py
  questions/              the 44 questions with keys and passages
  recordings/             the recorded answers, so that anyone can replay them
  scripts/                check_snapshot.py, fetch_snapshot.py, build_snapshot.py
  tests/                  pytest tests (20, no model download needed)

A README template:

README.md (outline)

# guidance-assistant

## What it is
A cited Q&A assistant over 137 GOV.UK pages on rights at work (snapshot of 2026-10-09),
for volunteer advisers. Not legal advice; not a government service.

## Results (44 fixed questions, recorded 2026-10-09)
- Search: rerank hit@5 0.944, MRR 0.872 (BM25 0.778, 0.549).
- Answers, chat-small + rerank: 34/36 right, 8/8 abstained, 0 shown but wrong.
- The model alone: 31/36 by the keys, no sources, answered 7 of 8 uncovered questions.
- chat-strong: same scores on a 22-question sample, about 18 times the cost.

## Design choices
- Official Content API, one fetch, SHA-256 checked snapshot; OGL v3.0 attribution.
- Chunks keep the GOV.UK address with #anchor and the dates.
- The model gives passage IDs; the application writes the sources.

## Limits
(one snapshot, one topic, one author's questions, England, Scotland and Wales only)

How to describe it in an interview, in two sentences: “I built a cited assistant over a checked snapshot of 137 GOV.UK pages; code checks every citation, and the application writes each source with its date. On 44 fixed questions it answered 34 of 36 right and abstained on all 8 uncovered ones.”

Go further: each of these is a good next step, with one main change.

• Dates on sections. Read the dates in headings such as “Rates between 1 April 2024 and 31 March 2026” into effective dates. Filter by the question’s date, and measure Q27 and the dated questions again.
• A refresh. Fetch the same paths again in a few months. Compare the checksums, list the changed pages, and run the evaluation on both snapshots. Which recorded answers are now out of date?
• A second nation. Add the Northern Ireland pages from nidirect (check their licence first) with a “nation” label, and filter by it as the course filtered by access.
• A live model. Run the 44 questions with a local model through Ollama, and compare “shown but wrong” with the recorded models.

Knowledge check

Check your understanding

A colleague says: 'The model alone got 31 of 36 right. The pages only add 3 more right answers, so the search is not worth the work.' What does this leave out? Use numbers from this case study.

Show the answer

It counts right answers only. Without the pages, no answer has a source or a date. So an adviser cannot tell the 31 right answers from the wrong one; Q33 sounded just as sure. On the 8 questions that the guidance does not cover, the model alone answered 7 from its own knowledge. Two examples: a sick pay rate for Northern Ireland, and personal advice about a settlement agreement. With the pages and rerank, it abstained on all 8, showed no wrong answer, and linked every claim to a dated section. For an advice centre, a checkable answer and an honest “the guidance does not say” matter more than three more right answers.

Check your understanding

The current minimum wage table was ranked 6th, after two older tables of the same page. Why did no wrong rate reach the adviser, and what change would fix the search itself?

Show the answer

Each passage reached the model with its section heading, for example “Rates before 1 April 2021”. The prompt forbids guessing. Seeing no current table, chat-small abstained instead of quoting an old rate. To fix the search, give each history section the dates in its heading as effective dates. Then filter by the question’s date before the top 5, as the course filtered policy versions. Then measure the dated and table questions again, because the dated questions need their own date.

Check your understanding

Why does sources.py look up each cited ID in the index instead of asking the model to write the page address?

Show the answer

A model can write an address that looks real but points to another page, or to a page that is not in the snapshot. The IDs can be checked: the citation checks reject an ID that does not exist or was not in the context. The application writes the title, the address and the dates from the index. So every source line is the page that the passage came from, with its real dates. The model is still responsible for the claims, and the checks flag claims whose numbers or words are not in the passage.

Recap

• Check the licence of the exact pages, and its conditions: attribution, and no suggestion of official status.
• Fetch once through the official API, and keep a snapshot with checksums and the fetch date of every page.
• Keep the way back: blocks with section and anchor, chunks with stable IDs and addresses.
• Pages hold their own history: headings and dates must travel with every passage.
• The model gives claims and IDs; the application writes the sources, the dates and the notice.
• Better retrieval gave 6 more right answers; the strong model gave none, at 18 times the cost.

You built an assistant that a volunteer can check page by page. The next case study searches technical documentation, where exact names and paraphrases both matter.

Next: Search technical documentation