Answer from versioned regulations

What you will be able to do

Build an assistant that answers questions about two US air passenger regulations on a given date. It searches only the versions in force on that date and cites them with their dates. It says when a date is outside its data. You measure what the date filter changes: in search, in the answers, and in the citations.

Case studyPortfolio piece

Dataset

14 CFR Parts 250 and 254 at every version date: 44 section versions, eCFR, fetched on 2026-10-09

Licence

Public domain (US government work)

Time

About 3.5 hours

Runs in

VS Code, Python 3.12+

Skills used

• Point-in-time versions
• Effective dates from an official API
• A snapshot with checksums
• Chunks that carry their version
• A date filter before the top k
• A citation check for the version
• Abstention outside the snapshot
• recall@k with and without the filter
• Cost and latency

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter has the shape of the course’s policy-assistant. It holds the snapshot, the dated questions, the recorded answers and the parts you built in the course. You write the three files that are new here: versions, chunks, and the sources with their version check. The finished project is there to compare. You need no account: the mock replays the answers that real models gave.

Before you startWhat to know and to have readyAbout 210 minutes

What you need to know

This case study applies the whole course to rules that change. The lessons it uses most:

• Effective dates and stable chunk IDs, from Preserve metadata.
• The date filter before the top k, from Lexical, dense and hybrid search.
• The version in force and conflicting evidence, from Handle difficult evidence.
• Retiring a version by its dates, from Updates and deletions.
• recall@k and the answer measures, from Retrieval measures and Answer measures.

What you need on your computer

• Python 3.12 or newer, VS Code and a terminal, as in the course.
• About 2 GB of free disk space: 0.9 GB for the packages and 1 GB for the two local models. If you did the course, the models are already in your Hugging Face cache.

Tested on macOS (Apple silicon) with Python 3.12.13. The versions are the course’s: sentence-transformers 6.1.0, torch 2.14.1, pydantic 2.14.0, pytest 9.1.1. The Windows commands follow the course’s commands, but they are not tested.

The situation

Rosa Mendes leads the case workers of a small help line for air passengers in Chicago. People call months after a trip: “I was bumped from a full flight in June 2022. What did the airline owe me?” The answer is in a US federal regulation, 14 CFR Part 250. Its amounts change every few years, and a complaint must use the rule of the day of the flight.

Kofi Asante, a volunteer developer, offers to build an assistant that finds the right passage. Rosa agrees on three conditions. Every answer must name the version it used and its dates, so that a case worker can open the same text. For a date that the assistant cannot cover, it must say so instead of using the nearest version. And it must never look like legal advice: the help line writes complaints, and a lawyer decides the hard cases.

Rosa, Kofi and the help line are invented. The regulations, their versions and their dates are real.

The new idea: the version in force on a date

Think of a bus timetable. The company prints a new one every season and keeps the old ones in a drawer. “When was the last bus on 10 June 2022?” has one right answer, from the timetable of that season. If you search the whole drawer, you find six timetables that look almost the same, and the newest is not the right one.

The course met this with Larkfield’s returns policy, versions 3 and 4. A regulation is the same problem, larger: every section changes on its own dates. The text of one section between two change dates is a section version. The one whose dates include the question’s date is the version in force. A service that shows the text as it was on any past date gives point-in-time versions. Two sections of our data over ten years:

Two sections of 14 CFR over time, as the snapshot holds them. Each dot is the same question asked for another flight date.

Text description of the diagram

1. Section 250.5(a)(2) caps what an airline pays a bumped passenger who arrives one to two hours late. It was $675 until 2021-01-12, $775 from 2021-04-13, and $1,075 from 2025-01-22.
2. Section 254.4 sets the lowest baggage liability limit: $3,500, then $3,800, then $4,700, on the same dates.
3. Twice, an amendment was published three months before it came into force (orange windows). In those windows the old amount still applied.
4. The same question gets a different right answer for each flight date. For March 2015 the snapshot holds no version at all.

The orange windows are the trap. In December 2024 the new amount, $1,075, was already published, but the rule in force was still $775. A search that finds “the newest text” gives the wrong answer for any flight in that window.

What you will build

You will build regulations-assistant, a small Python project. The picture shows one question from start to end.

The design of regulations-assistant. Search and answers come from the course; the versions, the coverage check and the version check are new.

Text description of the diagram

1. A snapshot of the two parts on every date they changed becomes 44 section versions, then 134 chunks. Each chunk keeps its ID, paragraph, version dates and eCFR address.
2. A question comes with a date. If the date is before the snapshot’s history, the assistant says so and calls no model.
3. Otherwise, the date filter keeps only the versions in force on that date, before search chooses the top 5.
4. The model answers in claims with passage IDs, and names the version.
5. Code checks every citation, including that its version was in force on the date.
6. The answer card shows each source with its section, its version’s dates and its eCFR address.

The data

The regulations come from the eCFR, the Electronic Code of Federal Regulations, published by the Office of the Federal Register and the Government Publishing Office. We chose two short parts of Title 14. Part 250, Oversales, sets who an airline must ask first when a flight is full, and what it must pay a passenger it bumps. Part 254, Domestic baggage liability, sets the lowest limit on an airline’s liability for lost or late bags. Their amounts are reviewed on the same two-year cycle.

The licence, and what it asks of us

The CFR is a work of the US government, so it is in the public domain in the United States. Anyone may copy and adapt it, with no attribution required; we name the source under every answer anyway. One fact matters more than the licence: the eCFR is not an official legal edition of the CFR. The official texts are the annual CFR and the Federal Register. That becomes a rule of the project, in Responsible use.

We also looked at Canada’s Air Passenger Protection Regulations on the Justice Laws Website. Two of its four versions exist as XML under the Open Government Licence – Canada. The two older ones exist only as web pages under other terms. So we chose the eCFR, which gives every version through one documented API.

A snapshot, fetched once

The eCFR has a point-in-time API. Ask for a part “on” a date, and it returns the XML of that day. Another call lists every section version and its date. The script scripts/fetch_snapshot.py asked once for each date on which a section changed. It also asked for the day the eCFR was up to date to. That was 16 requests, one every 1.2 seconds. We kept every response exactly as it came, with its address, time and SHA-256 fingerprint, in regulations/.

Three things we learned on the way are now facts of the project:

• The history starts on 2017-01-01. The API returns nothing for 31 December 2016. Many texts were in force earlier, but the snapshot cannot show that. So the first version of each section says “2017-01-01 or earlier”.
• The eCFR was up to date to 2026-10-07, two days before we fetched it. For any later date, a new change could be missing. Even the course’s “today”, 2026-10-09, is after that date.
• The last versions are complete. The file of 2026-10-07 is byte for byte the file of 2025-01-22, for both parts: nothing changed in between.

First checks

We read the versions before writing any search. Three findings shape the project:

• Published is not in force. On 2021-01-13 and 2024-10-24, the eCFR added a line to the changed sections: “Link to an amendment published at 89 FR 84818, Oct. 24, 2024.” The new text came into force three months later. The line is a warning, not the new rule.
• A real error inside a version. From 2021-04-13 to 2021-08-01, § 250.5(b)(3), for flights to another country, said $1,350. In the same version, (a)(3) and the passenger notice in § 250.9 said $1,550. A correction changed (b)(3) on 2021-08-02.
• Rules that did not exist yet. § 250.7 was “[Reserved]” until 2021-01-12, then its heading was “xxx” for three months, then it became the rule on passengers already on board. A question about 2019 must not get the 2021 rule.

The questions

Before any search, Rosa and Kofi fix what a good answer is, as in Corpus and questions. The file questions/questions.jsonl has 34 questions, each with a date (as_of): the day of the flight, or the day of the question. Each has a reference answer, the relevant passages (section, version and paragraph), and keys: regular expressions that a right answer must match. The must_not patterns hold the amounts of the other versions, so an answer with the wrong version’s amount fails.

Kind	How many	Example	A right answer
Dated	15	Bumped on 10 June 2022, rebooked 90 minutes late: the most the airline had to pay?	$775 (version 2021-08-02 to 2024-10-23)
Pending	3	The same, on 20 December 2024	$775: the $1,075 amendment was published, not in force
Correction	1	Chicago to London on 1 June 2021, no flight within four hours: the limit?	$1,350, the text of § 250.5(b)(3) then
Absent	2	On 15 April 2019, could the airline remove a passenger whose boarding pass was scanned?	The version in force has no rule on it
Stable	6	What counts as a large aircraft in the baggage rules?	More than 60 seats (same in every version)
Before	2	Bumped on 10 March 2015: the most the airline had to pay?	The snapshot holds no version on that date
After	1	Bumped on 15 February 2027, no flight within two hours: the limit?	$2,150 in the latest version, with a warning
Not covered	4	How long can an airline keep passengers on the tarmac?	The two parts do not say

Groups ask one question on several dates. Q01 to Q05 and Q28 are the six dots of the timeline above: the same words, six flight dates, four different right answers.

The author wrote every question, answer and key by reading the versions; no model wrote or labelled a question. A script checks that each relevant passage exists and is in force on the question’s date, and that each key matches its reference answer. After reading the first answers, we changed one check. Q16’s must_not now also catches an answer that starts with “Yes”: a key had matched the words “did not state” in one. Then we froze the set.

Set up the project

The steps use VS Code and its terminal. Every command runs in the project folder.

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder regulations-assistant.

2. In VS Code, choose File > Open Folder, and open regulations-assistant. Then open a terminal: Terminal > New Terminal.

3. Make and activate a virtual environment:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1

   Linux

   python3 -m venv .venv
   source .venv/bin/activate

   The prompt starts with (.venv). If VS Code asks whether to use the new environment for this folder, choose Yes.

4. Install the course’s versions. On Linux, install the CPU-only PyTorch first: python -m pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu.

   python -m pip install -r requirements.txt

5. Run the tests, and check the snapshot:

   python -m pytest
   python scripts/check_snapshot.py

Output (end of pytest, then the last lines of check_snapshot.py)

======================== 5 passed, 13 skipped in 0.44s =========================
OK  versions_part-254.json             SHA-256 9196296c6050…
OK  sources.csv                        SHA-256 6c625a911adc…
17 files checked | fetched 2026-10-09 from the eCFR API | history from 2017-01-01 | up to date to 2026-10-07

What this tells us. The packages work, and the snapshot is the one we tested: every load compares the files with regulations/SHA256SUMS first, and a test changes $4,700 to $7,400 in a copy and checks that the load stops. The 13 skipped tests are for the three files that you write. Each one runs as soon as its file exists.

Look at the snapshot and at the questions whose amendment was published but not in force:

python -m regulations_assistant snapshot
python -m regulations_assistant questions --kind pending

Output

14 CFR Part 250 (Oversales): 7 files, on 2017-01-01, 2021-01-13, 2021-04-13, 2021-08-02, 2024-10-24, 2025-01-22, 2026-10-07
14 CFR Part 254 (Domestic baggage liability): 6 files, on 2017-01-01, 2021-01-13, 2021-04-13, 2024-10-24, 2025-01-22, 2026-10-07
16 files | fetched 2026-10-09 | history from 2017-01-01 | eCFR up to date to 2026-10-07 | checksums OK (regulations/SHA256SUMS)
Q02 2021-02-15 pending     A passenger was denied boarding involuntarily on an oversold domestic flight on…
Q04 2024-12-20 pending     A passenger was denied boarding involuntarily on an oversold domestic flight on…
Q14 2024-11-28 pending     On a domestic flight on 28 November 2024 with a large aircraft, what was the lo…
3 questions

Part 1: from dated copies to versions

The snapshot holds each part as it was on several dates. The search needs something else: each section’s versions, each with the dates it was in force. In Preserve metadata, the effective dates came from an inventory. Here we derive them from the copies. Walk the dates in order. When a section’s text differs from the previous date, a new version starts, and the old one ends the day before.

The eCFR’s XML has a DIV8 element for each section, with its heading, paragraphs (P), flush paragraphs (FP), tables, footnotes, and the notice of a published amendment (XREF). The source notes (CITA) list the Federal Register documents that made the text; they stay out of the searchable text. Python’s standard library reads XML, so the project needs no new package. Create regulations_assistant/versions.py:

regulations_assistant/versions.py

"""Section versions: what each section said, and from when to when it was in force.

The snapshot has each part as it was on several dates. This file turns those copies into versions:
1. Parse a part (eCFR XML) into sections, and each section into blocks: paragraphs, tables,
   headings inside a section, and the eCFR's notice of a published amendment ("Link to an
   amendment published at 89 FR 84818, Oct. 24, 2024."). The source note (CITA) and the
   approval note (APPRO) are kept out of the text. Each block keeps its top-level paragraph
   label, "(a)", "(b)"..., so that a citation can name it.
2. Walk the dates in order. When a section's text differs from the day before, a new version
   starts on that date, and the previous version ends the day before.
3. The first version starts on the first date of the snapshot (2017-01-01, where the eCFR's history
   starts): it may have been in force earlier, but the snapshot cannot say. The last version has no
   end date: it is still in force on the day the eCFR was up to date to (current_to).

`coverage()` says whether a question's date is inside what the snapshot can answer.
"""

import re
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from datetime import date, timedelta

TOP_LABEL = re.compile(r"^\(([a-z])\)\s")     # "(a) Subject to ..." starts paragraph (a)
CONTAINERS = {"DIV", "EXTRACT"}                 # wrappers: read what is inside them


def clean(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


@dataclass(frozen=True)
class Block:
    kind: str           # paragraph, table, heading, notice
    text: str
    paragraph: str      # the top-level label, for example "(b)"; "" before the first label


@dataclass(frozen=True)
class SectionText:
    section: str        # 250.5
    heading: str        # § 250.5 Amount of denied boarding compensation for ...
    blocks: tuple
    amended_by: str     # the source note: the Federal Register documents that made this text

    @property
    def text(self) -> str:
        return "\n".join(b.text for b in self.blocks)


@dataclass(frozen=True)
class SectionVersion:
    section: str
    part: str
    heading: str
    effective_from: str     # YYYY-MM-DD
    effective_to: str       # YYYY-MM-DD, or "" when still in force on current_to
    blocks: tuple = field(repr=False)
    amended_by: str = ""
    pending: str = ""       # the eCFR's notice of an amendment published but not yet in force

    @property
    def text(self) -> str:
        return "\n".join(b.text for b in self.blocks)

    def in_force(self, as_of: str) -> bool:
        return self.effective_from <= as_of and (not self.effective_to or as_of <= self.effective_to)

    @property
    def url(self) -> str:
        """The eCFR page of this section as it was on the first day of this version."""
        return f"https://www.ecfr.gov/on/{self.effective_from}/title-14/section-{self.section}"


def _blocks(element, out: list, label: list) -> None:
    for child in element:
        tag = child.tag
        if tag in CONTAINERS:
            _blocks(child, out, label)
        elif tag == "P" or tag.startswith("FP") or tag == "FTNT":     # FP: a flush paragraph; FTNT: a footnote
            text = clean("".join(child.itertext()))
            m = TOP_LABEL.match(text)
            if tag == "P" and m and (not label[0] or ord(m.group(1)) == ord(label[0][1]) + 1 or m.group(1) == "a"):
                label[0] = f"({m.group(1)})"
            if text:
                out.append(Block("paragraph", text, label[0]))
        elif tag == "TABLE":
            rows = [" | ".join(clean("".join(td.itertext())) for td in tr.iter("TD")) for tr in child.iter("TR")]
            out.append(Block("table", "\n".join(r for r in rows if r.strip(" |")), label[0]))
        elif tag in ("HD1", "HD2", "HD3"):
            out.append(Block("heading", clean("".join(child.itertext())), label[0]))
        elif tag == "XREF" and "amendment published" in "".join(child.itertext()):
            out.append(Block("notice", clean("".join(child.itertext())), label[0]))


def parse_part(xml: bytes) -> dict[str, SectionText]:
    """One part on one date: {section number: its heading, blocks and source note}."""
    root = ET.fromstring(xml)
    sections = {}
    for div in root.iter("DIV8"):
        blocks: list = []
        _blocks(div, blocks, [""])
        cita = div.find("CITA")
        sections[div.get("N")] = SectionText(div.get("N"), clean("".join(div.find("HEAD").itertext())),
                                             tuple(blocks), clean("".join(cita.itertext())) if cita is not None else "")
    return sections


def day_before(day: str) -> str:
    return (date.fromisoformat(day) - timedelta(days=1)).isoformat()


def build_versions(snapshot) -> list[SectionVersion]:
    """Every version of every section in the snapshot, in order of section and date."""
    versions = []
    for part in sorted({f.part for f in snapshot.files if f.kind == "part"}):
        current: dict[str, tuple[str, SectionText]] = {}     # section -> (from, text) of the open version
        for day in snapshot.dates(part):
            sections = parse_part(snapshot.xml(part, day))
            for number in list(current):                     # a section that disappears ends the day before
                if number not in sections:
                    start, old = current.pop(number)
                    versions.append(_version(part, old, start, day_before(day)))
            for number, now in sections.items():
                if number in current and (current[number][1].heading, current[number][1].text) == (now.heading, now.text):
                    continue                                 # same text: the same version goes on
                if number in current:
                    start, old = current[number]
                    versions.append(_version(part, old, start, day_before(day)))
                current[number] = (day, now)
        for start, text in current.values():
            versions.append(_version(part, text, start, ""))
    return sorted(versions, key=lambda v: (_order(v.section), v.effective_from))


def _version(part: str, s: SectionText, start: str, end: str) -> SectionVersion:
    pending = next((b.text for b in s.blocks if b.kind == "notice"), "")
    return SectionVersion(s.section, part, s.heading, start, end, s.blocks, s.amended_by, pending)


def _order(section: str) -> list:
    return [int(x) if x.isdigit() else x for x in re.split(r"(\d+)", section)]


def in_force(versions: list[SectionVersion], section: str, as_of: str) -> SectionVersion | None:
    return next((v for v in versions if v.section == section and v.in_force(as_of)), None)


def coverage(as_of: str, snapshot) -> str:
    """'before': the snapshot holds no version on that date; 'after': the date is later than the day
    the eCFR was up to date to, so a newer change may be missing; 'inside' otherwise."""
    if as_of < snapshot.history_start:
        return "before"
    if as_of > snapshot.current_to:
        return "after"
    return "inside"

Three choices matter. A version is named by its first day, so its name is also its start date. The notice “Link to an amendment published at …” is kept as its own block, marked notice: the model must see it, and pending says that a change is coming. And coverage() is a separate question from the filter: “can the snapshot answer for this date at all?”

Run the tests, then look at one section over time:

python -m pytest
python -m regulations_assistant versions 254.4

Output (the end of pytest, then the versions)

========================= 9 passed, 9 skipped in 0.07s =========================
254.4   2017-01-01 to 2021-01-12 |   71 words | § 254.4 Carrier liability.
254.4   2021-01-13 to 2021-04-12 |   83 words | § 254.4 Carrier liability.  (amendment published, not yet in force)
254.4   2021-04-13 to 2024-10-23 |   71 words | § 254.4 Carrier liability.
254.4   2024-10-24 to 2025-01-21 |   83 words | § 254.4 Carrier liability.  (amendment published, not yet in force)
254.4   2025-01-22 to today      |   71 words | § 254.4 Carrier liability.
5 versions of 1 section

What this tells us. One short section has five versions; three of them are the rule with a new amount, and two are the old rule with a notice added. Over both parts, versions finds 44 versions of 18 sections. A test compares them with the eCFR’s own list of versions: the same 44 sections and dates, so our comparison of texts missed no change and invented none.

Part 2: chunks that carry their version

Cut each version into chunks as in Chunk deliberately: between top-level paragraphs, at most 200 words, never inside a block. Two things are new. The chunk ID hashes the section, the version’s first day and the positions. So the same paragraph in six versions gets six IDs, and a citation always points to one version. And each chunk carries its version’s dates and the eCFR address of that version, https://www.ecfr.gov/on/2021-04-13/title-14/section-250.5, which opens the text as it was that day.

Create regulations_assistant/chunk.py:

regulations_assistant/chunk.py

"""Chunks of section versions, with stable IDs and the version's dates, as in the course's policy assistant.

A short section version is one chunk. A long one (250.5 has about 1,000 words) is cut between its
top-level paragraphs, (a), (b), ..., into chunks of at most MAX_WORDS words. A paragraph that is
longer than that is cut between its blocks; a block (one sub-paragraph, or a table) is never cut.
Every chunk starts with the section's heading, so "(b) ... $1,550" still says which section it is.

The chunk ID is the first 12 hexadecimal characters of SHA-256 over
"section|effective_from|start|end": two versions of the same paragraph get two IDs, so a citation
always points to one version. Every chunk carries the version's dates; the date filter and the
citation checks use them.
"""

import hashlib
from dataclasses import asdict, dataclass

from .versions import SectionVersion

MAX_WORDS = 200


def chunk_id(section: str, effective_from: str, start: int, end: int) -> str:
    return hashlib.sha256(f"{section}|{effective_from}|{start}|{end}".encode("utf-8")).hexdigest()[:12]


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    section: str            # 250.5
    part: str               # 250
    heading: str            # § 250.5 Amount of denied boarding compensation ...
    version: str            # a version is named by its first day (= effective_from)
    effective_from: str
    effective_to: str       # "" = still in force on the day the eCFR was up to date to
    position: int
    start: int              # character positions in the version's text
    end: int
    paragraphs: str         # the top-level paragraphs it holds, for example "(a),(b)"
    text: str               # what is indexed and shown to the model
    url: str                # the eCFR page of the section on the version's first day
    pending: str            # the eCFR's notice of a published amendment not yet in force, or ""

    def to_dict(self) -> dict:
        return asdict(self)

    def in_force(self, as_of: str) -> bool:
        return self.effective_from <= as_of and (not self.effective_to or as_of <= self.effective_to)


def words(spans: list) -> int:
    return sum(len(block.text.split()) for block, _, _ in spans)


def chunks_of(version: SectionVersion, max_words: int = MAX_WORDS) -> list[Chunk]:
    spans, pos = [], 0                   # (block, start, end) in the version's text
    for block in version.blocks:
        spans.append((block, pos, pos + len(block.text)))
        pos += len(block.text) + 1       # the blocks are joined with "\n"
    paragraphs: list[list] = []          # the blocks of each top-level paragraph; a notice joins the next
    for span in spans:
        if paragraphs and (span[0].paragraph == paragraphs[-1][-1][0].paragraph
                           or paragraphs[-1][-1][0].kind == "notice"):
            paragraphs[-1].append(span)
        else:
            paragraphs.append([span])
    parts: list[list] = [[]]
    for paragraph in paragraphs:
        if parts[-1] and words(parts[-1]) + words(paragraph) > max_words:
            parts.append([])             # the paragraph does not fit: start a new chunk
        for span in paragraph:           # a paragraph too long for one chunk is cut between blocks
            if parts[-1] and words(parts[-1]) + words([span]) > max_words:
                parts.append([])
            parts[-1].append(span)
    return [_make(version, part, i) for i, part in enumerate(parts)]


def _make(version: SectionVersion, spans: list, position: int) -> Chunk:
    start, end = (spans[0][1], spans[-1][2]) if spans else (0, 0)   # a reserved section has no text
    labels = list(dict.fromkeys(block.paragraph for block, _, _ in spans if block.paragraph))
    body = "\n".join(block.text for block, _, _ in spans)
    return Chunk(
        chunk_id=chunk_id(version.section, version.effective_from, start, end), section=version.section,
        part=version.part, heading=version.heading, version=version.effective_from,
        effective_from=version.effective_from, effective_to=version.effective_to, position=position,
        start=start, end=end, paragraphs=",".join(labels), text=f"14 CFR {version.heading}\n{body}".strip(),
        url=version.url, pending=version.pending)


def chunk_all(versions: list[SectionVersion]) -> list[Chunk]:
    return [c for v in versions for c in chunks_of(v)]

Run the tests, build the index, and look at the chunks of one section:

python -m pytest
python -m regulations_assistant ingest
python -m regulations_assistant chunks 254.4

Output

======================== 13 passed, 5 skipped in 0.12s =========================
index index.sqlite: 134 chunks from 44 section versions | embeddings intfloat/multilingual-e5-small | 27.9 s
db8de54f5d91 2017-01-01 to 2021-01-12 #0 -             77 words | On any flight segment using large aircraft,…
cb303a8b8934 2021-01-13 to 2021-04-12 #0 -             89 words | Link to an amendment published at 86 FR 253…
e9c55bed3e23 2021-04-13 to 2024-10-23 #0 -             77 words | On any flight segment using large aircraft,…
50c05ac5e1c7 2024-10-24 to 2025-01-21 #0 -             89 words | Link to an amendment published at 89 FR 848…
ef9d00e64081 2025-01-22 to today      #0 -             77 words | On any flight segment using large aircraft,…
5 chunks

What this tells us. Five versions, five chunks, five IDs, and almost the same text: only the amount differs, plus the notice in two of them. To a search engine, they are near-twins. The median chunk has 153 words. ingest took 27.9 seconds on our Mac, mostly loading the e5 model.

Part 3: search, with and without the date filter

The index holds every version of every section. The date filter of Lexical, dense and hybrid search keeps only the chunks in force on the question’s date. It runs before search chooses the top 5. The search command can turn it off. Try Q03’s question, a flight on 10 June 2022, both ways:

python -m regulations_assistant search "A passenger was denied boarding involuntarily on an oversold domestic flight on 10 June 2022. The airline rebooked them on a flight that arrived 90 minutes after their original flight. What is the most the airline had to pay as denied boarding compensation?" --as-of 2022-06-10

Then run the same command with --as-of none.

A real search (Q03, rerank). Without the filter, all five places go to versions that ended in 2021; with it, all five are in force on the flight date.

Text description of the diagram

Without the filter, the reranker’s top 5 holds four chunks of versions that ended in April 2021 and one that ended in January 2021. None was in force on 10 June 2022, and the right version is not among them. The model saw the dates in the headers and abstained. With the filter, all five chunks are in force on that date. They include § 250.5(a) of the version from 2021-08-02, and the model answered $775.

The old versions win the top places because they are not wrong about words: they describe the same rule, with another amount. The reranker cannot know that the date in the question makes them irrelevant. Measure it on all 26 questions that have relevant passages, with each search method, filter on and off:

python -m regulations_assistant eval --method rerank
python -m regulations_assistant eval --method rerank --no-date-filter

Output

rerank, top 5, date filter on: hit@5 0.923 | recall@5 0.865 | precision@5 0.462 | MRR 0.827 (26 questions with relevant passages)
  chunks not in force on the question's date, in the top 5: 0 (in 0 of 34 questions); first chunk from another version of the right paragraph: 0
rerank, top 5, date filter off: hit@5 0.731 | recall@5 0.500 | precision@5 0.169 | MRR 0.415 (26 questions with relevant passages)
  chunks not in force on the question's date, in the top 5: 132 (in 33 of 34 questions); first chunk from another version of the right paragraph: 12

A chunk counts as relevant only if it comes from the right version: the same paragraph of another version is not a hit. The other methods, with --method bm25, dense and hybrid, give:

Method	hit@5, filter on	hit@5, filter off	MRR on / off	Top-5 chunks not in force, filter off (of 170)
BM25	0.885	0.731	0.673 / 0.476	115
Dense (e5)	0.885	0.538	0.731 / 0.332	136
Hybrid (RRF)	0.846	0.577	0.708 / 0.396	121
Rerank	0.923	0.731	0.827 / 0.415	132

• Rerank, with the date filter
• Rerank, without the date filter (dashed)

recall@k of the reranker on the 26 questions with relevant passages, with and without the date filter (snapshot of 2026-10-09, e5, the course's reranker). Relevant means the right version.

Text description of the chart

Two lines show recall at k = 1, 3, 5 and 10. With the date filter: 0.481, 0.788, 0.865, 1.000. Without it: 0.173, 0.365, 0.500, 0.635. The line without the filter is lower at every k, and the gap does not close at k = 10.

What this tells us. Without the filter, about four of every five top-5 places go to versions not in force on the question’s date. Every method loses, and dense search most: to an embedding model, six versions of one paragraph mean the same thing. The gap stays at k = 10, so more passages do not fix it. With the filter, the reranker finds the right version for 24 of 26 questions, in about 0.5 seconds each. Its two misses (Q22, Q23) are stable sections that other passages pushed out.

Part 4: a baseline, the model alone

Before you trust the snapshot to help, measure what the model knows without it, as in What retrieval-augmented generation is. Ask chat-small Q04, a flight on 20 December 2024, with no passages:

python -m regulations_assistant ask Q04 --closed-book

Output

Question (2024-12-20): A passenger was denied boarding involuntarily on an oversold domestic flight on 20 December 2024. The airline rebooked them on a flight that arrived 90 minutes after their original flight. What is the most the airline had to pay as denied boarding compensation?
The model alone, without the regulations:
For a 90-minute arrival delay, the airline owed 200% of the passenger’s one-way fare, capped at **$1,075**.

gpt-6-luna-2026-09-22 | 106 tokens in, 329 out | 3.112 s

This answer is confident and wrong. $1,075 was published on 24 October 2024, but it came into force on 22 January 2025; on 20 December 2024 the rule was $775. The model knows the new amount and does not know when it started.

Over the whole set, the model alone knows a lot: by the keys, 20 of the 26 answerable questions right. We read the 6 misses. Three use the wrong version (Q04; $1,350 for September 2021, Q10; the old form number for 2023, Q21). One misses the 2021 rule on boarded passengers (Q19), one gives the wrong section as its reason (Q17), and one is empty: hidden reasoning used all 2,000 tokens (Q15). It also answered all 8 questions that the snapshot cannot answer.

One of those 8 answers teaches the most: Q28, a flight on 10 March 2015. The model alone said $650, with no source. The Federal Register shows that this is right. The limit rose to $675 on 25 August 2015. Our snapshot, which starts in 2017, cannot know that. Keep this in mind for Part 6.

Part 5: answers that name their version

The answer pipeline is the course’s ask(), with two additions. Before the search, coverage() decides whether the date is inside the snapshot; before 2017-01-01, the assistant answers without a model. After the answer, a new check runs: every cited passage must be in force on the question’s date. The checks of Answer and cite cannot see this: “$775”, cited to the 2021 version for a flight in 2025, passes all five.

The application, not the model, writes each source line from the index: the section, the paragraph, the version’s dates and the eCFR address. Create regulations_assistant/sources.py:

regulations_assistant/sources.py

"""Show an answer the way a case worker should see it, and check that it cites the right version.

The model gives claims and passage IDs. The application writes everything else from the index:
- every source with its section, the dates of its version, and the eCFR address of that version;
- a line about the question's date when the snapshot cannot be sure (after the day the eCFR was up
  to date to: a newer change would be missing);
- the notice: what this is not, and that the eCFR is not an official legal edition.

`check_versions` is the citation check that this case study adds: every cited passage must come
from a version in force on the question's date. The course's checks (cite.py) would pass a claim
that cites the $775 version for a flight in 2025; this one flags it as wrong_version.
"""

from .snapshot import ATTRIBUTION

NOTICE = ("Question date {as_of}. This is what the regulation's text said, not legal advice. "
          "Check the eCFR page and the Federal Register before you rely on it.")
AFTER = ("Note: {as_of} is after {current_to}, the day the eCFR was up to date to when this snapshot was made. "
         "A change made after {current_to} is not in it.")


def check_versions(answer, store, as_of: str) -> list[str]:
    """The cited IDs whose version was not in force on as_of (in order of first citation)."""
    wrong = []
    for cid in answer.cited():
        chunk = store.chunk(cid)
        if chunk is not None and not chunk.in_force(as_of):
            wrong.append(cid)
    return wrong


def dates(chunk) -> str:
    return f"in force {chunk.effective_from} to {chunk.effective_to or 'today'}"


def render(answer, checks: list, store, as_of: str, snapshot) -> str:
    head = [AFTER.format(as_of=as_of, current_to=snapshot.current_to), ""] if as_of > snapshot.current_to else []
    foot = ["", NOTICE.format(as_of=as_of), ATTRIBUTION]
    if not answer.answerable:
        return "\n".join(head + [f"Answer: {answer.answer}"] + foot)
    sources = [c for c in (store.chunk(i) for i in answer.cited()) if c is not None]
    number = {c.chunk_id: i for i, c in enumerate(sources, 1)}
    wrong = set(check_versions(answer, store, as_of))
    lines = head + [f"Answer: {answer.answer}", ""]
    for claim, check in zip(answer.claims, checks):
        refs = "".join(f"[{number[i]}]" for i in claim.chunk_ids if i in number) or "[no source]"
        problems = list(check.problems) + [f"wrong_version:{i}" for i in claim.chunk_ids if i in wrong]
        lines.append(f"- {claim.text} {refs}" + (f"  (check: {'; '.join(problems)})" if problems else ""))
    lines += ["", f"Sources (question date {as_of}):"]
    for i, c in enumerate(sources, 1):
        flag = "" if c.chunk_id not in wrong else f"  NOT IN FORCE ON {as_of}"
        lines.append(f"[{i}] 14 CFR {c.section} {c.paragraphs}, version {dates(c)}{flag}")
        lines.append(f"    {c.url}")
    return "\n".join(lines + foot)

Run the tests, then ask Q04 with the snapshot:

python -m pytest
python -m regulations_assistant ask Q04

Output (recorded answer, chat-small)

============================== 18 passed in 0.14s ==============================
Question (2024-12-20): A passenger was denied boarding involuntarily on an oversold domestic flight on 20 December 2024. The airline rebooked them on a flight that arrived 90 minutes after their original flight. What is the most the airline had to pay as denied boarding compensation?
Date filter: 2024-12-20
Passages: 1c4d6d940b9a, a17194a3d78a, 6dda423ba161, 2f1842e9ccb9, 503f2a66ff16 (1680 tokens, estimated)

Answer: Under the versions in force from 2024-10-24 to 2025-01-21, a 90-minute arrival delay falls in the more-than-one-but-less-than-two-hour category. The regulation specifies compensation of at least 200% of the fare or $775, whichever is lower, so $775 is the highest amount specified by that formula; the passenger’s fare is not provided.

- The applicable category is an alternate arrival more than one hour but less than two hours after the original planned arrival. [1][2]
- For domestic travel in that category, the regulation specifies compensation of at least 200% of the fare or $775, whichever is lower. [1][2]

Sources (question date 2024-12-20):
[1] 14 CFR 250.5 (a), version in force 2024-10-24 to 2025-01-21
    https://www.ecfr.gov/on/2024-10-24/title-14/section-250.5
[2] 14 CFR 250.9 (b), version in force 2024-10-24 to 2025-01-21
    https://www.ecfr.gov/on/2024-10-24/title-14/section-250.9

Question date 2024-12-20. This is what the regulation's text said, not legal advice. Check the eCFR page and the Federal Register before you rely on it.
Source: eCFR (ecfr.gov), 14 CFR Parts 250 and 254, fetched 2026-10-09. US government work, public domain. The eCFR is not an official legal edition of the CFR.

gpt-6-luna-2026-09-22 | 1927 tokens in, 915 out | 7.286 s

What this tells us. The passage of § 250.5 that the model saw starts with “Link to an amendment published at 89 FR 84818”. The model did not use the coming amount. It named the version and its dates, as rule 4 of the prompt asks. Both sources link to the text as it was on 2024-10-24. The model alone said $1,075 for the same question.

Now a date before the snapshot:

python -m regulations_assistant ask Q28

Output

Question (2015-03-10): A passenger was denied boarding involuntarily on an oversold domestic flight on 10 March 2015. The airline rebooked them on a flight that arrived 90 minutes after their original flight. What is the most the airline had to pay as denied boarding compensation?
No search and no model call: the date is outside the snapshot.

Answer: The snapshot holds no version in force on 2015-03-10: its history starts on 2017-01-01. The rule may have existed then, but this assistant cannot show it.

Question date 2015-03-10. This is what the regulation's text said, not legal advice. Check the eCFR page and the Federal Register before you rely on it.
Source: eCFR (ecfr.gov), 14 CFR Parts 250 and 254, fetched 2026-10-09. US government work, public domain. The eCFR is not an official legal edition of the CFR.

This is the abstention of Handle difficult evidence, decided by code instead of the model. “No version in the snapshot” is not “no rule”: the answer says that the rule may have existed.

Common mistake

The answer is right, for another date

Symptom. A case worker writes a complaint with the amount the assistant gave. The airline replies that the amount was not in force on the day of the flight.

Cause. The citation checks passed: the ID was real, it was in the context, and the amount was in the passage. But the passage came from another version. Nothing checked the version against the date.

Fix. Filter by the date before the top k, so that other versions cannot be cited. Check every cited passage against the date in code (check_versions), and show each source’s version and dates on the answer card. A wrong version is then visible to the reader, not only to a log.

Part 6: what goes wrong without the date filter

With --no-date-filter, the project runs a naive assistant, with no date filter and no coverage check. We recorded chat-small on all 34 questions this way. Start with Q03, the search of Part 3:

python -m regulations_assistant ask Q03 --no-date-filter

Output (part)

Date filter: off
Passages: 5077dd776912, 69e13aa7108b, 6f13357b3c16, 74b46264229e, 4123ef37981b (1680 tokens, estimated)

Answer: The regulations in the passages do not say what the maximum denied boarding compensation was under the version in force on 2022-06-10.

The model did not use an old amount. It read the dates in the headers, saw that no passage was in force on 10 June 2022, and abstained. As with Larkfield’s policies, the dates in the header do much of the work, but the right answer is lost. Q05 failed the same way.

Now Q28, the flight of March 2015, without the coverage check:

python -m regulations_assistant ask Q28 --no-date-filter

Output (part)

Answer: Under 14 CFR 250.9, in the version in force from 2017-01-01 or earlier to 2021-01-12, a 90-minute arrival delay falls in the tier requiring 200% of the fare, capped at $675. Therefore, the most the airline had to pay was $675.

- For domestic travel, alternate transportation arriving more than one hour but less than two hours after the original planned arrival time requires compensation of 200% of the fare, capped at $675, under the version in force from 2017-01-01 or earlier to 2021-01-12. [1]  (check: wrong_version:69e13aa7108b)
- A 90-minute delay is more than one hour and less than two hours, so the maximum compensation was $675. [1]  (check: wrong_version:69e13aa7108b)

Sources (question date 2015-03-10):
[1] 14 CFR 250.9 (b), version in force 2017-01-01 to 2021-01-12  NOT IN FORCE ON 2015-03-10

This is a real wrong-version answer, worse than the model alone. The header said “2017-01-01 or earlier”, and the model read it as “also in 2015”. The true limit then was $650. Only the coverage check in code prevents this; here the version check caught it, and the card says “NOT IN FORCE”. The baggage question for June 2016 (Q29) got $3,500 the same way, right only by luck.

The third case is quieter. For the flight to London on 1 June 2021 (Q09), § 250.5(b) of that version was pushed out of the top 5. The model answered from § 250.9’s notice, in force too, with $1,550: the real error of the first checks. With the filter, it saw both texts and answered $1,350 from § 250.5. Neither answer mentioned the conflict, so a person must, as in conflicting evidence.

Part 7: a fair comparison

All runs below use the same 34 questions, the same snapshot, prompt answer_v1 and the top 5. The mock replays the answers recorded on Azure:

python -m regulations_assistant eval --answers
python -m regulations_assistant eval --answers --no-date-filter
python -m regulations_assistant eval --answers --method bm25

Output (the first run)

chat-small, rerank, top 5, answer_v1, date filter on: 34 questions
  answerable, correct:          24/26
  answerable, abstained:        2/26
  no answer expected, correct:  6/8
  shown but wrong:              2  (Q16, Q33)
  citations of the right version: 33/33 (100.0%)
  claims flagged by the checks: 3 of 39
  model calls:                  32 (the other questions were answered without one)
  tokens in / out (median):     1794 / 328
  latency median / p95:         2.86 / 4.43 s (recorded)
  cost per question (median):   US$0.00035

chat-small, 34 questions	Rerank, filter on	Rerank, filter off	BM25, filter on
Answerable, correct	24/26	20/26	20/26
Answerable, abstained	2	5	5
No answer expected, correct	6/8	4/8	7/8
Shown but wrong	2 (Q16, Q33)	5 (Q09, Q16, Q28, Q29, Q33)	2 (Q09, Q33)
Citations of the right version	33/33	23/25	23/23
Latency, median / p95 (recorded)	2.86 / 4.43 s	2.61 / 4.22 s	2.84 / 4.96 s
Cost per question (median)	US$0.00035	US$0.00030	US$0.00032

What this tells us. The date filter gave 4 more right answers at the same model and cost, kept every citation on the right version, and turned three shown-but-wrong answers into right answers or honest abstentions. Without it, the losses are mostly abstentions: the right version was crowded out, and the headers stopped the model from using a wrong one.

Two answers are still shown but wrong. For Q16 (2020), both models said “Yes”: § 250.9 says the airline pays, so they inferred that it must offer unasked. That rule came only in 2021. For Q33, a flight to Paris, the model gave the domestic minimum, $4,700. Search had missed § 254.2, which limits Part 254 to domestic flights. These are failures from Find where it fails: an inference the text does not make, and a missed scope passage.

Now compare the models. We recorded chat-strong on a sample: the 17 questions with an even number, which include every kind. Compare it with chat-small on the same 17:

python -m regulations_assistant eval --answers --ids Q02,Q04,Q06,Q08,Q10,Q12,Q14,Q16,Q18,Q20,Q22,Q24,Q26,Q28,Q30,Q32,Q34 --model chat-strong

On the 17 even questions	chat-small, filter on	chat-strong, filter on	chat-small, filter off	chat-strong, filter off
Answerable, right	10/12	11/12	9/12	9/12
No answer expected, correct	4/5	4/5	3/5	4/5
Shown but wrong	1 (Q16)	1 (Q16)	2 (Q16, Q28)	1 (Q28)
Citations of the right version	13/13	14/14	10/11	9/10
Latency, median / p95 (recorded)	2.68 / 4.16 s	4.34 / 4.86 s	2.73 / 4.93 s	3.86 / 7.39 s
Cost per question (median)	US$0.00031	US$0.00574	US$0.00030	US$0.00556

What this tells us. The strong model got one more right: Q24, which needs two passages. It made the same Q16 inference, and without the filter it also said $675 for 2015. So the date filter helped more than the stronger model, at no extra cost. chat-strong costs about 18 times more per question; with no clear gain, chat-small with the filter is the choice. With the reranker (about 0.5 s) and the model (about 2.9 s), a case worker waits about 3.4 seconds.

The final design, end to end

The final design uses prompt answer_v1, chat-small, rerank top 5, the date filter and the coverage check. It checks the version of every citation. Here is one question from start to end, for a date after the snapshot:

python -m regulations_assistant ask Q30

Output (recorded answer, chat-small)

Question (2027-02-15): A passenger was denied boarding involuntarily on an oversold domestic flight on 15 February 2027. The airline offered no other flight arriving within two hours of the original one. What was the compensation limit?
Date filter: 2027-02-15
Passages: 29aa7740d1df, cde04502622d, 6dda423ba161, 56caaf943f74, 786f4a9fd5cb (1696 tokens, estimated)

Note: 2027-02-15 is after 2026-10-07, the day the eCFR was up to date to when this snapshot was made. A change made after 2026-10-07 is not in it.

Answer: In the version in force from 2025-01-22 to today, compensation is at least 400% of the fare, capped at $2,150 (whichever is lower).

- For an involuntary denial on a domestic oversold flight with no alternate transportation planned to arrive less than two hours after the original flight, compensation is at least 400% of the fare or $2,150, whichever is lower. [1]

Sources (question date 2027-02-15):
[1] 14 CFR 250.9 (b), version in force 2025-01-22 to today
    https://www.ecfr.gov/on/2025-01-22/title-14/section-250.9

Question date 2027-02-15. This is what the regulation's text said, not legal advice. Check the eCFR page and the Federal Register before you rely on it.
Source: eCFR (ecfr.gov), 14 CFR Parts 250 and 254, fetched 2026-10-09. US government work, public domain. The eCFR is not an official legal edition of the CFR.

gpt-6-luna-2026-09-22 | 1882 tokens in, 281 out | 2.606 s

The answer uses the latest version and names its first day. The application adds the note. § 250.5(e) says that the department reviews the amounts every two years. A case worker must check the eCFR for a flight in 2027.

To ask a question of your own, you need a live model. The mock knows only the recorded questions, and says so: “No recording for this request”. Copy .env.example to .env and choose a free local model with Ollama, or your own key, as in the course. Your answers will differ from the recorded ones; the filter, the checks and the sources work the same way.

Insights and limits

• The filter works in search. Without it, 132 of 170 top-5 places went to versions not in force, and right answers fell from 24 to 20.
• Dates in the header limit the damage. Inside the snapshot the models abstained rather than use a wrong version.
• Outside the snapshot, code decides. The wrong-version answers came from dates before 2017.
• The model alone knows amounts, not dates: $1,075 for December 2024, and no source for anything.
• A version can be wrong. In 2021, two sections in force disagreed; a person must settle it.
• Limits. One person wrote the 34 questions. Only 26 have passages, so one question moves a retrieval measure by about 0.04. Two short parts of one title. Answers recorded once; another run can differ. The automatic checks are a net with holes, so we read every answer.

Responsible use

• Not legal advice. The assistant says what one version’s text said, not what a passenger should get. Real cases depend on the fare, the ticket and the reason for the denial. A person checks anything that matters, and a lawyer decides the hard cases.
• Always show the version and its dates. Every source links to the eCFR text of the version’s first day. A reader can then see a wrong version, even when every other check passes.
• The eCFR is not the official text, and it can lag. The official texts are the annual CFR and the Federal Register. The snapshot was up to date to 2026-10-07. Refresh it, evaluate again, and check the Federal Register for amendments not yet in the eCFR.
• Say what the snapshot cannot cover: dates before 2017, baggage on flights abroad, delays, refunds and tarmac rules. The assistant abstains; a case worker should know why.
• Personal data. Keep callers’ names and bookings out of logs, and out of a hosted model without consent and a contract that covers it.

Make it a portfolio piece

This project shows a skill that many RAG projects lack: answers that are right for a date. It has an official source and versions checked against the publisher’s own list. It has a measured filter, a version check in code, and an honest comparison with cost.

What to put in the repository:

regulations-assistant/
  README.md               what it is, what it is not, how to run and check it, the results
  DATASET.md              source, licence, fetch date, history start, up-to-date date, checksums, limits
  LICENSE                 MIT, for the code
  regulations/            the snapshot (public domain, with the source named in README and DATASET.md)
  regulations_assistant/  the code, including versions.py, chunk.py and sources.py
  questions/              the 34 dated questions with keys and passages
  recordings/             the recorded answers, so that anyone can replay them
  scripts/                check_snapshot.py, fetch_snapshot.py, build_snapshot.py
  tests/                  pytest tests (18, no model download needed)

A README template:

README.md (outline)

# regulations-assistant

## What it is
A cited Q&A assistant over 14 CFR Parts 250 and 254 at every version date (eCFR snapshot of
2026-10-09), for passenger case workers. Every answer names its version. Not legal advice.

## Results (34 dated questions, recorded 2026-10-09)
- Search: rerank hit@5 0.923 with the date filter, 0.731 without; 132 of 170 top-5 places
  went to versions not in force without it.
- Answers, chat-small + rerank: 24/26 right with the filter, 20/26 without;
  33/33 citations of the right version.
- The model alone: 20/26 by the keys, wrong version for a published-not-in-force amount.
- chat-strong: 11/12 against 10/12 on a 17-question sample, about 18 times the cost.

## Design choices
- Official point-in-time API, one fetch, SHA-256 checked; versions checked against the eCFR's list.
- Chunk IDs include the version's first day; the date filter runs before the top k.
- A coverage check before the model; a version check on every citation.

## Limits
(one snapshot, two parts, history from 2017, one author's questions, the eCFR is not official)

How to describe it in an interview: “I built an assistant that answers questions about a regulation on a given date. With a date filter, right answers rose from 20 to 24 of 26, and all 33 citations named the right version.”

Go further: each of these is a good next step, with one main change.

• A refresh. Fetch the versions list again later, add any new date, rebuild, and evaluate. Which recorded answers are out of date?
• The conflict, in code. When two passages in force give different amounts, show both and flag the answer, as Q09 needs.
• Canada. Use the two XML versions of the Air Passenger Protection Regulations (SOR/2019-150) on Justice Canada’s GitHub. Keep the attribution line of the Open Government Licence – Canada.
• Scope passages. Pin § 250.2 and § 254.2 into every context, and measure Q22 and Q33 again.

Knowledge check

Check your understanding

Without the date filter, the models rarely used an old amount for a date inside the snapshot. A colleague concludes that the filter is not needed: the dates in the headers are enough. What does this leave out? Use numbers from this case study.

Show the answer

It leaves out what the headers cannot do: put the right version into the top 5. Without the filter, 132 of 170 top-5 places went to versions not in force, and recall@5 fell from 0.865 to 0.500. The model then abstained instead of answering (Q03, Q05), so right answers fell from 24 to 20 of 26. The headers also failed for dates before the snapshot: “2017-01-01 or earlier” led to $675 for 2015, where $650 was right. The filter keeps old versions out of the budget and out of the citations; the headers let a reader see the dates.

Check your understanding

Q04 asks about a flight on 20 December 2024. The passage of § 250.5 in force that day starts with 'Link to an amendment published at 89 FR 84818, Oct. 24, 2024.' Why is $775 right and $1,075 wrong, and which parts of the project protect the answer?

Show the answer

The amendment was published on 24 October 2024 but came into force on 22 January 2025. On 20 December 2024 the version in force still said $775; the notice only warns that a change is coming. The date filter keeps the version from 2025-01-22, with $1,075, out of the search. Rule 4 of the prompt says that a published notice does not change the rule before the new version is in force. The version check would flag a citation of the 2025 version. The model alone, with no passages, said $1,075.

Check your understanding

For the flight of 10 March 2015 (Q28), why does the final design answer without calling a model, and why is that answer not 'there was no rule'?

Show the answer

The snapshot’s history starts on 2017-01-01, so no version in it can be in force in 2015. A model given the 2017 version as the nearest text answered $675, citing a version not in force. The true amount then was $650. So code decides: coverage() returns “before”, and the assistant says that it cannot show the rule. It does not say that there was no rule. Part 250 existed in 2015, and the source note of § 250.5 lists earlier amendments. Only the snapshot is too short.

Recap

• A regulation changes section by section. Derive each section’s versions from point-in-time copies, and check them against the publisher’s list.
• Put the version’s first day in the chunk ID, and its dates in every chunk.
• Filter by the question’s date before the top k: without it, four of five top-5 places went to versions not in force.
• Decide in code what the snapshot cannot cover: before its history, no answer; after its date, a warning.
• Check every citation’s version against the date, and show the version and its dates on every answer.
• Published is not in force: a notice of an amendment is not the new rule.

You built an assistant whose answers a case worker can check against the text of the right day. The final assignment asks you to build and measure an assistant of your own.

Next: Final assignment