Capstone on a public document collection

What you will be able to do

Take the capstone’s method to a real collection: the GOV.UK Service Manual’s 20 guides on agile delivery, under an open licence. Scope it for one user and one decision, and reserve the held-out questions before any answer. Compare a baseline with three alternatives fairly, and run the held-out set once. Then show a failure honestly, and write the README that makes it a portfolio piece.

Case studyPortfolio piece

Dataset

GOV.UK Service Manual: the 20 guides of the topic Agile delivery

Licence

Open Government Licence v3.0

Time

About 4 hours

Runs in

VS Code, Python 3.12+, Docker

Skills used

• Scope for one user
• Licence and data card
• A snapshot with checksums
• Held-out questions first
• A rule written before measuring
• Fair comparison
• Honest failure analysis
• Portfolio README

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is the course’s finished doc-assistant, made ready for one public collection. It holds the collection’s files, 60 questions, and a real model’s answers, recorded by the course team. You need no key and no money. The finished project is the result, for when you are stuck. The code is under the MIT licence; the guides are Crown copyright under the Open Government Licence.

Before you startWhat to know and to have readyAbout 240 minutes

What you need to know

This case study repeats the whole course on new documents. It uses these lessons most:

• Problem and audience, Acceptance criteria and Plan the evidence
• Synthetic data pack
• Run held-out tests, Inspect error categories and Document trade-offs

Hybrid search and the reranker come from Lexical, dense and hybrid search and Reranking and query changes in RAG: Building AI That Uses Your Data.

What you need on your computer

• Python 3.12 or newer, Git, VS Code, curl and Docker Desktop.
• About 5 GB of free disk space: the packages (about 1 GB, with PyTorch), the embedding model and PostgreSQL.
• An internet connection for the install and for one optional check against GOV.UK.

Tested on macOS (Apple M4, 16 GB, Docker Desktop, PostgreSQL 18.6, Python 3.12.13). The Windows commands are written, not tested. Every time in this case study comes from the author’s computer, which other work shared: your times will differ.

The situation

Ruth is the delivery manager of a small digital team in a public body. Every month, new people join: user researchers, developers, a product manager. Most of them are new to government. They all ask the same questions: “How long should a discovery take?”, “When do we book the assessment?”, “What is a show and tell?”

The answers are in the GOV.UK Service Manual. Its topic Agile delivery has 20 guides, from Core principles of agile to Retiring your service. Ruth gives new joiners a reading list today. They do not read it all, and they mix up the guides. Last month a joiner told the team that an alpha lasts “4 to 8 weeks”. That is the discovery guide’s number; the alpha guide says 6 to 8.

The team and its people are invented for this case study. The guides are real.

The objective. Ruth must decide one thing: should new joiners use an assistant as their first stop, instead of the reading list? She says yes only with evidence. The assistant answers from the 20 guides and cites a guide and a section for every claim. When the guides do not answer, it says so. Its held-out results must meet the thresholds that you write before any answer.

What you will build

The case study in one picture. The two locked boxes are fixed before any answer: the documents and the held-out questions.

Text description of the diagram

Read it from left to right. The guides come from GOV.UK once, as a snapshot. The 60 questions split into 25 locked held-out questions and 35 tuning questions. You compare versions on the tuning questions only. The chosen version meets the held-out questions once.

Part	What you do	Evidence you keep
1	Scope: one user, one decision, non-goals; the thresholds and the rule for a change	docs/project-brief.md, docs/acceptance-criteria.md
2	The collection: licence, snapshot, conversion, data card	data/dataset.md, the datapack output
3	Reserve the held-out questions	evaluation/heldout.lock, a commit
4	The baseline on the tuning questions	the failures, read one by one
5	Three alternatives, one measure, one rule	a comparison table
6	The held-out run, once	reports/heldout-baseline.json, docs/evaluation-report.md
7	New input, end to end	a new guide, uploaded, indexed and cited
8	The README	the portfolio piece

Set up the project

The steps use VS Code and its terminal. Every command runs in the project folder.

Stop the course's containers first

This project uses the usual ports: PostgreSQL on 5432, the API on 8000 and the practice identity provider on 8400. In your doc-assistant folder, run docker compose --profile app down (it keeps the data) before you start.

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder, for example ~/projects. You get the folder public-collection-capstone.

2. In VS Code, choose File > Open Folder, and open public-collection-capstone. Then open a terminal: Terminal > New Terminal.

3. Make and activate a virtual environment, and install the packages:

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   Windows (PowerShell)

   py -3.12 -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt

   On Linux without a GPU, install the CPU build of PyTorch first, as the top of requirements.txt says.

4. Copy the settings file. Then change the password in it: POSTGRES_PASSWORD, and the same password in the two URLs.

   macOS / Linux

   cp .env.example .env

   Windows (PowerShell)

   Copy-Item .env.example .env

5. Make the practice identity provider’s keys, then start PostgreSQL:

   python -m idp init
   docker compose up -d --wait db

   idp init comes first: it writes two secrets into .env, and Compose refuses to start without them.

6. Run the tests:

   python -m pytest

   .................................................................ssss... [ 64%]
   .......................................                                  [100%]
   107 passed, 4 skipped in 38.71s

What this tells us. The course’s tests pass on this project too. They run on the course’s synthetic two-shop pack, which now lives in tests/fixtures/pack/. Why keep it? The access and isolation code needs two tenants and staff-only documents to be tested, and a public collection has neither. The four skipped tests check the held-out set, which you reserve in Part 3.

Part 1: scope it

Start with the five lines of Problem and audience, written with Ruth. The finished project has them in docs/project-brief.md:

• User: Ruth, and the new joiners of her team.
• Decision: does the onboarding pack send new joiners to the assistant first, instead of the reading list?
• Documents: the 20 guides of the topic Agile delivery, as one fixed copy.
• Success: a short answer in a few seconds, with a guide and a section for every claim, or “the guides do not say”.
• Non-goals: no other topics of the manual, no advice, no live copy of GOV.UK, and one tenant.

Why is one tenant enough? The worked capstone has two shops, because each shop’s documents are private, and a leak between them is the worst failure. Here every document is public, and one team uses them. A second tenant would protect nothing. The isolation code stays in the product, and its tests still run on the synthetic pack. In the README, you do not claim isolation for this collection: you say it does not apply.

Then the acceptance criteria, before any answer. They are the worked capstone’s thresholds, so the two projects can be compared:

ID	Measured on the held-out questions	Threshold
Q1	correct answers on answerable questions	at least 75%
Q2	abstains when the guides do not answer	100%
Q3	citation validity	100%
Q4	follows a hidden instruction	not measured: the guides contain none
Q5	median time, question to answer	under 5 s
C1	model cost per question	under US$0.001

One more line belongs here, and it is new: the rule for choosing a change. You will try alternatives on the tuning questions. Write now what “better” means, so that a lucky run cannot choose for you:

On the 35 tuning questions, first live pass, an alternative becomes the candidate only if it gets at least 2 more right than the baseline. It must also get no unanswerable question wrong that the baseline got right, and add no more than 0.5 s to the median time.

Why 2 and not 1? In Run held-out tests, two identical live runs differed by one question. A rule that accepts one question accepts noise.

Part 2: the collection

The licence first

Before you read a single guide, check that you may copy it and publish your copy. The author compared two collections:

	GOV.UK Service Manual, Agile delivery	Open Source Guides (GitHub)
Licence	Open Government Licence v3.0	CC BY 4.0
Size	20 guides, about 19,100 words	14 articles, about 40,000 words
Checkable facts	many: 15-minute stand-ups, 6 to 8-week alphas, email addresses	few: mostly advice
A real difficulty	four phase guides with the same headings; one contradiction	overlapping articles

Both licences allow a copy with an attribution. The GOV.UK guides won, because a question about them can be marked right or wrong. A key like \b15\b|fifteen can check “How long should our stand-up last?”. No key can check “Be kind to new contributors”.

A snapshot of a living source

Think of a notice board in an office. Anyone may change it at any time. If you want to quote it in a report, you photocopy it and write the date on the copy. Your report is then about the copy, and anyone can check it.

GOV.UK is that notice board: six of the 20 guides were updated in 2026 alone. Now imagine that you download the guides a month after the author. One guide has a new paragraph, so its passages move. Their IDs change, the recorded answers no longer match, and a held-out key may no longer be true. You and the author measured two different collections without knowing it.

The fix is a snapshot: the exact files, fetched once, kept with a checksum (a SHA-256) for each file. Everything downstream belongs to that snapshot: the documents, the passages, the recordings and the held-out lock. A second script compares the snapshot with today’s GOV.UK, so you know when it has aged. You may keep a copy only if the licence allows it, and the Open Government Licence does.

A snapshot pins everything that depends on the text. The check tells you when the source has moved on.

Text description of the diagram

On the left is GOV.UK, which can change any day. One fetch makes the snapshot: the exact files with a SHA-256 for each. From the snapshot, every computer gets the same text, so the same passages and passage IDs. That keeps two things true: the recorded answers match, and the held-out keys still describe the text. A dotted arrow back to GOV.UK is the check, which lists what changed since the fetch.

When do you need a snapshot? When the source changes without version numbers, and your evaluation or your recordings depend on the exact words. It costs disk space, and the copy ages. Say in the data card how old it is, and refresh it on purpose, with new recordings and a new held-out run.

Convert and check

The starter has the snapshot in data/snapshot/. It holds the topic page and its 20 guides, as the GOV.UK Content API returned them, and one more guide for Part 7. scripts/collection.py turns each guide into a Markdown document with the metadata that the assistant reads:

python -m scripts.collection prepare
python -m scripts.datapack

Snapshot: 22 of 22 files match data/snapshot/SHA256SUMS
Wrote 20 documents to data/westbrook/documents/ (19,135 words), last updated 2016-05-23 to 2026-05-22.
Wrote data/westbrook/inventory.csv and data/SHA256SUMS (24 files, SHA-256 of the list 1a5f6f7cd1a5...).
Checksums: 24 of 24 files match SHA256SUMS
westbrook: 20 documents, 197 sections, formats {'md': 20}, languages {'en': 20}
  staff-only 0; with more than one version: none; with an end date 0; supplier documents: none
Every document parses.

What this tells us. All 22 files of the snapshot match their checksums. The 20 documents parse, with 197 sections. The SHA-256 of the list of documents starts with 1a5f6f7cd1a5, the same as the author’s, so your collection is the same, byte for byte. There is no staff-only document and no second version: a public collection has neither.

The conversion had one surprise. The guides use three levels of heading, and the course’s reader knows two. Agile methods: an introduction has a section “Scrum” twice, under two different parents. Two citations would look the same. So the script names a third-level section after its parent:

data/westbrook/documents/agile-methodologies.md (part)

## Popular agile methods explained: Scrum

Scrum is the most commonly used agile method.

Now see how old the snapshot is. This command needs the internet:

python -m scripts.collection check

20 of 20 guides are the same as the snapshot; 0 changed.

Write the result in the data card. The finished project’s card, data/dataset.md, follows the course’s template. It names the source, the licence and the attribution sentence, the contents, the real difficulties, the questions and the limits. Its key line: “Nothing was planted: these are the difficulties of the real collection.”

Part 3: reserve the held-out questions

The 60 questions in evaluation/questions.jsonl were written by one person, by reading the guides. No model wrote, answered or labelled them. Each names the sections that answer it. There are six kinds:

Kind	How many	Example
answerable	24	“How long should our daily standup last?”
phase	12	“How long does an alpha usually last?” (the four phase guides compete)
paraphrase	7	“Hardly anyone uses our old online service any more. Should we keep running it?”
definition	6	“What is a blocker?”
conflict	2	“Do we need spend controls approval before we start a discovery?”
unanswerable	9	“How many points are in the Service Standard?” (another topic of the manual)

The conflict questions are real. How the discovery phase works still says “check if you need spend controls approval”. Getting approval to spend money on a service, updated in April 2026, says that spend controls were replaced on 1 April 2026. The newer guide applies.

Now reserve the held-out set, before you see a single answer. This is the step of Plan the evidence:

python -m scripts.holdout reserve

Held out: 25 questions (westbrook 25). Kinds: answerable 10, conflict 1, definition 2, paraphrase 3, phase 5, unanswerable 4.
Tuning set: 35 questions (westbrook 35).
Wrote evaluation/heldout.jsonl, dev.jsonl and heldout.lock. Commit them now, before any tuning.

Commit the three files at once, with the rest of the project:

git init
git add .
git commit -m "The collection, the questions and the held-out set, before any answer"

What this tells us. The split is stratified, so the held-out set has every kind. The lock’s fingerprint starts with 75f6adf284e4, the same as the author’s: the split is repeatable. The four skipped tests now run and pass. From here on, the 25 held-out questions are closed until Part 6.

Part 4: the baseline

Load the collection into PostgreSQL, as one team’s documents:

python -m scripts.load_pack

Identity: 1 tenants, 2 people, 2 memberships
westbrook: 20 documents uploaded, 0 already there
Indexing with intfloat/multilingual-e5-small@614241f ...
Indexed 20 documents in 9.7 s (failed 0).
  westbrook: 20 documents, 225 passages, 0 failed

The baseline is the course’s finished settings: hybrid search, the top 5 passages, and the prompt guide_v1. That prompt is the course’s answer_v2 with the words about shops changed to guides; the rules did not change. You wrote it before any answer, so it is part of the setup, not tuning.

Run the 35 tuning questions. Look at these answers as often as you like:

python -m scripts.evaluate --set dev --show-failures

hybrid, guide_v1, recorded: 35 questions | correct 0.971 | retrieval hit 0.967 | citations valid 1.000 | abstained when it should 1.0 | injection obeyed 0 of 0 | median 11 ms | US$0.000202 per question
  answerable       14  correct 1.0  hit 1.0
  conflict          1  correct 1.0  hit 1.0
  definition        4  correct 1.0  hit 1.0
  paraphrase        4  correct 0.75  hit 0.75
  phase             7  correct 1.0  hit 1.0
  unanswerable      5  correct 1.0  hit None
  WRONG G49 (westbrook, paraphrase, hit False): The guides do not say whether you should keep running the service because hardly anyone uses it.

What this tells us. 34 of 35 are right, every citation is valid, and every unanswerable question is refused. The one failure is G49, a paraphrase: “Hardly anyone uses our old online service any more. Should we keep running it?” The answer says that the guides do not say. They do: How the live phase works says to retire a service that so few users need that it is not cost effective. hit False says that the search did not find that section, so the model could only refuse.

The time in this output is not the assistant’s speed: the answers are recorded. In the author’s live run, the median question took about 1.6 s, and the model call was most of it.

Part 5: three alternatives, one measure

The failure points at the search, so the alternatives are search modes. All three are in the course’s code: dense (vectors only), dense_rerank and hybrid_rerank (a cross-encoder rereads 20 candidates). First, the free measure: did the search find a relevant section?

python -m scripts.evaluate --set dev --retrieval-only --show-failures --config RETRIEVAL_MODE=dense

Run it four times, with dense, hybrid, dense_rerank and hybrid_rerank:

dense, guide_v1, recorded: retrieval hit 1.000 (30 of 30 questions with relevant passages), median search time 8.1 ms
hybrid, guide_v1, recorded: retrieval hit 0.967 (29 of 30 questions with relevant passages), median search time 9.3 ms
  missed G49 (westbrook, paraphrase)
dense_rerank, guide_v1, recorded: retrieval hit 1.000 (30 of 30 questions with relevant passages), median search time 363.5 ms
hybrid_rerank, guide_v1, recorded: retrieval hit 1.000 (30 of 30 questions with relevant passages), median search time 362.0 ms

Then the measure that the rule uses: correct answers. Run the full evaluation with each alternative, for example:

python -m scripts.evaluate --set dev --show-failures --config RETRIEVAL_MODE=dense

• Baseline (hybrid)
• Alternatives

The author's live run, first pass. Every alternative fixed the same one question; the rule asks for two.

Text description of the chart

The baseline, hybrid, got 34 of 35 right in the first pass and 34 in the second. Dense, dense with the reranker, and hybrid with the reranker each got 35 of 35. “p2” is the second pass, and “+R” means with the reranker. The axis starts at 30 so that the one-question difference is visible.

Version (tuning set, live pass 1)	Correct	Search found a relevant section	Median search time
hybrid (baseline)	34 of 35	29 of 30	about 9 ms
dense	35 of 35	30 of 30	about 7 ms
dense_rerank	35 of 35	30 of 30	about 340 ms
hybrid_rerank	35 of 35	30 of 30	about 350 ms

What this tells us. Every alternative found G49’s section and answered it, and nothing else changed. Dense was even a little faster. So is dense better? Apply the rule that you wrote in Part 1. It needs 2 more right answers, and each alternative has 1. No candidate: the baseline stays.

That can feel wrong: G49 failed in both of the baseline’s passes, so it is not noise. But it is one question, and at 34 of 35 the tuning set cannot show a difference of one. The honest record: “Dense fixed G49, a paraphrase. Not enough evidence to switch. Watch paraphrases.” With no candidate, the held-out set runs once, for the baseline.

Part 6: the held-out run, once

python -m scripts.evaluate --set heldout --show-failures --out reports/heldout-baseline.json

hybrid, guide_v1, recorded: 25 questions | correct 0.880 | retrieval hit 0.905 | citations valid 1.000 | abstained when it should 1.0 | injection obeyed 0 of 0 | median 12 ms | US$0.000201 per question
  answerable       10  correct 0.8  hit 0.9
  conflict          1  correct 1.0  hit 1.0
  definition        2  correct 1.0  hit 1.0
  paraphrase        3  correct 1.0  hit 1.0
  phase             5  correct 0.8  hit 0.8
  unanswerable      4  correct 1.0  hit None
  WRONG G01 (westbrook, answerable, hit False): The guides do not say how long a daily standup should last.
  WRONG G13 (westbrook, answerable, hit True): Show work that’s in progress, not yet started and done. Also track progress against goals and KPIs, risks and issues, blockers, deadlines and dependen
  WRONG G30 (westbrook, phase, hit False): The guides do not say whether members of the public can use your service during alpha.
Wrote reports/heldout-baseline.json

Read it against the acceptance criteria. The author’s two live passes gave the same result. Here it is with the live time and cost:

Measure	Threshold	Pass 1	Pass 2	Met?
Correct, answerable questions (Q1)	at least 75%	18 of 21	18 of 21	yes
Abstains when the guides do not answer (Q2)	100%	4 of 4	4 of 4	yes
Citation validity (Q3)	100%	100%	100%	yes
Median time, question to answer (Q5)	under 5 s	about 1.3 s	about 1.7 s	yes
Model cost per question (C1)	under US$0.001	US$0.000201	US$0.000201	yes

Every criterion is met: 85.7% of answerable questions are right, against a threshold of 75%. But do not stop at the scorecard. Read the three wrong answers, as Inspect error categories taught:

• G13, the team wall: the key was too strict. The answer is right: “Show work that’s in progress, not yet started and done.” But the key not started|upcoming does not match “not yet started”. You do not change a reserved key after a run. The score stays 22, and the report says that a person reads 23 of 25 as right.
• G30, the public in alpha: the search missed. The opening text of the alpha guide says: “Alpha services should not be available for the public to use.” The search did not bring that passage, and the assistant said “The guides do not say.”
• G01, the stand-up: the search missed. The next section looks at it closely.

One failure, honestly: the 15-minute stand-up

G01 is the easiest question in the bank: “How long should our daily standup last?” The answer has a section of its own in Agile tools and techniques. It says: “keep it to 15 minutes or less”. Yet the assistant said that the guides do not say. Ask it yourself, and look at the passages:

python -m scripts.ask "How long should our daily standup last?"

Ruth (owner) at Westbrook digital team asks: How long should our daily standup last?
Passages found: ca47f49114ab how-the-beta-phase-works v4, 9a823f56a408 how-the-discovery-phase-works v4, 01df8cd661a8 core-principles-agile v1, 2bbb0c5e9905 how-the-live-phase-works v4, 5b1eb08f23b8 retiring-your-service v2
Answer (not answerable): The guides do not say how long a daily standup should last.
recorded chat-small | 3300 ms | tokens 1265 in, 80 out | US$0.000167 | trace 041008fb094ca06d9a6ab4115f9e7b78

Why G01 failed, from the author's diagnosis of the loaded project. The right section was first in one list and missing from the other, so passages that both lists found pushed it to sixth.

Text description of the diagram

Read the picture from left to right. The vector search put Daily standup first. The keyword search did not have it in its top 20. Fusion gives each passage 1 ÷ (60 + rank) from each list and adds the votes. A first place in one list is worth 1/61 = 0.0164. Two places, 9th and 1st, are worth 0.0309. So five passages that both lists found came first, and Daily standup came sixth. The model sees only the top 5.

This is reciprocal rank fusion, from Lexical, dense and hybrid search and Inspect error categories. There, two weak votes beat one strong vote too. Why did the keyword search miss the section? It ranks a passage by how many of the question’s words it has, and how close they are. It does not know that “standup” is rare, while “how”, “long” and “should” are in almost every guide. So passages full of common words came first. G30 and G49 fell below the cut the same way. Each right section was high in one list and low or missing in the other, so it came 7th and 8th. The author found this by listing the rank of the right section in each list. That is error analysis of failures already counted: it changes no score.

Show this failure in a demo. It is safe: the assistant refused rather than invented a number. But it is not helpful: a new joiner would think that the manual says nothing about stand-ups. On this collection, the search is the weak part, not the model.

What happens next? G01 and G30 move to the tuning set, where you may now study them. New held-out questions, written by someone who has not seen the results, replace them. Then dense search, or more passages per question (TOP_K), gets a fair test, with enough questions to pass the rule. The evaluation report, docs/evaluation-report.md in the finished project, lists all of this.

Part 7: new input, end to end

Ruth’s team books its alpha assessment, and she wants the assistant to answer about it. The guide What happens at a service assessment is in another topic of the manual, so the assistant cannot answer yet:

python -m scripts.ask "How long does a service assessment last?"

Ruth (owner) at Westbrook digital team asks: How long does a service assessment last?
Passages found: 9d919d981425 how-the-beta-phase-works v4, 684964e4dfdb running-your-service-in-a-sustainable-way v2, 3f71a3a020f2 retiring-your-service v2, b598b7feab08 how-the-beta-phase-works v4, 5f355a76d6f1 running-your-service-in-a-sustainable-way v2
Answer (not answerable): The guides do not say how long a service assessment lasts.
recorded chat-small | 3308 ms | tokens 1459 in, 31 out | US$0.000161 | trace 206a36fb03f9af69d265c9fb19e1b0dc

The snapshot includes that guide, in data/snapshot/new-input/. Convert it, then upload it through the real product path: the API, as Ruth, with a practice token. Use three terminals, all with .venv active.

1. Convert the guide:

   python -m scripts.collection convert data/snapshot/new-input/how-service-assessments-work.json

   Wrote data/new-input/how-service-assessments-work.md: What happens at a service assessment (last updated 2026-09-07).

2. In terminal 2, start the practice identity provider: python -m idp. In terminal 3, start the API: fastapi run doc_assistant/main.py.

3. Back in terminal 1, upload the guide as Ruth:

   macOS / Linux

   TOKEN=$(python -m idp token usr-ruth)
   curl -s -X POST http://127.0.0.1:8000/v1/tenants/westbrook/documents -H "Authorization: Bearer $TOKEN" -F "file=@data/new-input/how-service-assessments-work.md"

   Windows (PowerShell)

   $TOKEN = python -m idp token usr-ruth
   curl.exe -s -X POST http://127.0.0.1:8000/v1/tenants/westbrook/documents -H "Authorization: Bearer $TOKEN" -F "file=@data/new-input/how-service-assessments-work.md"

   {"document":{"document_id":"16cac3b5-966c-42c0-b260-a8ddae5ad9ed","doc_key":"how-service-assessments-work","version":"8","title":"What happens at a service assessment","doc_type":"guide","language":"en","access":"public","effective_from":"2026-09-07","effective_to":"","file_name":"how-service-assessments-work.md","size_bytes":6710,"sha256":"8b81216955d12ac83f81656f9209f01ee745640216cb95d77ae5c8c623ffbf54","status":"uploaded","error":null,"attempts":0,"chunk_count":0,"uploaded_by":"usr-ruth","uploaded_at":"2026-10-11T18:37:12.838893+00:00","indexed_at":null,"links":{"self":"/v1/tenants/westbrook/documents/16cac3b5-966c-42c0-b260-a8ddae5ad9ed"}},"duplicate":false}

   The answer says "status":"uploaded": the worker indexes the guide in the background. Your IDs and times will differ.

4. Wait a few seconds for the worker. In the author’s run, the document list then showed 21 guides indexed, the new one with 11 passages. Ask again:

   python -m scripts.ask "How long does a service assessment last?"

   Ruth (owner) at Westbrook digital team asks: How long does a service assessment last?
   Passages found: 36c81f3bbff0 how-service-assessments-work v8, a3d94db45906 how-service-assessments-work v8, dcb51899a15d how-service-assessments-work v8, 9d919d981425 how-the-beta-phase-works v4, d8911bff8979 how-service-assessments-work v8
   Answer (answerable): A service assessment usually lasts about 4 hours.
     - Service assessments usually last about 4 hours. [What happens at a service assessment, start, v8]
   recorded chat-small | 3383 ms | tokens 1503 in, 54 out | US$0.000177 | trace 917ba9a10cbfa23d206857995c3abb60

What this tells us. The new guide went through the same path as every document: the upload, the job states, the worker, eleven new passages. The same question then got a cited answer from the new guide. Nothing else changed: no restart and no new code. The answers in this part are recordings too, made before and after the same upload.

Last, run the course’s smoke test against the running API: python scripts/smoke.py http://127.0.0.1:8000, with SMOKE_TOKEN set to the token. It ends with 7 of 7 checks passed.. A made-up team gets 404. Stop both servers with Ctrl+C.

The recommendation

For Ruth: yes, make it the first stop, with two conditions. The assistant met every threshold on questions that nobody tuned it on. It got 18 of 21 answerable questions right, refused every unanswerable question, and every citation was valid. It took about 1.3 to 1.7 seconds and US$0.0002 a question. Its failures were refusals, not inventions. The conditions:

1. Open the cited section. The answer is short; the guide is the authority. The page shows the passage with one click.
2. “The guides do not say” is not the last word. On this collection, the search sometimes misses a section that exists (G01, G30, G49). When the assistant refuses a basic question, search GOV.UK too. Then tell Ruth, so that the question joins the next tuning set.

Insights and limits

• A rule written first is worth more than a chart. Without it, a one-question gain would have looked like an improvement.
• The tuning set can be too easy. At 34 of 35, it could not separate the versions. More questions, and harder ones, especially paraphrases, would make the next comparison fairer.
• On a public collection, the risk moves. There is no tenant to leak and no hidden instruction. The risk is a wrong guide, an old guide, or a missed section.
• Limits of this study. One person wrote the 60 questions and knows the guides. 25 held-out questions means that one question is 4 points. It covers one topic of one manual, on one shared laptop, on macOS only.

Responsible use

• Guidance, not advice. GOV.UK says that it does not publish advice. Say the same in your product: the assistant points to the guide, and people decide.
• Attribution is a condition of the licence. Keep the licence’s attribution sentence and the source address in the README and the data card. Do not use the GOV.UK crown or logos: the licence excludes them.
• Do not imitate government. Your portfolio page must not look like GOV.UK or speak for the Government Digital Service. Name it as your project.
• Say how old it is. The answers describe the snapshot. Give its date in the README, and run python -m scripts.collection check before a demo.
• Be polite to the source. The fetch script waits between requests. Never point a load test at GOV.UK.

Make it a portfolio piece

This project shows what employers look for. It has a real collection with a checked licence, an evaluation planned before the results, a fair comparison and explained failures. The finished project’s README.md is the model.

What to put in the repository:

public-collection-capstone/
  README.md                     the user and decision, the result, the recommendation, how to run it, the limits
  LICENSE                       MIT, for the code
  data/dataset.md               the source, the licence and attribution, the snapshot, the conversion, the limits
  data/snapshot/                the exact files used, with SHA256SUMS
  docs/project-brief.md         one user, one decision, the non-goals, why one tenant
  docs/acceptance-criteria.md   the thresholds and the rule for a change, written first
  docs/evaluation-report.md     the held-out result, the three failures, the comparison
  evaluation/                   the 60 questions, the held-out set and its lock
  recordings/                   the recorded answers: anyone can rerun the evaluation without a key
  reports/                      the held-out result, and the live runs with their times and costs
  scripts/collection.py         prepare, check, convert

1. Push the repository to GitHub. Check that .env and .idp/ are not in it: .gitignore keeps them out.
2. Put the result in the README’s first lines: 22 of 25 held-out questions, the thresholds met, and the recommendation with its two conditions.
3. Link docs/evaluation-report.md, and name the three failures in the README, not only in the report.
4. From a fresh clone, run the quick start of the README. Then say in the README that you did.

This is one of the two case studies that you may choose for the certificate. To make it your own, take another topic of the Service Manual (Service assessments has 8 guides), or another collection with an open licence. Write new questions and reserve them first. Record your own answers with a live model, and keep the rule.

Knowledge check

Check your understanding

Every alternative got 35 of 35 tuning questions right, and the baseline 34. Why did the baseline stay, and was that the right call?

Show the answer

The rule, written before any answer, asks for at least 2 more right answers, because two identical live runs can differ by one question. Each alternative was 1 better (G49 only). Under the rule, keeping the baseline was right. The honest record says that dense search fixed a paraphrase question, and that the evidence was too small to switch. The next step is more and harder tuning questions, not a quiet switch.

Check your understanding

The assistant answered 'The guides do not say' to 'How long should our daily standup last?', although a section says '15 minutes or less'. Is this a model failure or a search failure, and how do you know?

Show the answer

A search failure. The evaluation marks the question hit False: no relevant passage reached the model. python -m scripts.ask lists the passages that the model saw, and the Daily standup section is not among them. With only those passages, the model was right to refuse. The fix belongs in the search, tested on the tuning set.

Check your understanding

A friend wants to skip the snapshot: 'Just fetch the guides from GOV.UK each time, so they are always up to date.' What breaks, and when is their idea right?

Show the answer

When a guide changes, its passages and passage IDs change. The recorded answers no longer match, and the held-out keys may no longer be true. Two people who ran the same commands then measured different collections. Their idea is right for a product in use, which must follow GOV.UK. Then refresh on purpose: a new snapshot, new recordings or live answers, and a new held-out run. Put the snapshot’s date in the README.

Recap

• You scoped the product for one user and one decision, with non-goals and one tenant. You wrote the thresholds and the rule for a change before any answer.
• You checked the licence, kept a snapshot with checksums, converted it, and wrote the data card.
• You reserved 25 held-out questions before any answer; the split matched the author’s lock.
• Three alternatives each fixed one tuning question. The rule asked for two, so the baseline stayed.
• The held-out run met every threshold: 18 of 21 answerable, 4 of 4 refusals, valid citations. The failures were two search misses and one strict key.
• A new guide went in through the API and was cited at once. The README tells the whole story.

Next, the second case study: mark a weak capstone against the rubric.

Next: Review a weak capstone