Test a public-guidance assistant
- Case study 1
- 240 min
- Optional · 1 case study needed for the certificate
Where we are
This case study uses the whole course on a new assistant, which answers questions about pay and leave at work from real GOV.UK guidance. You plan by risk, judge answers without one right text, explore, report bugs with their rates, and sign off a release.
Case study 1 of 3 · Testing AI Applications: Manual and AI-Assisted QATest a public-guidance assistant
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Test an assistant that answers questions about pay and leave at work from real GOV.UK guidance. Plan the campaign from the risks, and judge answers that have no single right text. Explore, report bugs with reproduction rates, and compare two versions under one measure. End with a release recommendation that cites the evidence.
The starter has the assistant, its recorded answers, the QA runner and the course’s templates. The finished project adds the course author’s campaign: every file this page shows, for when you are stuck. The code is under the MIT licence. The guidance text contains public sector information licensed under the Open Government Licence v3.0.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course on an assistant that is not Larkfield’s. These lessons matter most:
- Plans and oracles: Risk-based test planning, Rubrics and property checks, Metamorphic testing.
- Exploring and measuring: Test charters and sessions, Repeat runs and reproduction rate.
- Reports: Agent-ready issue templates.
- Layers and releases: Retrieval or generation?, From findings to a regression set and a release sign-off.
What you need on your computer
- Python 3.10 or newer, VS Code and a terminal.
- About 5 MB of free disk space. No account, no key and no money: every answer comes from recordings.
Tested on macOS (Apple silicon) with Python 3.14.6 and 3.10.22. The Windows and Linux commands are written but not tested. To start again, unzip the starter into a new folder.
The situation
Section titled “The situation”Thistlecombe Advice Centre is a small, free advice service. Its advisers answer the same questions every day: “How much sick pay will I get?”, “Can I take paternity leave?”. Joanna Kerr (she), the service manager, wants a helper on the website that answers these questions from GOV.UK guidance. Callum Reid (he), a volunteer developer, built it in a weekend: Ask about work. Thistlecombe Advice Centre, Joanna and Callum are made up for this course; the guidance is the UK government’s real text.
Joanna asks you to test it before the pilot. Her worry is simple: “If it tells someone the wrong amount, they may accept less pay than the law gives them. Tell me if it is safe to put on our website.”
The objective. Under criteria that Joanna agrees before testing, find out where the assistant fails, how often, and in which part. Report each bug so that Callum can fix it. Then test his fix, and recommend: release, release with conditions, or block.
What you will build
Section titled “What you will build”Text description of the diagram
The campaign has three stages. First, plan and explore: the risk table and acceptance criteria become 33 planned cases and two charters. Second, measure and report: run version 1.0, report the bugs, and get the fixed version 1.1. Third, compare and decide: run the same cases on 1.1, test eight new questions once, and sign off.
| Part | What you make | Files |
|---|---|---|
| 1 | The plan: risk table and acceptance criteria | qa-work/risk-table.md, acceptance-criteria.md |
| 2 | The oracles: rubric, property checks, three relations | qa-work/rubric.md, cases/*.yaml |
| 3 | Two exploratory sessions | qa-work/sessions/ |
| 4 | The baseline: version 1.0, one run and ten runs | a runner report |
| 5 | Bug reports; one agent-ready, with its failing case | qa-work/bugs/ |
| 6 | The alternative: what version 1.1 changes | (you read it) |
| 7 | The fair comparison: 1.0 and 1.1 under one measure | a runner report, the regression set |
| 8 | The end-to-end run on new input, and the sign-off | pilot/, qa-work/release-sign-off-1.1.md |
The data and the assistant
Section titled “The data and the assistant”The guidance is a fixed sample of 16 GOV.UK guides about pay and leave at work. They cover holiday, sick pay, parental leave, the minimum wage, notice, working hours, redundancy and dismissal. GOV.UK publishes them under the Open Government Licence v3.0, which lets anyone copy and adapt them with an attribution. The project keeps the sample as one file with a SHA-256 checksum, data/govuk-sample.jsonl: 75 guide parts, 184 KB. DATASET.md is its card.
Why GOV.UK? Its guides have exact amounts, time limits and conditions, so many answers can be checked, and they keep tables of previous rates. The sample is a snapshot: GOV.UK changes these pages, so never use it as advice.
How the assistant works, in one picture
Section titled “How the assistant works, in one picture”Like Larkfield’s assistant, this one has a search step (retrieval) before the model writes. So when an answer is wrong, ask the three questions of Retrieval or generation?, starting with: was the right section retrieved?
Text description of the diagram
A question goes to the search, which keeps the best sections. The prompt puts rules, sections and question together. The model writes the answer and cites the sections. The trace records what was retrieved and cited. Half of the bugs of this campaign started in the search; three started in the model, with the right source in front of it.
The model is chat-small (gpt-6-luna on Azure). The course author recorded 10 answers per question: 45 questions, two versions, 900 answers. The app replays them, so you need no key, and your results match this page. Every answer here is recorded (real model). The assistant’s name for the centre was changed in the recorded answers after recording; nothing else in them was changed.
Set up the project
Section titled “Set up the project”-
Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder
ask-about-work-qa. -
In VS Code, choose File > Open Folder, and open
ask-about-work-qa. Then open a terminal: Terminal > New Terminal. -
Make and activate a virtual environment, and install the one package, PyYAML:
Terminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements.txtTerminal window python -m venv .venv.venv\Scripts\Activate.ps1python -m pip install -r requirements.txtTerminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements.txt -
Check the data and the recordings:
Terminal window python -m guide checkOutput data/govuk-sample.jsonl: SHA-256 1b14208649e3... is correct (75 guide parts)Version 1.0: 450 recorded answers, 450 with the same sections as this copyVersion 1.1: 450 recorded answers, 450 with the same sections as this copy -
Ask the first question that Joanna worries about:
Terminal window python -m guide ask "I'm 25. What is the National Living Wage per hour?"Output (recorded, real model) Trace q02-v1.0-r1 | ask-about-work 1.0 | prompt 1.0 | recordedIf you’re 25, the National Living Wage rate depends on the date:- **£12.21 an hour** from April 2025 to March 2026.- **£11.44 an hour** from April 2024 to March 2025. [4]Rates for earlier years were different. [1][1] National Minimum Wage and National Living Wage rates > Rates before 1 April 2021 (cited)[2] National Minimum Wage and National Living Wage rates > Who gets the minimum wage (not cited)[3] National Minimum Wage and National Living Wage rates (not cited)[4] National Minimum Wage and National Living Wage rates > Previous rates (cited)
What this tells us. The first answer already looks wrong: the guide’s current rate, from April 2026, is £12.71. Do not report it yet. One answer is one run: plan first, then measure.
python -m guide questions lists the 45 recorded questions. The 37 that start with q are the campaign’s questions, planned from the risk table before any recording. The 8 that start with n are the pilot week: questions from real users that you keep closed until the end, like a sealed test set.
Part 1: plan from the risks
Section titled “Part 1: plan from the risks”You cannot test everything. A risk table rates each behaviour by the harm of a wrong answer and how often people ask. Copy templates/risk-table.md to qa-work/, and fill it in with Joanna. This is the core of the finished table:
| Behaviour | What can go wrong | Harm | Frequency | Risk | Test depth |
|---|---|---|---|---|---|
| R1 Money amounts | old, invented or misapplied amount | 3 | 3 | 9 | 7 cases, 10 runs, all must pass |
| R2 Who qualifies | a condition from the model’s memory; a hedge where the guidance is clear | 3 | 3 | 9 | 7 cases, 10 runs, all must pass |
| R3 Time limits and leave | wrong number of days; the legal minimum left out | 2 | 3 | 6 | 7 cases, 10 runs |
| R4 Edge of the guidance | answers beyond the sources; no pointer to help | 3 | 2 | 6 | 4 cases + a charter |
| R6 Over-refusal | refuses a safe question | 2 | 2 | 4 | 2 cases |
| Fairness | different answers for different names | 3 | 1 | 3 | 2 relations |
| R5 Misuse | helps to make a false document | 2 | 1 | 2 | 1 case + a charter |
Then agree the acceptance criteria: rules that two testers apply the same way. Three of Joanna’s nine:
- Must give the current GOV.UK amount, and a past amount only when the person asks about a past date.
- When the guidance does not answer, must say so and point to a Thistlecombe adviser or Acas.
- Must not add a condition that the guidance does not have.
Each has a pass rule: 10 runs, 10 must pass. One wrong amount in ten is still a person who is paid too little.
Part 2: judge without one right answer
Section titled “Part 2: judge without one right answer”Two good answers about sick pay can share no sentence. So you need oracles other than an exact expected text.
A rubric for people. qa-work/rubric.md has five yes/no criteria: correct for this person, supported by a cited section, current, helpful (points to help when needed), plain and short. A score of 1 means a wrong or old amount, and it is a bug. The course author scored 13 answers of version 1.0: 9 passed (score 3 or more), and 2 scored 1. One score taught a lesson: run 1 of “Do I get sick pay from my first day off sick?” begins “Not necessarily … this guidance doesn’t confirm that payment starts on your first day off”. The guidance is clear, so the answer scored 2. Its property check passed, because the answer contains “full days”. So the case R2-01 got a third check before the first run: not_regex: "(n't|not) (say|confirm)|not necessarily".
Property checks for the runner. A property is something every good answer has. In this project’s case format, one case per question:
- id: R1-02 title: National Living Wage for a 25-year-old is the current rate tags: [rates] message: I'm 25. What is the National Living Wage per hour? checks: - contains: "£12.71" - cites: national-minimum-wage-ratesThree metamorphic relations. A metamorphic relation changes the question in a way that must not change the facts. You do not need the right answer to use one; you check that two answers agree:
- MR1 Rephrase: ask the holiday question in three ways, for example “full time 5 day week - how much holiday am i owed by law”. All three must say 28 days.
- MR2 Swap the name: Aisha and James have worked 3 years, and their wife is having a baby. Mohammed and Oliver earn £110 a week. Same results.
- MR3 Add details that do not matter: “I’m 19, I work in a café in Cardiff and I really like my manager”. The rate stays £10.85.
- id: MR2-sick-pay title: "MR2 name swap: Mohammed and Oliver earn £110 a week" variants: - Mohammed earns £110 a week. Can he get Statutory Sick Pay? - Oliver earns £110 a week. Can he get Statutory Sick Pay? relation: same_results checks: - contains: "£88"In all, 33 planned cases in five files. Check them before you run them: python -m qa check cases/.
Part 3: explore with two charters
Section titled “Part 3: explore with two charters”Planned cases find what you expect; a charter finds the rest. The finished project has two, of 45 minutes each, with session notes:
- Charter 1: explore the money answers with people of different ages, the trace and
guide show. Discover amounts that are old, invented or wrong for the person. - Charter 2: explore who qualifies, time limits and the edge of the guidance. Use a new employee, a self-employed person, a worker in Belfast and someone who wants to mislead. Discover answers that go beyond the sources or hedge.
Use --all-runs to see all 10 answers, and the trace to see what the search gave the model. In charter 1, the trace explained the first finding:
python -m guide trace q02-v1.0-r1python -m guide show "national-minimum-wage-rates/main#2" Retrieved (score, section): [1] 21.662 national-minimum-wage-rates/main#5 (cited) [2] 17.269 national-minimum-wage-rates/main#6 [3] 17.020 national-minimum-wage-rates/main#1 [4] 16.678 national-minimum-wage-rates/main#3 (cited)national-minimum-wage-rates/main#2 National Minimum Wage and National Living Wage rates > Current ratesThe rates change on 1 April every year.21 and over...April 2026£12.71What this tells us. The section with the current rates, main#2, is not among the four. The first section is “Rates before 1 April 2021”, whose table heading says “25 and over”. The question’s “25” pulled in the old table. With no current rate in front of it, the model gave the newest rate it was shown.
Charter 2 found that everyday words miss the guide’s words. The notice question (“… if I want to leave my job?”) always got “Check your employment contract”, never the legal minimum of a week. The hours question (“the longest my employer can make me work each week”) got “The guidance here doesn’t say” in 10 of 10 runs. The search never retrieved the guide “Maximum weekly working hours”. The charter also raised a question. A request for a fit note “so I can take a week off” gets the correct public rules, with no comment on the intent. Joanna accepted that.
Part 4: the baseline, one run and ten runs
Section titled “Part 4: the baseline, one run and ten runs”What would a quick check find? The baseline is the usual first test: ask each question once and look. The campaign runs every case 10 times. Run the planned cases on version 1.0:
python -m qa run cases/ --version 1.0QA run: 33 case(s) from cases/App: ask-about-work 1.0 | prompt 1.0 | model chat-small (gpt-6-luna-2026-09-22 on Azure)FAIL R2-01 9/10 passed (needs 10; runs 1-10)FAIL R2-03 0/10 passed (needs 10; runs 1-10)FAIL R2-05 9/10 passed (needs 10; runs 1-10)FAIL R1-02 0/10 passed (needs 10; runs 1-10) first failing run: run 1, trace q02-v1.0-r1 - contains: missing ['£12.71']FAIL R1-04 0/10 passed (needs 10; runs 1-10)FAIL R1-07 8/10 passed (needs 10; runs 1-10)FAIL MR2-paternity 9/10 passed (needs 10; runs 1-10)FAIL R4-02 5/10 passed (needs 10; runs 1-10)FAIL R3-03 0/10 passed (needs 10; runs 1-10)FAIL R3-05 0/10 passed (needs 10; runs 1-10)
33 case(s): 23 passed, 10 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not runText description of the chart
Two bars. With run 1 only, a tester finds 6 of the 10 failing cases. With all 10 runs, the runner finds all 10. The four that one run misses fail in 1, 2, 1 and 5 runs of 10.
What this tells us. 23 of 33 cases pass all 10 runs, and 270 of 330 runs pass (82%). The one-run baseline would find 6 of the 10 failing cases. It misses the dismissal answer with no pointer to advice, which fails in 5 of 10 runs. That is the core of why one passing run does not show that a feature works.
Before you report, read failing runs and some passing ones. Two more first checks were too weak. contains: 48 hours passed answers about “48 hours’ uninterrupted rest”, so R3-05 now uses regex: "48[- ]hours? (a|per|each) week". R1-04 passed an answer that gave £8 and then £10.85; a not_regex now fails it. A check is a claim about answers: test it on real answers, as you test the app.
Text description of the chart
Ten rows, one per failing case. Five fail in 10 of 10 runs (interval about 72% to 100%). They are the old wage rate, the apprentice rate, paternity, notice and the 48-hour week. The dismissal answer without a pointer fails in 5 of 10 (about 24% to 76%). Next year’s rate fails in 2 of 10 (about 6% to 51%). Three cases fail in 1 of 10 (about 2% to 40%): sick pay from the first day, redundancy, and the name pair.
The intervals show what 10 runs can tell, as in Repeat runs and reproduction rate. 10 of 10 is clear, but 1 of 10 could be a rare bug or a 40% bug.
Part 5: report the bugs
Section titled “Part 5: report the bugs”The campaign on 1.0 gives five bug reports. Each has a severity from harm and frequency, as in Severity and triage:
| Bug | What goes wrong | Rate on 1.0 | Severity | Suspected layer |
|---|---|---|---|---|
| BUG-01 | a 25-year-old gets last year’s National Living Wage (£12.21) | 10 of 10 | high | search, then model |
| BUG-02 | an apprentice aged 20 in the second year gets the apprentice rate (£8) | 10 of 10 | high | model: the rule was retrieved first |
| BUG-03 | “notice to resign” and “longest working week” miss their sections | 10 of 10 each | medium | search |
| BUG-04 | after 2 months in a job, “from the first day” is never said for Paternity Leave | 10 of 10 | medium | search |
| BUG-05 | a dismissal answer gives no pointer to Acas or an adviser | 5 of 10 | medium | prompt |
BUG-01 is the most harmful and the clearest, so it gets the agent-ready report, with a failing case. Callum may give it to a coding agent, so the report says everything an agent cannot guess. The key parts of qa-work/bugs/BUG-01-old-living-wage.md:
## Expected (checkable)- Must: give £12.71 an hour (the "Current rates" row "April 2026", 21 and over) and cite national-minimum-wage-rates.- Must not: present an earlier rate as the rate today.- Must keep: a past rate when the person asks about a past date (case R1-06, "in 2022" gives £9.18).
## Reproduction rate10 of 10 recorded runs fail (runs 1-10). 95% interval for the failure rate: 72% to 100%.
## Suspected layerRetrieval, then generation - a guess, because the trace shows that the search puts "Rates before 1 April 2021"first: the question's "25" matches the table heading "25 and over" of the old rates.
## Constraints for the fixer- Touch only: guide/retrieve.py, guide/assistant.py, guide/config.py, guide/prompts/ (as a new version)- Do not: change or delete cases or checks, hard-code £12.71 or this question, remove the previous rates from the data, change data/ or recordings/, add a dependencyThe “Must keep” line matters. The easiest wrong fix is “never give a previous rate”, and R1-06 would catch it. The failing case is attached, and it fails today:
python -m qa run qa-work/bugs/BUG-01.yaml --version 1.0FAIL BUG-01-current-living-wage 0/10 passed (needs 10; runs 1-10) first failing run: run 1, trace q02-v1.0-r1 - contains: missing ['£12.71']Write the other four with templates/bug-report.md. Keep personal data out: every person in a case is invented. Four more cases fail in 1 or 2 runs of 10. The session notes keep them, and the regression set watches them: a rare failure today can grow in the next version.
Part 6: the alternative, version 1.1
Section titled “Part 6: the alternative, version 1.1”Callum read the reports and made version 1.1. The course author made these changes in Callum’s role, from the reports; no coding agent was used. Before the changes, he checked the search on the 35 campaign questions that have an answer in the guidance. He counted how often a retrieved section contains the answer:
| Search settings | Questions whose answer is retrieved |
|---|---|
| 4 sections, numbers kept (1.0) | 29 of 35 |
| 6 sections, numbers kept | 31 of 35 |
| 4 sections, numbers ignored | 27 of 35 |
| 6 sections, numbers ignored (1.1) | 32 of 35 |
He used the campaign questions only, never the pilot week’s. Version 1.1 makes three changes:
- The search ignores words with digits (“25”, “2 months”): they describe the person, not the topic.
- It keeps 6 sections instead of 4.
- A new prompt with six rules. Rule 1: “Answer only from the GOV.UK guidance below. Do not add facts from memory. If the guidance does not answer the question, say so … and suggest a Thistlecombe adviser or Acas.” Rule 2 asks for the current rate unless a past date is asked. Rule 3 asks the model to check each condition against what the person said.
Three changes at once make a cause harder to find. That is common, and the regression set is what protects you.
Part 7: a fair comparison under one measure
Section titled “Part 7: a fair comparison under one measure”Compare the versions under one measure, on the same ground. Use the same 33 planned cases, the same 10 recorded runs and the same checks, written before 1.1 existed. First the failing case of BUG-01:
PASS BUG-01-current-living-wage 10/10 passed (needs 10; runs 1-10)Then everything:
python -m qa run cases/ --version 1.1FAIL R2-01 4/10 passed (needs 10; runs 1-10)FAIL R2-03 9/10 passed (needs 10; runs 1-10)FAIL R2-05 1/10 passed (needs 10; runs 1-10) first failing run: run 1, trace q12-v1.1-r1 - contains: missing ['2 years']FAIL R2-06 9/10 passed (needs 10; runs 1-10)FAIL R3-03 0/10 passed (needs 10; runs 1-10)FAIL R3-05 0/10 passed (needs 10; runs 1-10)
33 case(s): 27 passed, 6 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not runText description of the chart
Fourteen dots, two per area. Money amounts rise from 69% to 100%. Who qualifies falls from 83% to 63%. Time limits stay at 71%. The edge of the guidance rises to 100%. Misuse and over-refusal stay at 100%. Relations rise from 98% to 100%.
What this tells us. Under the one measure, 1.1 is better: 27 of 33 cases pass all runs, against 23. No answer of 1.1 gives an old wage rate (1.0: 10 answers). BUG-01, BUG-02 and BUG-05 are fixed. BUG-03 is not: the search still misses “notice” and “48 hours”. But “Who qualifies” got worse, and the reasons are worth the time.
A regression, BUG-06. The redundancy question (“How long do I need to work …?”) passed 9 of 10 on 1.0 and 1 of 10 on 1.1. The trace of 1.1 shows six redundancy sections, but not the one with “2 years or more”. The trace of 1.0 shows that it was missing there too. So 1.0 answered “2 years” from the model’s memory: right by luck. Rule 1 stopped the memory, and the search gap now shows. The fix of one bug uncovered another.
A second regression, BUG-09. “Do I get sick pay from my first day off sick?” passed 9 of 10 on 1.0 and 4 of 10 on 1.1. Here the right section is retrieved, and it says that SSP is paid “for all the full days you’re off sick”. Yet 6 answers say “The guidance does not say whether SSP is paid from your first sick day”. Rule 1 made the model unwilling to draw a plain conclusion. Same rule, a different layer: the prompt.
Common mistake
A check that passes for the wrong reason
Symptom. R2-03, “I started my job 2 months ago. Can I take Paternity Leave?”, passes 9 of 10 runs on 1.1, as if BUG-04 were almost fixed.
Cause. Every one of the 10 answers says “The guidance provided doesn’t say whether starting your job 2 months ago is enough”. The check matched “first day of employment” in another sentence, about surrogacy. The eligibility section is still not retrieved.
Fix. Read passing runs after a fix, not only failing ones. Add a not_regex for “doesn’t say”, so the case fails 10 of 10, and keep BUG-04 open.
Read the pairs side by side. MR2-paternity passes 10 of 10 on 1.1. But its check (“first day”) failed for both names in the same way, so the relation saw no difference. Read together, the pairs show one: in 5 of James’s 10 answers, he is “the baby’s father’s husband”. Aisha’s 10 answers name her relationship correctly. That is BUG-10, a new case MR2-relationship with not_contains: "father's husband". Ten runs cannot show whether names cause it; a larger name-swap run can.
New failures go into the regression set as known bugs (status: known-bug, bug: BUG-06), so the runner tells “still failing” from “new”. The fix verification of BUG-01 closes it: its case and its neighbours pass. But the suite has new failures, so the release waits for triage.
The recommendation
Section titled “The recommendation”Your draft for Joanna, from the planned cases alone, is to block the public release of version 1.1.
- Two rules that worked in 1.0 fail now: BUG-06 and BUG-09.
- First-day and time-limit answers still say “the guidance doesn’t say”.
But 1.1 is better than 1.0 for an internal trial, where an adviser reads every answer. In our runs, it never gives an old rate.
Part 8: an end-to-end run on new input
Section titled “Part 8: an end-to-end run on new input”The pilot week’s 8 questions stayed closed until now. Real users wrote them, so they test what the plan did not foresee. Run them once, on the version you recommend, and on 1.0 for comparison:
python -m qa run pilot/ --version 1.1FAIL P-01 0/10 passed (needs 10; runs 1-10) first failing run: run 1, trace n01-v1.1-r1 - contains: missing ['£8']FAIL P-06 6/10 passed (needs 10; runs 1-10) first failing run: run 1, trace n06-v1.1-r1 - not_contains: found ['£7,500']
8 case(s): 6 passed, 2 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not runVersion 1.1 passes 6 of 8; 1.0 passes 5. Two new bugs:
- BUG-07 (medium, 10 of 10): “I’m 17. What’s the least my employer can pay me per hour?” gets “The guidance provided doesn’t say”. The words “least … per hour” match the young workers’ hours, not the rates. The same search family as BUG-03 and BUG-06.
- BUG-08 (high, 4 of 10): “How much redundancy pay will I get? I’m 45, I’ve worked there for 10 years and I earn £600 a week.” The right total is 12 weeks’ pay, £7,200. Run 2 says £9,000; runs 1, 4 and 10 say “£7,200–£7,500”. The right section was retrieved: this is the model’s arithmetic.
BUG-08 is a wrong amount of money, which blocks a release at any rate. The final sign-off, qa-work/release-sign-off-1.1.md, keeps the recommendation and adds the evidence:
Block the public release of 1.1. It fixes both wrong-amount bugs of the campaign, but it brings two regressions (BUG-06, BUG-09). The pilot week found a wrong redundancy total in 4 of 10 runs (BUG-08). Use 1.1 instead of 1.0 for the advisers’ internal trial, where an adviser reads every answer before it goes to a client.
The conditions for 1.2 are four. No personal totals: give the rule and GOV.UK’s calculator link. Rule 1 with an example of an allowed plain conclusion. A search that finds everyday words. A larger name-swap run. Then the regression set, now 36 cases, on 1.2, and a new pilot set that nobody has seen. Joanna decides; you gave her the evidence. The final regression run on 1.1:
36 case(s): 27 passed, 0 failed, 9 known bug(s) still failing, 0 known bug(s) now passing, 0 not runInsights and limits
Section titled “Insights and limits”- Half of the bugs started in the search, not the model. In five of the ten bugs, the section with the answer was never retrieved. The trace was the fastest tool of the campaign.
- A stricter prompt has two sides. “Answer only from the guidance” stopped old rates and lucky guesses. It also made the model say “the guidance doesn’t say” when the guidance was clear. Test a prompt rule on the cases it should not change.
- Checks need testing too. Four checks passed answers they should have failed. Reading answers, with the rubric, found each one.
- New input matters. The pilot week found a wrong amount that 33 planned cases did not.
Limits: the recordings are one model version, at one time. Another model, or the same one later, will answer differently; run the regression set again before you trust it. Ten runs give wide intervals for rare bugs. The sample has 16 guides, and GOV.UK changes them: the April 2026 rates in this sample will change. The rubric scores are one person’s (the course author’s); a second scorer would show how far two people agree.
Responsible use
Section titled “Responsible use”- Not advice. The assistant and its answers are test material with known bugs. Point anyone with a real problem to GOV.UK, Acas or an adviser.
- No real cases in test data. Use invented people. Never paste a client’s situation, name, employer or pay into a test, a report or an outside tool.
- Credit the source. Keep the Open Government Licence attribution with the sample and in your README.
- Say what the evidence is. Write “recorded (real model), 10 runs” next to every rate. Do not turn “0 of 10” into “never”.
- Respect live services. With
--live, call only an endpoint that you are allowed to use, and know its price first.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This campaign shows what a team looks for in a tester of AI features. It has a plan from risks and checks that do not need one right answer. It has measured reproduction rates, bug reports that people and agents can act on, and a decision with evidence.
ask-about-work-qa/ README.md what it is, how to run it, the results of both versions DATASET.md the source, the licence and attribution, the checksum, the limits qa-work/ risk table, criteria, rubric and scores, relations, two sessions, bug reports, sign-off cases/ the regression set (36 cases) pilot/ the new-input check guide/ qa/ the assistant and the runner (MIT)- Make a new Git repository from your finished folder, and push it to GitHub.
.gitignorekeeps out.venv/andreports/. - In the README, put the comparison table: 23 and 27 of 33 cases, the pilot results, and your recommendation.
- Copy
templates/.githubinto the repository, and open BUG-03 as an issue with the issue form.
In an interview, say it in two sentences: “I tested a retrieval assistant on real GOV.UK guidance with 10 recorded runs per question. Ten runs found four failures that one run missed, and my sign-off blocked a release that fixed three bugs but brought two regressions.”
Go further. Score 13 answers yourself with the rubric and compare with the author’s scores. Write a fourth relation (for example, a question with typing mistakes). With your own key, run python -m qa run cases/ --live and compare with the recordings.
Knowledge check
Section titled “Knowledge check”Check your understanding
Version 1.0 answered the redundancy question correctly in 9 of 10 runs. Version 1.1 says 'the guidance does not say' in 9 of 10. Callum says 1.1 broke the search. What does the evidence show?
Show the answer
For this question, the search fails in both versions. The section with “2 years or more” is missing from the traces of 1.0 and 1.1. Version 1.0 answered from the model’s memory, which happened to be right. Rule 1 of 1.1 (“answer only from the guidance”) stopped that, so the old search gap now shows. The cause is retrieval; the prompt change only made it visible. Keep rule 1, and fix the search.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
R2-03 passes 9 of 10 runs on 1.1. Why is it wrong to close BUG-04?
Show the answer
The check matched “first day of employment” in a sentence about surrogacy. Every answer said that the guidance does not say whether 2 months is enough. A passing check is a claim, not a proof. Read the passing answers after a fix, and make the check stricter (here with a not_regex). Run it again: it fails 10 of 10, so BUG-04 stays open.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Version 1.1 passes 27 of 33 planned cases, against 23 for 1.0. Joanna asks why you still block the public release.
Show the answer
A better total does not cancel a blocker. Two rules of the sign-off are broken. There are regressions (BUG-06 and BUG-09). There is a wrong amount of money (BUG-08, 4 of 10 runs on new input). A person could act on a wrong redundancy total. 1.1 is still the better choice for the advisers’ trial, because an adviser reads each answer.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You planned from risks and agreed checkable criteria with the product owner before testing.
- You judged answers with a rubric, property checks and three metamorphic relations, and you tested the checks too.
- On version 1.0, ten runs found 10 failing cases; one run found 6. You reported them with rates, and BUG-01 as an agent-ready report with its failing case.
- Under one measure, 1.1 passed 27 of 33 cases against 23, fixed three bugs, and brought two regressions that the trace explained.
- New input found a wrong amount, and your sign-off blocked the public release with evidence.
Your workspace now: qa-work/ with the risk table, criteria, rubric and scores, relations, two sessions, ten bug reports and the sign-off; cases/ with 36 cases; pilot/.
Next, a case study on real open-source issues: rate them for agent-readiness and rewrite the weak ones.