Final assignment
- After the case studies
- about 3 hr
Where we are
In this course, you learned to test an AI assistant from plan to release: judge replies with rules, explore, measure how often a failure happens, report it so that a person or a coding agent can fix it, and protect the fix with a regression set. Now a new release comes in, and nobody has tested it yet. In this assignment, you use all of it on your own.
What you will be able to do
Use the whole course on a release that nobody has tested yet. You plan, explore, measure, report and protect, as Esme would at work. At the end, you give Omar a recommendation that he can check line by line.
Before you startWhat to know and to have ready
What you need to know
- Modules 1 to 7. Each part below names the lessons that it uses.
- You do not need the case studies for this assignment. The certificate asks for at least one of them.
What you need on your computer
- Your course folder
ai-qa, with the practice app inpractice-appand its virtual environment. You need Python 3.10 or newer. - Your workspace at the end of Module 7, with your regression set in
workspace/regression/. If you are not sure, copy the files ofworkspace/snapshots/end-of-m07intoworkspace. That snapshot is the course’s workspace at the end of Module 7. - The folder
ai-qa/releases/2.4.0-rc. It came with the course folder.
No account, no key and no money. Everything runs on the simulated model, so every result that you report is simulated (practice app).
Tested on macOS (Apple silicon) with Python 3.14.6. The Windows and Linux commands follow Module 1; they were not run for this page.
The situation
Section titled “The situation”It is Thursday afternoon. Tomás sends Esme a release candidate, 2.4.0-rc1, with these notes:
- New prompt support-v9: shorter answers.- Fixed: answers to French questions are in French.- Fixed: the assistant always asks before it cancels an order, also when the customer insists.- Fixed: safe gardening questions (for example about weed killer) are no longer refused.- Fixed: the assistant no longer shares its instructions.- Changed: the confirmation step for actions was rewritten.- Changed: refund amounts are now computed in one place (simpler code, same rules).- Includes the fixes of 2.3.1.Omar wants to release on Monday. “Is it better than what customers have now? Can we ship it?” he asks. Grace adds: “And show me that it still follows my acceptance criteria.”
You are Esme. Every line in the notes is a claim, not evidence. A “fixed” line may be true only for the case that Tomás tried. A “changed” line, and the new prompt, can break rules that nobody meant to touch. This candidate has several new problems, of different kinds. Some are easy to see, and some appear only on some runs. This page does not say what they are, or where.
Test the behaviour, as a tester does. Your work is marked on evidence from runs: exports, traces, reproduction rates and failing cases. You could compare the code of the two releases, but that misses the point of the exercise. A finding without run evidence does not count.
The workflow
Section titled “The workflow”The picture shows the order of the work. The green boxes are the parts that you submit.
Text description of the diagram
The candidate arrives with its release notes. First, you run your regression set on 2.3.0, the baseline. Second, you plan: the risk table, rubrics, property checks and at least three metamorphic relations. Third, you run the set on the candidate and check each claimed fix with neighbours. Fourth, you run two exploratory sessions, each with its own charter. Fifth, you repeat runs to measure a reproduction rate for every finding. Sixth, you write five bug reports, one of them agent-ready with a failing case. Seventh, you add a case per bug to the regression set and run it on both versions. Eighth, you write the sign-off. Omar decides.
Set up the candidate
Section titled “Set up the candidate”Keep the final work in its own folder, workspace/final, so that it does not mix with the module work.
-
Open a terminal in
~/projects/ai-qa/practice-appand activate the virtual environment, as in Meet the bug-hunt app. -
Make the folders for your submission, and copy your regression set into it:
Terminal window mkdir -p ../workspace/final/{plan,notes,bugs/evidence,cases,regression,signoff}cp ../workspace/regression/regression.yaml ../workspace/final/regression/Terminal window "plan","notes","bugs\evidence","cases","regression","signoff" | ForEach-Object { New-Item -ItemType Directory -Force "..\workspace\final\$_" }Copy-Item ..\workspace\regression\regression.yaml ..\workspace\final\regression\Terminal window mkdir -p ../workspace/final/{plan,notes,bugs/evidence,cases,regression,signoff}cp ../workspace/regression/regression.yaml ../workspace/final/regression/ -
Check the copied set:
Terminal window python -m qa check ../workspace/final/regressionWith the course’s set, the output starts with
15 valid case(s): reg-smoke, reg-cancel-asks-first, …. Your count can differ. -
Move to the candidate’s folder, check its versions and reset its data:
Terminal window cd ../releases/2.4.0-rcpython -m helpdesk versionpython -m helpdesk resetTerminal window cd ..\releases\2.4.0-rcpython -m helpdesk versionpython -m helpdesk resetTerminal window cd ../releases/2.4.0-rcpython -m helpdesk versionpython -m helpdesk resetlarkfield-helpdesk 2.4.0-rc1 | model sim-1 (simulated) | prompt support-v9 | app date 2026-11-16The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.Two of the three versions changed: the app and the prompt. The model is still sim-1.
-
Send one message and export it, to check that the evidence works:
Terminal window python -m helpdesk ask "Where is my order LK-200101?" --seed 1python -m helpdesk export c-0001 -o ../../workspace/final/bugs/evidence/setup-check.jsonOrder LK-200101 (1 x HT-550 corded hedge trimmer): Processing, placed on 2026-11-14.[conversation c-0001 | trace tr-0001-1 | seed 1]Exported c-0001 to ../../workspace/final/bugs/evidence/setup-check.json -
Go back to
practice-app(cd ../../practice-app, orcd ..\..\practice-appon Windows). Run the set on 2.3.0, your baseline, and keep the report:Terminal window python -m qa run ../workspace/final/regression --report-dir ../workspace/final/reportsWith the course’s set, the last lines are:
15 case(s): 7 passed, 0 failed, 8 known bug(s) still failing, 0 known bug(s) now passing, 0 not runReport: ../workspace/final/reports/qa-YYYYMMDD-HHMMSS/report.md
Now the candidate is yours. Run the same set on it, and keep this report next to the baseline’s:
python -m qa run ../workspace/final/regression --app ../releases/2.4.0-rc --report-dir ../workspace/final/reportsCheck that the second line names 2.4.0-rc1, then read every line. This page does not show that output.
For the web app, run python -m helpdesk serve in the candidate’s folder. If 2.3.0 is still running on port 8730, stop it first, or add --port 8731. Run reset in the candidate’s folder before each session.
What you deliver
Section titled “What you deliver”Four required items. Each one uses lessons and templates that you know. Copy the templates from ai-qa/templates (also on GitHub).
| Item | Put it in | Templates | Lessons that taught it |
|---|---|---|---|
| 1. A risk-based test plan with rubrics, property checks and at least three metamorphic relations | final/plan/, cases in final/cases/ |
risk-table.md, acceptance-criteria.md, rubric.md, metamorphic-relations.md |
Risk-based test planning, Rubrics and property checks, Metamorphic testing |
| 2. Notes from at least two exploratory sessions, with charters | final/notes/ |
test-charter.md, session-notes.md |
Test charters and sessions, Personas, edge cases and hostile inputs |
| 3. Five bug reports with reproduction rates; at least one agent-ready, with a failing case | final/bugs/, exports in final/bugs/evidence/, the case in final/cases/ |
bug-report.md, agent-ready-bug-report.md, failing-case.yaml |
Repeat runs and reproduction rate, Capture the evidence, A bug report template for AI behaviour, Severity and triage, Agent-ready issue templates, Turn a bug into a failing case |
| 4. A regression set that runs with the runner, and a release recommendation that cites its results | final/regression/, final/reports/, final/signoff/release-2.4.0-rc.md |
release-sign-off.md, fix-verification-checklist.md |
From findings to a regression set and a release sign-off, Verify the agent’s fix |
For charters about retrieval, actions and safety, use Module 6: Retrieval or generation?, Tools, actions and permissions, Prompt injection and data leaks and Harmful, biased and over-cautious output.
Also write a short final/README.md: a list of your files, what each one holds, and a section AI assistance. Say which tools you used, for what, and what they got wrong.
A time plan
Section titled “A time plan”About three hours. Split it over two or three sittings if you want; the time boxes still help.
| Minutes | Step | Output |
|---|---|---|
| 15 | Set up, read the release notes, run the baseline | final/reports/ (2.3.0) |
| 30 | Update the risk table for this release; rubrics, property checks, relations | final/plan/, first cases |
| 15 | Run the set on the candidate; check each claimed fix with two neighbours | final/reports/ (2.4.0-rc), notes |
| 60 | Two sessions of 30 minutes, each with its own charter | final/notes/ |
| 30 | Repeat runs for every finding; five bug reports, one agent-ready | final/bugs/, final/cases/ |
| 30 | Grow the set, run it on both versions, write the sign-off and the README | final/regression/, final/signoff/ |
Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all four must be demonstrated. You can improve an item and send it again. If no reviewer is available to you, use the table on your own work, or ask a colleague.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. Risk-based test plan | The risk table rates harm and use, and puts the areas that this release changed higher. Each high row has a rubric, a property check or a case. At least three metamorphic relations, each with a case that runs. | The table is there, but the release’s changes do not affect it. Or only two relations. Or a rubric line says “good answer”, which two testers would apply differently. | No ratings. Or relations that change the meaning of the input, so a different answer is correct. |
| 2. Two exploratory sessions | Two charters with a target, resources and a question, written before the session. Notes with a line per message, trace IDs and seeds. A debrief that names findings, open questions and what was not covered. | Notes without trace IDs. Or a second “session” that repeats the first charter. | No charters. Or notes written afterwards from memory. |
| 3. Five bug reports | Each has the exact input, versions, the trace ID, checkable Must / Must not lines, a severity, and a rate as “x of y runs, seeds …”. Most are new in 2.4.0-rc, of different kinds. The agent-ready report has an export and a failing case. That case fails on the candidate and passes where the old version was right. | A rate without the number of runs. Or a failing case that also fails on 2.3.0 for a bug that the candidate brought. Or five variants of one bug. | “Sometimes wrong” with no rate. Or the failing case passes on the candidate. |
| 4. Regression set and recommendation | The set runs with python -m qa run. It has a case per new bug and the earlier guards. The report files of both runs are there: 2.3.0 and the candidate. Every FIXED? has its neighbours. The sign-off follows the three questions. It cites the counts and the report files, and says what was not tested. |
The recommendation is right, but it cites no report. Or you closed FIXED? cases without neighbours. | No run on the candidate. Or you deleted or weakened a guard to make the set pass. |
Not accepted: a recommendation to approve while your own evidence shows a failed guard or a new critical bug. Critical means safety, a data leak, wrong money or an action without consent. Item 4 is then “not yet demonstrated”, however strong the rest is.
What a strong submission looks like
Section titled “What a strong submission looks like”- It checks every line of the release notes and says which claims hold, which hold only in part, and how it knows. Each fix has at least two neighbours: another wording, another customer, another language.
- It finds several new problems, of different kinds, not five versions of one. Each has a rate from enough runs for its frequency.
- It separates new from old. A bug that was already in 2.3.0 is not a reason to block a release that fixes other things. A rule that worked in 2.3.0 and fails now is a regression, at any rate.
- Its sign-off reads in two minutes. The recommendation comes first, then the evidence table, the blockers and what was not tested. Omar can check every number in the report files without running anything.
- It labels every result “simulated (practice app)”. It says that a real model would need its own recorded runs.
Common mistake
A claimed fix accepted on its own case
Symptom: the runner shows FIXED? for a case, and the sign-off says “fixed”.
Cause: FIXED? means “this exact case passes now”. A change can make the case pass without removing the cause, as Verify the agent’s fix showed.
Recover: run neighbours that the fix must also handle. Run the guards that sit next to the changed rule. Read what changed in the trace. Then write “fixed”, “fixed in part” or “not fixed”, with the evidence.
Common mistake
A guard that fails on the baseline
Symptom: a new guard case fails on 2.3.0 as well as on the candidate.
Cause: it tests a bug that 2.3.0 already had, not a rule that worked.
Recover: a guard must pass on the baseline. Give the case a bug: and status: known-bug line and its own bug report. Or keep it as a note for a later release.
How to submit
Section titled “How to submit”Make a ZIP file of workspace/final, without the hidden .helpdesk folders and without virtual environments. Its README.md comes first. Send the ZIP file to your reviewer, or put the folder in a repository that your reviewer can read. Use test data only: the practice app has no real customers, so your files must not have any either.
Recap
- The release notes are claims. Your plan, sessions and regression set turn them into evidence.
- Every finding gets a rate from enough runs, and every new bug gets a saved case.
- The sign-off follows three questions: did a guard fail, is a new critical bug open, are the criteria met?
- If you have not done one yet, choose a case study for the certificate: Test a public-guidance assistant, Agent-ready reports from real open-source issues or A safety test pass with open attack sets.
Your workspace now: everything from Module 7, plus final/ with your plan, notes, bug reports, cases, regression set, reports, sign-off and README.