Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

In this course, you learned to test an AI assistant from plan to release: judge replies with rules, explore, measure how often a failure happens, report it so that a person or a coding agent can fix it, and protect the fix with a regression set. Now a new release comes in, and nobody has tested it yet. In this assignment, you use all of it on your own.

What you will be able to do

Use the whole course on a release that nobody has tested yet. You plan, explore, measure, report and protect, as Esme would at work. At the end, you give Omar a recommendation that he can check line by line.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

  • Modules 1 to 7. Each part below names the lessons that it uses.
  • You do not need the case studies for this assignment. The certificate asks for at least one of them.

What you need on your computer

  • Your course folder ai-qa, with the practice app in practice-app and its virtual environment. You need Python 3.10 or newer.
  • Your workspace at the end of Module 7, with your regression set in workspace/regression/. If you are not sure, copy the files of workspace/snapshots/end-of-m07 into workspace. That snapshot is the course’s workspace at the end of Module 7.
  • The folder ai-qa/releases/2.4.0-rc. It came with the course folder.

No account, no key and no money. Everything runs on the simulated model, so every result that you report is simulated (practice app).

Tested on macOS (Apple silicon) with Python 3.14.6. The Windows and Linux commands follow Module 1; they were not run for this page.

It is Thursday afternoon. Tomás sends Esme a release candidate, 2.4.0-rc1, with these notes:

releases/2.4.0-rc/RELEASE-NOTES.md
- New prompt support-v9: shorter answers.
- Fixed: answers to French questions are in French.
- Fixed: the assistant always asks before it cancels an order, also when the customer insists.
- Fixed: safe gardening questions (for example about weed killer) are no longer refused.
- Fixed: the assistant no longer shares its instructions.
- Changed: the confirmation step for actions was rewritten.
- Changed: refund amounts are now computed in one place (simpler code, same rules).
- Includes the fixes of 2.3.1.

Omar wants to release on Monday. “Is it better than what customers have now? Can we ship it?” he asks. Grace adds: “And show me that it still follows my acceptance criteria.”

You are Esme. Every line in the notes is a claim, not evidence. A “fixed” line may be true only for the case that Tomás tried. A “changed” line, and the new prompt, can break rules that nobody meant to touch. This candidate has several new problems, of different kinds. Some are easy to see, and some appear only on some runs. This page does not say what they are, or where.

Test the behaviour, as a tester does. Your work is marked on evidence from runs: exports, traces, reproduction rates and failing cases. You could compare the code of the two releases, but that misses the point of the exercise. A finding without run evidence does not count.

The picture shows the order of the work. The green boxes are the parts that you submit.

A flow in three rows, read left to right. 2.4.0-rc arrives with its release notes. 1. Baseline: run your regression set on 2.3.0. 2. Plan: risk table, rubrics, property checks, at least 3 metamorphic relations. 3. Run the set on 2.4.0-rc; check each claimed fix with neighbours. 4. Two exploratory sessions, each with its own charter. 5. Repeat runs: a reproduction rate for every finding. 6. Five bug reports; one agent-ready, with a failing case. 7. A case per bug; run the set on both versions. 8. Sign-off: approve, approve with conditions, or block. Last: Omar decides. Steps 2, 4, 6, 7 and 8 have green borders: they are submitted.
The assignment in eight steps. Green borders: the parts that you submit.
Text description of the diagram

The candidate arrives with its release notes. First, you run your regression set on 2.3.0, the baseline. Second, you plan: the risk table, rubrics, property checks and at least three metamorphic relations. Third, you run the set on the candidate and check each claimed fix with neighbours. Fourth, you run two exploratory sessions, each with its own charter. Fifth, you repeat runs to measure a reproduction rate for every finding. Sixth, you write five bug reports, one of them agent-ready with a failing case. Seventh, you add a case per bug to the regression set and run it on both versions. Eighth, you write the sign-off. Omar decides.

Keep the final work in its own folder, workspace/final, so that it does not mix with the module work.

  1. Open a terminal in ~/projects/ai-qa/practice-app and activate the virtual environment, as in Meet the bug-hunt app.

  2. Make the folders for your submission, and copy your regression set into it:

    Terminal window
    mkdir -p ../workspace/final/{plan,notes,bugs/evidence,cases,regression,signoff}
    cp ../workspace/regression/regression.yaml ../workspace/final/regression/
  3. Check the copied set:

    Terminal window
    python -m qa check ../workspace/final/regression

    With the course’s set, the output starts with 15 valid case(s): reg-smoke, reg-cancel-asks-first, …. Your count can differ.

  4. Move to the candidate’s folder, check its versions and reset its data:

    Terminal window
    cd ../releases/2.4.0-rc
    python -m helpdesk version
    python -m helpdesk reset
    larkfield-helpdesk 2.4.0-rc1 | model sim-1 (simulated) | prompt support-v9 | app date 2026-11-16
    The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.

    Two of the three versions changed: the app and the prompt. The model is still sim-1.

  5. Send one message and export it, to check that the evidence works:

    Terminal window
    python -m helpdesk ask "Where is my order LK-200101?" --seed 1
    python -m helpdesk export c-0001 -o ../../workspace/final/bugs/evidence/setup-check.json
    Order LK-200101 (1 x HT-550 corded hedge trimmer): Processing, placed on 2026-11-14.
    [conversation c-0001 | trace tr-0001-1 | seed 1]
    Exported c-0001 to ../../workspace/final/bugs/evidence/setup-check.json
  6. Go back to practice-app (cd ../../practice-app, or cd ..\..\practice-app on Windows). Run the set on 2.3.0, your baseline, and keep the report:

    Terminal window
    python -m qa run ../workspace/final/regression --report-dir ../workspace/final/reports

    With the course’s set, the last lines are:

    15 case(s): 7 passed, 0 failed, 8 known bug(s) still failing, 0 known bug(s) now passing, 0 not run
    Report: ../workspace/final/reports/qa-YYYYMMDD-HHMMSS/report.md

Now the candidate is yours. Run the same set on it, and keep this report next to the baseline’s:

Terminal window
python -m qa run ../workspace/final/regression --app ../releases/2.4.0-rc --report-dir ../workspace/final/reports

Check that the second line names 2.4.0-rc1, then read every line. This page does not show that output.

For the web app, run python -m helpdesk serve in the candidate’s folder. If 2.3.0 is still running on port 8730, stop it first, or add --port 8731. Run reset in the candidate’s folder before each session.

Four required items. Each one uses lessons and templates that you know. Copy the templates from ai-qa/templates (also on GitHub).

Item Put it in Templates Lessons that taught it
1. A risk-based test plan with rubrics, property checks and at least three metamorphic relations final/plan/, cases in final/cases/ risk-table.md, acceptance-criteria.md, rubric.md, metamorphic-relations.md Risk-based test planning, Rubrics and property checks, Metamorphic testing
2. Notes from at least two exploratory sessions, with charters final/notes/ test-charter.md, session-notes.md Test charters and sessions, Personas, edge cases and hostile inputs
3. Five bug reports with reproduction rates; at least one agent-ready, with a failing case final/bugs/, exports in final/bugs/evidence/, the case in final/cases/ bug-report.md, agent-ready-bug-report.md, failing-case.yaml Repeat runs and reproduction rate, Capture the evidence, A bug report template for AI behaviour, Severity and triage, Agent-ready issue templates, Turn a bug into a failing case
4. A regression set that runs with the runner, and a release recommendation that cites its results final/regression/, final/reports/, final/signoff/release-2.4.0-rc.md release-sign-off.md, fix-verification-checklist.md From findings to a regression set and a release sign-off, Verify the agent’s fix

For charters about retrieval, actions and safety, use Module 6: Retrieval or generation?, Tools, actions and permissions, Prompt injection and data leaks and Harmful, biased and over-cautious output.

Also write a short final/README.md: a list of your files, what each one holds, and a section AI assistance. Say which tools you used, for what, and what they got wrong.

About three hours. Split it over two or three sittings if you want; the time boxes still help.

Minutes Step Output
15 Set up, read the release notes, run the baseline final/reports/ (2.3.0)
30 Update the risk table for this release; rubrics, property checks, relations final/plan/, first cases
15 Run the set on the candidate; check each claimed fix with two neighbours final/reports/ (2.4.0-rc), notes
60 Two sessions of 30 minutes, each with its own charter final/notes/
30 Repeat runs for every finding; five bug reports, one agent-ready final/bugs/, final/cases/
30 Grow the set, run it on both versions, write the sign-off and the README final/regression/, final/signoff/

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all four must be demonstrated. You can improve an item and send it again. If no reviewer is available to you, use the table on your own work, or ask a colleague.

Required item Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Risk-based test plan The risk table rates harm and use, and puts the areas that this release changed higher. Each high row has a rubric, a property check or a case. At least three metamorphic relations, each with a case that runs. The table is there, but the release’s changes do not affect it. Or only two relations. Or a rubric line says “good answer”, which two testers would apply differently. No ratings. Or relations that change the meaning of the input, so a different answer is correct.
2. Two exploratory sessions Two charters with a target, resources and a question, written before the session. Notes with a line per message, trace IDs and seeds. A debrief that names findings, open questions and what was not covered. Notes without trace IDs. Or a second “session” that repeats the first charter. No charters. Or notes written afterwards from memory.
3. Five bug reports Each has the exact input, versions, the trace ID, checkable Must / Must not lines, a severity, and a rate as “x of y runs, seeds …”. Most are new in 2.4.0-rc, of different kinds. The agent-ready report has an export and a failing case. That case fails on the candidate and passes where the old version was right. A rate without the number of runs. Or a failing case that also fails on 2.3.0 for a bug that the candidate brought. Or five variants of one bug. “Sometimes wrong” with no rate. Or the failing case passes on the candidate.
4. Regression set and recommendation The set runs with python -m qa run. It has a case per new bug and the earlier guards. The report files of both runs are there: 2.3.0 and the candidate. Every FIXED? has its neighbours. The sign-off follows the three questions. It cites the counts and the report files, and says what was not tested. The recommendation is right, but it cites no report. Or you closed FIXED? cases without neighbours. No run on the candidate. Or you deleted or weakened a guard to make the set pass.

Not accepted: a recommendation to approve while your own evidence shows a failed guard or a new critical bug. Critical means safety, a data leak, wrong money or an action without consent. Item 4 is then “not yet demonstrated”, however strong the rest is.

  • It checks every line of the release notes and says which claims hold, which hold only in part, and how it knows. Each fix has at least two neighbours: another wording, another customer, another language.
  • It finds several new problems, of different kinds, not five versions of one. Each has a rate from enough runs for its frequency.
  • It separates new from old. A bug that was already in 2.3.0 is not a reason to block a release that fixes other things. A rule that worked in 2.3.0 and fails now is a regression, at any rate.
  • Its sign-off reads in two minutes. The recommendation comes first, then the evidence table, the blockers and what was not tested. Omar can check every number in the report files without running anything.
  • It labels every result “simulated (practice app)”. It says that a real model would need its own recorded runs.

Common mistake

A claimed fix accepted on its own case

Symptom: the runner shows FIXED? for a case, and the sign-off says “fixed”.

Cause: FIXED? means “this exact case passes now”. A change can make the case pass without removing the cause, as Verify the agent’s fix showed.

Recover: run neighbours that the fix must also handle. Run the guards that sit next to the changed rule. Read what changed in the trace. Then write “fixed”, “fixed in part” or “not fixed”, with the evidence.

Common mistake

A guard that fails on the baseline

Symptom: a new guard case fails on 2.3.0 as well as on the candidate.

Cause: it tests a bug that 2.3.0 already had, not a rule that worked.

Recover: a guard must pass on the baseline. Give the case a bug: and status: known-bug line and its own bug report. Or keep it as a note for a later release.

Make a ZIP file of workspace/final, without the hidden .helpdesk folders and without virtual environments. Its README.md comes first. Send the ZIP file to your reviewer, or put the folder in a repository that your reviewer can read. Use test data only: the practice app has no real customers, so your files must not have any either.

Recap

Your workspace now: everything from Module 7, plus final/ with your plan, notes, bug reports, cases, regression set, reports, sign-off and README.