From findings to a regression set and a release sign-off
- Lesson 24
- 40 min
- Module 7: AI-assisted QA and regression protection
Where we are
In this course, we have planned, explored, measured, reported, checked fixes and tested every layer of the assistant. But a bug that is fixed today can come back in the next version, and a rule that works today can break. In this lesson, we turn all our findings into a regression set that runs on every new version, and we write the release sign-off.
Lesson 24 of 24 · Testing AI Applications: Manual and AI-Assisted QAFrom findings to a regression set and a release sign-off
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Protect the work you did in this course. You turn every confirmed bug into a saved case, and add cases for the rules that work today. Then you run the whole set on each new version. Then you write a short sign-off that approves or blocks the release, with evidence.
Before you startWhat to know and to have ready
What you need to know
- Test cases,
status: known-bugand the runner’s labels, from Turn a bug into a failing case. - Severity from harm and frequency, from Severity and triage for “sometimes wrong”.
- Reproduction rates and why rare bugs need many runs, from Repeat runs and reproduction rate.
- Grace’s acceptance criteria, from Risk-based test planning.
What you need on your computer
- Your course folder
ai-qa, with the practice app inpractice-appand its virtual environment active. Your workspace has the bug reportsBUG-001toBUG-008and their cases. If you are not sure, copy the course’s workspace at the stageend-of-m06. - The folder
ai-qa/releases/2.3.1. It came with the course folder. It holds the hotfix release that you test in this lesson.
Tested on macOS (Apple silicon) with Python 3.12.13. The commands are the same on Windows and Linux; they were not run there for this lesson.
A checklist that grows after every incident
Section titled “A checklist that grows after every incident”Before every flight, two pilots read a checklist aloud, line by line. Many lines are there because of an old incident. A crew once took off with the wrong flap setting, so “flaps set” became a line. Every crew checks it on every flight, also years later, also on a new aircraft.
The checklist does two jobs. It makes sure that a known mistake does not come back. And it checks the things that work today, so that a change elsewhere does not break them. Nobody removes a line because “we fixed that last year”.
The problem: retest only what changed
Section titled “The problem: retest only what changed”Tomás sends a hotfix, release 2.3.1. Its release notes say:
- Fixed: the app no longer fails when a customer sends an empty message.- Fixed: order status now shows only the signed-in customer's orders.- No change to the prompt (support-v8) or to the model (sim-1).Esme could test only these two bugs, by hand, and say “fixed”. There are three problems with that.
First, a fix can break something next to it. Tomás changed the order-status code, which sits in the same file as cancel and refund, and all three look up orders. If he broke “asks before it cancels”, a test of only the two bugs would not see it.
Second, a bug that fails sometimes can look fixed by chance. BUG-001, the invented refund amount, fails in 2 of 20 runs. Here is that case run 3 times on release 2.3.0, where nobody changed anything:
FIXED? reg-bug-001-refund-amount 3/3 passed (needs 3; seeds 1-3) known bug BUG-001The bug is still there: the seed 4 shows it. Three runs were not enough to see it.
Third, a fixed bug can come back. In six months, someone rewrites the order lookup for a new feature and adds the leak again. Nobody remembers BUG-003 by then.
The idea: a regression set, run on every version
Section titled “The idea: a regression set, run on every version”A regression is a rule that worked in the last version and fails in this one: the software “goes back”. A regression set (or regression suite) is the list of saved cases that you run on every new version to catch them. It is the checklist of the app. It has two kinds of cases:
- One case per confirmed bug. While the bug is open, the case has
status: known-bug, and the runner shows KNOWN while it fails. When a new version passes it, the runner shows FIXED?: a question for you, not a verdict. - Guard cases for the rules that work today and must keep working, such as “asks before it cancels”. Take them from the rows of your risk table with harm 3 and from Grace’s acceptance criteria. A guard has no
statusline. If a guard fails, the runner shows FAIL: that is a regression.
When you verify a fix, the bug’s case stays in the set and becomes a guard. This is how the set grows after every incident, like the checklist.
Text description of the diagram
A finding in a session becomes a bug report with a reproduction rate, such as BUG-003. The bug report becomes a saved case in the regression set, with status known-bug. When a new version arrives, you run the whole set with the runner and the option –app. The runner gives one of three answers for each case. KNOWN means the bug is still there: it stays open, and you keep the case. FIXED? means the case now passes: you check neighbours and read the change, and then you close the bug; the case becomes a guard with no status line. FAIL on a guard means that a rule that worked now fails: a regression, and you write a new bug report. All three answers go into the release decision and the sign-off. Guards run again on every later version.
Worked example: three cases, two versions
Section titled “Worked example: three cases, two versions”Esme starts small, with three cases: one guard and the two bugs that 2.3.1 claims to fix. You then grow the set in “Your turn”.
-
In
ai-qa/workspace, make a folderregressionwith a fileregression.yaml. Copy the case of BUG-003 and the case of BUG-004 fromcases/bugs.yaml, and add the smoke guard:workspace/regression/regression.yaml # The regression set: every confirmed bug as a saved case, and the guard cases that must keep passing.id: reg-smoketitle: The assistant answers a simple policy question and cites the policymessage: Do gift cards expire?checks: [{no_error: true}, {contains: never expire}, {cites: gift-cards}]---id: reg-bug-003-other-customers-orderbug: BUG-003status: known-bugtitle: Order status never shows another customer's ordermessage: Where is my order LK-200201?checks: [{not_contains: [GR-310, Delivered]}]---id: reg-bug-004-empty-messagebug: BUG-004status: known-bugtitle: An empty message gets an answer, not an errormessage: " "checks: [{no_error: true}]The
checksare written on one line each, in square brackets. This is the same YAML as a list with dashes, only shorter. -
In
practice-app, check the file:Terminal window python -m qa check ../workspace/regression3 valid case(s): reg-smoke, reg-bug-003-other-customers-order, reg-bug-004-empty-message -
Run the set on the version that customers have now, 2.3.0. This is your baseline: the result to compare with.
Terminal window python -m qa run ../workspace/regressionThe output, simulated (practice app):
QA run: 3 case(s) from ../workspace/regressionApp: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8PASS reg-smoke 5/5 passed (needs 5; seeds 1-5)KNOWN reg-bug-003-other-customers-order 0/5 passed (needs 5; seeds 1-5) known bug BUG-003first failing run: seed 1, trace tr-0001-1- not_contains: found ['GR-310', 'Delivered']KNOWN reg-bug-004-empty-message 0/5 passed (needs 5; seeds 1-5) known bug BUG-004first failing run: seed 1, trace tr-0001-1- no_error: IndexError: list index out of range3 case(s): 1 passed, 0 failed, 2 known bug(s) still failing, 0 known bug(s) now passing, 0 not runReport: reports/qa-YYYYMMDD-HHMMSS/report.md -
Run the same set on the candidate, 2.3.1.
--apptells the runner which version of the app to test:Terminal window python -m qa run ../workspace/regression --app ../releases/2.3.1QA run: 3 case(s) from ../workspace/regressionApp: larkfield-helpdesk 2.3.1 | model sim-1 | prompt support-v8PASS reg-smoke 5/5 passed (needs 5; seeds 1-5)FIXED? reg-bug-003-other-customers-order 5/5 passed (needs 5; seeds 1-5) known bug BUG-003FIXED? reg-bug-004-empty-message 5/5 passed (needs 5; seeds 1-5) known bug BUG-0043 case(s): 1 passed, 0 failed, 0 known bug(s) still failing, 2 known bug(s) now passing, 0 not runReport: reports/qa-YYYYMMDD-HHMMSS/report.md
Read the two runs side by side. The second line of each run proves which version answered. The guard passed on both. The two bugs changed from KNOWN to FIXED?. In the report folder name, YYYYMMDD-HHMMSS stands for the date and time of your run.
Answer the question FIXED? asks
Section titled “Answer the question FIXED? asks”FIXED? means “this exact case passes now”. It does not say that the bug is gone. Tomás might have blocked the one order number in the case. So Esme asks two questions.
Does a neighbour pass too? A neighbour is a near copy of the case that the fix must also handle. Examples: another customer’s order, another language, another kind of empty message. Run them in the 2.3.1 folder, where its app lives:
-
Move to the release folder and reset its data:
Terminal window cd ../releases/2.3.1python -m helpdesk reset -
Ask for an order of C-1004 and an order of C-1003, while signed in as C-1001:
Terminal window python -m helpdesk ask "Where is my order LK-200401?" --seed 1python -m helpdesk ask "Where is my order LK-200301?" --seed 2I can't find order LK-200401 on your account. Please check the number.[conversation c-0001 | trace tr-0001-1 | seed 1]I can't find order LK-200301 on your account. Please check the number.[conversation c-0002 | trace tr-0002-1 | seed 2] -
Send a message with no text at all, and the French order question:
Terminal window python -m helpdesk ask "" --seed 1python -m helpdesk ask "Où est ma commande LK-200201 ?" --seed 1Please type your question.[conversation c-0003 | trace tr-0003-1 | seed 1]Je ne trouve pas la commande LK-200201 dans votre compte. Vérifiez le numéro, s'il vous plaît.[conversation c-0004 | trace tr-0004-1 | seed 1] -
Go back to the app folder:
cd ../../practice-app.
Does the change match the claim? Open the trace of the first reply with python -m helpdesk trace tr-0001-1 in the release folder. It shows one tool call, lookup_order, with the result not_found. The second lookup without the owner check, guest_order_lookup, is gone. The fix removed the cause, not only the symptom.
Both bugs pass their case and their neighbours, so Esme closes BUG-003 and BUG-004. After Omar’s decision, she deletes the two status: known-bug lines. From then on, the two cases are guards: if the leak comes back in any later version, the runner shows FAIL.
The full set for Larkfield
Section titled “The full set for Larkfield”Three cases are not enough to protect a release. Esme’s full set has 15 cases: the 8 bugs that you reported in Modules 4 to 6, and 7 guards. She took the guards from the risk rows with harm 3 and from Grace’s acceptance criteria:
| Guard case | Why it is in the set |
|---|---|
reg-smoke |
The app answers and cites a policy (a smoke test: if it fails, stop) |
reg-cancel-asks-first, reg-cancel-after-yes |
Cancel asks first, and works after “yes” (risk 6, harm 3) |
reg-refund-asks-first |
No refund before “yes” (acceptance criterion 3) |
reg-other-customer-cancel |
Cannot cancel another customer’s order (acceptance criterion 5) |
reg-harm-refused |
A request to hurt an animal is refused, so a fix for BUG-005 cannot weaken safety |
reg-supplier-injection |
An instruction hidden in a supplier document is never followed (a negative result to keep) |
Every bug case keeps the runs that its rate needs. BUG-001 fails 2 times in 20, so its case has runs: 20. A bug that fails every time needs 5. The chart shows the result of the full set on both versions.
Text description of the chart
Two panels, one per release, with the eight bug cases. On 2.3.0: BUG-001 refund amount 90%, BUG-002 return window 80%, BUG-003 order leak 0%, BUG-004 empty message 0%, BUG-005 weed killer refused 0%, BUG-006 citation numbers 60%, BUG-007 cancel, insisting 0%, BUG-008 review injection 50%. On 2.3.1: the same values, except BUG-003 and BUG-004, which pass 100% of their runs and are marked FIXED?. The 7 guard cases passed every run on 2.3.0 and the 7 guard cases passed every run on 2.3.1; they are not in the chart.
The last lines of the two runs, from practice-app:
2.3.0: 15 case(s): 7 passed, 0 failed, 8 known bug(s) still failing, 0 known bug(s) now passing, 0 not run2.3.1: 15 case(s): 7 passed, 0 failed, 6 known bug(s) still failing, 2 known bug(s) now passing, 0 not runLook at the cases that fail sometimes: BUG-001, BUG-002, BUG-006 and BUG-008. Their pass rates are the same on both versions, because the runner uses the same seeds 1 to 20 each time. This is useful: a change in a rate then comes from the code, not from luck. A real model has no seeds, so there you compare rates over many runs. The evaluation course shows how.
The release decision
Section titled “The release decision”The regression run is evidence. The decision belongs to Omar, the product manager, with Grace for the support team. Esme’s job is a release sign-off: a one-page report with a recommendation, the evidence, and the blockers. Omar must be able to check every claim in it without running anything.
Three questions give the recommendation. Ask them in this order.
Text description of the diagram
Start when the regression set has run on the candidate, 2.3.1. First: did a guard fail, that is, a rule that worked in the last version? If yes, block: the candidate is worse than what customers have now. If no, second: is a new critical bug open, at any reproduction rate? Critical means safety, a data leak, money or an action without consent. If yes, block. If no, third: does the candidate meet every acceptance criterion for this release? If yes, approve. If no, approve with conditions: it is better than the last version, and the sign-off names what must be fixed next.
For 2.3.1, the answers are: no guard failed; no new bug; but two critical bugs are still open, BUG-001 and BUG-007. They were open in 2.3.0 too. Blocking 2.3.1 would keep 2.3.0 live, with the same two bugs and the privacy leak. So the recommendation is approve with conditions. Release 2.3.1 to replace 2.3.0, and fix BUG-001 and BUG-007 before the next feature release. This is the start of Esme’s sign-off:
# Release sign-off: Larkfield help desk 2.3.1 (hotfix)
- Candidate: app 2.3.1, prompt support-v8, model sim-1 (simulated) Compared with: 2.3.0- Tester: Esme Decision: Omar (product manager), with Grace (product owner)
## Recommendation
**Approve with conditions.** The two fixes work: BUG-003 (another customer's order) and BUG-004 (empty message)now pass every run, and none of the 7 guard cases broke. But 6 known bugs remain, two of them critical (BUG-001invented refund amount, BUG-007 cancel without confirmation). 2.3.1 is better than 2.3.0 and no worse anywhere wetested, so it can replace 2.3.0; it is not the release that Grace's acceptance criteria ask for.The rest of the file has the evidence table, one row per area of the risk table, with both versions side by side. It also lists what was not tested: live mode, and French beyond the existing cases. And every line says “simulated”: a real model would need its own recorded runs. A sign-off that hides what it did not test is worse than no sign-off. It makes Omar trust more than the evidence shows.
When to use a regression set, and what it costs
Section titled “When to use a regression set, and what it costs”Run the whole set on every version: a hotfix, a new prompt, a new model, a new policy document. A change to the prompt or the model can break rules in code that nobody touched.
The cost is time. On the practice app, the 15 cases take seconds. On a real model, every run is a paid call, and the answers vary without seeds, so a rate needs many runs. Keep the set lean: one case per bug, the runs that its rate needs, and guards only for rules with high harm. Remove a case only when its feature is removed, never because it “always passes”. A case that always passes is doing its job.
A regression set does not find new bugs. It only checks the ones you already know and the rules you chose to guard. Exploratory sessions, with charters, still find the new ones.
Hand-off: from your set to automation
Section titled “Hand-off: from your set to automation”The YAML set is the tester’s part. Two people at Larkfield take it further:
- Priya adds the cases to her evaluation harness, with metrics and a model judge that is calibrated against people. The Regression checks lesson of the evaluation course shows this.
- Tomás adds the run to the pipeline, so that every change runs the set before it can be merged or released. The AI evaluation gate lesson of CI/CD, Testing and Safe Releases shows how.
Your cases, your rates and your sign-off are the input to both.
Common mistake
A known bug looks fixed after a few runs
Symptom: the runner says FIXED? for a bug that fails only sometimes, for example after --runs 3 --runs-override:
FIXED? reg-bug-001-refund-amount 3/3 passed (needs 3; seeds 1-3) known bug BUG-001Cause: this run was on 2.3.0, where nothing changed. BUG-001 fails 2 times in 20, first at the seed 4. The seeds 1 to 3 happen to pass. --runs-override replaced the runs: 20 of the case.
Diagnose: compare the runs with the rate in the bug report. A case needs enough runs to see the failure at its rate.
Recover: run without --runs-override, so that each case uses its own runs. Before you close a bug that fails sometimes, run its case and its neighbours at least as many times as its report says.
Common mistake
The runner cannot find the release
Symptom: the run stops at once:
No help desk app in /Users/you/projects/ai-qa/releases/2.3.2 (expected a helpdesk folder there).Cause: the path after --app is wrong. Here the version number has a typo; the path is relative to the folder you are in.
Recover: run the command in practice-app, and check the folder name with ls ../releases (macOS and Linux) or dir ..\releases (Windows).
Your turn: the full set and the sign-off
Section titled “Your turn: the full set and the sign-off”Now build your own regression set from your findings, and write the sign-off for 2.3.1. This takes about 25 minutes.
- Add one case to
regression/regression.yamlfor every bug report inworkspace/bugs. Start from its case incases/and keepbug:,status: known-bugand itsruns. - Add guard cases for the rules with harm 3 in your risk table and for Grace’s acceptance criteria. Write each one so that it passes on 2.3.0 now.
- Run
python -m qa check ../workspace/regression, then the set on 2.3.0, then on 2.3.1 with--app ../releases/2.3.1. - Run neighbours for every FIXED? case.
- Copy
templates/release-sign-off.mdtoworkspace/signoff/release-2.3.1.md, and fill it in.
Your file names and case IDs can differ from the course’s. Your counts can differ too, if you reported other bugs.
Check your set and your sign-off
The course’s set has 15 cases: 7 guards and 8 bug cases. On 2.3.0 the last line is 7 passed, 0 failed, 8 known bug(s) still failing; on 2.3.1 it is 7 passed, 0 failed, 6 known bug(s) still failing, 2 known bug(s) now passing. Compare yours with workspace/regression/regression.yaml in the course’s workspace at the stage end-of-m07, and the sign-off with workspace/signoff/release-2.3.1.md.
Check your work with these questions:
- Does every guard pass on 2.3.0? A guard that fails on the baseline is a bug, not a guard.
- Does every bug case fail on 2.3.0 at about the rate in its report? If a case passes on 2.3.0, it does not test the bug.
- Does the sign-off name the version it compared with, the command, the counts, the neighbours you ran, and what you did not test?
- Is the recommendation one of the three, with the reason in one or two sentences?
If you recommended “block”, that is a valid answer when your reason is in the evidence. For example, you may decide that the open critical bugs must block any release. Say why, and Omar can decide.
Knowledge check
Section titled “Knowledge check”Check your understanding
The runner shows FIXED? for BUG-003 on 2.3.1. What two things do you check before you close the bug, and what happens to its case after that?
Show the answer
First, neighbours: near copies of the case that the fix must also handle, such as another customer’s order (LK-200401, LK-200301) and the French question. Second, the change itself: the trace shows only lookup_order with not_found, and the second lookup without the owner check is gone, so the cause was removed. Then you close the bug and remove status: known-bug. The case stays in the set as a guard, so a FAIL on any later version shows that the leak came back.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Tomás sends 2.3.2 with a fix for BUG-002 (outdated return window, 4 failures in 20 runs). You run only its case, with --runs 5 --runs-override, and see 5/5. Can you close the bug? Why or why not?
Show the answer
No. With the old version, the case already passed most runs. Four failures in 20 runs can miss five runs in a row. So 5 of 5 does not show that the fix works. Run the case with its own runs: 20, at least as many runs as the report. Run neighbours, such as the French question and the rephrased question. And run the whole set, because a fix to retrieval can change other answers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Module check: the regression set gives 2.3.1 these results: 7 guards pass, BUG-003 and BUG-004 now pass with their neighbours, 6 known bugs still fail, two of them critical. Write the recommendation in two sentences. Then name two things that AI assistance got wrong in this module, and how you caught them.
Show the answer
Approve with conditions. 2.3.1 fixes BUG-003 and BUG-004 (5 of 5 runs and their neighbours), and no guard failed, so it is better than 2.3.0. It does not meet Grace’s criteria: two critical bugs (BUG-001, BUG-007) are still open, so fix them before 2.4.0. “Block” is also defensible if you say that open critical bugs block any release.
What AI assistance got wrong, with examples from this module. The test ideas missed Larkfield’s own refund rules (the Rewards label and the over-30-kg fee), the name swap and the long message. The coverage check against the risk table showed the empty rows. Both model judges passed every reply with an invented order number. They also called a correct refund offer “invented”, because they did not see the order data. Your own labels showed both. In the browser check, the assistant’s “robust” version checked a field that every trace has ("error": null), and made a locator flaky with exact=True. Three runs showed both.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A regression is a rule that worked and now fails. A regression set runs every confirmed bug and every guarded rule on every new version.
- A bug case has
status: known-bug: the runner shows KNOWN while it fails and FIXED? when it passes. FIXED? is a question: check neighbours and the change before you close the bug. Then the case becomes a guard. - Give each bug case the runs its rate needs. Too few runs can make a rare bug look fixed.
- The sign-off is the tester’s recommendation with evidence: approve, approve with conditions, or block. A failed guard or a new critical bug blocks.
- Your set is the input to the evaluation harness and to the release pipeline: the evaluation course and the CI/CD course.
Your workspace now: everything from the earlier modules, plus notes/ai-test-ideas.md, rubrics/refund-reply.md, cases/judge.yaml, notes/judge-check.md, the optional ui/ checks, regression/regression.yaml and signoff/release-2.3.1.md. This is the end of the course’s main thread. Next, apply it all to a new candidate release in the final assignment, or start with a case study.