Severity and triage for "sometimes wrong"
- Lesson 11
- 30 min
- Module 4: Report bugs people can act on
Where we are
In the last lesson, we captured evidence that reproduces each failure. Now we have several bugs, and some appear in only 1 run of 10. Is a rare bug serious? Can the team close it as “cannot reproduce”? In this lesson, we rate severity from harm and frequency together, and we agree a priority in triage.
Lesson 11 of 24 · Testing AI Applications: Manual and AI-Assisted QASeverity and triage for "sometimes wrong"
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Decide how serious a bug is when it appears only in some runs, and what the team does about it. You rate severity from harm and frequency together, so that a rare but serious failure is not closed as “cannot reproduce”. Then you agree a priority with the developer and the product owner.
Before you startWhat to know and to have ready
What you need to know
- The evidence kit and the exports from Capture the evidence.
- Harm and the risk table, from Risk-based test planning.
- How to run cases many times with the QA runner, and what a reproduction rate is, from Repeat runs and reproduction rate.
What you need on your computer
- The app in
ai-qa/practice-app, with the virtual environment active and PyYAML installed for the QA runner. - Your workspace
ai-qa/workspace, with the foldercasesand your exports inbugs/evidence.
Tested on macOS (Apple silicon) with Python 3.14.6. The Windows commands are the same as on macOS here, but they were not run.
A busy emergency room
Section titled “A busy emergency room”In an emergency room, a nurse sees each patient first. This is triage. The nurse does not treat patients in the order they arrive. A man with a cut finger came first, but a woman with chest pain goes before him. The chest pain may be nothing, and it may stop while she waits. The nurse still does not send her home because “the pain is gone now”. It came once, and the harm if it comes back is too high.
Two questions decide the order: how bad is it if we wait, and how sure are we that it is there? A rare symptom with a high harm still goes first.
The problem: a rare bug, closed after three tries
Section titled “The problem: a rare bug, closed after three tries”Esme sends BUG-001, the refund offer with the wrong amount, to Tomás. He tries it three times, with the seeds 1, 2 and 3:
for s in 1 2 3; do python -m helpdesk ask "I want to return the hose reel from order LK-200102" --seed $s; doneAll three replies are the same, simulated (practice app):
I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 51.05 dollars (the item price minus 12.95 dollars return shipping). Reply yes to confirm.51.05 dollars is correct. Tomás writes “cannot reproduce, 3 of 3 OK” and wants to close the bug. But Esme found the wrong amount with seed 4, and her export replays it. Who is right?
Both runs are real. The refund offer fails in only some runs, and Tomás’s three runs were all good ones. Esme measured the rate with the QA runner. She put one case per bug in workspace/cases/bugs.yaml (you write this file at the end of the lesson), and ran it:
python -m qa run ../workspace/cases/bugs.yamlQA run: 2 case(s) from ../workspace/cases/bugs.yamlApp: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8KNOWN bug-001-refund-amount 18/20 passed (needs 20; seeds 1-20) known bug BUG-001 first failing run: seed 4, trace tr-0001-1 - contains: missing ['51.05']KNOWN bug-002-return-window-version 16/20 passed (needs 20; seeds 1-20) known bug BUG-002 first failing run: seed 2, trace tr-0001-1 - contains: missing ['45 days'] - not_cites: cited ['returns-policy@3#return-window']
2 case(s): 0 passed, 0 failed, 2 known bug(s) still failing, 0 known bug(s) now passing, 0 not runReport: reports/qa-<date>-<time>/report.mdThe refund offer failed in 2 of 20 runs: about 1 run in 10. KNOWN means that the case belongs to a reported bug (status: known-bug) and still fails. With 100 runs (--runs 100 --runs-override), the counts are 11 of 100 for the refund and 30 of 100 for the return window.
How likely was Tomás’s result? If a bug appears in 1 run of 10, each run passes with a chance of 90%. Three runs in a row all pass with a chance of 0.9 × 0.9 × 0.9, which is 72.9%. So 73% of the time, three tries show nothing. The chart shows this chance for other rates.
Text description of the chart
A column chart with one column for each reproduction rate. 1 in 2: 13%. 3 in 10, the return window: 34%. 1 in 6: 58%. 1 in 10, the refund offer, highlighted: 73%. 1 in 50: 94%. The rarer the failure, the more likely three tries miss it.
For a bug that appears 1 time in 2, three clean tries would be surprising. For a bug that appears 1 time in 50, three clean tries tell you almost nothing: 94% of the time, you see no failure. “Cannot reproduce” after three tries is not evidence that the bug is gone. It is evidence that three tries are too few.
And the refund bug is about money. If the team closes it, some customers agree to 79.00 dollars and get 51.05 dollars. A rare failure with that harm must not fall off the list because it is rare.
The idea: severity from harm and frequency together
Section titled “The idea: severity from harm and frequency together”Severity is how bad a bug is: what it costs the customer and the business when it happens, and how often it happens. It is a fact about the bug, and the tester proposes it from the evidence.
Priority is how soon the team fixes it. It is a decision, and the team makes it in triage. It uses the severity, but also other things: the release date, the cost of the fix, a workaround.
Testers know both words. For AI features, severity needs one change: a bug has a rate, not a yes or no. So severity combines two ratings.
- Harm: the cost of the worst credible wrong answer, as in the risk table. High: money, privacy, safety, or an action without the customer’s consent. Medium: wrong information that misleads. Low: style, tone, an error message, or no answer.
- Frequency: how often the failure appears when a customer sends the trigger. This is the reproduction rate, measured with enough runs. Rare: fewer than 1 run in 10. Sometimes: 1 in 10 to 1 in 2. Often: more than half the runs.
The severity matrix puts the two together. The picture shows it, with four findings placed on it.
Text description of the diagram
Frequency goes across the top. Rare is fewer than 1 run in 10, sometimes is 1 in 10 to 1 in 2, and often is more than half the runs. Harm goes down the side. High is money, privacy, safety or an action without consent. Medium is wrong information that misleads. Low is style, tone, an error message or no answer.
The top row is critical in all three cells, with the note “rare is still critical”. The middle row is medium, high, high. The bottom row is low, low, medium.
Four findings are placed on the grid. The refund offer with a wrong amount, 2 of 20 and 11 of 100 runs, simulated: high harm, sometimes, so critical. The return window with the expired policy, 4 of 20 and 30 of 100 runs, simulated: medium harm, sometimes, so high. The Nunavut answer that starts with “No”, 2 of 10 runs, recorded from a real model: medium harm, sometimes, so high. The warranty answer that sends the customer to “the shop where you bought it”, 1 of 10 runs, recorded: low harm, sometimes, so low.
Read the top row first. With high harm, every frequency is critical. This is the rule that protects the refund bug: rare does not lower the severity of a bug that costs money or leaks data. In the other rows, frequency matters. A misleading answer in 3 runs of 10 is worse than the same answer in 1 run of 50.
Why “the worst credible wrong answer”? Because the rate tells you how often, and harm must tell you how bad. If you rate the harm of the average reply, a bug that is usually fine always looks harmless.
The matrix works the same for a real model. The two dashed boxes come from the course’s recorded runs of a real model (Azure, the support assistant’s prompt, 10 runs per question). Asked “Do you deliver to Nunavut?”, the real model started 2 of 10 replies with “No. Large items aren’t delivered to Nunavut”. Larkfield does deliver other items there, so a customer may give up. Asked about the hose reel’s warranty, 1 of 10 replies said “contact the shop where you bought it”. That shop is Larkfield. The reply has no wrong fact, so the harm is low.
A tiny worked example
Section titled “A tiny worked example”Esme rates her two bugs, with every value written down:
| Bug | Harm, and why | Runs | Frequency | Severity |
|---|---|---|---|---|
| BUG-001 refund offer | high: money; the customer agrees to 79.00 dollars and gets 51.05 | 2 of 20 (11 of 100) | sometimes (about 1 in 10) | critical |
| BUG-002 return window | medium: a customer may give up a valid return | 4 of 20 (30 of 100) | sometimes (about 3 in 10) | high |
BUG-001 fails less often than BUG-002, but its severity is higher, because its harm is higher. Sort a backlog by frequency alone, and the refund bug comes last.
How many runs before “cannot reproduce”?
Section titled “How many runs before “cannot reproduce”?”Sometimes a bug really is gone, for example after a fix. How many clean runs does it take before you can say so? It depends on the rate you reported. The Repeat runs and reproduction rate lesson showed how to find the number. This table gives the runs you need to see the failure at least once, with a chance of 90%, if it is still there:
| Reported rate | Runs for a 90% chance to see it |
|---|---|
| 1 in 2 | 4 |
| 3 in 10 | 7 |
| 1 in 6 | 13 |
| 1 in 10 | 22 |
| 1 in 50 | 114 |
So Larkfield’s team agrees one rule. Close a bug as “cannot reproduce” only after the runs in this table, with the same versions and settings, and no failure. For BUG-001, that is 22 runs, not 3. Before that, its status stays open, with the note “more runs needed”.
What does it cost? In the practice app, 100 runs take less than a second. With a real model, each run is a paid call and takes a second or two. The course’s 300 recorded calls cost about 4 US cents on the model used. Prices change: check your provider’s current page. For a bug that costs money or leaks data, the runs are cheap compared with the harm.
Triage with the developer and the product owner
Section titled “Triage with the developer and the product owner”Severity is the tester’s proposal. Priority is a team decision, made in a short triage meeting. At Larkfield, three people come:
- Esme (QA) brings the evidence: the kit, the runs, and the proposed severity.
- Tomás (AI engineer) brings the likely cause, the cost of a fix, and the risk that the fix breaks something else.
- Grace (support lead, the product owner) brings the business impact. How many customers send this trigger? What does a wrong answer cost? Is there a workaround?
The picture shows the path of a finding through triage.
Text description of the diagram
A finding starts with x of y runs and its evidence. The first question is: are there enough runs to know the rate, 20 to 100? If not, the finding goes back for more runs, with the same versions and settings. It is never closed as “cannot reproduce” at this point.
With enough runs, Esme rates the harm of the worst credible wrong answer and reads the severity from the matrix. Then the triage meeting: Esme brings the evidence, Tomás the cause and the cost, Grace the business impact. The meeting ends with one of four outcomes. The team fixes the bug before the release, or in a later release. Or it accepts the risk, with a written reason and an owner. Or it closes the item as not a bug, with the reason.
Larkfield uses four priorities. P0: stop, block the release now. P1: fix before the release. P2: fix in the next release. P3: fix when time allows.
In the meeting, Tomás asks: “BUG-001 is 1 in 10. Can it wait?” Grace answers: “Rare is not OK for money. A customer who agrees to 79 dollars and gets 51 calls us angry, and that call costs more than the fix.” The bug is critical, P1, fix before the release. BUG-002 is high: customers ask about returns every day, and Grace wants it fixed before the release too, P1.
Esme also records the item that Grace closed at the session debrief in Test charters and sessions: an item name in English inside a French reply. The product catalogue is in English for now, so it is not a bug. A closed item always has a reason, written where the team can find it.
When to use the matrix, and what it costs
Section titled “When to use the matrix, and what it costs”Use the matrix for every finding, so that two testers rate the same bug the same way. It takes a minute per bug, once you have the rate. It does not replace the meeting. The matrix proposes the severity; people decide the priority, and sometimes they disagree with the matrix for a good reason. Write that reason down.
Do not rate severity from one or two runs. Measure first: a rate from 3 runs can move a bug from “often” to “rare”, as the failure case below shows. And do not spend the meeting on labels. If Esme and Tomás argue for ten minutes between “high” and “critical”, the bug needs more evidence, not more talk.
Common mistake
Five clean runs, and the runner says FIXED?
Symptom: you re-run the bug cases with only 5 runs, starting at seed 3. Nothing in the app changed (simulated, practice app):
python -m qa run ../workspace/cases/bugs.yaml --runs 5 --runs-override --seed-start 3KNOWN bug-001-refund-amount 4/5 passed (needs 5; seeds 3-7) known bug BUG-001 first failing run: seed 4, trace tr-0001-1 - contains: missing ['51.05']FIXED? bug-002-return-window-version 5/5 passed (needs 5; seeds 3-7) known bug BUG-002The return-window bug looks fixed. With seeds 1 to 5, the same case had failed once.
Cause: 5 runs are too few for a bug that fails about 3 times in 10. Seeds 3 to 7 happen to be good ones. The question mark in FIXED? is the runner’s warning: a known bug that passes needs checking.
Diagnose: compare the number of runs with the table above. For 3 in 10, you need 7 runs for a 90% chance to see the failure.
Recover: run the case with its own number of runs (20), or with 100. Never close a bug, or lower its severity, on a run that is shorter than the table says.
Common mistake
The backlog is sorted by how often
Symptom: the team fixes the bugs that fail most often first. The refund bug, 1 in 10, waits at the bottom. Meanwhile, it reaches customers.
Cause: frequency was used as severity. Harm was not rated at all.
Recover: rate the harm of each bug first, with the product owner, and read the severity from the matrix. Any bug with high harm goes to the top row, however rare.
Your turn: measure and rate your bugs
Section titled “Your turn: measure and rate your bugs”Measure the rates of BUG-001 and BUG-002, then write the triage notes. This takes about 15 minutes.
-
In
workspace/cases, make a filebugs.yamlwith this content.bug:links each case to a report, andstatus: known-bugtells the runner that the bug is reported and not fixed yet:workspace/cases/bugs.yaml # One case per reported bug (Module 4). status: known-bug -> the runner shows KNOWN while it fails, FIXED? when it passes.id: bug-001-refund-amounttitle: The refund offer gives the amount from the orderbug: BUG-001status: known-bugmessage: I want to return the hose reel from order LK-200102checks:- contains: "51.05"runs: 20---id: bug-002-return-window-versiontitle: The return window comes from the policy in forcebug: BUG-002status: known-bugmessage: How many days do I have to return an unused item?checks:- contains: 45 days- not_cites: returns-policy@3runs: 20 -
Check the file:
Terminal window python -m qa check ../workspace/cases/bugs.yaml2 valid case(s): bug-001-refund-amount, bug-002-return-window-version -
Run it. Your output must match the run in “The problem” above:
Terminal window python -m qa run ../workspace/cases/bugs.yaml -
In
workspace/notes, make a filetriage.md. Copy the severity matrix into it as a table. -
Add a table with one row for BUG-001, one for BUG-002, and one for a finding from your first bug hunt. Use these columns: Bug, Harm, Frequency (runs), Severity, Proposed priority, Question for the product owner.
-
If your hunt finding has fewer than 20 runs, add a case for it to a scratch file and run it 20 times first.
Check your triage notes
Your rows for the two bugs should say:
| Bug | Harm | Frequency (runs) | Severity | Proposed priority |
|---|---|---|---|---|
| BUG-001 invented refund amount | high (money) | 2 of 20 | critical | P1 |
| BUG-002 expired returns policy | medium | 4 of 20 | high | P1 |
For your own finding, check three things. Is the harm the worst credible wrong answer, not the usual one? Is the frequency based on at least 20 runs, with the seeds written down? And does your severity come from the matrix, or do you have a written reason for a different one?
A good question for Grace names a business fact that you do not know. For example: “How many customers ask about returns of garden chemicals in spring?” That answer changes the priority, not the severity.
The workspace snapshot for the end of this module has Esme’s full triage notes, with all five bugs of the module, in workspace/notes/triage.md.
Knowledge check
Section titled “Knowledge check”Check your understanding
A bug appears in 1 run of 10. Tomás runs it 3 times and sees no failure. Predict how likely that was, and how many runs he needs for a 90% chance to see it.
Show the answer
Each run passes with a chance of 90%. Three passing runs have a chance of 0.9 × 0.9 × 0.9, about 73%. Three clean runs are the usual result, not a surprise. For a 90% chance to see the failure at least once, he needs 22 runs. Until then, the bug stays open with “more runs needed”, not “cannot reproduce”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In 1 of 40 runs, the assistant shows a test customer the delivery address of another customer. What is the severity, and why? What would you tell a developer who says 'it is very rare'?
Show the answer
Critical. Showing another customer’s address is a privacy failure: high harm. With high harm, the matrix says critical for every frequency, so 1 in 40 does not lower it. Rare means that fewer customers are hurt, not that the hurt is smaller. At Larkfield’s volume of chats, 1 in 40 still reaches real customers, and one leak can be a legal problem.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The weed killer question is refused in 20 of 20 runs. The harm is low (no answer, the customer can call). What is its severity? Suppose Grace gives it priority P1 in spring. Is that a mistake?
Show the answer
Low harm and often (20 of 20) give medium severity. P1 is not a mistake. Severity is a fact about the bug; priority is a team decision that also uses other facts. In spring, many customers ask about garden chemicals, so the refusal reaches many people and fills the phone line. Grace knows that; the matrix does not. The team writes the reason next to the priority.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- “Cannot reproduce” after a few tries is weak evidence. A 1-in-10 bug passes 3 tries 73% of the time.
- Severity is how bad a bug is; priority is how soon the team fixes it. The tester proposes severity; triage decides priority.
- Severity combines harm (the worst credible wrong answer) and frequency (the reproduction rate, from enough runs). With high harm, every frequency is critical.
- Close a bug as “cannot reproduce” only after enough runs for its reported rate: 22 for 1 in 10.
- Triage brings the evidence (tester), the cause and cost (developer) and the business impact (product owner) together. Every decision, including “not a bug”, has a written reason.
Your workspace now: workspace/cases/bugs.yaml with the cases of BUG-001 and BUG-002, and workspace/notes/triage.md with the matrix and your first rows. Next, you write the bug reports themselves, with a template.