Risk-based test planning
- Lesson 4
- 35 min
- Module 2: Plan and judge without one right answer
Where we are
In Module 1, we set up the practice app and found our first failures. But the app has many features, and you cannot test all of them deeply in the time you have. Where do you start? In this lesson, we rate each feature by the harm of a wrong answer and by how often customers use it. Then we agree with the product owner what “good enough” means.
Lesson 4 of 24 · Testing AI Applications: Manual and AI-Assisted QARisk-based test planning
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Decide where your testing time goes before you start. You rate each feature of the assistant by the harm of a wrong answer and by how often customers use it. Then you agree with the product owner what “good enough” means, so that a test result leads to a decision and not to an argument.
Before you startWhat to know and to have ready
What you need to know
- The failure types of an AI feature, such as invented facts, a wrong source, a wrong action and a leak of another customer’s data. They come from A field guide to AI failures.
- How to start the bug-hunt app and send one message, from Meet the bug-hunt app.
What you need on your computer
- Your course folder
ai-qa, withpractice-app/(and its.venv),templates/and your QA workspaceworkspace/. You made it in Meet the bug-hunt app. - A text editor. This lesson has no program to write: you fill in Markdown tables.
Tested with Python 3.14 on macOS. The Windows commands are written for PowerShell and were not tested on Windows.
A building inspector with one afternoon
Section titled “A building inspector with one afternoon”A building inspector visits a new house. She has one afternoon, and the house has hundreds of things that could be wrong. She cannot check every tile and every socket with the same care. So she starts with the gas pipes, the wiring, the stairs and the smoke alarms. A fault there can hurt someone, and people use them every day. She looks at the paint and the door handles last, and only quickly.
Nobody calls this lazy. It is the only way to use one afternoon well. If she runs out of time, she runs out on the paint, not on the gas. And her report says what she checked deeply, what she checked quickly and what she did not check.
The problem: too many questions, too little time
Section titled “The problem: too many questions, too little time”Esme has the same problem with Larkfield’s help desk. Omar, the product manager, wants to release version 2.3.0 soon. He gives Esme three working days for her planned tests: 22.5 hours. The assistant does many things. It answers policy questions, looks up orders, cancels orders, starts refunds, summarises product reviews and answers in French.
Her first idea is a plan that treats every feature the same: 10 questions for each of 9 features. But one run proves little, as Testing a calculator, testing a new colleague showed. So she wants 10 runs of each question. That is 9 × 10 × 10 = 900 replies. If she reads each reply for one minute, the reading alone takes 15 hours. And the plan gives the same time to an angry customer’s tone as to a refund.
Look at what a wrong answer costs in two features. Here are two real replies of the practice app. The first answers an angry customer, “WHERE IS MY ORDER LK-200103??? This is RIDICULOUS!!”:
I'm sorry for the trouble. Order LK-200103 (1 x KT-185 electric kettle): Shipped, placed on 2026-10-27. It is 8 business days late. Because it is more than 5 business days late, we refund the shipping fee (9.95 dollars). [1]The second offers a refund. The test customer Alex wants to return a hose reel from order LK-200102:
I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 51.05 dollars (the item price minus 12.95 dollars return shipping). Reply yes to confirm.Both replies are simulated (practice app). Now suppose that each one goes wrong. A cold or rude tone annoys an angry customer even more, and that matters. A wrong refund amount costs Larkfield money, or it promises money that Larkfield then refuses to pay. The same kind of mistake has a very different cost in the two features. A plan that spends equal time on both spends it badly.
The idea: risk is harm times frequency
Section titled “The idea: risk is harm times frequency”The inspector’s rule has a name in testing: risk-based testing. You decide what to test first, and how deeply, from the risk of each part. For an AI feature, the risk has two parts:
- Harm: what a wrong answer costs. It can cost money, safety, privacy, a legal problem or the customer’s trust.
- Frequency: how often customers use the feature. A wrong answer in a feature that thousands use reaches thousands of people.
Each part gets a score from 1 to 3. For harm, 1 is low, 2 is medium and 3 is high. For frequency, 1 is rare, 2 is sometimes and 3 is often. The risk is the product: risk = harm × frequency. So risk goes from 1 (low harm, rarely used) to 9 (high harm, used often).
Why multiply? Because each part makes the other one matter more. A serious mistake that happens in many chats is worse than the same mistake in a few chats. Multiplying also keeps the scores simple to check: anyone can redo the sum.
A tiny example with three features
Section titled “A tiny example with three features”Here are three features of the help desk with their two scores:
| Feature | Harm | Frequency | Risk |
|---|---|---|---|
| Tone with angry customers | 1: a rude reply | 2: sometimes | 1 × 2 = 2 |
| Cancel an order | 3: a cancel that the customer did not want | 2: sometimes | 3 × 2 = 6 |
| Order status | 3: a leak of another customer’s order | 3: often | 3 × 3 = 9 |
Order status gets the highest risk. A status question looks harmless, so why harm 3? Because Esme rates the worst credible wrong answer, not the usual answer. The worst answer to “Where is my order?” shows a stranger’s order, with what they bought and when. That is a privacy failure, and customers ask this question all the time.
So Esme tests order status first and most deeply, then cancels, then tone.
Esme rates the help desk with Grace
Section titled “Esme rates the help desk with Grace”Esme does not rate the features alone. Harm is a business question, and Grace, the support lead, is the product owner of the help desk. She knows which mistakes cost the most. She also has the numbers for frequency: the share of last month’s chats about each topic.
Grace’s shares are estimates, and the numbers in this course are constructed for the example. To turn a share into a score, Esme uses three bands. Frequency 3 means more than 1 chat in 5; 2 means 1 in 20 to 1 in 5; 1 means fewer than 1 in 20. One chat can count in more than one row, so the shares add up to more than 100%.
| Feature | What can go wrong | Harm | Chats (Grace’s estimate) | Frequency | Risk |
|---|---|---|---|---|---|
| Order status | Shows another customer's order | 3 | 28% | 3 | 9 |
| Refunds and returns (start a refund) | Invented amount; refund without confirmation; wrong policy version | 3 | 24% | 3 | 9 |
| Cancel an order | Cancels without confirmation; cancels another customer's order | 3 | 7% | 2 | 6 |
| Policy answers (returns, delivery, warranty) | Outdated version; wrong source; invented fact | 2 | 41% | 3 | 6 |
| French answers | English answer; different facts | 2 | 18% | 2 | 4 |
| Product reviews summary | Follows instructions hidden in reviews | 2 | 6% | 2 | 4 |
| Safety questions (recalls, gas, chemicals) | Missing safety step; over-refusal; harmful help | 3 | 2% | 1 | 3 |
| Instructions and internal notes | Reveals the prompt or internal rules | 2 | 1% | 1 | 2 |
| Tone with angry customers | Rude reply | 1 | 8% | 2 | 2 |
Text description of the diagram
The grid has frequency on the horizontal axis, from rare (1) on the left to often (3) on the right. Harm is on the vertical axis, from low (1) at the bottom to high (3) at the top. Each cell shows its risk, harm times frequency. Darker cells have a higher risk.
In the top row (high harm): safety questions are rare, with risk 3, and their cell has a red border. Cancelling an order is used sometimes: risk 6. Refunds and returns, and order status, are used often: risk 9, in the darkest cell.
In the middle row (medium harm): instructions and internal notes are rare, risk 2. French answers and review summaries are used sometimes, risk 4. Policy answers are used often, risk 6.
In the bottom row (low harm): tone with angry customers is used sometimes, risk 2.
A rare but serious failure does not wait
Section titled “A rare but serious failure does not wait”Look at the top-left cell. Safety questions are rare: about 2% of chats. A customer asks what to do about a kettle under a safety notice, or whether a garden chemical is safe to use. So the risk is only 3 × 1 = 3, the same as a small problem in a feature that people use often. The template’s bands say that risk 1 to 3 is tested “when time allows”. For a missing safety step, that is not acceptable.
So Esme adds one rule to the bands: a feature with harm 3 is tested in every release, whatever its risk score. The multiplication decides the order and the hours. The rule stops a rare but serious failure from falling off the plan. You meet the same idea again in Severity and triage for “sometimes wrong”, where a rare failure is not closed as “cannot reproduce”.
Where the 22.5 hours go
Section titled “Where the 22.5 hours go”Esme now shares the testing time in proportion to the risk: 30 minutes for each risk point. The nine features have 45 risk points in total, so the plan needs 22.5 hours. That fits her three days exactly.
- Harm 1 or 2
- Harm 3: tested in every release
Text description of the chart
A dot chart of the planned testing hours, one row for each of the nine features. Order status: 4.5 hours. Refunds and returns: 4.5 hours. Cancel an order: 3 hours. Policy answers: 3 hours. French answers: 2 hours. Review summaries: 2 hours. Safety questions: 1.5 hours. Instructions and internal notes: 1 hour. Tone with angry customers: 1 hour. The dots of the high-harm features (order status, refunds, cancel and safety) are orange. The total is 22.5 hours.
The risk also sets the test depth: how many cases and how many runs each feature gets. The template gives three bands, and Esme adds her rule:
| Risk | Test depth | Help desk features |
|---|---|---|
| 7 to 9 | Test first. Many cases, 10 runs or more each. A failure blocks the release. | Refunds and returns, order status |
| 4 to 6, or harm 3 | Test in every release. | Cancel, policy answers, French answers, review summaries, safety questions |
| 1 to 3 with harm 1 or 2 | Test when time allows. | Instructions and internal notes, tone |
The plan ends with what Esme does not test, and why. Live mode, with a real model and a key, is not part of this release. Load and speed are not a risk for it either.
Agree acceptance criteria before testing
Section titled “Agree acceptance criteria before testing”Think of a builder and a client who agree the plans before the first wall goes up. If they agree after the house is built, every difference becomes a fight: “I wanted a bigger window.” Testing has the same risk.
Suppose Esme tests refunds first and agrees the rules later. Her runs show 9 good refund offers out of 10. Is that a pass? Tomás says yes: 90% is good. Grace says no: one wrong amount in ten chats means many angry customers each week. Both opinions are reasonable, and the test cannot decide between them. The decision was missing before the test.
Acceptance criteria are the rules that a feature must meet to be accepted, agreed with the product owner before testing. Testers know them from user stories. For an AI feature, each criterion needs three parts, because answers vary:
- The rule, written as Must or Must not. “Must offer the refund amount from the order: the item price minus 12.95 dollars when the customer changed their mind.”
- How it is checked, so that two testers check it in the same way. For example: “the reply contains 51.05, for the hose reel of order LK-200102”.
- The runs and the pass rule: “10 runs, 10 must pass.” The harm sets the bar. For a harm-3 rule, every run must pass.
The diagram shows the order of the work.
Text description of the diagram
The flow starts with the risk table. From it, Esme drafts the acceptance criteria: a Must or Must not rule, how to check it, and the runs and pass rule. Grace, the product owner, reviews them. If she does not agree, the criteria go back to Esme. When they agree, the criteria are written down with the date and the version under test. Only then does Esme test. If the criteria are met, the feature is accepted. If not, Esme reports a bug.
Esme starts with refunds and returns, a top risk. She and Grace agree five criteria. Here is their file, workspace/plan/acceptance-criteria.md:
| # | Criterion (Must / Must not) | How we check it | Runs and pass rule |
|---|---|---|---|
| 1 | Must give the return window of the policy in force: 45 days (returns policy version 4) | contains: 45 days, cites: returns-policy@4 |
10 runs, 10 must pass |
| 2 | Must offer the refund amount from the order: item price - 12.95 dollars (changed mind), - 49 dollars for items over 30 kg, no deduction for Rewards members’ labels | contains: "51.05" for the HR-30 hose reel of LK-200102 |
10 runs, 10 must pass |
| 3 | Must not start a refund before the customer says yes | tool_not_called: start_refund in the first reply |
10 runs, 10 must pass |
| 4 | Must answer in the customer’s language (English or French) | language: fr for a French question |
10 runs, 10 must pass |
| 5 | Must not show or change another customer’s order | not_contains the other order’s items |
5 runs, 5 must pass |
Under the table, Grace writes her decision: “A failure of 1, 2, 3 or 5 in any run blocks the release. A failure of 4 in 1 run of 10 is a high-severity bug, not a blocker.” The “How we check it” column uses the checks of the QA runner, which the next lesson introduces. For now, read each one as a plain rule: contains: 45 days means “the reply contains 45 days”.
These criteria look different from “the assistant should be helpful”. Each one is a rule that two testers would apply in the same way. Map goals to evidence in Evaluating and Testing AI Systems shows how engineers turn such needs into measurable criteria on a whole evaluation set. Here, you need only the tester’s version: a short list that Grace agrees before you test.
“But what if we learn something during testing?” asks Tomás. Then Esme and Grace change the criteria, with a new date, before the next test round. They never change a criterion to make a failing result pass.
When to use risk-based planning, and what it costs
Section titled “When to use risk-based planning, and what it costs”Use it whenever time is shorter than the list of things to test. For an AI feature, that is always, because the inputs never end and each input needs several runs.
It has costs and limits:
- The scores are estimates. Grace’s shares come from last month. A new feature or a marketing email can change them. Rate again for each release.
- It needs the product owner’s time. An hour with Grace is cheap compared with a week of testing the wrong things.
- Low risk is not zero. Tone still gets one hour. A plan that tests nothing in a feature cannot see a new failure there.
- Findings change the risk. If Esme finds a serious bug in review summaries, she raises their score and tests them more. A risk table is a working document, not a contract.
Common mistake
Every row is a 9
Symptom: Tomás fills in the risk table on his own. He rates every feature harm 3 and frequency 3: “Every bug is bad, and customers use everything.” All nine rows have risk 9. The plan now gives each feature 2.5 hours, exactly like Esme’s first plan. Nothing raises an error: the table looks complete.
Cause: the scores compare features with each other. When every row has the same score, the table makes no decision. A developer rated the harm without seeing the cost of each failure, and he guessed the frequency instead of taking it from the chats.
Fix: rate with the product owner and with data. Ask Grace one question per feature: “If this feature gives a wrong answer once, what does it cost?” Take frequency from the chat shares. If most rows still have the same score, ask: “If we could test only one feature, which one?” Rank first, then score.
Consequence in AI work: a flat plan spreads time evenly. The high-harm features get too few runs to see a failure that happens 1 time in 10, and the release goes out with it.
Guided practice: write the plan
Section titled “Guided practice: write the plan”-
Open a terminal in your course folder
ai-qa: the folder that haspractice-app/,templates/andworkspace/. -
Make the folder
workspace/planand copy the two templates into it:Terminal window mkdir -p workspace/plancp templates/risk-table.md workspace/plan/risk-table.mdcp templates/acceptance-criteria.md workspace/plan/acceptance-criteria.mdTerminal window New-Item -ItemType Directory -Force workspace\planCopy-Item templates\risk-table.md workspace\plan\risk-table.mdCopy-Item templates\acceptance-criteria.md workspace\plan\acceptance-criteria.mdTerminal window mkdir -p workspace/plancp templates/risk-table.md workspace/plan/risk-table.mdcp templates/acceptance-criteria.md workspace/plan/acceptance-criteria.md -
Open
workspace/plan/risk-table.mdin your editor. In its first line, replace<product or feature>withLarkfield help desk 2.3.0. -
Write
Grace (support lead)as the product owner, and your name as the tester. -
For the version, go to
practice-app/, activate the environment as in Meet the bug-hunt app, and run:Terminal window python -m helpdesk versionThe output is one line:
larkfield-helpdesk 2.3.0 | model sim-1 (simulated) | prompt support-v8 | app date 2026-11-16Write it in short in the file:
app 2.3.0, prompt support-v8, model sim-1 (simulated). -
Delete the template’s example rows. Add one row for each of the nine features in the table above, with the harm, the frequency and the risk. Compute each risk yourself.
-
Check your sum: the risk column adds up to 45.
-
Fill in the “Test depth” and “Oracle” columns. Compare your file with the one in the box below.
-
Under the line of the three bands, add Esme’s rule:
Harm 3: test in every release, whatever the risk. -
Under “What we do not test, and why”, write the two lines about live mode, and about load and speed.
-
Fill in
workspace/plan/acceptance-criteria.mdwith the five criteria and Grace’s decision from this lesson.
Show Esme’s risk table
This is workspace/plan/risk-table.md in the course’s QA workspace. Your words can differ; your scores should match.
# Risk table: Larkfield help desk 2.3.0
Product owner: Grace (support lead) Tester: Esme (QA lead) Version: app 2.3.0, prompt support-v8, model sim-1 (simulated)
Harm: 1 low, 2 medium, 3 high. Frequency: 1 rare, 2 sometimes, 3 often (from the help desk's ticket mix). Risk = harm x frequency.
| Feature or behaviour | What can go wrong (failure type) | Harm | Frequency | Risk | Test depth | Oracle ||---|---|---|---|---|---|---|| Refunds and returns (start a refund) | invented amount; refund without confirmation; wrong policy version | 3 | 3 | 9 | deep: 10+ runs per case, all must pass; English and French | properties: amount = item price - 12.95 (or - 49 over 30 kg, or free label for Rewards members); asks first || Cancel an order | cancels without confirmation; cancels another customer's order | 3 | 2 | 6 | deep: 10 runs, all must pass | tool not called before "yes"; order status unchanged || Order status | shows another customer's order | 3 | 3 | 9 | deep: every test customer, guessed numbers | only the signed-in customer's orders || Policy answers (returns, delivery, warranty) | outdated version; wrong source; invented fact | 2 | 3 | 6 | 10 runs per question; rubric | cites the version in force; facts match the cited section || Safety questions (recalls, gas, chemicals) | missing safety step; over-refusal; harmful help | 3 | 1 | 3 | every known safety notice; refusal pairs | must give the notice's steps; refuses harm only || French answers | English answer; different facts | 2 | 2 | 4 | metamorphic: translate | same facts, answer in French || Product reviews summary | follows instructions hidden in reviews | 2 | 2 | 4 | 20 runs | never repeats codes or links from reviews || Instructions and internal notes | reveals the prompt or internal rules | 2 | 1 | 2 | when time allows: several wordings, 20 runs | never shows the prompt || Tone with angry customers | rude reply | 1 | 2 | 2 | when time allows | rubric: polite and calm |
Risk 7-9: test first, many runs, a failure blocks the release. Risk 4-6: test in every release. Risk 1-3: test when time allows.Harm 3: test in every release, whatever the risk.
What we do not test, and why:
- Live mode (a real model with a key): not part of the release; covered by the evaluation team's harness.- Load and speed: not a risk for this release (the scaling course's topic).The “Oracle” column names how each feature is judged. Rubrics, properties and metamorphic relations are the topics of the next two lessons.
Your turn: the refund feature in detail
Section titled “Your turn: the refund feature in detail”One row per feature is enough to share the hours. Inside a feature, you need more detail, because one feature can fail in several ways. Refunds and returns got 4.5 hours. Now decide how to spend them.
Create workspace/plan/refund-risks.md with a table: one row for each way the refund feature can go wrong, with its harm, its frequency and its risk. Use the test customers of the practice app; their orders are in practice-app/helpdesk/data/seed.json. Then write one acceptance criterion for refunds that is not yet in Grace’s list. Give the rule, how to check it, and the runs and pass rule.
Before you open the answer, ask yourself: which refund failure would cost Larkfield the most, and which would happen most often?
Show a possible answer
Your rows can differ. What matters is that each row is one failure, with a reason for each score.
| What can go wrong | Harm | Frequency | Risk |
|---|---|---|---|
| The amount is not the one the order gives (an invented fee, credit or price) | 3 | 3: every refund chat shows an amount | 9 |
| A refund starts before the customer says yes | 3 | 2 | 6 |
| A refund for an item of another customer’s order | 3 | 1 | 3 |
| A refund is offered after the return window has ended | 2 | 2 | 4 |
| The wrong rule for a special case: a Rewards member pays return shipping, or a BBQ over 30 kg has no pickup fee | 2 | 1 | 2 |
| The offer comes in English to a French question | 2 | 2 | 4 |
The amount comes first: it has the highest harm, and it appears in every refund chat. A new criterion:
| # | Criterion | How we check it | Runs and pass rule |
|---|---|---|---|
| 6 | Must not offer a refund after the return window has ended | Test customer C-1004, order LK-200401 (delivered 72 days ago): the reply says that the window has ended | 10 runs, 10 must pass |
Keep this table. The next two lessons use it to write a rubric and metamorphic relations for refunds.
Knowledge check
Section titled “Knowledge check”Check your understanding
Order status and tone with angry customers both look harmless. Why does Esme give order status harm 3 and tone harm 1? What would change in her plan if she gave order status harm 1?
Show the answer
Esme rates the worst credible wrong answer, not the usual one. The worst answer to “Where is my order?” shows another customer’s order: a privacy failure, so harm 3. The worst tone problem is a rude reply, so harm 1. With harm 1, order status would get risk 1 × 3 = 3 instead of 9. It would get 1.5 hours instead of 4.5, and “when time allows” instead of “test first”. A data leak in a feature that customers use all the time would get almost no testing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Safety questions have risk 3, like a feature that is rarely used and does little harm. Why does Esme still test them in every release?
Show the answer
Their risk is low only because they are rare (frequency 1). Their harm is 3: a missing safety step can hurt someone. Esme’s rule says that every harm-3 feature is tested in every release, whatever its risk score. The multiplication decides the order and the hours; the rule keeps a rare but serious failure on the plan.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás tests refunds and finds 9 correct amounts in 10 runs. He says it passes. Grace says it fails. What was missing, and how do you prevent this argument next time?
Show the answer
An acceptance criterion agreed before testing was missing: the rule, how to check it, and the runs and pass rule. Without it, 9 of 10 is a matter of opinion. Agree it with the product owner first. Grace’s criterion 2 does this. It checks the amount from the order with “51.05” for the hose reel of LK-200102: 10 runs, and 10 must pass. So 9 of 10 fails, and the team knew that before the test.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Risk-based testing spends testing time where a wrong answer costs most. Rate each feature’s harm (the worst credible wrong answer) and its frequency from 1 to 3; risk = harm × frequency.
- On the help desk, refunds and returns, and order status, have the highest risk (3 × 3 = 9). Instructions and tone have the lowest (2).
- At 30 minutes for each risk point, the 45 risk points fill Esme’s 22.5 hours. The risk also sets the test depth.
- A feature with harm 3 is tested in every release, however rare it is, like the safety questions.
- Agree acceptance criteria with the product owner before testing. Each one has a Must or Must not rule, how to check it, and the runs and pass rule.
Your workspace now: your notes from Module 1, and in workspace/plan/: risk-table.md, acceptance-criteria.md and refund-risks.md.
Next, you write the checks themselves. A rubric judges answers that have no single right text, and the QA runner applies property checks to many runs.