Skip to content
Français

A bug report template for AI behaviour

  • Lesson 12
  • 35 min
  • Module 4: Report bugs people can act on

Where we are

In the last two lessons, we collected evidence and rated how serious each bug is. But evidence in your notes helps nobody until it is in a report that a developer can follow alone. In this lesson, we write that report with the course's template, and we test it on a reader who was not there.

What you will be able to do

Write a bug report that a developer who was not in your session can act on alone. You use the course’s bug report template. It has a summary, the steps, the actual result, the expected result as checkable rules, the reproduction rate and a suspected layer. And you keep secrets and personal data out of it.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa: the app in practice-app, the templates in templates, your work in workspace.
  • A terminal in ai-qa/practice-app, with the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.14.6. The Windows commands were written, not run.

A friend asks for your cake recipe. You write: “Some flour, some sugar, three eggs. Bake until done.” You make this cake every week, so it is enough for you. Your friend’s cake comes out flat. Which flour? How much? How hot is the oven? What does “done” look like?

A good recipe works for a person who never saw you cook. It says “250 g of flour” and “180 °C for 25 minutes”. It also says how to check the result: “the top is golden, and a knife comes out clean”. Your friend can follow it alone, and knows without asking whether the cake is right.

A bug report is a recipe for a failure. The developer must make the same failure happen, and know when it is fixed.

The problem: a report only its writer understands

Section titled “The problem: a report only its writer understands”

Look again at BL-104 from the backlog: “Sometimes the bot says 30 days for returns. I think it should be 45 now. Not sure.” For its writer, a support team lead, it was clear. She saw the chats. For Tomás, it is a flat cake. He cannot run it: there are no steps. He cannot check a fix: “should be 45 now, not sure” is not a rule. And he cannot judge how urgent it is: “sometimes” is not a rate.

The picture puts BL-104 next to the report that Esme wrote for the same bug. Each part of BL-104 becomes a part of the new report. Four parts of the new report have nothing in BL-104 at all.

Two columns. Left, BL-104 from the backlog, in four quotes: 'Return days wrong?', 'Sometimes', 'the bot says 30 days for returns', and 'I think it should be 45 now. Not sure.' Right, BUG-002, the same bug rewritten, in eight parts. Arrows: 'Return days wrong?' becomes the summary, the return window answer uses the expired policy, 4 of 20 runs. 'Sometimes' becomes the reproduction rate, 4 of 20 runs, seeds 2, 8, 11, 14, simulated. 'the bot says 30 days' becomes the actual result, the reply word for word, source version 3, trace tr-0001-1. 'should be 45 now, not sure' becomes the expected result: must say 45 days and cite version 4, must not say 30 days. A dashed red box, 'not in BL-104 at all: the developer cannot reproduce the bug without these four', points to the versions and settings (app 2.3.0, support-v8, sim-1, seed 2, C-1001), the steps (reset, then the exact question with --seed 2), the evidence (the export evidence/bug-002-seed2.json) and the suspected layer (retrieval, a guess: both versions were found).
The same bug, as the backlog reported it and as Esme rewrote it. Every vague phrase becomes something a developer can run or check.
Text description of the diagram

On the left, BL-104 has four phrases. On the right, BUG-002 has eight parts.

“Return days wrong?” becomes the summary: the return window answer uses the expired policy, in 4 of 20 runs. “Sometimes” becomes the reproduction rate: 4 of 20 runs, seeds 2, 8, 11 and 14, simulated. “The bot says 30 days for returns” becomes the actual result: the reply word for word, its source and the trace ID. “I think it should be 45 now. Not sure.” becomes the expected result. It must say 45 days and cite version 4. It must not say 30 days.

Four parts have nothing in BL-104, shown with a dashed red box. The versions and settings: app 2.3.0, prompt support-v8, model sim-1, seed 2, test customer C-1001. The steps: reset, then the exact question with –seed 2. The evidence: the export evidence/bug-002-seed2.json. And the suspected layer: retrieval, a guess, because the search found both versions. Without these four, a developer cannot reproduce the bug.

The support lead was not careless: nobody gave her a form. A template fixes that.

The idea: one template, one question per section

Section titled “The idea: one template, one question per section”

A bug report template is a fixed list of sections. Each section answers one question that the developer would otherwise ask you. You fill every section, every time, so nothing depends on memory.

The course’s template is in your course folder: ai-qa/templates/bug-report.md. It is free to copy and change, also at work (licence CC0). Its sections, and the question each one answers:

Section The developer’s question
Title (the summary) What goes wrong, where, and how often?
Version, test customer Which build answered, and as whom?
Severity, priority How bad is it, and how soon do we fix it?
Steps to reproduce How do I make it happen?
Actual result What exactly did you see?
Expected result How do I know when it is fixed?
Reproduction rate How often, from how many runs?
Evidence What can I replay?
Notes Where do you think it starts, and why?

Two sections are new compared with a report for ordinary software. The reproduction rate is there because an AI feature fails in some runs and not others. The suspected layer is there because the same wrong reply can start in the search, the prompt, the model or the tools.

Esme turns BL-104 and her evidence kit into a report. Write it with her.

  1. Copy the template into your workspace, with a name that says the bug’s ID and what goes wrong:

    Terminal window
    cp ../templates/bug-report.md ../workspace/bugs/BUG-002-outdated-returns-policy.md
  2. Open the file in your editor. The comments <!-- ... --> explain each part. Delete them when you are done.

Now fill it, section by section.

The title is the summary. Use the pattern: what goes wrong, where, how often. “Return days wrong?” names a topic. “The return window answer sometimes uses the expired returns policy (30 days, version 3), 4 of 20 runs” names the behaviour, the feature and the rate. In a list of 50 bugs, a developer knows this one from its title.

Version, test customer, severity and priority. Copy the versions and settings from your kit. Write the severity with its reason from the matrix: “high: medium harm (a customer may give up a valid return); sometimes (4 of 20 runs)”. Leave the priority as “set in triage” until the meeting decides it.

Steps to reproduce. Start with reset, so that the developer has the same data as you. Then paste the exact command, with the exact words and the seed. Never retype the question.

Actual result. Paste the reply word for word, or the part that matters, with its source. Then add the trace ID and what the trace shows: the fields that explain the failure, not all 60 lines.

Expected result, as checkable rules. This is the section that most reports get wrong. “It should give the correct return period” is not a rule: two people can disagree about “correct”. Write rules with Must and Must not that anyone checks the same way:

  • Must: say “45 days” and cite the returns policy, version 4.
  • Must not: say “30 days”, or cite version 3, which ended on 31 October 2026.

These are the same rules that you put in cases/bugs.yaml: contains: 45 days and not_cites: returns-policy@3. A rule that a person can check is also a rule that the runner can check.

Reproduction rate. Write x of y runs, the failing seeds and the label, simulated or recorded. Add the case ID, so that anyone can run it again.

Evidence. The export and the trace ID. Say that the replay gives the same reply.

Notes: the suspected layer. The trace often points to a part of the assistant, and that saves the developer time. Write it as a guess, with the reason: “Suspected layer (a guess): retrieval. The search returns the expired version 3 with the same score as version 4.” The layers are the parts of the assistant: retrieval (the search), the prompt, the model, the tools and permissions, and the app code. If your guess is wrong, the word “guess” and the reason let the developer see why, and move on. Retrieval or generation? shows how to prove the layer with evidence.

The picture shows the finished report, with the job of each part.

The report BUG-002, shortened, with a note beside each part. Title: 'BUG-002: The return window answer sometimes uses the expired returns policy (30 days, version 3), 4 of 20 runs'; note: summary, what goes wrong, where, and how often. Version: app 2.3.0, prompt support-v8, model sim-1 (simulated), app date 2026-11-16; test customer C-1001; note: versions, settings, test customer, which build and how to run it again. Severity: high, medium harm, sometimes; priority P1, set in triage; note: severity from harm and frequency, priority comes from triage. Steps: python -m helpdesk reset, then python -m helpdesk ask with the exact question and --seed 2; note: reset first, then the exact input and the seed, copy, do not retype. Actual result: 'within 30 days of delivery' [1], source [1] returns policy version 3, trace tr-0001-1 retrieved version 4 and version 3; note: word for word, with the trace ID and what the trace shows. Expected: must say 45 days and cite version 4; must not say 30 days or cite version 3; note: rules that two people check the same way. Reproduction rate: 4 of 20 runs, seeds 2, 8, 11, 14, simulated; note: x of y runs, which seeds, simulated or recorded. Evidence: export evidence/bug-002-seed2.json; note: the export to attach, replays the same reply. Notes: suspected layer, a guess, retrieval, because both versions were found with the same score; note: a labelled guess at the layer, with the reason.
The anatomy of a bug report for AI behaviour: BUG-002, shortened, with the job of each part.
Text description of the diagram

From top to bottom: the title is the summary, which says what goes wrong, where and how often. The version lines give the three versions, the app date and the test customer: which build answered, and how to run it again. Severity comes from harm and frequency; the priority comes from triage.

The steps start with reset, then give the exact input and the seed, copied and not retyped. The actual result is word for word, with the trace ID and what the trace shows. The expected result is a set of rules that two people check the same way.

The reproduction rate gives x of y runs, the failing seeds, and whether the runs were simulated or recorded. The evidence is the export to attach, which replays the same reply. The notes give a labelled guess at the layer, with the reason.

Here is the whole report. Your file should look like this.

workspace/bugs/BUG-002-outdated-returns-policy.md
# BUG-002: The return window answer sometimes uses the expired returns policy (30 days, version 3), 4 of 20 runs
From backlog report BL-104 ("Return days wrong?").
- **Version:** app 2.3.0, prompt support-v8, model sim-1 (simulated); app date 2026-11-16
- **Test customer:** C-1001
- **Severity:** high: medium harm (a customer may give up a valid return or miss 15 days of return rights); sometimes (4 of 20 runs)
- **Priority:** P1 (triage, see `notes/triage.md`)
## Steps to reproduce
1. `python -m helpdesk reset`
2. `python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2`
## Actual result
"Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps." Source [1]: Returns policy, version 3: Return window. (trace tr-0001-1)
The trace lists two retrieved sections with the same score, 7.926: returns policy version 4 (in force from 2026-11-01) and version 3 (effective to 2026-10-31). `cited` is `returns-policy@3#return-window`. No tool was called.
## Expected result
- Must: say "45 days" and cite the returns policy, version 4.
- Must not: say "30 days" or cite version 3 for a question asked on or after 1 November 2026.
## Reproduction rate
4 of 20 runs (seeds 2, 8, 11, 14), simulated (practice app); 30 of 100 with seeds 1-100. Case `cases/bugs.yaml` → `bug-002-return-window-version`. The same happens in French with seed 2.
## Evidence
- Conversation export: `evidence/bug-002-seed2.json` (replay after a reset: 1 of 1 same reply)
- Trace: tr-0001-1
## Notes
Suspected layer (a guess): retrieval. The search returns the expired version 3 with the same score as version 4, so the reply can use either. Workaround for support agents: quote the returns policy, version 4.

A bug report travels. It goes into a tracker that many people read, into chat channels, and sometimes to an outside company or a coding agent. Anything in it can leak. So a report holds test data only.

Here is a report as a new tester might write it from a real chat at work. It is constructed for this lesson: the person, the email, the order and the key are invented.

A constructed example: what not to write
Customer Maria Lopez (maria.lopez@example.com, order LK-90412) asked how long
her refund takes to reach her card. The bot did not give the time.
I ran it again in live mode with:
HELPDESK_LIVE_KEY=sk-live-7f3a... python -m helpdesk ask "..." --live

Two things must go. The customer’s name, email and order are personal data: the report does not need them, and she did not agree to share them. And the key is a secret: anyone who reads it can use the account and spend its money.

The safe version shows the same finding with a test customer, in the practice app:

The same finding, with test data
Test customer C-1001 asks "How long does a refund take to reach my card?"
The reply gives no time for a card: "Refunds to a gift card appear at once. [1]"
(simulated, practice app). Steps: python -m helpdesk reset, then
python -m helpdesk ask "How long does a refund take to reach my card?" --seed 1

Follow four rules:

  • Reproduce with a test customer. If a real chat showed the failure, find the same failure with the app’s test customers, and report that. Describe the real case by its shape, for example “an order number of another customer”, not by its values.
  • Never paste a key, password or token. Name the setting instead: “live mode, with HELPDESK_LIVE_KEY set”. The practice app never writes the key into a trace or an export, but your terminal history and screenshots can contain it.
  • Check your attachments. Open each export and screenshot before you attach it. An export from a real system can contain real messages.
  • If a secret leaked, say so at once. Do not only delete it from the report. Tell the owner, so that they replace the key.

The completion check: the cold-reader test

Section titled “The completion check: the cold-reader test”

How do you know that your report works? Give it to a cold reader: a person who was not in your session, with nothing but the report. The picture shows the test.

A flow. Your report, and nothing else, goes to a developer who was not in your session. They follow the steps: reset, then the exact input. Question: do they see the actual result? If no: find what was missing (a version, the seed, the customer, the exact words, the data state), add it to the report, and give it to them again. If yes, second question: can they say pass or fail from the expected result alone? If no: rewrite the expected result as Must and Must not rules, and try again. If yes: the report is ready for triage.
The cold-reader test. A report passes when a person who was not there sees the failure, and can say from the report alone whether a reply passes.
Text description of the diagram

Your report, and nothing else, goes to a developer who was not in your session. They follow the steps: reset, then the exact input. If they do not see the actual result, find what was missing: a version, the seed, the customer, the exact words, or the data state. Add it, and try again.

If they see the actual result, the second question is whether they can say pass or fail from the expected result alone. If not, rewrite the expected result as Must and Must not rules, and try again. When both answers are yes, the report is ready for triage.

If no colleague is free, be your own cold reader. Close your terminal and open a new one. Read only the report, not your notes. Run its steps exactly as written. Esme did this for BUG-002:

Terminal window
python -m helpdesk reset
python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2
The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.
Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.
Sources:
[1] Returns policy, version 3: Return window (/policies/returns-policy/v3#return-window)
[conversation c-0001 | trace tr-0001-1 | seed 2]

The reply breaks both rules of the expected result: it says 30 days, and it cites version 3. The report passes the test.

When to use the full template, and what it costs

Section titled “When to use the full template, and what it costs”

A good report takes 10 to 15 minutes once you have the evidence kit. That is less than one round of “Which version? Which seed? What did you type?” in a chat with the developer. Use the full template for every bug that leaves your session notes.

Keep each section short, and write one bug per report. If you find two failures in one reply, write two reports and link them to each other.

Common mistake

Two bugs in one report

Symptom: Esme asks the return question in French with seed 2. The reply is in English, and it says 30 days, citing version 3 (simulated, practice app). Her first report covers both. Tomás fixes the search, the reply now says 45 days, and he closes the report. The reply is still in English.

Cause: one report held two failures with two causes: the search (the expired policy) and the language of the reply. The fix of one closed the report for both.

Recover: write one report per failure, each with its own expected rules, rate and suspected layer. Link them: “Seen together with BUG-002 at seed 2.” Each bug then closes only when its own rules pass.

Common mistake

The expected result says 'the right answer'

Symptom: the report says “Expected: the bot should give the right refund.” Tomás changes the reply, and Esme reopens the bug, because the amount is still wrong in some runs. They argue about what “right” means.

Cause: the expected result was not checkable. Two people read it differently, and nobody could test it.

Recover: rewrite it with Must and Must not rules and real values: “Must: offer 51.05 dollars (64.00 minus 12.95 return shipping). Must not: add a fee or a credit that is not in a policy or the order.” Then it is clear when the bug is fixed, for both of you and for the runner.

The backlog in practice-app/backlog/ has four weak reports from support agents. You rewrote BL-104 with Esme. Now rewrite BL-101 on your own, as BUG-003. It is the main task of this module. Then complete the module’s set of reports. This takes about 45 minutes.

practice-app/backlog/BL-101.md
# BL-101: Order lookup broken!!
Reported by: support agent (chat shift)
Customer saw the wrong order in the chat. Very bad. Please fix ASAP.
  1. Read BL-101. Write down what it says, and what it does not say.
  2. Find the failure with the test customers. Their orders are in practice-app/helpdesk/data/seed.json. Ask about orders as one test customer: your own order numbers, and the numbers of other customers.
  3. When you see the failure, capture the evidence kit. Export it to ../workspace/bugs/evidence/.
  4. Measure the rate. Add a case bug-003-... to workspace/cases/bugs.yaml, run it, and note x of y runs.
  5. Copy the template to ../workspace/bugs/ as BUG-003- plus a short name. Fill every section. Write “Rewritten from backlog report BL-101” under the title.
  6. Rate the severity with the matrix, and add a row to workspace/notes/triage.md.
  7. Do the cold-reader test: a new terminal, the report only.

Then write the other reports of this module the same way:

  • BUG-004 from BL-103 (“The chat crashed. Error 500.”). Which inputs from Personas, edge cases and hostile inputs could crash a chat? Try them in the web app and in the terminal.
  • BUG-001, the refund offer, from the evidence kit that you made in Capture the evidence.
  • BUG-005, the refusal of the weed killer question from your first bug hunt.

Optional stretch: report the finding of Metamorphic testing: the rephrased return question (“What is the time limit for sending back something I haven’t used?”) never gets the return window. Give the steps for both wordings, so that the developer sees one pass and one fail, and attach both exports. Use your own ID: the course’s next reports, BUG-006 to BUG-008, come in Module 6.

Leave BL-102 for now. Its kind of failure comes in Prompt injection and data leaks.

Check your reports

Check each report against this list. Every answer must be yes.

  1. Does the title say what goes wrong, where, and how often (x of y runs)?
  2. Are the app, prompt and model versions there, with the app date, the seed and the test customer?
  3. Do the steps start with reset, and give the exact input, copied?
  4. Is the actual result word for word, with a trace ID and the trace fields that explain it?
  5. Is every expected rule a Must or Must not that the runner could check?
  6. Is the rate based on at least 20 runs (or 5 of 5 for a bug that fails every time), with the label “simulated (practice app)”?
  7. Is the suspected layer written as a guess, with a reason from the trace?
  8. Is there an export in bugs/evidence, and does it replay?
  9. Is every name, order and account a test one, with no key or password anywhere?
  10. Did a cold reader, or you in a new terminal, see the failure from the report alone?

For BUG-003, look at the trace closely. It shows two tool calls. Which one went wrong, and what does that say about the layer?

For BUG-004, the web app shows one thing and the terminal shows another. Put both in the actual result. Its replay says turn 1: same error and 1 of 1 replies are the same. An error that happens again is reproduced, not different, so the export does reproduce it.

The workspace snapshot for the end of this module has Esme’s five reports in workspace/bugs/, her cases in workspace/cases/bugs.yaml and her triage notes. Compare them with yours after you finish, not before.

Check your understanding

A report says 'Expected: the bot should tell the customer the right return period.' Rewrite it as checkable rules for the practice app on its app date, 16 November 2026.

Show the answer
  • Must: say “45 days” and cite the returns policy, version 4.
  • Must not: say “30 days”, or cite the returns policy, version 3 (it ended on 31 October 2026).

Two people check these rules the same way, and the runner can too: contains: 45 days and not_cites: returns-policy@3.

Check your understanding

A colleague's draft report has a real customer's name and email, a screenshot from the live help desk, and the steps contain the live key. What do you change, and is deleting the key from the report enough?

Show the answer

Reproduce the failure with a test customer, and write the report with that customer’s ID, order and export. Describe the real case by its shape only (“an order of another customer”), and remove the name, email and screenshot. Replace the key in the steps with the name of the setting, for example “live mode, with HELPDESK_LIVE_KEY set”. Deleting it is not enough: the key was already shared, so tell its owner at once, so that they replace it.

Check your understanding

Module check: what in your report lets a developer who was not in your session reproduce the bug alone, and how do you check that it works?

Show the answer

The steps start with reset and give the exact input, copied, with the seed and the test customer. The versions (app, prompt, model) and the app date say which build to use. The export replays the same reply, and the trace ID shows what happened inside. The expected result has Must and Must not rules, so the developer can see the failure and know when it is fixed. The reproduction rate says how many runs to expect before it appears. To check it, give the report to a cold reader, or open a new terminal, read only the report, and run its steps. If the actual result appears and the rules decide pass or fail, the report works.

Recap

  • A bug report is a recipe for a failure: a person who was not there must make it happen and know when it is fixed.
  • Each section of the template answers one question that the developer would otherwise ask you.
  • Write the expected result as Must and Must not rules. A rule that a person can check, the runner can check too.
  • Write the suspected layer as a guess, with the reason from the trace.
  • Test data only: no real customers, no keys, passwords or tokens. Reproduce real cases with test customers.
  • The cold-reader test proves the report: a new terminal, the report only, the same failure.

Your workspace now: five reports in workspace/bugs/, BUG-001 to BUG-005. BUG-003 and BUG-004 are rewritten from the backlog. Their exports are in workspace/bugs/evidence/, one case per bug is in workspace/cases/bugs.yaml, and workspace/notes/triage.md has all five. Next, you see what happens when a coding agent, not a person, reads your report.

Next: What a coding agent needs