Capture the evidence
- Lesson 10
- 30 min
- Module 4: Report bugs people can act on
Where we are
In Module 3, we found failures and measured how often they happen. But a failure that you saw is not yet a failure that a developer can see: the seed, the versions and the data all change the reply, and a reset deletes the trace. In this lesson, we collect the six pieces of evidence that let anyone see the failure again.
Lesson 10 of 24 · Testing AI Applications: Manual and AI-Assisted QACapture the evidence
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Collect everything a developer needs to see the failure you saw, before it disappears. You learn the six pieces of evidence for a failure of an AI feature, and where each one comes from in the bug-hunt app. Then you prove that your evidence reproduces the failure.
Before you startWhat to know and to have ready
What you need to know
- How to send a message, read a trace, export, replay and reset, from Meet the bug-hunt app.
- What a reproduction rate is, from Repeat runs and reproduction rate.
- The refund offer that gave a wrong amount with seed 4, from Rubrics and property checks. You capture its evidence at the end of this lesson.
What you need on your computer
- Your course folder
ai-qa, with the app inai-qa/practice-appand your notes inai-qa/workspace. - A terminal in
ai-qa/practice-app, with the virtual environment active. The prompt starts with(.venv).
Tested on macOS (Apple silicon) with Python 3.14.6. The Windows commands were written, not run.
Back at the garage
Section titled “Back at the garage”Remember the car with a noise on some mornings, from Repeat runs and reproduction rate. Your count, “3 of the last 10 mornings”, told the mechanic how often to expect it. But one drive around the block still shows nothing.
A good mechanic asks more questions. When does the noise come: on a cold morning, at high speed, when you brake? What exactly does it sound like? Did anything change, such as new tyres? Better still, you recorded the noise on your phone, with the speed on the screen. Now she can make the car do it again, and she knows what to listen for.
The count said how often. What was still missing was the evidence: the exact conditions, and a record of what happened.
The problem: “cannot reproduce”
Section titled “The problem: “cannot reproduce””Larkfield’s help desk has a backlog of bug reports from the time before Esme joined. Support agents wrote them. This one came from a support team lead:
# BL-104: Return days wrong?
Reported by: support team lead
Sometimes the bot says 30 days for returns. I think it should be 45 now. Not sure.Tomás, the AI engineer, tries it. He types the question in his own words. Then, after he reads the report again, he types it in the customer’s words. Neither try shows the bug, so he closes the report as “cannot reproduce”. The picture shows his two tries next to the run that Esme made.
Text description of the diagram
The first run is Esme’s. She sends “How many days do I have to return an unused item?” with seed 2. The reply says “within 30 days of delivery” and cites the returns policy, version 3. This is the bug, and it appears every time with these words and this seed.
The second run is Tomás’s first try. He types “How long do I have to return something?”, also with seed 2. The reply is “I could not find this in Larkfield’s policies, so I do not want to guess.” He missed the bug because the exact input was not in the report.
The third run is Tomás’s second try. He types the exact words, but with no seed, so the app chooses seed 22928. The reply says “within 45 days of delivery” and cites version 4, which is correct. He missed the bug because the seed was not in the report.
Look at the second run. Same seed, other words, a different reply. The assistant searches the policies with the words of the question. Other words found other sections, so the reply took another path. In the third run, the words were right, but the seed was not. The return question fails in about 3 runs of 10, so a random seed usually shows a correct reply.
Tomás did nothing wrong. He tested what the report gave him, and it gave him almost nothing. “Cannot reproduce” here means “the report did not say how”. In ordinary software, the same input usually gives the same result, so “the bot says 30 days” might be enough. In an AI feature, the exact words, the seed, the versions and the data all change the reply. Each one that is missing is another way to miss the bug.
The idea: an evidence kit
Section titled “The idea: an evidence kit”When you see a failure, you collect an evidence kit: everything another person needs to make the assistant do it again, and to see why. In the bug-hunt app, it has six pieces. Each piece answers one question that a developer asks.
Text description of the diagram
The six pieces, with their values for the return-window failure:
- The exact input, “How many days do I have to return an unused item?”, copied from the chat and not retyped. It answers: what did you send?
- The versions: app 2.3.0, prompt support-v8, model sim-1 (simulated). It answers: which build answered?
- The settings: seed 2, app date 2026-11-16, test customer C-1001. It answers: how do I get the same run?
- The output, word for word: “within 30 days of delivery”, with source [1], the returns policy, version 3. It answers: what did you see?
- The trace ID, tr-0001-1, and what the assistant retrieved or called. It retrieved version 4 and version 3, called no tool, and cited version 3. It answers: what happened inside?
- The conversation export, evidence/bug-002-seed2.json, which holds pieces 1 to 5 in one file. Its replay gives 1 of 1 same reply. It answers: can I run it myself?
Why each piece?
- The exact input. The search uses the words. Retype a question from memory and you test another question, as Tomás did. Copy it from the chat or the trace.
- The versions. A release has three: the app, the prompt (the instructions to the model) and the model. A bug in prompt support-v8 can be gone in support-v9, or new. Without versions, “it works now” can mean “it works in another build”.
- The settings. The seed fixes sim-1’s random choices. The app date decides which policy version is in force. The test customer decides which orders the assistant can see.
- The output, word for word. “It said something wrong about returns” is a memory. “Within 30 days of delivery”, with the source, is a fact that anyone can check.
- The trace ID, and what the assistant retrieved or called. The reply shows what the customer saw. The trace shows why: which policy sections the search found, which tools ran with which values, and which source the reply used. Quote the fields that matter, not all 60 lines.
- The conversation export. One JSON file with all of the above, for every turn. A developer can replay it on their own computer.
The trace and the export are the app’s flight recorder, as you saw in Meet the bug-hunt app. This lesson is about using them on purpose, at the moment of a failure, and checking that the result really reproduces it.
Worked example: Esme captures BL-104
Section titled “Worked example: Esme captures BL-104”Esme starts again from the backlog report. She already knows from her session notes that seed 2 shows the 30-day reply. Now she captures the kit, piece by piece. Run the steps with her in ai-qa/practice-app.
-
Reset the test data, so that your IDs are the same as hers:
Terminal window python -m helpdesk resetThe store is reset: 4 test customers, 8 orders, 8 reviews, no conversations. -
Send the question with the customer’s exact words and seed 2:
Terminal window python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2The output, simulated (practice app):
Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.Sources:[1] Returns policy, version 3: Return window (/policies/returns-policy/v3#return-window)[conversation c-0001 | trace tr-0001-1 | seed 2]This gives pieces 1, 3 and 4: the input, the seed and the output. The last line also gives the conversation ID and the trace ID.
-
Show the trace:
Terminal window python -m helpdesk trace tr-0001-1The output is about 60 lines. These are the lines that Esme quotes in her notes:
trace tr-0001-1 (part) "app_version": "2.3.0","prompt_version": "support-v8","model": "sim-1","settings": { "seed": 2, "today": "2026-11-16", ... },"customer_id": "C-1001","retrieved": [{ "doc_id": "returns-policy", "version": 4, "section": "Return window","score": 7.926, "effective_from": "2026-11-01", "effective_to": null },{ "doc_id": "returns-policy", "version": 3, "section": "Return window","score": 7.926, "effective_from": "2025-03-01", "effective_to": "2026-10-31" },...],"tool_calls": [],"cited": [ "returns-policy@3#return-window" ]This gives pieces 2 and 5. The search found both versions with the same score, 7.926. The reply cited version 3, which ended on 31 October. No tool was called, so the tool code is not involved.
Esme writes the five pieces in her notes before she does anything else. Then she makes the sixth piece, the export, and proves that it works.
-
Make a folder for the evidence in your workspace:
Terminal window mkdir -p ../workspace/bugs/evidenceTerminal window New-Item -ItemType Directory -Force ..\workspace\bugs\evidenceTerminal window mkdir -p ../workspace/bugs/evidence -
Export the conversation into it. Give the file a name that says which bug and which seed:
Terminal window python -m helpdesk export c-0001 -o ../workspace/bugs/evidence/bug-002-seed2.jsonExported c-0001 to ../workspace/bugs/evidence/bug-002-seed2.jsonForward slashes work on Windows too, so this command is the same on every system.
-
Reset, as a developer with a clean copy of the app would start:
Terminal window python -m helpdesk reset -
Replay the export:
Terminal window python -m helpdesk replay ../workspace/bugs/evidence/bug-002-seed2.jsonturn 1: same reply1 of 1 replies are the same (seed 2, new conversation c-0001).
“1 of 1 replies are the same” is the proof. After a reset, with nothing but the file, the app gave the 30-day reply again. Esme now knows that her evidence is complete, before she sends it to anyone.
Why does she call the bug BUG-002? Her first finding, the refund amount, will be BUG-001. You write that one at the end of this lesson. Esme gives each bug one ID and uses it in every file: the export, the report and later the test case.
Check the export, not your memory
Section titled “Check the export, not your memory”Open bug-002-seed2.json in your editor and find each piece of the kit. The table shows where each one is.
| Piece | Where it is in the export |
|---|---|
| 1. Exact input | messages: the entry with "role": "user", its text |
| 2. Versions | app_version, prompt_version, model, and mode (sim or live) |
| 3. Settings | settings (seed, app date) and customer_id |
| 4. Output | messages: the entry with "role": "assistant" |
| 5. Trace | traces: the full trace of each reply, with retrieved, tool_calls and cited |
| 6. The export itself | the file, with started_at and exported_at |
If a piece is missing from the export, it is missing from your evidence.
Three places to get an export
Section titled “Three places to get an export”You do not always find a failure with ask. Each way of testing in the app gives you an export:
- In the terminal:
python -m helpdesk export c-... -o file.json, with the conversation ID from the last line of the reply. Inchat, type/export. - In the web app: select Export in the top bar. The browser downloads
c-0001.json(or the conversation’s ID). Move it intoworkspace/bugs/evidenceand rename it. - From the QA runner: every failing run of a case is already saved in the report folder, in
reports/qa-.../conversations/. You used these in Rubrics and property checks. Copy the one you need intoworkspace/bugs/evidence.
When to capture, and what it costs
Section titled “When to capture, and what it costs”Capture at the moment you see the failure. Evidence disappears. reset deletes every conversation and trace. A random seed that you did not note is gone. And a new release can change the reply. The cost is small: two commands and a file of a few kilobytes.
Capture for failures, not for every run. A passing run needs a line in your session notes with its seed, not an export. Keep exports for the runs that you report.
With a real model, the export is a record, not a replay. A real model does not give the same reply twice, even with the same settings. In the course’s recorded runs of a real model, 11 of 30 questions gave 10 different replies in 10 runs. So for a real assistant, collect the same kit for several failing runs, and say how many runs you made. The pieces stay the same: exact input, versions, settings, output, trace and export.
Real data is the exception. At Larkfield, every conversation in the app uses test customers. A conversation from the real help desk contains real names, orders and addresses. Do not attach it as it is. Reproduce the failure with a test customer, and attach that export. A bug report template for AI behaviour shows what to keep out of a report.
Common mistake
The export stops with FileNotFoundError
Symptom: the export prints a long error that ends like this:
FileNotFoundError: [Errno 2] No such file or directory: '../workspace/bugs/evidence/bug-002-seed2.json'Cause: the folder workspace/bugs/evidence does not exist yet. export writes a file, but it does not make folders.
Recover: make the folder (the first step of the export above), then run the same export again. Nothing is lost: the conversation is still in the app until you reset.
Common mistake
The replay says DIFFERENT reply
Symptom: you replay the export of a cancellation, and the replay shows this (simulated, practice app):
turn 1: DIFFERENT reply before: Do you want me to cancel order LK-200101 (HT-550 corded hedge trimmer)? Reply yes to confirm. now: Order LK-200101 is Cancelled, so it can no longer be cancelled. You can return the items after delivery.turn 2: DIFFERENT reply before: Done: order LK-200101 is cancelled. now: I could not find this in Larkfield's policies, so I do not want to guess. The help desk can help: 1-800-555-0142.0 of 2 replies are the same (seed 7, new conversation c-0002).Cause: the conversation changed the test data: it really cancelled order LK-200101. The replay ran on the changed data, so the assistant found a cancelled order.
Diagnose: look for a tool call that changes data (cancel_order, start_refund) in the export’s traces.
Recover: replay with the data as it was at the start:
python -m helpdesk replay --reset c-0001.jsonturn 1: same replyturn 2: same reply2 of 2 replies are the same (seed 7, new conversation c-0001).The data state is part of the settings. In your report, start the steps with python -m helpdesk reset, so that the developer starts with the same data as you.
Your turn: capture the refund finding
Section titled “Your turn: capture the refund finding”In Rubrics and property checks, the refund offer for the hose reel gave a wrong amount with seed 4. Capture its evidence kit as BUG-001. This time, the conversation has two turns: the offer, then “yes”. So the kit must also show what the assistant called when the customer said yes. This takes about 10 minutes.
- Reset the test data.
- Start a chat with seed 4:
python -m helpdesk chat --seed 4. - Type
I want to return the hose reel from order LK-200102and press Enter. - Type
yes. Then type/traceto see the trace of that reply. - Type
/quit. - Export the conversation to
../workspace/bugs/evidence/bug-001-seed4.json. - Reset, then replay the export. If it changed data, add
--reset. - In your notes, write the six pieces of the kit for BUG-001. For piece 5, write what the trace of turn 2 shows.
Check your capture
The chat (simulated, practice app), shortened:
assistant> I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 79.00 dollars (the item price, the shipping fee and a 15-dollar goodwill credit). Reply yes to confirm.[trace tr-0001-1]
assistant> Done: the return is started (refund 51.05 dollars). We email the return label within 1 business day.[trace tr-0001-2]The trace of turn 2, tr-0001-2, has one tool call:
"input": "yes","tool_calls": [ { "name": "start_refund", "args": { "order_id": "LK-200102", "sku": "HR-30", "confirmed": true }, "result": { "ok": true, "refund_id": 1, "order_id": "LK-200102", "sku": "HR-30", "amount": 51.05 } }]The replay after a reset gives 2 of 2 replies are the same (seed 4, new conversation c-0001). The refund changed the test data, but the replay started from fresh data, so the plain replay works too. replay --reset gives the same result.
Piece 5 is the important one. The reply offered 79.00 dollars. The tool started a refund of 51.05 dollars, the correct amount (64.00 minus 12.95 return shipping). So the customer agreed to one amount and gets another. Without the trace of turn 2, you would report only “a wrong amount”. With it, you can show that the reply and the action disagree, and that the refund tool computed the right amount.
Your kit:
- The input as in step 3, then “yes”.
- App 2.3.0, prompt support-v8, model sim-1 (simulated).
- Seed 4, app date 2026-11-16, test customer C-1001.
- The two replies above.
- Traces tr-0001-1 and tr-0001-2, with the tool call.
- The export
bug-001-seed4.json.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás has your seed, 2, but he types the return question in his own words. Predict: does he see the 30-day reply? Explain why the exact words are part of the evidence.
Show the answer
Probably not. In this lesson, “How long do I have to return something?” with seed 2 gave “I could not find this in Larkfield’s policies”. The assistant searches the policies with the words of the question. Other words find other sections, so the reply takes another path, even with the same seed. Copy the exact input from the chat or the trace. Never retype it from memory.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
You export a conversation in which the assistant cancelled an order. Your replay says '0 of 2 replies are the same'. Is your evidence wrong? What do you do?
Show the answer
The evidence is fine. The conversation changed the test data, and the replay ran on the changed data: the order was already cancelled. Replay with python -m helpdesk replay --reset file.json, which starts from fresh data. In the report, start the steps with python -m helpdesk reset, because the data state is part of the settings.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
You see a wrong reply in the web app and you did not type a seed. Name the six pieces of evidence that you collect, and say where you get each one in the web app.
Show the answer
- The exact input: copy your message from the chat. 2. The versions: they are in the trace and the export (app, prompt, model). 3. The settings: the seed and conversation shown under the reply, the test customer in the top bar, and the app date in the trace. 4. The output: copy the reply word for word, with its sources. 5. The trace: select the trace ID under the reply, and note what it retrieved, which tools it called and what it cited. 6. The export: select Export, then move the file into
workspace/bugs/evidence. Finally, reset and replay the export to check that it gives the same reply.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- “Cannot reproduce” often means “the report did not say how”. For an AI feature, the exact words, the seed, the versions and the data all change the reply.
- An evidence kit has six pieces: the exact input, the versions, the settings, the output word for word, the trace ID with what the assistant retrieved or called, and the export.
- Capture at the moment of the failure:
resetdeletes conversations and traces. - Prove the kit: reset, then replay the export. “Same reply” means a developer can see what you saw. Use
replay --resetwhen the conversation changed data. - The trace shows what the reply cannot: in BUG-001, the reply offered 79.00 dollars but the refund tool started 51.05.
Your workspace now: workspace/bugs/evidence/ with bug-002-seed2.json (the return window) and bug-001-seed4.json (the refund offer), and the six pieces of each kit in your notes. Next, you decide how serious each bug is, and what to do when it appears only sometimes.