Skip to content
Français

Capture the evidence

  • Lesson 10
  • 30 min
  • Module 4: Report bugs people can act on

Where we are

In Module 3, we found failures and measured how often they happen. But a failure that you saw is not yet a failure that a developer can see: the seed, the versions and the data all change the reply, and a reset deletes the trace. In this lesson, we collect the six pieces of evidence that let anyone see the failure again.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Collect everything a developer needs to see the failure you saw, before it disappears. You learn the six pieces of evidence for a failure of an AI feature, and where each one comes from in the bug-hunt app. Then you prove that your evidence reproduces the failure.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa, with the app in ai-qa/practice-app and your notes in ai-qa/workspace.
  • A terminal in ai-qa/practice-app, with the virtual environment active. The prompt starts with (.venv).

Tested on macOS (Apple silicon) with Python 3.14.6. The Windows commands were written, not run.

Remember the car with a noise on some mornings, from Repeat runs and reproduction rate. Your count, “3 of the last 10 mornings”, told the mechanic how often to expect it. But one drive around the block still shows nothing.

A good mechanic asks more questions. When does the noise come: on a cold morning, at high speed, when you brake? What exactly does it sound like? Did anything change, such as new tyres? Better still, you recorded the noise on your phone, with the speed on the screen. Now she can make the car do it again, and she knows what to listen for.

The count said how often. What was still missing was the evidence: the exact conditions, and a record of what happened.

Larkfield’s help desk has a backlog of bug reports from the time before Esme joined. Support agents wrote them. This one came from a support team lead:

practice-app/backlog/BL-104.md
# BL-104: Return days wrong?
Reported by: support team lead
Sometimes the bot says 30 days for returns. I think it should be 45 now. Not sure.

Tomás, the AI engineer, tries it. He types the question in his own words. Then, after he reads the report again, he types it in the customer’s words. Neither try shows the bug, so he closes the report as “cannot reproduce”. The picture shows his two tries next to the run that Esme made.

Three runs of the practice app, each from an input to its reply. Esme: the exact words 'How many days do I have to return an unused item?' with seed 2 give 'within 30 days of delivery', source Returns policy, version 3: the bug, shown every time with these words and this seed. Tomás, his own words 'How long do I have to return something?' with seed 2, gives 'I could not find this in Larkfield's policies, so I do not want to guess.' Missed: the exact input was not in the report. Tomás, the exact words with no seed (the app chose seed 22928), gives 'within 45 days of delivery', source Returns policy, version 4. Missed: the seed was not in the report.
One bug, three runs. Only the run with the exact words and a failing seed shows it. Simulated (practice app).
Text description of the diagram

The first run is Esme’s. She sends “How many days do I have to return an unused item?” with seed 2. The reply says “within 30 days of delivery” and cites the returns policy, version 3. This is the bug, and it appears every time with these words and this seed.

The second run is Tomás’s first try. He types “How long do I have to return something?”, also with seed 2. The reply is “I could not find this in Larkfield’s policies, so I do not want to guess.” He missed the bug because the exact input was not in the report.

The third run is Tomás’s second try. He types the exact words, but with no seed, so the app chooses seed 22928. The reply says “within 45 days of delivery” and cites version 4, which is correct. He missed the bug because the seed was not in the report.

Look at the second run. Same seed, other words, a different reply. The assistant searches the policies with the words of the question. Other words found other sections, so the reply took another path. In the third run, the words were right, but the seed was not. The return question fails in about 3 runs of 10, so a random seed usually shows a correct reply.

Tomás did nothing wrong. He tested what the report gave him, and it gave him almost nothing. “Cannot reproduce” here means “the report did not say how”. In ordinary software, the same input usually gives the same result, so “the bot says 30 days” might be enough. In an AI feature, the exact words, the seed, the versions and the data all change the reply. Each one that is missing is another way to miss the bug.

When you see a failure, you collect an evidence kit: everything another person needs to make the assistant do it again, and to see why. In the bug-hunt app, it has six pieces. Each piece answers one question that a developer asks.

Six cards, each with a piece of evidence, its value for the return-window failure, and the question it answers. 1, the exact input: 'How many days do I have to return an unused item?', copied from the chat, not retyped. It answers: what did you send? 2, the versions: app 2.3.0, prompt support-v8, model sim-1 (simulated). It answers: which build answered? 3, the settings: seed 2, app date 2026-11-16, test customer C-1001. It answers: how do I get the same run? 4, the output, word for word: 'within 30 days of delivery', source [1] Returns policy, version 3. It answers: what did you see? 5, the trace ID and what it retrieved or called: tr-0001-1, retrieved version 4 and version 3, no tool calls, cited version 3. It answers: what happened inside? 6, the conversation export: evidence/bug-002-seed2.json, pieces 1 to 5 in one file, replay gives 1 of 1 same reply. It answers: can I run it myself?
The evidence kit for one failure, filled in for the return-window reply with seed 2. Simulated (practice app).
Text description of the diagram

The six pieces, with their values for the return-window failure:

  1. The exact input, “How many days do I have to return an unused item?”, copied from the chat and not retyped. It answers: what did you send?
  2. The versions: app 2.3.0, prompt support-v8, model sim-1 (simulated). It answers: which build answered?
  3. The settings: seed 2, app date 2026-11-16, test customer C-1001. It answers: how do I get the same run?
  4. The output, word for word: “within 30 days of delivery”, with source [1], the returns policy, version 3. It answers: what did you see?
  5. The trace ID, tr-0001-1, and what the assistant retrieved or called. It retrieved version 4 and version 3, called no tool, and cited version 3. It answers: what happened inside?
  6. The conversation export, evidence/bug-002-seed2.json, which holds pieces 1 to 5 in one file. Its replay gives 1 of 1 same reply. It answers: can I run it myself?

Why each piece?

  • The exact input. The search uses the words. Retype a question from memory and you test another question, as Tomás did. Copy it from the chat or the trace.
  • The versions. A release has three: the app, the prompt (the instructions to the model) and the model. A bug in prompt support-v8 can be gone in support-v9, or new. Without versions, “it works now” can mean “it works in another build”.
  • The settings. The seed fixes sim-1’s random choices. The app date decides which policy version is in force. The test customer decides which orders the assistant can see.
  • The output, word for word. “It said something wrong about returns” is a memory. “Within 30 days of delivery”, with the source, is a fact that anyone can check.
  • The trace ID, and what the assistant retrieved or called. The reply shows what the customer saw. The trace shows why: which policy sections the search found, which tools ran with which values, and which source the reply used. Quote the fields that matter, not all 60 lines.
  • The conversation export. One JSON file with all of the above, for every turn. A developer can replay it on their own computer.

The trace and the export are the app’s flight recorder, as you saw in Meet the bug-hunt app. This lesson is about using them on purpose, at the moment of a failure, and checking that the result really reproduces it.

Esme starts again from the backlog report. She already knows from her session notes that seed 2 shows the 30-day reply. Now she captures the kit, piece by piece. Run the steps with her in ai-qa/practice-app.

  1. Reset the test data, so that your IDs are the same as hers:

    Terminal window
    python -m helpdesk reset
    The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.
  2. Send the question with the customer’s exact words and seed 2:

    Terminal window
    python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2

    The output, simulated (practice app):

    Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.
    Sources:
    [1] Returns policy, version 3: Return window (/policies/returns-policy/v3#return-window)
    [conversation c-0001 | trace tr-0001-1 | seed 2]

    This gives pieces 1, 3 and 4: the input, the seed and the output. The last line also gives the conversation ID and the trace ID.

  3. Show the trace:

    Terminal window
    python -m helpdesk trace tr-0001-1

    The output is about 60 lines. These are the lines that Esme quotes in her notes:

    trace tr-0001-1 (part)
    "app_version": "2.3.0",
    "prompt_version": "support-v8",
    "model": "sim-1",
    "settings": { "seed": 2, "today": "2026-11-16", ... },
    "customer_id": "C-1001",
    "retrieved": [
    { "doc_id": "returns-policy", "version": 4, "section": "Return window",
    "score": 7.926, "effective_from": "2026-11-01", "effective_to": null },
    { "doc_id": "returns-policy", "version": 3, "section": "Return window",
    "score": 7.926, "effective_from": "2025-03-01", "effective_to": "2026-10-31" },
    ...
    ],
    "tool_calls": [],
    "cited": [ "returns-policy@3#return-window" ]

    This gives pieces 2 and 5. The search found both versions with the same score, 7.926. The reply cited version 3, which ended on 31 October. No tool was called, so the tool code is not involved.

Esme writes the five pieces in her notes before she does anything else. Then she makes the sixth piece, the export, and proves that it works.

  1. Make a folder for the evidence in your workspace:

    Terminal window
    mkdir -p ../workspace/bugs/evidence
  2. Export the conversation into it. Give the file a name that says which bug and which seed:

    Terminal window
    python -m helpdesk export c-0001 -o ../workspace/bugs/evidence/bug-002-seed2.json
    Exported c-0001 to ../workspace/bugs/evidence/bug-002-seed2.json

    Forward slashes work on Windows too, so this command is the same on every system.

  3. Reset, as a developer with a clean copy of the app would start:

    Terminal window
    python -m helpdesk reset
  4. Replay the export:

    Terminal window
    python -m helpdesk replay ../workspace/bugs/evidence/bug-002-seed2.json
    turn 1: same reply
    1 of 1 replies are the same (seed 2, new conversation c-0001).

“1 of 1 replies are the same” is the proof. After a reset, with nothing but the file, the app gave the 30-day reply again. Esme now knows that her evidence is complete, before she sends it to anyone.

Why does she call the bug BUG-002? Her first finding, the refund amount, will be BUG-001. You write that one at the end of this lesson. Esme gives each bug one ID and uses it in every file: the export, the report and later the test case.

Open bug-002-seed2.json in your editor and find each piece of the kit. The table shows where each one is.

Piece Where it is in the export
1. Exact input messages: the entry with "role": "user", its text
2. Versions app_version, prompt_version, model, and mode (sim or live)
3. Settings settings (seed, app date) and customer_id
4. Output messages: the entry with "role": "assistant"
5. Trace traces: the full trace of each reply, with retrieved, tool_calls and cited
6. The export itself the file, with started_at and exported_at

If a piece is missing from the export, it is missing from your evidence.

You do not always find a failure with ask. Each way of testing in the app gives you an export:

  • In the terminal: python -m helpdesk export c-... -o file.json, with the conversation ID from the last line of the reply. In chat, type /export.
  • In the web app: select Export in the top bar. The browser downloads c-0001.json (or the conversation’s ID). Move it into workspace/bugs/evidence and rename it.
  • From the QA runner: every failing run of a case is already saved in the report folder, in reports/qa-.../conversations/. You used these in Rubrics and property checks. Copy the one you need into workspace/bugs/evidence.

Capture at the moment you see the failure. Evidence disappears. reset deletes every conversation and trace. A random seed that you did not note is gone. And a new release can change the reply. The cost is small: two commands and a file of a few kilobytes.

Capture for failures, not for every run. A passing run needs a line in your session notes with its seed, not an export. Keep exports for the runs that you report.

With a real model, the export is a record, not a replay. A real model does not give the same reply twice, even with the same settings. In the course’s recorded runs of a real model, 11 of 30 questions gave 10 different replies in 10 runs. So for a real assistant, collect the same kit for several failing runs, and say how many runs you made. The pieces stay the same: exact input, versions, settings, output, trace and export.

Real data is the exception. At Larkfield, every conversation in the app uses test customers. A conversation from the real help desk contains real names, orders and addresses. Do not attach it as it is. Reproduce the failure with a test customer, and attach that export. A bug report template for AI behaviour shows what to keep out of a report.

Common mistake

The export stops with FileNotFoundError

Symptom: the export prints a long error that ends like this:

FileNotFoundError: [Errno 2] No such file or directory: '../workspace/bugs/evidence/bug-002-seed2.json'

Cause: the folder workspace/bugs/evidence does not exist yet. export writes a file, but it does not make folders.

Recover: make the folder (the first step of the export above), then run the same export again. Nothing is lost: the conversation is still in the app until you reset.

Common mistake

The replay says DIFFERENT reply

Symptom: you replay the export of a cancellation, and the replay shows this (simulated, practice app):

turn 1: DIFFERENT reply
before: Do you want me to cancel order LK-200101 (HT-550 corded hedge trimmer)? Reply yes to confirm.
now: Order LK-200101 is Cancelled, so it can no longer be cancelled. You can return the items after delivery.
turn 2: DIFFERENT reply
before: Done: order LK-200101 is cancelled.
now: I could not find this in Larkfield's policies, so I do not want to guess. The help desk can help: 1-800-555-0142.
0 of 2 replies are the same (seed 7, new conversation c-0002).

Cause: the conversation changed the test data: it really cancelled order LK-200101. The replay ran on the changed data, so the assistant found a cancelled order.

Diagnose: look for a tool call that changes data (cancel_order, start_refund) in the export’s traces.

Recover: replay with the data as it was at the start:

Terminal window
python -m helpdesk replay --reset c-0001.json
turn 1: same reply
turn 2: same reply
2 of 2 replies are the same (seed 7, new conversation c-0001).

The data state is part of the settings. In your report, start the steps with python -m helpdesk reset, so that the developer starts with the same data as you.

In Rubrics and property checks, the refund offer for the hose reel gave a wrong amount with seed 4. Capture its evidence kit as BUG-001. This time, the conversation has two turns: the offer, then “yes”. So the kit must also show what the assistant called when the customer said yes. This takes about 10 minutes.

  1. Reset the test data.
  2. Start a chat with seed 4: python -m helpdesk chat --seed 4.
  3. Type I want to return the hose reel from order LK-200102 and press Enter.
  4. Type yes. Then type /trace to see the trace of that reply.
  5. Type /quit.
  6. Export the conversation to ../workspace/bugs/evidence/bug-001-seed4.json.
  7. Reset, then replay the export. If it changed data, add --reset.
  8. In your notes, write the six pieces of the kit for BUG-001. For piece 5, write what the trace of turn 2 shows.
Check your capture

The chat (simulated, practice app), shortened:

assistant> I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 79.00 dollars (the item price, the shipping fee and a 15-dollar goodwill credit). Reply yes to confirm.
[trace tr-0001-1]
assistant> Done: the return is started (refund 51.05 dollars). We email the return label within 1 business day.
[trace tr-0001-2]

The trace of turn 2, tr-0001-2, has one tool call:

trace tr-0001-2 (part)
"input": "yes",
"tool_calls": [
{
"name": "start_refund",
"args": { "order_id": "LK-200102", "sku": "HR-30", "confirmed": true },
"result": { "ok": true, "refund_id": 1, "order_id": "LK-200102", "sku": "HR-30", "amount": 51.05 }
}
]

The replay after a reset gives 2 of 2 replies are the same (seed 4, new conversation c-0001). The refund changed the test data, but the replay started from fresh data, so the plain replay works too. replay --reset gives the same result.

Piece 5 is the important one. The reply offered 79.00 dollars. The tool started a refund of 51.05 dollars, the correct amount (64.00 minus 12.95 return shipping). So the customer agreed to one amount and gets another. Without the trace of turn 2, you would report only “a wrong amount”. With it, you can show that the reply and the action disagree, and that the refund tool computed the right amount.

Your kit:

  1. The input as in step 3, then “yes”.
  2. App 2.3.0, prompt support-v8, model sim-1 (simulated).
  3. Seed 4, app date 2026-11-16, test customer C-1001.
  4. The two replies above.
  5. Traces tr-0001-1 and tr-0001-2, with the tool call.
  6. The export bug-001-seed4.json.

Check your understanding

Tomás has your seed, 2, but he types the return question in his own words. Predict: does he see the 30-day reply? Explain why the exact words are part of the evidence.

Show the answer

Probably not. In this lesson, “How long do I have to return something?” with seed 2 gave “I could not find this in Larkfield’s policies”. The assistant searches the policies with the words of the question. Other words find other sections, so the reply takes another path, even with the same seed. Copy the exact input from the chat or the trace. Never retype it from memory.

Check your understanding

You export a conversation in which the assistant cancelled an order. Your replay says '0 of 2 replies are the same'. Is your evidence wrong? What do you do?

Show the answer

The evidence is fine. The conversation changed the test data, and the replay ran on the changed data: the order was already cancelled. Replay with python -m helpdesk replay --reset file.json, which starts from fresh data. In the report, start the steps with python -m helpdesk reset, because the data state is part of the settings.

Check your understanding

You see a wrong reply in the web app and you did not type a seed. Name the six pieces of evidence that you collect, and say where you get each one in the web app.

Show the answer
  1. The exact input: copy your message from the chat. 2. The versions: they are in the trace and the export (app, prompt, model). 3. The settings: the seed and conversation shown under the reply, the test customer in the top bar, and the app date in the trace. 4. The output: copy the reply word for word, with its sources. 5. The trace: select the trace ID under the reply, and note what it retrieved, which tools it called and what it cited. 6. The export: select Export, then move the file into workspace/bugs/evidence. Finally, reset and replay the export to check that it gives the same reply.

Recap

  • “Cannot reproduce” often means “the report did not say how”. For an AI feature, the exact words, the seed, the versions and the data all change the reply.
  • An evidence kit has six pieces: the exact input, the versions, the settings, the output word for word, the trace ID with what the assistant retrieved or called, and the export.
  • Capture at the moment of the failure: reset deletes conversations and traces.
  • Prove the kit: reset, then replay the export. “Same reply” means a developer can see what you saw. Use replay --reset when the conversation changed data.
  • The trace shows what the reply cannot: in BUG-001, the reply offered 79.00 dollars but the refund tool started 51.05.

Your workspace now: workspace/bugs/evidence/ with bug-002-seed2.json (the return window) and bug-001-seed4.json (the refund offer), and the six pieces of each kit in your notes. Next, you decide how serious each bug is, and what to do when it appears only sometimes.

Next: Severity and triage for "sometimes wrong"