Trace failures

What you will be able to do

Find out why a task failed by reading what the run did: its model calls, tool results, proposal, approval and writes, in order. You find the first event that went wrong, name its kind, and decide where the fix belongs, without guessing what the model was thinking.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Task success and the saved scores, from Task success measures.
• The events in a run’s state and the trace command, from Explicit task state and Checkpoint and resume.

What you need on your computer

• The project after the last lesson, with results/agent-chat-small-r1-all.json. python -m pytest prints 64 passed.

This lesson adds no file to the project. Every trace is a real recording of gpt-6-luna as chat-small or gpt-6.1-sol as chat-strong on Azure, replayed by the project’s code. In the steps, you approve some proposals as Amira: a person’s choice, simulated. One section shows three new answers from chat-small, recorded for this lesson. Tested with Python 3.12.13 on macOS.

A parcel that never arrived

You ordered a parcel, and it did not arrive. You open the tracking page:

1. Picked up from the shop, Monday 08:12.
2. Arrived at the depot, Monday 15:40.
3. Out for delivery, Tuesday 07:55.
4. Delivery attempted: address not found, Tuesday 11:20.
5. Returned to the depot, Tuesday 17:05.

You know where it went wrong: at event 4. The parcel left the depot correctly. The driver could not find the address. Now you can check the address on the order, which was the real mistake.

You did not need to know what the driver was thinking. If you call him a week later and ask, he will give you a story, and it may sound right. But the scans are a record of what happened at the time. His story is new text, written after.

The problem: you know the score, not the reason

The last lesson told you which tasks failed: 9 of 70 for the chat-small agent in repeat 1, every proposal approved as it is. T-90104 and T-90305 were among them. It did not tell you why. Without the reason, you cannot choose the fix. Is it the prompt, a tool, the policy text or the approval screen?

Tomás has an idea: “Ask the model why it did it. Or turn on its reasoning and read that.” Both sound reasonable. Both fail, for two reasons.

First, the reasoning is not there to read. chat-small runs with reasoning_effort: "none", so it used 0 reasoning tokens in its 149 answered calls of repeat 1. chat-strong reasons on its own: in repeat 1, it used 518 reasoning tokens in 17 of its 134 answered calls. Azure counts them in the bill and does not return them. You pay for that text and you cannot read it.

Second, an explanation that you ask for later is new text, not a record. The section Ask the model why below shows three real examples.

The idea: read the trace, find the first wrong event

You have read traces since Workflow and agent boundaries. Here is the full definition. A trace is the ordered record of what a run did that you can observe. It holds each model call with what it asked for, and each tool call with its result. It also holds the plan, the proposal, the approval, each write, each reconciliation and the stop. The project keeps it in the task state, as a list of events, and trace prints it.

To find why a task failed, walk the trace from the start. At each event, ask one question: was this right, given what the run knew at this point? The first event where the answer is “no” is the first wrong event. Its kind tells you where to look for the fix:

Kind of first wrong event	Example	Where the fix belongs
A tool call that is refused or fails, and the run cannot recover	the order service is down, and the run guesses	the tool contract, the loop’s rules, or the policy’s “a person” rule
A missing or wrong read	no get_payments on a double charge	the prompt’s instructions, the tool descriptions
A decision against the evidence	the order is 12 days late, and the model applies the rule for “lost”	the policy text, the prompt, the model
A wrong proposal that the approval lets through	a reship on a German ticket, approved	the approval screen and the approver’s checklist
A write that fails or happens twice	a timeout after the write, then a second call	the executor: operation IDs, reconciliation

Events after the first wrong one are often wrong too, but they are effects. Fix the first one.

Here is a trace with every kind of event, including a failure and an approval. It is T-90603, a double charge, where the refund call times out after the refund was made (simulated):

The trace of T-90603 as a timeline: model calls, tool results, the approval, a write that fails and the reconciliation. Every arrow is one event that the project records. The model decisions are recordings of gpt-6-luna as chat-small; the timeout is simulated.

Text description of the diagram

T-90603, a double charge. Step 1: the model asks for get_order; the order service answers ok. Step 2: the model asks for get_payments; the payment service answers with two captured charges of 45.95. Step 3: the model asks to finish with a refund of 45.95 on the later payment, rule W7. The loop checks the proposal, and the run waits for approval.

Amira sees the exact action and approves it. The executor writes the refund with an operation ID. The payment service times out, but the refund was made (simulated). The executor reconciles: it looks up the operation ID, finds the refund, and does not send it again. The run stops with done, with one refund.

Read it with the question. Every event is right, given what the run knew. The timeout is a failure event, but not a wrong one: the executor did the right thing next. A trace with a failure in it is not a failed task.

A tiny worked example: T-90104, five events

T-90104 is a late watering can. The customer writes that it “shipped weeks ago”. Grace’s policy expects resolve: refund the 9.95 shipping fee under W2, because the order is more than 5 business days late. Here is its trace:

1. Run the ticket with its trace:

   python -m resolver run T-90104 --trace

Run T-90104-agent-chat-small-1: agent, chat-small, ticket T-90104, status done
2 model call(s), 4565 tokens in, 179 out, US$0.00055, 2.996 s waiting for the model

  1  model     chat-small: get_order({"order_id":"LK-793996"})  (2163 in, 21 out, 1.05 s)
  1  tool      get_order({"order_id": "LK-793996"}) -> ok
  2  model     chat-small: finish({"outcome":"ask_customer","rule":"W3","…)  (2402 in, 158 out, 1.946 s)
  2  proposal  ask_customer (W3): no change
  2  stop      finished

Walk it event by event:

1. Model asks for get_order LK-793996. Right: the ticket names the order.
2. Tool: ok. Right. The result says: shipped 16 September, business_days_late 12, shipping fee 9.95, refunds empty. That is every fact W2 needs, computed in code.
3. Model asks to finish with ask_customer, rule W3. Wrong. W3 is for an order more than 15 business days late. It is 12.
4. Proposal: ask_customer. The same decision, accepted by the schema: it is a valid resolution, only not the right one.
5. Stop: finished.

The first wrong event is number 3: a decision against the evidence. The tool did its job; the model misread the policy. The proposal also carries the model’s stated reason. It says that the W2 refund “is due”, and then that “no evidence confirms whether that fee was already refunded”. But the order showed "refunds": []. The stated reason does not match the evidence, so it cannot be the record of the decision. The trace is.

Explore the traces

The explorer holds seven real runs, event by event. Three are failures, and four went right, some with a failure event inside. The approvals are Amira’s, in this lesson’s steps.

Explore

Where did the run first go wrong?

T-90305 (needs person), customer C-52445: “Hallo, meine Bestellung LK-642445 ist beschädigt angekommen. Der Deckel des Komposters ist gebrochen. Foto im Anhang.” Attachment: IMG_3381.jpg.

Expected by Grace's policy: hand_to_person (H5): no change. Run T-90305-agent-chat-small-1: gpt-6-luna as chat-small, recorded, replayed by the project's own code. Approved by Amira (agent), in the lesson's steps.

Status

done

Model calls

2 (4,591 tokens in)

Tool calls

1

Changes written

2

T-90305, event 9 of 9. Task success: no, with 2 unexpected change(s) in the database. First wrong event: the proposal at step 2. The ticket is in German, so H5 says: hand it to a person. The model read German well and applied W5. Then the approval let it through, and two changes reached the database that the policy did not allow. Two layers failed: the decision, then the person's check.

1. 1 Model step 1

   Model asks for get_order(LK-642445)

   chat-small: 2172 tokens in, 21 out, 1.137 s.

2. 2 Tool step 1 worked

   Tool get_order(LK-642445): ok

   status delivered, shipped 2026-10-02, delivered 2026-10-06 (3 days ago), shipping fee 9.95, refunds 0. Items: COMPOST x 1, 64.00, in stock.

3. 3 Model step 2

   Model asks to finish: resolve (W5)

   chat-small: 2419 tokens in, 209 out, 2.485 s.

4. 4 Proposal step 2 wrong

   Proposal: resolve (W5): reship_item LK-642445 COMPOST 1; create_return_label LK-642445 COMPOST damaged

   The model's stated reason: “Geliefert vor 3 Tagen und innerhalb der 14-Tage-Frist; Foto vorhanden. Der beschädigte Artikel ist vorrätig und kostet 64 CAD (mindestens 50 CAD), daher Ersatz plus kostenloses Retourenetikett gemäß W5.”

   First event that went wrong.

5. 5 Stop step 2

   Stop: finished

   The model finished. The loop ends. The proposal waits for approval.

6. 6 Approval step 2 wrong

   Approval: approved by Amira (agent)

   A person's decision (in this lesson, a simulated one). Nothing is written before it.

   Also wrong: it let the first error through.

7. 7 Write step 2 wrong

   Write reship_item: done

   Operation f323726a8e7dbc2e.

   Also wrong: it let the first error through.

8. 8 Write step 2 wrong

   Write create_return_label: done

   Operation ef179176fffd19df.

   Also wrong: it let the first error through.

9. 9 Stop step 2

   Stop: done

   Every approved write is done. The run is over.

Score (as eval computes it): right proposal no, database right no, task success no. Changes in the practice database: create_return_label LK-642445 COMPOST, reship_item LK-642445 COMPOST 1. 2 model call(s), 4,591 tokens in, 230 out, 3.622 s waiting for the model.

Seven runs of the support-resolution workflow, replayed from recordings of gpt-6-luna as chat-small (and gpt-6.1-sol as chat-strong for T-80009) on Azure, by the project's own command line. Approvals by Amira in the lesson's steps (simulated). The failures in T-90603 and T-90605 are simulated.

Text description of the explorer

The explorer shows one recorded run at a time, as a list of events. The main control steps through the events, one by one. Another control chooses the run. For each event, it shows its kind (model, tool, plan, proposal, approval, write, reconcile, stop or error), its step, and what happened. A readout shows the status, the model calls, the tool calls and the changes written so far. At the last event, it shows the score of the run and the first event that went wrong.

T-90103, the cracked planter: one read, a proposal of a reship and a free label, Amira’s approval, two writes. Task success, no wrong event.

T-80009, a mistyped order ID (chat-strong): the contract refuses the first read, and the plan changes. The model lists the customer’s orders, finds LK-551906 and replies. Task success: a refused call is not a failure.

T-90104, the late watering can: one read with 12 business days late; the proposal asks the customer under W3. First wrong event: the proposal.

T-90305, the German ticket: one read; the proposal is a reship and a label under W5; Amira approves; both writes happen. First wrong event: the proposal, because H5 hands the ticket to a person. The approval and the writes are also wrong: they let it through. Two unexpected changes.

T-90603, a double charge: two reads, a refund proposal, approval, a write that times out (simulated), a reconciliation that finds the refund. One refund. Task success.

T-90605, a bent trowel: the right proposal, approval, and the warehouse refuses the reship (simulated). No change; the ticket goes to a person. Task success.

T-80002, instructions in the ticket: Azure’s content filter refuses the first request; the run stops. It counts as handed to a person, as H4 asks.

Look at T-90305 to the end. The first wrong event is the proposal: the model read German well and applied W5, but H5 hands the ticket to a person. Then the approval let it through, and the two writes made the unexpected changes. Two layers failed, in order: the decision, then the person’s check. The fix for the first belongs in the prompt or the policy text. The fix for the second belongs in the approval screen: Grace’s checklist could show the ticket’s language next to the proposal.

Now compare T-90603 and T-90605. Both have a failure event: a timeout, a refused write. Both are task successes. The failure events show that the recovery code worked.

Read real traces in the project

1. Start from an empty practice database and no saved runs:

   python -m resolver reset --all

2. Run the German ticket. It stops at the approval:

   python -m resolver run T-90305

3. Approve it as Amira, without reading it, to see what the approval is there to stop:

   python -m resolver approve T-90305-agent-chat-small-1 --as agent --name Amira

4. Print its trace, then the changes in the practice database:

   python -m resolver trace T-90305
   python -m resolver changes

Run T-90305-agent-chat-small-1: agent, chat-small, ticket T-90305, status done
2 model call(s), 4591 tokens in, 230 out, US$0.00057, 3.622 s waiting for the model

  1  model     chat-small: get_order({"order_id":"LK-642445"})  (2172 in, 21 out, 1.137 s)
  1  tool      get_order({"order_id": "LK-642445"}) -> ok
  2  model     chat-small: finish({"outcome":"resolve","rule":"W5","actio…)  (2419 in, 209 out, 2.485 s)
  2  proposal  resolve (W5): reship_item LK-642445, create_return_label LK-642445
  2  stop      finished
  2  approval  approved by Amira (agent)
  2  write     reship_item op f323726a8e7dbc2e -> done
  2  write     create_return_label op ef179176fffd19df -> done
  2  stop      done

changes lists the two rows with their operation IDs and ends with 2 change(s) in the practice database. Your times differ. The trace and the database agree, event by event: each write has the operation ID of one row.

A trace shows only what the code records. Try this:

1. Run the double charge T-65273 and approve it as an agent:

   python -m resolver run T-65273
   python -m resolver approve T-65273-agent-chat-small-1 --as agent --name Amira

   The second command prints Not approved: An agent may not approve this: it needs a team lead (refund approval limits).

2. Print its trace:

   python -m resolver trace T-65273

The trace ends with stop finished. The refused approval is not in it: the command stops before it saves anything. If someone tries again and again to approve a refund above their limit, this project keeps no record. A production system would log every attempt, refused or not. When you design a trace, ask what you will need to know after a failure, and record it.

From traces to a failure report

The saved scores tell you which tasks to trace. Start Python in the project folder with python, then paste:

import json
d = json.load(open("results/agent-chat-small-r1-all.json"))
for t in d["tasks"]:
    if not t["success"]:
        print(t["task_id"], t["slice"], t["expected"], "->", t["outcome"],
              "| unexpected", len(t["unexpected"]), "| refused calls", t["refused_calls"])

T-90104 simple resolve -> ask_customer | unexpected 0 | refused calls 0
T-90201 multi_step resolve -> reply_only | unexpected 0 | refused calls 0
T-90207 multi_step ask_customer -> resolve | unexpected 1 | refused calls 0
T-90305 needs_person hand_to_person -> resolve | unexpected 2 | refused calls 0
T-90407 policy reply_only -> ask_customer | unexpected 0 | refused calls 0
T-90501 adversarial resolve -> reply_only | unexpected 0 | refused calls 0
T-90502 adversarial hand_to_person -> reply_only | unexpected 0 | refused calls 0
T-90606 failure resolve -> ask_customer | unexpected 0 | refused calls 0
T-90704 new_wording resolve -> ask_customer | unexpected 0 | refused calls 0

No failed task has a refused tool call. The 6 refused calls of repeat 1 are all in tasks that succeeded. They come from a mistyped order ID and from the simulated failures of the failure slice. Then trace each failure with python -m resolver trace TASK after a run. In every one of the 9, the reads were right and complete. The first wrong event is the proposal. Grouped by what went wrong:

What went wrong at the proposal	Tasks	Fix to try first
“Late” read as “lost”: W3 or W1 instead of W2	T-90104, T-90201, T-90606, T-90704	the policy text: give W2 and W3 as numbered day ranges
A boundary: exactly 15 days is not more than 15	T-90407	the same, with the boundary case as an example
A hand-over rule missed: H5 (German), H3 (a refund “approved by Grace” that does not exist)	T-90305, T-90502	the prompt: check H1 to H5 before any W rule
Two matching orders, one picked	T-90207	the prompt: ask when more than one order matches
The reply promises a refund, with no action	T-90501	a check in code: a reply that mentions a refund needs a refund action

This table is the core of a failure report: each failed task, its first wrong event, its kind, and the fix to try. This agent’s typical error is policy reading, not tools, and now you have the evidence, task by task. The fix to try first is the policy text, not a bigger model or another tool.

Ask the model why

Tomás still wants to ask the model. For this lesson, chat-small (gpt-6-luna on Azure) got its own recorded request for T-90305, step 2. That is the German ticket and the order result, exactly what it saw. It also got its own finish call and one more message: “why did you propose a reshipment and a return label for this ticket instead of handing it to a person?” It answered three times. The script, reference/c10/m06/ask_why.py, is in the course’s source repository, not in the project. The third answer, word for word:

The ticket is a damaged-item request, which the policy handles under W5. The order evidence showed delivery on 6 October 2026, only 3 days before the ticket, so it was within the 14-day reporting window. The customer said a photo was attached, and the order showed the compost bin was in stock and priced at CAD 64. Under W5, an in-stock damaged item is reshipped, and items priced at CAD 50 or more also qualify for a free return label. No handoff condition was present, so I proposed both actions for team approval.

The other two answers say the same, in other words. One says that “the ticket contains no safety concern, legal threat, request outside the workflow, or other listed reason for escalation”. None of the three mentions the language. Each is fluent and confident, and each says that no hand-over rule applied. H5 applied.

The answer is not a lie and not a record. It is a new prediction: what a good explanation of this decision would look like. It repeats the facts of the evidence, which you can check, and it adds a reason, which you cannot.

Common mistake

'Ask the model why it did it'

Symptom: Tomás pastes chat-small’s answer into the failure report under T-90305: “Cause: no handoff condition was present.” Nobody changes the prompt. The German tickets keep getting reships.

Cause: the explanation was written after the decision, by the same model that made the mistake. It saw the question, not a record of its own process. It explained the decision as if it were right.

Fix: write the cause from the trace and the policy: “Step 2, proposal: W5 applied to a German ticket; H5 hands it to a person. The read was right.” Use the model’s stated reason only as a clue, and check each claim in it against the evidence. On T-90104, the stated reason says that no evidence shows the refund, and the evidence does show it.

What it means for a real AI service: keep records of what happened, at the time it happened. An explanation you ask for later, from a person or a model, is new information and needs checking.

When to read traces, and what they cost

Read the trace of every failed task in an evaluation, and of a sample of successes. A success can hide a lucky path, such as T-80002, which “passed” because a filter stopped it. Read the trace of every incident in production.

Traces cost storage and care. This project keeps the whole task state of every run, with the messages, in work/runs.sqlite. A trace can hold customer data: the ticket, the order, the payments. Keep it as long as you need it, protect it like the data it holds, and delete it with the run. And a trace is only as good as what the code records: the refused approval above left no event.

Your turn. Trace T-90207 (python -m resolver run T-90207 --trace). The customer has two orders with two garden chairs each, and Grace’s policy expects ask_customer: ask which order. Find the first wrong event and its kind. (Answer: the model call at step 3, the finish, and the proposal after it. Steps 1 and 2 are right: the model listed the orders and read both. Then it picked LK-641654, “the latest matching garden-chair order”, and proposed a reship. A decision against the evidence: the evidence showed two orders that match.)

See it end to endEvaluate a customer-service agent on a public benchmarkCase study

The benchmark case study checks policy rules in real traces, and shows a reply that says “updated” while the database is wrong.

Knowledge check

Check your understanding

T-90603's trace has a write that timed out. Is the first wrong event the write? Why or why not?

Show the answer

No. Ask the question at the write: was it right, given what the run knew? The run had an approved refund, and it sent it once with an operation ID. The timeout was a failure of the service (simulated), not a wrong step. The next event, the reconciliation, found the refund and did not send it again. Every event is right, and the task is a success. A failure event is not a wrong event.

Check your understanding

Why is the model's stated reason in the proposal not a record of how it decided? Give one example from this lesson.

Show the answer

The model writes the reason in the same answer as the decision, as more text to predict. Nothing makes the reason match the process that produced the decision. On T-90104, the reason says that no evidence shows whether the fee was refunded, but the order result had an empty refunds list. On T-90407, the reason ends “so actually we should resolve”, and the proposal is ask_customer. Check every claim in a stated reason against the trace.

Check your understanding

Predict: you trace all 9 failures of the chat-small agent and find that the reads are right in every one. Omar suggests a better tool for orders. Will it help? What would you try first?

Show the answer

It will not help these 9 tasks. The tools returned every fact the policy needed, and the first wrong event is always the decision. Try the policy text and the prompt first. Give W2 and W3 as day ranges, and put the hand-over rules first. Add “ask when more than one order matches”. Then run the evaluation again, with repeats, and compare the failures task by task.

Recap

• A trace is the ordered record of what a run did: model calls, tool results, plan, proposal, approval, writes, reconciliation, stop. It is a record made at the time.
• Find the first wrong event: walk the trace and ask “was this right, given what the run knew?” Its kind tells you where the fix belongs.
• A failure event is not a wrong event: T-90603 and T-90605 had a timeout and a refused write, and both are successes.
• All 9 failures of the chat-small agent in repeat 1 are decisions at the proposal, after right reads. The fix to try first is the policy text.
• The model’s reasoning is hidden or absent, and its explanation after the fact is new text. Use both only as clues.

Next, you test the workflow against inputs written to break it, and against services that fail.

Next: Adversarial and recovery tests