Skip to content
Français

Read a run

  • Lesson 12
  • 35 min
  • Module 4: Find agent failures in Grafana

Where we are

In Module 3, we made the workflow record every model call, tool call, write and approval as a span, and we joined the two traces of a run with a link. But having traces is not the same as reading them. Sam asks which step was slow and which step cost the money, and the totals cannot say. In this lesson, we read the waterfall of one agent run, step by step. We start with a parcel's tracking page.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Read the waterfall of one agent run: what each row is, which step was slow, and which step cost the money. Find where the run waited and which step made a change. Then read your own run the same way.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project ~/projects/observable-workflow as Module 3 left it: the same as the snapshot end-of-m03. python -m pytest -q ends with 86 passed, 1 skipped, 1 deselected.
  • A terminal in the project, with the virtual environment active.

This lesson adds no file. You need no account and no Docker. Grafana is optional. Tested on macOS (Apple silicon) with Python 3.12.13. The commands are the same on Windows and Linux, and the course did not run them there.

You order a lamp. The shop gives you a tracking page. It has one line per event. “Picked up, Monday 14:02.” “Arrived at the depot, Monday 18:40.” “Out for delivery, Wednesday 08:15.” “Delivered, Wednesday 11:30.” You never see the lorry, but you can read the trip.

You read it in a fixed way. The lines are in time order. The space between two lines tells you where the parcel waited: a whole day at the depot, and nothing happened. If the parcel came late, you do not blame “the trip”. You find the line where the time went.

A trace of an agent run is the same kind of page, with one difference: you see it as a picture. Each step is a bar on a time line.

In Module 3, run R-1014-040 became a trace. This is the double refund of ticket T-90104. Its first lines give the totals:

Run R-1014-040 | ticket T-90104 | chat-strong (agent_structured) | release 2.1.0 | language en
2 model call(s), 1 read(s), 2 write call(s), 4586 tokens in, 265 out, US$0.01182

Sam, who approved the refund, asks three questions. “Which step took the time? Which step cost the money? And why did the customer wait until the next morning?” The totals cannot answer any of them. US$0.01182 is a sum of two model calls. “2 write call(s)” does not say which one was the second refund. And the wait is not in the totals at all.

A total is like a parcel that “took three days”. To fix anything, you need the step.

A waterfall is the picture of one trace: one row per span, in time order. Each row has a bar on a shared time line. The bar starts when the span started, and its length is how long the span lasted. The name comes from the look: the bars step down and to the right, like water down a stair.

Five things in a waterfall need a name:

  • The offset is when a span started, counted from the start of its trace. 2.4 s means 2.4 seconds after the run began.
  • The duration is how long the span lasted. A long bar is a slow step.
  • The indent shows the parent. A row that is moved to the right is a child of the row above it. The first row, with no parent, is the root span: here, the whole agent run.
  • An error is a span with the status ERROR. The detail after it says why, for example timeout.
  • A gap is time with no span. Nothing ran. In an agent run, a gap is often a wait: for a person, or for the next message.

Two more things are special to agent runs. A retry is the same tool called again, after an error. Its idempotency key tells you whether it is the same operation or a new one. And a link joins the second trace of a run to the first one, across a gap.

Run R-1014-040 as a waterfall, one row per span, in time order. Trace 1, Wednesday 14 October 17:52:00, 6.6 s. Row 1, the run span invoke_agent support-resolver, resolve, rule W2, 6.56 s, a bar across the whole axis. Then chat chat-strong, 2160 in and 47 out, US$0.00479, 2.39 s. Then execute_tool get_order, ok, LK-793996, 0.00 s. Row 2, chat chat-strong, 2426 in and 218 out, US$0.00703, 4.17 s. Row 3, approval.request, needs agent, 0.00 s. Then 4, a dashed box: 15 h 25 min, no span, the run waits for a person. Trace 2, Thursday 15 October 09:17:12, 10.0 s. Row 5, approve support-resolution, approved by usr-sam, agent, limit 100, 10.01 s, with a span link back to approval.request. Row 6, in red, execute_tool request_refund, ERROR timeout, key 797fbb5bbb9f8a7e, 10.00 s. Row 7, execute_tool request_refund, done, key 797fbb5bbb9f8a7e-retry1, 0.00 s. Then execute_tool send_reply, sent, 0.01 s. Notes: 1, the run span is the agent's whole work in trace 1, and every other span of trace 1 is its child. 2, the most expensive step, US$0.00703 of US$0.01182, 59 percent, and trace 1's slowest step. 3, the run asks for a person and trace 1 ends. 4, a gap of 15 h 25 min with no span. 5, trace 2 starts with Sam's decision and links back to approval.request. 6, an error and the run's slowest step: the refund call timed out after 10.00 s. 7, a retry: the same refund with a new key, and a new key means a new refund.
The anatomy of a waterfall: run R-1014-040, both traces. Replayed model decisions; the timeout and the retry are a practice incident we caused on purpose.
Text description of the diagram

Trace 1 has five rows. The first is the run span, 6.56 seconds, the agent’s whole work. Under it are two model calls of 2.39 and 4.17 seconds, with the order lookup between them, and then the request for approval. The second model call cost US$0.00703, the most of any step. Then comes a dashed box: 15 hours 25 minutes with no span, while the run waited for a person. Trace 2 starts with Sam’s approval, linked back to the approval request. Its first refund call is red: an error, a timeout after 10.00 seconds. The next row is the same refund with a new key, done. The last row is the reply, sent. Each trace has its own time axis.

Worked example: read R-1014-040 row by row

Section titled “Worked example: read R-1014-040 row by row”

Print the whole waterfall of the double refund from the saved week:

Terminal window
python -m obs trace R-1014-040 --data data/telemetry/week
Run R-1014-040 | ticket T-90104 | chat-strong (agent_structured) | release 2.1.0 | language en
2 model call(s), 1 read(s), 2 write call(s), 4586 tokens in, 265 out, US$0.01182
Trace 1 f0433c4403e2767e1d69ece18a947526 Wed 14 Oct 17:52:00 6.6 s
0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2
0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out US$0.00479 stop
2.4 s | = | 0.00 s execute_tool get_order ok LK-793996
2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out US$0.00703 stop
6.6 s | =| 0.00 s approval.request needs agent
... waited 15 h 25 min for a person (this trace has a link to 78d246ee22e5f5ef in trace f0433c44...)
Trace 2 15ba0d9388d1b9e001af373f21c4b955 Thu 15 Oct 09:17:12 10.0 s
0.0 s |========================| 10.01 s approve support-resolution approved by usr-sam (agent, limit 100)
0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e
10.0 s | =| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e-retry1
10.0 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved

Read it in four passes, as you read the tracking page.

Pass 1: the shape. There are two traces with a gap between them. Trace 1 ends with approval.request, so the run asked a person. Trace 2 starts with approve, so a person decided. In this workflow, a second trace always means a person’s decision.

Pass 2: the slow step. Look for the longest bar under the root. In trace 1, the second model call took 4.17 s of 6.56 s: 4.17 ÷ 6.56 = 64%. In trace 2, the first refund call took 10.00 s, and it is red. That is the slowest step of the whole run, and it is an error. The model spans last as long as the recorded calls did (Module 2’s run clock). The 10 seconds are the simulated payment service’s timeout.

Pass 3: the expensive step. Only model calls cost money here. Each cost is tokens × the price of chat-strong in the agents course’s price table. That is US$2.00 per million tokens in, and US$10.00 per million out.

Model call Tokens in × price Tokens out × price Cost
1 2,160 × 2.00 ÷ 1,000,000 = 0.00432 47 × 10.00 ÷ 1,000,000 = 0.00047 US$0.00479
2 2,426 × 2.00 ÷ 1,000,000 = 0.00485 218 × 10.00 ÷ 1,000,000 = 0.00218 US$0.00703

The second call is the expensive step: 0.00703 ÷ 0.01182 = 59% of the run. It read more (the order details from get_order were added to the conversation) and it wrote more (the proposal and the draft reply). Output tokens cost five times as much as input tokens.

Pass 4: the gaps and the changes. The longest time in this run is not a bar. It is the gap: 15 h 25 min, from Wednesday 17:52 to Thursday 09:17. For the customer, this was the whole delay. A dashboard of span durations would never show it, because a gap is not a span. The workflow records the wait as an attribute instead (larkfield.approval.wait_s, from Module 3).

Then look for writes. Trace 2 has two request_refund rows. The first timed out. The second has a new key, 797fbb5bbb9f8a7e-retry1, and is done. Its parent is approve support-resolution, not a model call. So no model asked for the second refund. The workflow’s retry code sent it, with a new key, and the payment service made a second refund.

Now Sam has the answers. The slow step is the refund call that timed out. The expensive step is the second model call. The wait was 15 h 25 min for a person. And the step that caused the second refund is the retry, not the model.

Explore

Read a run: the slow step, the expensive step and the wait

Run
R-1014-040 · ticket T-90104 · chat-strong · release 2.1.0
Evidence
replayed decision; simulated timeout and naive retry
Outcome
resolved
Steps
2 model calls, 4 tool calls
Tokens in / out · cost
4586 / 265 · US$0.01182
Slowest step
execute_tool request_refund, 10.00 s (at Thu 09:17:12)
Most expensive step
chat chat-strong, US$0.00703 (59% of the run)
Wait for a person
15 h 25 min
Flags (obs)
duplicate_write

Choose a run, then open a row of the waterfall to read its attributes.

Open a row to read the span’s attributes.

Trace 1 · Wed 14 Oct 17:52:00 · 6.6 s

  1. invoke_agent support-resolver resolve · rule W26.56 s
    gen_ai.request.model
    chat-strong
    gen_ai.usage.input_tokens
    4586
    gen_ai.usage.output_tokens
    265
    larkfield.cost.usd
    0.011822
    larkfield.proposal.outcome
    resolve
    larkfield.proposal.rule
    W2
    larkfield.stop_reason
    finished
    larkfield.release
    2.1.0
  2. chat chat-strong 2160 in / 47 out · US$0.004792.39 s
    gen_ai.request.model
    chat-strong
    gen_ai.response.model
    gpt-6.1-sol-2026-09-29
    gen_ai.usage.input_tokens
    2160
    gen_ai.usage.output_tokens
    47
    gen_ai.response.finish_reasons
    stop
    larkfield.cost.usd
    0.00479
    larkfield.model.requested
    get_order(order_id)
  3. execute_tool get_order ok0.00 s
    gen_ai.tool.name
    get_order
    larkfield.tool.kind
    read
    larkfield.tool.status
    ok
    larkfield.order.id
    LK-793996
  4. chat chat-strong 2426 in / 218 out · US$0.007034.17 s
    gen_ai.request.model
    chat-strong
    gen_ai.response.model
    gpt-6.1-sol-2026-09-29
    gen_ai.usage.input_tokens
    2426
    gen_ai.usage.output_tokens
    218
    gen_ai.response.finish_reasons
    stop
    larkfield.cost.usd
    0.007032
    larkfield.model.requested
    finish(-)
  5. approval.request needs agent0.00 s
    larkfield.approval.needs_role
    agent
    larkfield.approval.limit
    100

waited 15 h 25 min for a person: a gap, not a span. Trace 2 starts with a link to approval.request in trace 1.

Trace 2 · Thu 15 Oct 09:17:12 · 10.0 s

  1. approve support-resolution approved by usr-sam (agent)10.01 s
    larkfield.approval.approver
    usr-sam
    larkfield.approval.role
    agent
    larkfield.approval.limit
    100
    larkfield.approval.wait_s
    55505.4
    larkfield.outcome
    resolved
    larkfield.writes_done
    1
    larkfield.release
    2.1.0
  2. execute_tool request_refund ERROR timeout · 9.95 RFD-LATE · key 797fbb5bbb9f8a7e10.00 s
    gen_ai.tool.name
    request_refund
    larkfield.tool.kind
    write
    larkfield.write.outcome
    timeout
    larkfield.idempotency_key
    797fbb5bbb9f8a7e
    larkfield.operation.id
    797fbb5bbb9f8a7e
    larkfield.order.id
    LK-793996
    larkfield.refund.amount
    9.95
    larkfield.refund.reason
    RFD-LATE
    error.type
    timeout
  3. execute_tool request_refund done · 9.95 RFD-LATE · key 797fbb5bbb9f8a7e-retry10.00 s
    gen_ai.tool.name
    request_refund
    larkfield.tool.kind
    write
    larkfield.write.outcome
    done
    larkfield.idempotency_key
    797fbb5bbb9f8a7e-retry1
    larkfield.operation.id
    797fbb5bbb9f8a7e
    larkfield.order.id
    LK-793996
    larkfield.refund.amount
    9.95
    larkfield.refund.reason
    RFD-LATE
  4. execute_tool send_reply sent0.01 s
    gen_ai.tool.name
    send_reply
    larkfield.tool.kind
    write
    larkfield.tool.status
    ok
    larkfield.reply.status
    sent

Each trace has its own time scale.

Runs of the saved week, with their real spans: replayed model decisions, simulated faults. Each trace has its own time scale; the wait is a gap.
Text description of the explorer

Choose one of four runs of the saved week. The readout gives the run, how its evidence was made, and its outcome. It also gives the steps, the tokens and the cost, the slowest step, the most expensive step and the wait for a person. Below it is the waterfall, one row per span. Open a row to read its attributes. R-1014-040, the double refund: the slowest step is execute_tool request_refund, 10.00 s at Thu 09:17:12. The most expensive is chat chat-strong, US$0.00703, 59% of the run. The wait is 15 h 25 min. Its second request_refund row has the key 797fbb5bbb9f8a7e-retry1 and its parent is the approval, not a model call. R-1012-034, the same ticket on chat-small: one trace, 3.00 s, US$0.00055, no wait; it asked the customer. For R-1018-010, on the local model gemma3:4b, the readout shows 7 model calls in 19.7 s. Its slowest step is a model call of 6.22 s. It has an error on get_payments, and no cost, because the local model is priced at US$0 (an assumption). R-1018-022: the slowest step is a failed get_order call that timed out after 10.00 s.

Open the second request_refund row of R-1014-040 in the explorer. Its attributes show larkfield.write.outcome: done and the new key. Then open the first one: error.type: timeout. The answer to the prediction is in those two rows.

On Monday, before Omar’s switch to chat-strong for hard tickets, the same ticket ran on chat-small. Read it:

Terminal window
python -m obs trace R-1012-034 --data data/telemetry/week
Run R-1012-034 | ticket T-90104 | chat-small (agent) | release 2.0.0 | language en
2 model call(s), 1 read(s), 0 write call(s), 4565 tokens in, 179 out, US$0.00055
Trace 1 3a6cd2cce257c77ab2a0ae8dd9d5841d Mon 12 Oct 15:31:37 3.0 s
0.0 s |========================| 3.00 s invoke_agent support-resolver ask_customer rule W3
0.0 s |======== | 1.05 s chat chat-small 2163 in / 21 out US$0.00023 tool_calls
1.1 s | = | 0.01 s execute_tool get_order ok LK-793996
1.1 s | ================| 1.95 s chat chat-small 2402 in / 158 out US$0.00032 tool_calls
3.0 s | =| 0.00 s execute_tool send_reply sent
Outcome: asked_customer

The shape is almost the same: two model calls with one lookup between them. But there is one trace, not two. The model chose rule W3 and asked the customer a question, so no approval and no write were needed. The run cost US$0.00055, about 22 times less than R-1014-040.

Is cheaper better here? No. By the course’s labels, this ticket needed a refund (rule W2), and chat-small asked the customer instead. The waterfall shows what the run did and what it cost. It does not show whether the decision was right. That needs a review or a label, as Module 7 shows.

Larkfield’s local model: slow, and “free”

Section titled “Larkfield’s local model: slow, and “free””

On the weekend, Tomás sent 30% of tickets to the local model gemma3:4b. Read one of its runs:

Terminal window
python -m obs trace R-1018-010 --data data/telemetry/week
Run R-1018-010 | ticket T-90602 | gemma3:4b (agent_structured) | release 2.1.1 | language en
7 model call(s), 3 read(s), 0 write call(s), 17582 tokens in, 542 out, US$0.00000
Trace 1 bf5b941801388658eaa36cf0bba32133 Sun 18 Oct 11:15:00 19.7 s
0.0 s |========================| 19.72 s invoke_agent support-resolver reply_only rule double charge
0.0 s |== | 1.68 s chat gemma3:4b 1898 in / 45 out US$0.00000 stop
1.7 s | = | 0.00 s execute_tool get_order ok LK-645112
1.7 s | == | 1.99 s chat gemma3:4b 2231 in / 41 out US$0.00000 stop
3.7 s | = | 0.20 s execute_tool get_payments ERROR service_unavailable LK-645112
3.9 s | === | 1.50 s chat gemma3:4b 2322 in / 42 out US$0.00000 stop
5.4 s | === | 1.69 s chat gemma3:4b 2412 in / 47 out US$0.00000 stop
7.1 s | = | 0.00 s execute_tool search_policy ok
7.1 s | === | 2.08 s chat gemma3:4b 2768 in / 42 out US$0.00000 stop
9.1 s | ======== | 6.22 s chat gemma3:4b 2858 in / 195 out US$0.00000 stop
15.4 s | ======| 4.36 s chat gemma3:4b 3093 in / 130 out US$0.00000 stop
19.7 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved

This run has seven model calls in 19.7 s. The slowest step is a model call of 6.22 s. The cost is US$0 in every row, because the course prices the local model at zero. That is an assumption: the computer, its power and its memory are not counted. So for this run, “the expensive step” can only mean time. With a price of zero, a cost panel would call the slowest model the cheapest one.

There is also a red row: get_payments failed with service_unavailable at 3.7 s, and no later get_payments call worked. Yet the run ended with a reply and the outcome resolved. Keep this row in mind. In the third lesson of this module, it has a name.

This part is optional, and it needs Docker. Grafana shows only what the workflow sends to it, so first send it a week. The week driver of The Collector replays the 316 runs with your project’s code, through the Collector in Grafana’s container:

  1. Start Grafana from the project folder: docker compose -f grafana/compose.yaml up -d. The first start also downloads the image: about 0.9 GB, 3.5 GB on disk. On the course team’s laptop, the stack was then ready in 4 to 52 seconds, the longest when the laptop was busy.
  2. Send the week: python -m scenarios week --otlp --out work/week. It replays the week at 1,000 times speed, so one practice day lasts about 86 seconds. On the course team’s laptop, it took 638 to 639 seconds in two runs.
  3. Open Grafana at http://localhost:3031.
  4. When you no longer need it, stop it and delete its data: docker compose -f grafana/compose.yaml down -v.

The same waterfall is in Grafana’s trace view. Open Explore, choose the data source Tempo and the query type TraceQL, and search for every trace of the ticket:

{ span.larkfield.ticket.id = "T-90104" }

Click a trace ID to open its waterfall. Grafana draws the same rows, bars and indents, and an overview line above them. It marks an error span with a red icon. Its filter buttons Errors and High latency do passes 2 and 4 for you: they keep only the red rows, or only the long ones. Grafana shows one trace at a time, so you follow the link from trace 2 to trace 1 yourself. Your times differ from the saved week’s: each replay has its own run clock.

Grafana's trace view for 'support-workflow: approve support-resolution', duration 10.03 s, 4 spans. The span list: approve support-resolution (10.03s) with a link icon at its right; under it execute_tool request_refund (10.02s) marked with a red error icon, execute_tool request_refund (5.68ms) and execute_tool send_reply (3.58ms).
Trace 2 of the double refund in Grafana's trace view, from the course team's run of the replayed week. The red row is the refund that timed out; the link icon on the first row opens trace 1. The start time is hidden: it showed the day of the course team's run.

Now make the double refund yourself, and read its waterfall before and after the approval.

  1. Run the ticket the way release 2.1.0 ran it:

    Terminal window
    python -m resolver run T-90104 --variant agent_structured --model chat-strong

    The last line:

    Nothing has changed yet. Next: python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent (or: reject)
  2. Read your run’s waterfall now, while it waits:

    Terminal window
    python -m obs trace T-90104
    Run T-90104-agent_structured-chat-strong-1 | ticket T-90104 | chat-strong (agent_structured) | release 2.0.0 | language en
    2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.00000
    Trace 1 … 6.6 s
    0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2
    0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out - stop
    2.4 s | = | 0.00 s execute_tool get_order ok LK-793996
    2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out - stop
    6.6 s | =| 0.00 s approval.request needs agent
    Outcome: waiting_approval

    One trace, and the outcome waiting_approval. Your trace ID and time differ. The cost shows -: your spans have no cost yet, because cost on spans arrives in Module 5. The saved week was recorded with it. The release says 2.0.0, the default of OBS_RELEASE.

  3. Approve it as Sam, with the slow payment service and the naive retry:

    Terminal window
    python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --name Sam --fault request_refund:timeout_after_write --naive-retry
    request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 2 call(s))
    Status: done
    Reply sent. Ticket: resolved.
  4. Read the waterfall again:

    Terminal window
    python -m obs trace T-90104

    The end of the output:

    ... waited 0 min 00 s for a person (this trace has a link to … in trace …)
    Trace 2 … 10.0 s
    0.0 s |========================| 10.00 s approve support-resolution approved by usr-sam (agent, limit 100)
    0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e
    10.0 s | =| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e-retry1
    10.0 s | =| 0.00 s execute_tool send_reply sent
    Outcome: resolved
  5. Do the four passes on your own output, and check five things. The shape is two traces. The slow step is the refund that timed out. The expensive step is the second model call (2,426 tokens in, 218 out). The gap is about zero, because you approved at once. And the second refund has the key ending in -retry1.

The steps are the same as the saved week’s, because the model decisions are the same recordings. Only the clock and the wait differ.

Common mistake

The waterfall has only one trace, and no refund

Symptom: python -m obs trace T-90104 shows trace 1 only, with Outcome: waiting_approval. There is no request_refund row, so you cannot find the second refund.

Cause: the run is still waiting for a person. Trace 2 does not exist until somebody approves or rejects the proposal. This is the waterfall of a run that has not finished.

Diagnose: look at the last row and the outcome. approval.request as the last row, and waiting_approval, mean “waiting”.

Recover: approve the run (step 3 above), then read the trace again. In real use, a run can wait for hours, as R-1014-040 did. Do not read a waiting run as a failed run.

Common mistake

A long bar in trace 2 looks shorter than a short bar in trace 1

Symptom: in a picture of a run with two traces, a 10-second bar looks no longer than a 6-second bar.

Cause: each trace has its own time scale. obs, Grafana and the explorer all scale a bar to its own trace. A shared scale would put 15 hours of gap between the traces, and every bar would be too thin to see.

Recover: compare durations by their numbers, not by their bars, across traces. Compare bars only inside one trace.

Your turn: find the slow step of another run

Section titled “Your turn: find the slow step of another run”

Change one condition: a run where a tool call, not a model call, is the slow step.

  1. Predict: run R-1018-022 is gemma3:4b on ticket T-90601, on Sunday evening. Its root span lasted 22.5 s. Will the slowest step be a model call or a tool call? How much of the 22.5 s will it take?

  2. Read it:

    Terminal window
    python -m obs trace R-1018-022 --data data/telemetry/week
  3. Do the four passes, and compare with your prediction.

Check your prediction

The course’s output (replayed decisions, a simulated order-service timeout):

Trace 1 cec6ea30d7d9124ef43d05a927af8e14 Sun 18 Oct 17:00:31 22.5 s
0.0 s |========================| 22.51 s invoke_agent support-resolver reply_only rule W1
0.0 s |== | 1.70 s chat gemma3:4b 1894 in / 45 out US$0.00000 stop
1.7 s | =========== | 10.00 s execute_tool get_order ERROR service_timeout LK-645001
11.7 s | ======= | 6.43 s chat gemma3:4b 1986 in / 202 out US$0.00000 stop
18.1 s | = | 0.00 s execute_tool get_customer_orders ok
18.1 s | =====| 4.38 s chat gemma3:4b 2253 in / 130 out US$0.00000 stop
22.5 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved

The slowest step is a tool call: get_order waited 10.00 s and then failed with service_timeout. That is 10.00 ÷ 22.51 = 44% of the run. The model then looked up the customer’s orders instead, and replied. There is one trace and no gap, because the reply needed no approval. The cost is US$0 (the local model’s assumed price). So in this run, the slow step is an error, and nothing in the final outcome says so.

When to read a waterfall, and what it costs

Section titled “When to read a waterfall, and what it costs”

Read a waterfall when you have one run and a question about it. Why was it slow? Why did it cost what it did? Why did it change what it changed? It is the most detailed record you have of a run’s path.

It is the wrong tool for a whole week. One careful read takes a few minutes, and the saved week has 316 runs in 434 traces. A waterfall also cannot tell you whether a decision was right. And it does not show the model’s words, because content capture is off. And a trace may be missing: the Collector’s tail sampling keeps some traces only. The next lesson searches many runs at once.

Check your understanding

In the waterfall of R-1014-040, which row is the slow step, which is the expensive step, and where is the customer's long wait? Give the number for each.

Show the answer

The slow step is the first execute_tool request_refund row in trace 2: 10.00 s, an error (timeout). The expensive step is the second chat chat-strong row in trace 1: US$0.00703 of US$0.01182, 59%. It had 2,426 tokens in and 218 out. The customer’s long wait is not a row. It is the gap of 15 h 25 min between the two traces, while the run waited for a person.

Check your understanding

Which span caused the second refund of R-1014-040, and how do you know that no model asked for it?

Show the answer

The second execute_tool request_refund span in trace 2, with the key 797fbb5bbb9f8a7e-retry1 and the outcome done. Its parent is approve support-resolution, the person’s decision, and it starts right after the first refund call timed out. No chat span comes between them. So the workflow’s retry code sent it, not the model. Its new key made the payment service treat it as a new refund.

Check your understanding

Run R-1018-010 cost US$0.00000. Why can you not call it the cheapest run of the weekend?

Show the answer

Its cost is zero only because the course prices the local model gemma3:4b at US$0, an assumption. The hardware, power and memory are not counted. By time, it was one of the slowest runs: seven model calls in 19.7 s. Without a real price for the local model, the cost of a run on it says nothing.

Recap

  • A waterfall is one row per span, in time order. Read the offset, the duration, the indent (the parent), errors and gaps.
  • Read in four passes: the shape (one trace or two), the slow step, the expensive step, then the gaps and the writes.
  • In R-1014-040, the slow step is the refund that timed out (10.00 s). The expensive step is the second model call (US$0.00703, 59%). The wait is a gap of 15 h 25 min. The retry with a new key, not the model, caused the second refund.
  • The local model’s cost is zero by assumption, so its “expensive step” can only be measured in time.

Your project is unchanged: the same as the snapshot end-of-m03. You can now read one run and find its slow step, its expensive step and its changes. But the saved week has 316 runs, and you cannot open them one by one. Next, you search runs by their structure with TraceQL. You look for many calls to one tool, a write without an approval, and an error followed by a resolved outcome.

Next: Search runs with TraceQL