Skip to content
Français

Find failures in public agent runs

  • Case study 1
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, we found failure patterns in the traces of Larkfield's workflow, with rules and queries that we wrote ourselves. But we wrote them for runs that we built, and we knew the answer for each run. In this case study, we try the same method on 456 runs of another team's agent, which someone else recorded and graded. We convert them to OpenTelemetry, read them with obs, and count which failures the traces can find.

What you will be able to do

Find out which failures of an AI agent its traces can show. You convert 456 real recorded runs of an open customer-service agent to OpenTelemetry, with the course’s names. You read them with obs, run the course’s failure rules, adapt the rules to this agent, and check every flag against the benchmark’s grade. Then you measure how many failed runs a reviewer finds with each method, test the method on another model’s runs, and write a recommendation.

Case studyPortfolio piece

Licence
MIT
Time
About 2.5 hours
Runs in
VS Code, Python 3.12+
Skills used
  • Convert recorded runs to OpenTelemetry
  • GenAI conventions
  • Reading runs with obs
  • Failure rules
  • Adapting rules to another agent
  • Hit rate and base rate
  • A review budget as one measure
  • New input
  • Recommendation

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

This case study is a project folder, not a notebook. It reads the runs with the course’s own obs, from your course project. It writes spans with the OpenTelemetry packages that the project already has. A notebook in Colab would have neither. The folder is small (about 1.4 MB) and adds no package.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

The page teaches two new ideas. You convert runs that another system recorded, and you compare ways to choose the runs that a person reviews.

What you need on your computer

  • The course project at the end of the course, in ~/projects/observable-workflow, with its virtual environment. If yours is broken, download the course’s snapshot end-of-m08 and follow its README.
  • Grafana is optional. Every step works with obs and the case study’s commands.

Tested on macOS (Apple silicon) with Python 3.12.13 and the course project’s packages. The Windows (PowerShell) and Linux commands are written, not run.

Larkfield wants an agent on its web shop that cancels and changes orders, and handles returns and exchanges. Amira (she) will build it. Priya (she) owns quality, and she asks the question first: “When this agent fails, will our traces show it?”

Tomás (he) thinks the answer is easy: “We have five failure rules. They will find its failures too.” Priya is not sure. The rules were written for the support workflow, and they were checked on a week that the course built, with an answer file. A new agent fails in its own ways.

So Priya proposes a rehearsal on public runs of a similar agent, which someone else recorded and graded. Convert them to the course’s traces, run the rules, and count what they find against the grade. Whatever the rules miss, the team must record in another way before launch.

You will build a failure report with evidence for 456 runs of an open agent. It has converted traces, a baseline, adapted rules and one fair measure. It also says what the traces cannot see, and it ends with a test on new input and a recommendation.

A flow from top to bottom. The fixed sample, 456 runs with SHA-256 checks, goes to convert, with the OpenTelemetry SDK. Convert writes two things: telemetry/design/traces.jsonl.gz, and truth.json, the grade, kept apart. The traces go to obs runs, trace and find, the course's tool unchanged, and, optionally, to Grafana with TraceQL. Then the rules: the course's, then the store's. Each flag is checked against the grade from truth.json. Then one measure: failed runs found in 92 reviews. Then the recommendation, and new input: another model's runs, which go back to convert.
The method of this case study. The grade stays in its own file, so that no rule can read it.
Text description of the diagram

The runs are converted to traces, read with obs and flagged by rules. The grade stays apart, in truth.json, and checks every flag. One measure compares the methods, and new input tests the recommendation.

  1. Download the starter project from the box at the top of this page. Unzip it into ~/projects, so that the folder public-agent-runs is next to observable-workflow.

  2. In VS Code, choose File > Open Folder, and open public-agent-runs. Then open a terminal: Terminal > New Terminal.

  3. Activate the course project’s virtual environment:

    Terminal window
    source ../observable-workflow/.venv/bin/activate

    If your course project is in another folder, set OBS_PROJECT to that folder (PowerShell: $env:OBS_PROJECT = "C:\path\to\observable-workflow").

  4. Run the tests: python -m pytest -q. The starter has three parts for you to finish, so six tests fail:

    FAILED tests/test_convert.py::test_tool_kinds_statuses_and_the_order_id - Ass...
    FAILED tests/test_rules.py::test_three_different_orders_are_not_a_loop - Asse...
    FAILED tests/test_rules.py::test_two_different_writes_are_not_a_duplicate - A...
    FAILED tests/test_rules.py::test_the_same_write_twice_is_a_duplicate - KeyErr...
    FAILED tests/test_rules.py::test_the_policys_second_lookup_is_not_a_hidden_error
    FAILED tests/test_rules.py::test_a_refused_change_that_was_never_repaired - K...
    6 failed, 13 passed in 0.28s
  5. Open a second terminal in the course project: cd ~/projects/observable-workflow, then activate the same environment. You run obs there, unchanged.

The finished project is in the same box, to compare with your work. Every output on this page comes from it.

The data is the published results of τ²-bench, an open benchmark of AI agents that talk with customers and use tools. Its retail domain is an online shop’s support desk. A customer wants to cancel or change an order, return or exchange items, or change an address. The agent has 16 tools and a written policy. A second model plays the customer and ends the chat when it is satisfied.

Priya needed four things: the time of every step, the tokens of every model call, the tool calls with their errors, and a grade. The first τ-bench’s runs have no times and no token counts. Other open sets are single decisions, or forbid commercial use. τ²-bench has all four. Its telecom runs are the data of Contain a runaway agent, in the agents governance course; its retail runs are closer to Larkfield.

The design runs are 456 runs of gpt-4.1: 114 tasks, each run 4 times. We study them and adapt the rules on them. The runs of Claude 3.7 Sonnet, o4-mini and gpt-4.1-mini on the same tasks are new input for the end. One run has these fields:

Field What it holds
id, task_id, trial the run; one of 114 tasks; each task was run 4 times
messages[] each message: who sent it (assistant, user, tool) and its time
in, out, cost, finish an agent’s model call: recorded token counts, the benchmark’s recorded cost, the finish reason
calls[] the tools that the model asked for: name and args
error, error_text a tool answered with an error, and its short text, such as Error: User not found
end how the customer ended the chat: stop, transfer or out_of_scope
grade the benchmark’s reward (1 or 0) and its checks: kept apart, never on a span

The benchmark gives reward 1 when two things are true. The shop’s database ends exactly as the task expects, and the agent told the customer what the task requires.

The unit is one run: one conversation, from the agent’s greeting to the customer’s last message. It becomes one trace.

The truth is the benchmark’s grade. It plays the role of the course week’s truth.json. A production system has no grade, so no rule may read it. The converter writes it to its own file, and only the check reads it.

The measure. Priya’s team reviews one run in five, as in Quality from live runs. Here that is 92 of 456 runs. A method is better when more of the 92 reviewed runs really failed. A rule’s flags tell the reviewer which runs to read first.

Think of a kitchen smoke alarm that rings every time someone makes toast. Soon, nobody listens. The useful question is: when it rings, how often is there a fire?

A flag works the same way. Its hit rate is the share of the runs it flags that really failed. Compare it with the base rate: the share of all runs that failed. A flag is useful only when its hit rate is clearly above the base rate. Otherwise, a reviewer does as well by choosing runs at random.

Ten runs, three of them failed: a base rate of 30%. Two reviews at random find 2 times 3 over 10, 0.6 failed runs on average. A rule flags 4 runs, and 2 of them failed: a hit rate of 50%. Two reviews of flagged runs find 2 times 2 over 4, 1.0 failed run on average.
The tiny example in tests/test_compare.py. With the same two reviews, the flagged runs find more failures, because the hit rate is above the base rate.
Text description of the diagram

Ten runs, three failed: the base rate is 30%. Two reviews at random find 0.6 failed runs on average. A rule flags four runs, two of which failed: a hit rate of 50%. Two reviews of flagged runs find 1.0 failed run on average.

agentruns/compare.py uses this arithmetic. The reviewer reads the flagged runs first, then others at random. If more runs are flagged than the budget, the reviewer picks among the flagged runs at random. The results are averages, so they do not depend on a random seed.

Think of copying an old paper logbook into your team’s spreadsheet. You put each fact in the column that everyone already uses, and you leave out what must not be there.

A converter does the same with recorded runs. The benchmark wrote each run as a list of messages, and our tools read traces. So agentruns/convert.py makes one trace per run with the OpenTelemetry SDK, and writes it as OTLP JSON, the format that obs and the Collector read.

Two columns joined row by row by arrows. On the left, one recorded run from the benchmark's file: the start at 15:00:00 with the agent's fixed greeting, no model call; the customer's message at 15:00:04; a model call of the agent at 15:00:05, 1,000 tokens in and 20 out, that asks for get_order_details; the tool's answer at 15:00:05.01, no error; a model call at 15:00:07, 1,500 in and 40 out, with text for the customer; the customer's STOP at 15:00:08. On the right, the trace: the root span invoke_agent retail-agent from 0 to 8.0 s, with larkfield.run.id R-007-0 and the outcome resolved; under it customer_turn from 0 to 4.0 s; chat gpt-4.1 from 4.0 to 5.0 s, input tokens 1000, finish tool_calls; execute_tool get_order_details from 5.0 to 5.01 s, kind read, status ok, order W1 and a digest; chat gpt-4.1 from 5.01 to 7.0 s, finish stop; customer_turn from 7.0 to 8.0 s, tau2.customer.end stop. At the bottom, the grade, reward 1 or 0, goes to truth.json, never on a span.
The tiny run of tests/helpers.py, with one tool call: one recorded run becomes one trace, one span per message, in time order. Each span lasts as long as the recorded step did. The grade goes to its own file.
Text description of the diagram

The run becomes the root span, from its start to its last message. Each customer message becomes a customer_turn span with the customer’s time. Each model call becomes a chat span with its tokens and finish reason, from the message before it to its answer. Each tool call becomes an execute_tool span with its kind, status, order ID and a digest of its arguments. The grade goes to truth.json.

The rules of the conversion are the course’s:

  • The GenAI names that the course project pins: invoke_agent, chat {model}, execute_tool {tool}, gen_ai.usage.input_tokens and the others.
  • The course’s own names where they mean the same thing, such as larkfield.run.id, larkfield.outcome and larkfield.tool.kind. obs was built for Larkfield and reads these names, so it reads the shop’s runs unchanged. The benchmark’s own facts get the prefix tau2..
  • No values that identify a person. The customers are made up, but the habit matters. A span keeps the argument names (email) and a digest of the values, never a name, an email address or a zip code. The same call twice has the same digest.
  • The recorded times. A chat span starts at the message before it and ends at the model’s answer. A customer_turn span is the time that the customer took to answer. It is our own span name, not a GenAI operation.
  1. Open agentruns/convert.py and find the first TODO (Part 1). A write needs its outcome, as in the course’s write spans:

    if k == WRITE:
    attrs[n.WRITE_OUTCOME] = "refused" if res.get("error") else "done"
  2. Find the second TODO (Part 1). A failed call needs an error status, with the tool’s short message:

    if res.get("error"):
    span.set_attribute(n.ERROR_TYPE, "tool_error")
    span.set_status(Status(StatusCode.ERROR, res.get("error_text", "")))
  3. Run the tests: python -m pytest -q. Two tests still fail: the rules of Parts 2 and 3.

  4. Convert the design runs:

    Terminal window
    python -m agentruns.convert design
    design (gpt-4.1-2025-04-14): 456 runs -> 12,142 spans in telemetry/design/traces.jsonl.gz
    the benchmark's grade -> telemetry/design/truth.json (118 of 456 runs failed; obs never reads it)

agentruns/data.py checks each file’s SHA-256 before it reads it. The conversion makes the same file every time: the span IDs come from a seed.

In your second terminal, in observable-workflow, list the last runs:

Terminal window
python -m obs runs --data ../public-agent-runs/telemetry/design --limit 5
run ticket started outcome model tools appr cost US$ secs model flags
R-109-3 task-109 Thu 11:38 resolved 12 6 0 0.06657 59.3 gpt-4.1-2025-04-14 duplicate_write
R-110-3 task-110 Thu 11:38 resolved 12 5 0 0.06665 64.1 gpt-4.1-2025-04-14 duplicate_write
R-111-3 task-111 Thu 11:38 handed_to_person 14 8 0 0.08078 66.0 gpt-4.1-2025-04-14 loop
R-112-3 task-112 Thu 11:38 resolved 16 4 0 0.07336 72.6 gpt-4.1-2025-04-14 -
R-113-3 task-113 Thu 11:38 resolved 8 6 0 0.03033 41.5 gpt-4.1-2025-04-14 duplicate_write,loop
456 run(s), 324 flagged (last 5 shown)

What this tells us. The course’s tool reads another team’s agent with no change: runs, outcomes, model calls, tools, costs and times. The costs are the benchmark’s recorded costs, not the course’s price table. The start times are the benchmark’s, read as UTC and shown in Larkfield’s time zone. And the last column is a warning: 324 of 456 runs are flagged.

All 456 runs ended normally: the simulated customer stopped the chat. 428 ended as resolved and 28 as handed_to_person. No run looped until the benchmark stopped it. So the loud failures of the course’s week are not here. If this agent fails, it fails calmly.

Terminal window
python -m obs trace R-002-0 --data ../public-agent-runs/telemetry/design
Trace 1 9cc6b0ee25dca4b7c7d22fc4a3cb15d5 Thu 05 Jun 11:33:58 36.2 s
0.0 s |========================| 36.20 s invoke_agent retail-agent finished
0.1 s |===== | 7.72 s customer_turn
7.8 s | == | 2.14 s chat gpt-4.1-2025-04-14 3531 in / 62 out US$0.00276 stop
…
13.2 s | === | 2.94 s chat gpt-4.1-2025-04-14 3895 in / 129 out US$0.00364 tool_calls
16.1 s | = | 0.00 s execute_tool list_all_product_types ok
16.1 s | = | 0.00 s execute_tool get_order_details ok #W6247578
16.1 s | = | 0.00 s execute_tool get_order_details ok #W9711842
16.1 s | = | 0.00 s execute_tool get_order_details ok #W4776164
16.1 s | = | 0.00 s execute_tool get_order_details ok #W6679257
16.1 s | = | 0.00 s execute_tool get_order_details ok #W2378156
…
27.9 s | == | 1.79 s chat gpt-4.1-2025-04-14 7225 in / 52 out US$0.00968 tool_calls
29.7 s | = | 0.00 s execute_tool return_delivered_order_items done #W2378156
29.7 s | === | 3.67 s chat gpt-4.1-2025-04-14 7820 in / 84 out US$0.01113 stop
33.4 s | ==| 2.82 s customer_turn
Outcome: resolved
Flag loop: the model asked for get_order_details(order_id) 5 times in one run

What this tells us. You read it as in Read a run. The customer took 7.7 seconds to write first. One model call asked for six tools at once, among them five orders of the customer. Then one return was accepted. The benchmark passed this run. But the course’s loop rule flags it: five calls of get_order_details with the argument order_id.

The course’s search for a failed read in a resolved run finds 46 runs:

Terminal window
python -m obs find --error-then-resolved --data ../public-agent-runs/telemetry/design
R-036-0 task-36 Thu 11:34 gpt-4.1-2025-04-14 a tool error, then resolved
R-035-0 task-35 Thu 11:34 gpt-4.1-2025-04-14 a tool error, then resolved
…
46 match(es) in 456 run(s)

Failed calls are common. Which of them matter?

A baseline: the course’s five rules, unchanged

Section titled “A baseline: the course’s five rules, unchanged”

Run them all, then check their flags against the grade:

Terminal window
python -m obs find all --data ../public-agent-runs/telemetry/design
R-002-0 task-2 Thu 11:33 gpt-4.1-2025-04-14 loop: the model asked for get_order_details(order_id) 5 times in one run
…
R-004-0 task-4 Thu 11:33 gpt-4.1-2025-04-14 duplicate_write: two writes with the same idempotency key None
…
477 match(es) in 456 run(s)

In public-agent-runs:

Terminal window
python -m agentruns.check --data design --rules course
456 runs, 118 failed by the benchmark's grade: base rate 26%. The course's rules:
rule flagged failed hit rate
loop 268 74 28%
duplicate_write 163 45 28%
no_action 31 6 19%
hidden_tool_error 15 7 47%
any rule 324 88 27%
Failed runs that a rule flagged: 88 of 118. Why they failed (truth.json):
…

What this tells us. The rules flag 324 runs, and their hit rate, 27%, is the base rate, 26%. A flag on 71% of the runs says nothing. Each rule carries an assumption about Larkfield’s workflow, and this agent breaks it:

Rule Larkfield’s assumption What this agent does
loop one order per ticket: the same tool with the same argument names three times is a loop a customer has several orders, so five get_order_details(order_id) calls are normal
duplicate_write every write has an idempotency key no write has a key, so “the same key” is “no key” on every pair of writes
no_action a reply is a send_reply tool call the reply is the model’s text answer
hidden_tool_error no later call of the same tool means the facts are missing the policy’s identity check: an email that fails, then a name and zip code that work
followed_injection the practice data marks its injected texts no text has a marker

The rules are not wrong. They are Larkfield’s, just as Contain a runaway agent found that one agent’s limits do not fit another agent.

agentruns/rules.py has the store’s rules. Each one reads the same Run objects as obs, and returns a sentence with span names and times, as in Agent failure patterns:

  • loop counts the same tool with the same arguments: the digest, not the names.
  • duplicate_write compares the tool and the digest, because there is no key.
  • hidden_tool_error counts the two identity lookups as one purpose.
  • no_action and followed_injection are dropped. A run with no change is often right here, because the customer only asked a question. And no text has a marker.
  1. Open agentruns/rules.py and find TODO (Part 2). The starter counts a tool with its argument names, as the course’s rule does. Count the digest of the values instead:

    return s.tool, s.get(n.ARGS_DIGEST)
  2. Run the tests: one still fails, the rule of Part 3.

Part 3: one rule from the store’s policy

Section titled “Part 3: one rule from the store’s policy”

The shop’s policy has rules that a trace can check without words. For example, “Exchange or modify order tools can only be called once per order”. We tried several of them on the design runs. Most of them fired on no run, or flagged runs that failed no more often than the base rate. One signal stood out: the store’s database refused a change, for example “Non-delivered order cannot be exchanged”. The run went on, and no later change of that kind on that order was accepted.

  1. Find TODO (Part 3), in refused_change. For each refused write, look for a later accepted write of the same tool on the same order:

    writes = _writes(run)
    for i, s in enumerate(writes):
    if s.get(n.WRITE_OUTCOME) != "refused":
    continue
    later = [x for x in writes[i + 1:] if x.tool == s.tool and x.get(n.ORDER_ID) == s.get(n.ORDER_ID)
    and x.get(n.WRITE_OUTCOME) == "done"]
    if not later:
    return (f"the store refused {s.tool} {s.get(n.ORDER_ID, '')} at {at(run, s)} "
    f"({s.status_message[:90]}), and no later {s.tool} on it was accepted").replace(" ", " ")
    return None
  2. Run the tests: the last line says 19 passed.

  3. Find the runs:

    Terminal window
    python -m agentruns.find refused_change --data design
    R-027-0 task-27 Thu 11:34 refused_change: the store refused exchange_delivered_order_items #W7181492 at +39.1 s (Error: Non-delivered order cannot be exchanged), and no later exchange_delivered_order_items on it was accepted
    R-030-0 task-30 Thu 11:34 refused_change: the store refused exchange_delivered_order_items #W2692684 at +22.7 s (Error: New item 3788616824 not found or available), and no later exchange_delivered_order_items on it was accepted
    …
    39 match(es) in 456 run(s), 39 run(s) flagged

Check one flag against the evidence. In observable-workflow, open the first run:

Terminal window
python -m obs trace R-027-0 --data ../public-agent-runs/telemetry/design
…
35.3 s | === | 3.85 s chat gpt-4.1-2025-04-14 6230 in / 113 out US$0.00818 tool_calls
39.1 s | = | 0.00 s execute_tool return_delivered_order_items done #W7181492
39.1 s | = | 0.00 s execute_tool exchange_delivered_order_items ERROR refused #W7181492
39.1 s | === | 5.07 s chat gpt-4.1-2025-04-14 6903 in / 138 out US$0.00973 stop
…
Outcome: handed_to_person

One model call asked for two changes to one order at the same time. The store accepted the return and refused the exchange. The return changed the order’s status, so the exchange was no longer possible. Only now, look at the grade, in public-agent-runs:

Terminal window
python -m agentruns.check --data design --run R-027-0
R-027-0 (task 27, trial 0): the benchmark's grade 0, failed: different_changes
changes made 1, expected 1; database as expected: no; customer told what they had to hear: yes
the course's rules: no flag
the store's rules: refused_change: the store refused exchange_delivered_order_items #W7181492 at +39.1 s (Error: Non-delivered order cannot be exchanged), and no later exchange_delivered_order_items on it was accepted

A true flag. Not every flag is. R-095-3 had the same refusal and passed: the agent had already exchanged that order once, and its second try changed nothing. A flag is a reason to look, not a verdict.

Part 4: a fair comparison under one measure

Section titled “Part 4: a fair comparison under one measure”

First, the store’s rules against the grade:

Terminal window
python -m agentruns.check --data design --rules store
456 runs, 118 failed by the benchmark's grade: base rate 26%. The store's rules:
rule flagged failed hit rate
refused_change 39 18 46%
hidden_tool_error 35 15 43%
duplicate_write 3 1 33%
loop 2 1 50%
any rule 45 21 47%
Failed runs that a rule flagged: 21 of 118. Why they failed (truth.json):
why failed flagged
wrong_arguments 50 9
different_changes 18 9
unasked_change 11 0
missing_change 13 1
no_change 13 2
right_changes_not_told 13 0

Then the measure, the same 92 reviews for every method:

Terminal window
python -m agentruns.compare --data design
design: review 92 of 456 runs (one in five); 118 failed by the grade. Failed runs found, on average:
how the reviewer chooses found hit rate
at random 23.8 26%
the course's rules first 25.0 27% (324 flagged: 92 of them at random)
the store's rules first 32.1 35% (45 flagged, then 47 at random)
456 recorded runs of gpt-4.1 in τ²-bench's retail domain; 118 failed by the benchmark's grade. Expected values: the average over every random choice.
Text description of the chart

Three bars. At random: 23.8 failed runs found in 92 reviews. The course’s rules first: 25.0. The store’s rules first: 32.1, with 45 runs flagged, 21 of them failed.

What this tells us.

  • The course’s rules do not beat a random sample: 25.0 against 23.8. They flag too much.
  • The store’s rules flag 45 runs, and 21 of them failed: a hit rate of 47%, nearly twice the base rate. With them first, the same 92 reviews find 32.1 failed runs instead of 23.8, about a third more.
  • But they flag only 21 of the 118 failed runs. Four failed runs in five leave no mark that a rule on the trace can see.

Now read truth.json for the 97 failed runs that no rule flagged:

The design runs (gpt-4.1), the store's rules. args: the right tool with other arguments; other: another change; unasked: a change the task did not ask for; missing: an expected change is missing; none: no change at all; untold: the customer was not told what they had to hear (agentruns/truth.py).
Text description of the chart

Six bars of failed runs that no rule flagged. Wrong arguments: 41. Other change: 9. Unasked change: 11. Missing change: 12. No change: 11. Not told: 13.

The largest group is the right tool with the wrong arguments: 50 failed runs, 41 of them not flagged. In 19 of the 50, only the new item differs: the agent chose another variant of the right product, such as another colour or size. It had looked up the product, and the item it chose was a real one. Only the customer’s words say which one they wanted.

Here is a failure that looks perfect in the trace:

Terminal window
python -m obs trace R-051-0 --data ../public-agent-runs/telemetry/design
…
9.3 s | = | 0.75 s chat gpt-4.1-2025-04-14 3932 in / 20 out US$0.00265 tool_calls
10.0 s | = | 0.00 s execute_tool get_order_details ok #W8855135
…
16.7 s | = | 0.83 s chat gpt-4.1-2025-04-14 4659 in / 26 out US$0.00377 tool_calls
17.5 s | = | 0.00 s execute_tool cancel_pending_order done #W8855135
17.5 s | === | 2.45 s chat gpt-4.1-2025-04-14 5135 in / 82 out US$0.00401 stop
20.0 s | ===| 1.88 s customer_turn
Outcome: resolved

The identity check, one lookup, one cancellation, a satisfied customer. The grade says it failed: different_changes. The task’s expected change is a return on another order, #W4689314, which had been delivered (data/runs/retail-tasks.jsonl.gz). In the original conversation, the customer asked to return a camera. The agent found a pending order instead, and the customer agreed to cancel it. The trace records every step correctly. It cannot know that the step was the wrong one.

Priya writes it for Amira, before the shop’s agent is built:

Four rows: a kind of failure, then the record that finds it. A call that failed, a change the store refused, a loop: 21 of 118 failed runs; the store's rules on the traces: review those runs first. A change that differs from what the customer asked, wrong arguments, another change or an unasked change: 79 of 118; record what the customer confirmed, as IDs, in a proposal event, then compare each write with it. The customer was not told what they had to hear: 13; a person reads the words, in sampled reviews, with content captured only for them. How many runs fail? Keep a random share of the reviews, because flagged-first reviews overcount the failures.
Each kind of failure needs its own record. The trace's structure finds the first kind only.
Text description of the diagram

Failed calls, refused changes and loops: the store’s rules, and review those runs first. A change that differs from the request: record the confirmed request as IDs and compare each write with it. A customer who was not told: a person reads the words. The failure rate: a random share of reviews.

  1. Write the agent’s own rules before launch, and measure their hit rate. Copy the method, not the rules: unchanged, the course’s rules stayed at the base rate; adapted, they reached 47%.
  2. Review flagged runs first. With the same 92 reviews, the team finds about a third more failures.
  3. Keep a random share of the reviews. Flagged-first reviews find more failures, so they overcount the failure rate. A random sample estimates it honestly.
  4. Record what the customer confirmed, as IDs. 79 of the 118 failures are a change that differs from the task’s request. A rule can see that only if the run records the request it acted on: the order, the items and the payment method. Larkfield’s support workflow records its proposal and a person’s approval. See Approvals, people and waiting time and Record decisions, approvals and changes.
  5. Let people read the words of a few runs. Whether the customer was told the right thing is in the text. Capture content only for the sampled reviews, as in What to record, and what not to.

Rules adapted on one model’s runs may fit only that model. Run the whole method on Claude 3.7 Sonnet’s 456 runs of the same tasks.

  1. Convert them:

    Terminal window
    python -m agentruns.convert claude-3.7-sonnet
    claude-3.7-sonnet (claude-3-7-sonnet-20250219): 456 runs -> 13,486 spans in telemetry/claude-3.7-sonnet/traces.jsonl.gz
    the benchmark's grade -> telemetry/claude-3.7-sonnet/truth.json (97 of 456 runs failed; obs never reads it)
  2. Read them with obs, in observable-workflow:

    Terminal window
    python -m obs runs --data ../public-agent-runs/telemetry/claude-3.7-sonnet --limit 3
    run ticket started outcome model tools appr cost US$ secs model flags
    R-111-3 task-111 Thu 12:00 resolved 18 9 0 0.52124 80.4 claude-3-7-sonnet-20250219 duplicate_write,loop
    R-112-3 task-112 Thu 12:00 resolved 23 10 0 0.67582 102.9 claude-3-7-sonnet-20250219 duplicate_write,loop
    R-113-3 task-113 Thu 12:00 resolved 14 6 0 0.30788 48.5 claude-3-7-sonnet-20250219 duplicate_write,loop
    456 run(s), 315 flagged (last 3 shown)
  3. Check the store’s rules, unchanged:

    Terminal window
    python -m agentruns.check --data claude-3.7-sonnet --rules store
    456 runs, 97 failed by the benchmark's grade: base rate 21%. The store's rules:
    rule flagged failed hit rate
    refused_change 32 20 62%
    hidden_tool_error 21 13 62%
    loop 5 5 100%
    any rule 34 22 65%
    Failed runs that a rule flagged: 22 of 97. Why they failed (truth.json):
    …
  4. Compare the methods on all four models:

    Terminal window
    python -m agentruns.convert o4-mini
    python -m agentruns.convert gpt-4.1-mini
    python -m agentruns.compare --data design claude-3.7-sonnet o4-mini gpt-4.1-mini --json results/compare.json
    …
    claude-3.7-sonnet: review 92 of 456 runs (one in five); 97 failed by the grade. Failed runs found, on average:
    how the reviewer chooses found hit rate
    at random 19.6 21%
    the course's rules first 21.0 23% (315 flagged: 92 of them at random)
    the store's rules first 32.3 35% (34 flagged, then 58 at random)
    o4-mini: review 92 of 456 runs (one in five); 130 failed by the grade. Failed runs found, on average:
    how the reviewer chooses found hit rate
    at random 26.2 29%
    the course's rules first 24.8 27% (274 flagged: 92 of them at random)
    the store's rules first 27.5 30% (19 flagged, then 73 at random)
    gpt-4.1-mini: review 92 of 456 runs (one in five); 155 failed by the grade. Failed runs found, on average:
    how the reviewer chooses found hit rate
    at random 31.3 34%
    the course's rules first 34.1 37% (356 flagged: 92 of them at random)
    the store's rules first 41.0 45% (101 flagged: 92 of them at random)

What this tells us.

  • On Claude 3.7 Sonnet, the store’s rules held. Their hit rate was 65% against a base rate of 21%. The same reviews found 32.3 failed runs instead of 19.6. Its five loops were all real failures.
  • On gpt-4.1-mini, they held too: 41.0 instead of 31.3.
  • On o4-mini, they nearly stopped working: 27.5 instead of 26.2. Most of its failures were wrong arguments (73 of 130). The store refused changes in only 15 of its runs. A rule finds only the failures that leave its signature.
  • On every model, the course’s rules, unchanged, stayed at the base rate.

So the recommendation stands, with one addition: measure the hit rate again for each new model, as a release check.

If you have Docker and 20 GB of free disk, search the runs with TraceQL, as in Search runs with TraceQL. grafana.md in the project has the steps and five queries. Start Grafana in observable-workflow (docker compose -f grafana/compose.yaml up -d), then send the runs with python -m agentruns.convert design --otlp. The converter moves the times so that the last run ends now. Stop Grafana with docker compose -f grafana/compose.yaml down -v. The course team ran these steps once. Grafana was ready after 14 seconds, and --otlp sent all 12,142 spans in about a second. Tempo then held 362 of the 456 runs: the Collector’s tail sampling keeps a trace with a write, an error or more than 30 seconds, and drops the others. The five queries of grafana.md returned 362 runs, 44 runs with a refused change, 68 runs with a failed call that ended resolved, 11 runs handed to a person, and 345 runs with a model call over 5 seconds. These are candidates, not failures: the rules in agentruns/rules.py keep fewer, because they also check what happened later in the run.

TraceQL finds the candidates, for example every refused change: { resource.service.name = "tau2-retail-agent" && span.larkfield.write.outcome = "refused" }. It cannot ask “was a later change of the same kind accepted?”. The rules in agentruns/rules.py do that.

Common mistake

The course project is not found

Symptom:

The course project is not in /Users/you/projects/observable-workflow. Put this folder next to observable-workflow, or set OBS_PROJECT to the course project's folder.

Cause: public-agent-runs is not next to observable-workflow, so the case study cannot import obs.

Recover: move the folder, or set OBS_PROJECT to your course project’s folder and run the command again.

  • A rule carries its workflow’s assumptions. On another agent, the course’s rules flagged 71% of the runs. List each rule’s assumption before you trust its count.
  • The hit rate decides whether a flag is useful. Compare it with the base rate every time. 88 of 118 failures flagged sounds good, until you see that 324 runs were flagged.
  • The structure finds only the failures that leave a structure. Refused changes, failed calls and loops are visible. A wrong choice among valid options, most of this agent’s failures, is not.
  • A benchmark, not a help desk. The customers are a model, and the tasks are a test set, each run 4 times. One task can fail the same way in all four trials, so the counts are not 456 independent cases.
  • The grade is one answer per task. A few tasks may accept more than one good outcome, and the benchmark counts only one.
  • Name the data honestly. These are another team’s benchmark runs. Present the hit rates as a rehearsal of the method, not as a measure of Larkfield’s agent.
  • Keep personal values off the spans, even when they are made up. The converter keeps argument names and digests. Do not add the values to make the search easier.
  • A flag is not a judgement of a person or a model. It chooses which runs a person reads first. The person decides.
  • Credit the source. The data is MIT-licensed. Keep its licence text and the attribution in your copy, as the dataset card does.

This project shows what operations teams look for. It has a converter to a standard, a baseline, rules checked against a grade, one fair measure and a test on new input.

What to put in the repository:

public-agent-runs/
README.md the question, how to run it, the results table and the recommendation
data/dataset.md the source, the licence, the checksums, the changes, the limits
data/runs/ the fixed sample and the τ²-bench licence text
LICENSE MIT, for your code
agentruns/ names, store, convert, truth, rules, find, check, compare
tests/ 19 tests
results/ your compare.json
  1. Make a new Git repository from your project folder, and push it to GitHub. .gitignore keeps out telemetry/ and results/: add results/compare.json with git add -f.
  2. Write the README’s results table from your own python -m agentruns.compare output, with the baseline and the new input.
  3. Say in the README that the project needs the course project (OBS_PROJECT), and that python -m pytest passes when it is there.

In two sentences, for an interview:

“I converted 456 open, graded runs of an AI customer-service agent to OpenTelemetry traces and checked failure rules against the grade. Rules adapted to the agent nearly doubled the hit rate and found a third more failures per review. The grade showed which failures need a record of the customer’s request instead.”

Go further: convert the airline domain’s runs of the same benchmark with python -I tools/make_sample.py, in a new empty folder, and adapt the store’s tool table. Or write a rule for “modify or exchange twice on one order”, and measure its hit rate. Or put a cost on a review and on a missed failure, and choose the budget that gives the lowest total cost.

Check your understanding

The course's five rules flagged 88 of the 118 failed runs. Why does Priya say that they found nothing useful?

Show the answer

Because they flagged 324 of the 456 runs, and only 27% of the flagged runs failed: the base rate is 26%. A reviewer who reads flagged runs first finds 25.0 failed runs in 92 reviews, about the same as at random (23.8). The rules carry Larkfield’s assumptions: one order per ticket, an idempotency key on every write, a send_reply tool. This agent breaks all three.

Check your understanding

R-051-0 cancelled a pending order, the customer agreed, and the run ended resolved. The grade says it failed. Why can no rule on its trace find it, and what record would let a rule find it?

Show the answer

Every step in the trace is correct: the identity check, the lookup, an accepted cancellation, a satisfied customer. The failure is the choice: the customer wanted to return a camera from another, delivered order. That is in the words, not in the structure. A record of the request as IDs (the order, the items, the change) would let a rule compare each write with it. Larkfield’s proposal and approval events are such a record.

Check your understanding

The store's rules raised the failures found per 92 reviews from 23.8 to 32.1 on gpt-4.1, and from 19.6 to 32.3 on Claude 3.7 Sonnet. On o4-mini, they found 27.5 instead of 26.2. What should Larkfield do with this result?

Show the answer

Measure the hit rate of its rules again at every change of model, as a release check. A rule finds only the failures that leave its signature. o4-mini’s failures were mostly wrong arguments, and the store refused few of its changes, so the rules had little to find. Keep a random share of the reviews, so that the failure rate stays honest, and record the confirmed request so that wrong arguments become visible.

Recap

  • A converter turns recorded runs from another system into traces. One run is one trace, with the course’s names and the recorded times. No personal values go on a span, and the grade has its own file.
  • The course’s rules, unchanged, flagged 71% of another agent’s runs at the base rate. Adapted, they reached a hit rate of 47% and found about a third more failures in the same 92 reviews.
  • Four failed runs in five left no structural mark: mostly the right tool with the wrong arguments. A record of the confirmed request, as IDs, and a few human reviews are what find them.
  • The adapted rules held on two of three other models. Measure the hit rate again at every change of model.

You can now read runs that another system recorded with your own tools. You can also say, with evidence, which failures their traces can show. Next, you measure the cost of a coding agent: many more steps, and a cost that depends on the path the agent takes.

Next: The cost of a coding agent