Find failures in public agent runs
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we found failure patterns in the traces of Larkfield's workflow, with rules and queries that we wrote ourselves. But we wrote them for runs that we built, and we knew the answer for each run. In this case study, we try the same method on 456 runs of another team's agent, which someone else recorded and graded. We convert them to OpenTelemetry, read them with obs, and count which failures the traces can find.
Case study 1 of 3 · Observability for AI Agents: Traces, Costs and Audit TrailsFind failures in public agent runs
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Find out which failures of an AI agent its traces can show. You convert 456 real recorded runs of an open customer-service agent to OpenTelemetry, with the course’s names. You read them with obs, run the course’s failure rules, adapt the rules to this agent, and check every flag against the benchmark’s grade. Then you measure how many failed runs a reviewer finds with each method, test the method on another model’s runs, and write a recommendation.
This case study is a project folder, not a notebook. It reads the runs with the course’s own obs, from your course project. It writes spans with the OpenTelemetry packages that the project already has. A notebook in Colab would have neither. The folder is small (about 1.4 MB) and adds no package.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- Spans, traces and the names on them, from Spans, traces and context and The GenAI conventions for agent runs.
- What stays off a span, from What to record, and what not to.
- Reading a waterfall with
obs trace, from Read a run. - The five failure patterns, their rules and the scenario truth, from Agent failure patterns.
- Sampled reviews, from Quality from live runs.
The page teaches two new ideas. You convert runs that another system recorded, and you compare ways to choose the runs that a person reviews.
What you need on your computer
- The course project at the end of the course, in
~/projects/observable-workflow, with its virtual environment. If yours is broken, download the course’s snapshotend-of-m08and follow its README. - Grafana is optional. Every step works with
obsand the case study’s commands.
Tested on macOS (Apple silicon) with Python 3.12.13 and the course project’s packages. The Windows (PowerShell) and Linux commands are written, not run.
The situation
Section titled “The situation”Larkfield wants an agent on its web shop that cancels and changes orders, and handles returns and exchanges. Amira (she) will build it. Priya (she) owns quality, and she asks the question first: “When this agent fails, will our traces show it?”
Tomás (he) thinks the answer is easy: “We have five failure rules. They will find its failures too.” Priya is not sure. The rules were written for the support workflow, and they were checked on a week that the course built, with an answer file. A new agent fails in its own ways.
So Priya proposes a rehearsal on public runs of a similar agent, which someone else recorded and graded. Convert them to the course’s traces, run the rules, and count what they find against the grade. Whatever the rules miss, the team must record in another way before launch.
What you will build
Section titled “What you will build”You will build a failure report with evidence for 456 runs of an open agent. It has converted traces, a baseline, adapted rules and one fair measure. It also says what the traces cannot see, and it ends with a test on new input and a recommendation.
Text description of the diagram
The runs are converted to traces, read with obs and flagged by rules. The grade stays apart, in truth.json, and checks every flag. One measure compares the methods, and new input tests the recommendation.
Set up the project
Section titled “Set up the project”-
Download the starter project from the box at the top of this page. Unzip it into
~/projects, so that the folderpublic-agent-runsis next toobservable-workflow. -
In VS Code, choose File > Open Folder, and open
public-agent-runs. Then open a terminal: Terminal > New Terminal. -
Activate the course project’s virtual environment:
Terminal window source ../observable-workflow/.venv/bin/activateTerminal window ..\observable-workflow\.venv\Scripts\Activate.ps1Terminal window source ../observable-workflow/.venv/bin/activateIf your course project is in another folder, set
OBS_PROJECTto that folder (PowerShell:$env:OBS_PROJECT = "C:\path\to\observable-workflow"). -
Run the tests:
python -m pytest -q. The starter has three parts for you to finish, so six tests fail:FAILED tests/test_convert.py::test_tool_kinds_statuses_and_the_order_id - Ass...FAILED tests/test_rules.py::test_three_different_orders_are_not_a_loop - Asse...FAILED tests/test_rules.py::test_two_different_writes_are_not_a_duplicate - A...FAILED tests/test_rules.py::test_the_same_write_twice_is_a_duplicate - KeyErr...FAILED tests/test_rules.py::test_the_policys_second_lookup_is_not_a_hidden_errorFAILED tests/test_rules.py::test_a_refused_change_that_was_never_repaired - K...6 failed, 13 passed in 0.28s -
Open a second terminal in the course project:
cd ~/projects/observable-workflow, then activate the same environment. You runobsthere, unchanged.
The finished project is in the same box, to compare with your work. Every output on this page comes from it.
The data
Section titled “The data”The data is the published results of τ²-bench, an open benchmark of AI agents that talk with customers and use tools. Its retail domain is an online shop’s support desk. A customer wants to cancel or change an order, return or exchange items, or change an address. The agent has 16 tools and a written policy. A second model plays the customer and ends the chat when it is satisfied.
Priya needed four things: the time of every step, the tokens of every model call, the tool calls with their errors, and a grade. The first τ-bench’s runs have no times and no token counts. Other open sets are single decisions, or forbid commercial use. τ²-bench has all four. Its telecom runs are the data of Contain a runaway agent, in the agents governance course; its retail runs are closer to Larkfield.
The design runs are 456 runs of gpt-4.1: 114 tasks, each run 4 times. We study them and adapt the rules on them. The runs of Claude 3.7 Sonnet, o4-mini and gpt-4.1-mini on the same tasks are new input for the end. One run has these fields:
| Field | What it holds |
|---|---|
id, task_id, trial |
the run; one of 114 tasks; each task was run 4 times |
messages[] |
each message: who sent it (assistant, user, tool) and its time |
in, out, cost, finish |
an agent’s model call: recorded token counts, the benchmark’s recorded cost, the finish reason |
calls[] |
the tools that the model asked for: name and args |
error, error_text |
a tool answered with an error, and its short text, such as Error: User not found |
end |
how the customer ended the chat: stop, transfer or out_of_scope |
grade |
the benchmark’s reward (1 or 0) and its checks: kept apart, never on a span |
The benchmark gives reward 1 when two things are true. The shop’s database ends exactly as the task expects, and the agent told the customer what the task requires.
Frame the problem
Section titled “Frame the problem”The unit is one run: one conversation, from the agent’s greeting to the customer’s last message. It becomes one trace.
The truth is the benchmark’s grade. It plays the role of the course week’s truth.json. A production system has no grade, so no rule may read it. The converter writes it to its own file, and only the check reads it.
The measure. Priya’s team reviews one run in five, as in Quality from live runs. Here that is 92 of 456 runs. A method is better when more of the 92 reviewed runs really failed. A rule’s flags tell the reviewer which runs to read first.
A new idea: hit rate and base rate
Section titled “A new idea: hit rate and base rate”Think of a kitchen smoke alarm that rings every time someone makes toast. Soon, nobody listens. The useful question is: when it rings, how often is there a fire?
A flag works the same way. Its hit rate is the share of the runs it flags that really failed. Compare it with the base rate: the share of all runs that failed. A flag is useful only when its hit rate is clearly above the base rate. Otherwise, a reviewer does as well by choosing runs at random.
Text description of the diagram
Ten runs, three failed: the base rate is 30%. Two reviews at random find 0.6 failed runs on average. A rule flags four runs, two of which failed: a hit rate of 50%. Two reviews of flagged runs find 1.0 failed run on average.
agentruns/compare.py uses this arithmetic. The reviewer reads the flagged runs first, then others at random. If more runs are flagged than the budget, the reviewer picks among the flagged runs at random. The results are averages, so they do not depend on a random seed.
Part 1: convert the runs to OpenTelemetry
Section titled “Part 1: convert the runs to OpenTelemetry”Think of copying an old paper logbook into your team’s spreadsheet. You put each fact in the column that everyone already uses, and you leave out what must not be there.
A converter does the same with recorded runs. The benchmark wrote each run as a list of messages, and our tools read traces. So agentruns/convert.py makes one trace per run with the OpenTelemetry SDK, and writes it as OTLP JSON, the format that obs and the Collector read.
Text description of the diagram
The run becomes the root span, from its start to its last message. Each customer message becomes a customer_turn span with the customer’s time. Each model call becomes a chat span with its tokens and finish reason, from the message before it to its answer. Each tool call becomes an execute_tool span with its kind, status, order ID and a digest of its arguments. The grade goes to truth.json.
The rules of the conversion are the course’s:
- The GenAI names that the course project pins:
invoke_agent,chat {model},execute_tool {tool},gen_ai.usage.input_tokensand the others. - The course’s own names where they mean the same thing, such as
larkfield.run.id,larkfield.outcomeandlarkfield.tool.kind.obswas built for Larkfield and reads these names, so it reads the shop’s runs unchanged. The benchmark’s own facts get the prefixtau2.. - No values that identify a person. The customers are made up, but the habit matters. A span keeps the argument names (
email) and a digest of the values, never a name, an email address or a zip code. The same call twice has the same digest. - The recorded times. A
chatspan starts at the message before it and ends at the model’s answer. Acustomer_turnspan is the time that the customer took to answer. It is our own span name, not a GenAI operation.
-
Open
agentruns/convert.pyand find the firstTODO (Part 1). A write needs its outcome, as in the course’s write spans:if k == WRITE:attrs[n.WRITE_OUTCOME] = "refused" if res.get("error") else "done" -
Find the second
TODO (Part 1). A failed call needs an error status, with the tool’s short message:if res.get("error"):span.set_attribute(n.ERROR_TYPE, "tool_error")span.set_status(Status(StatusCode.ERROR, res.get("error_text", ""))) -
Run the tests:
python -m pytest -q. Two tests still fail: the rules of Parts 2 and 3. -
Convert the design runs:
Terminal window python -m agentruns.convert designdesign (gpt-4.1-2025-04-14): 456 runs -> 12,142 spans in telemetry/design/traces.jsonl.gzthe benchmark's grade -> telemetry/design/truth.json (118 of 456 runs failed; obs never reads it)
agentruns/data.py checks each file’s SHA-256 before it reads it. The conversion makes the same file every time: the span IDs come from a seed.
Read the runs with obs
Section titled “Read the runs with obs”In your second terminal, in observable-workflow, list the last runs:
python -m obs runs --data ../public-agent-runs/telemetry/design --limit 5run ticket started outcome model tools appr cost US$ secs model flagsR-109-3 task-109 Thu 11:38 resolved 12 6 0 0.06657 59.3 gpt-4.1-2025-04-14 duplicate_writeR-110-3 task-110 Thu 11:38 resolved 12 5 0 0.06665 64.1 gpt-4.1-2025-04-14 duplicate_writeR-111-3 task-111 Thu 11:38 handed_to_person 14 8 0 0.08078 66.0 gpt-4.1-2025-04-14 loopR-112-3 task-112 Thu 11:38 resolved 16 4 0 0.07336 72.6 gpt-4.1-2025-04-14 -R-113-3 task-113 Thu 11:38 resolved 8 6 0 0.03033 41.5 gpt-4.1-2025-04-14 duplicate_write,loop456 run(s), 324 flagged (last 5 shown)What this tells us. The course’s tool reads another team’s agent with no change: runs, outcomes, model calls, tools, costs and times. The costs are the benchmark’s recorded costs, not the course’s price table. The start times are the benchmark’s, read as UTC and shown in Larkfield’s time zone. And the last column is a warning: 324 of 456 runs are flagged.
Explore what the decision needs
Section titled “Explore what the decision needs”How runs end
Section titled “How runs end”All 456 runs ended normally: the simulated customer stopped the chat. 428 ended as resolved and 28 as handed_to_person. No run looped until the benchmark stopped it. So the loud failures of the course’s week are not here. If this agent fails, it fails calmly.
One run, step by step
Section titled “One run, step by step”python -m obs trace R-002-0 --data ../public-agent-runs/telemetry/designTrace 1 9cc6b0ee25dca4b7c7d22fc4a3cb15d5 Thu 05 Jun 11:33:58 36.2 s 0.0 s |========================| 36.20 s invoke_agent retail-agent finished 0.1 s |===== | 7.72 s customer_turn 7.8 s | == | 2.14 s chat gpt-4.1-2025-04-14 3531 in / 62 out US$0.00276 stop… 13.2 s | === | 2.94 s chat gpt-4.1-2025-04-14 3895 in / 129 out US$0.00364 tool_calls 16.1 s | = | 0.00 s execute_tool list_all_product_types ok 16.1 s | = | 0.00 s execute_tool get_order_details ok #W6247578 16.1 s | = | 0.00 s execute_tool get_order_details ok #W9711842 16.1 s | = | 0.00 s execute_tool get_order_details ok #W4776164 16.1 s | = | 0.00 s execute_tool get_order_details ok #W6679257 16.1 s | = | 0.00 s execute_tool get_order_details ok #W2378156… 27.9 s | == | 1.79 s chat gpt-4.1-2025-04-14 7225 in / 52 out US$0.00968 tool_calls 29.7 s | = | 0.00 s execute_tool return_delivered_order_items done #W2378156 29.7 s | === | 3.67 s chat gpt-4.1-2025-04-14 7820 in / 84 out US$0.01113 stop 33.4 s | ==| 2.82 s customer_turn
Outcome: resolvedFlag loop: the model asked for get_order_details(order_id) 5 times in one runWhat this tells us. You read it as in Read a run. The customer took 7.7 seconds to write first. One model call asked for six tools at once, among them five orders of the customer. Then one return was accepted. The benchmark passed this run. But the course’s loop rule flags it: five calls of get_order_details with the argument order_id.
Tool errors
Section titled “Tool errors”The course’s search for a failed read in a resolved run finds 46 runs:
python -m obs find --error-then-resolved --data ../public-agent-runs/telemetry/designR-036-0 task-36 Thu 11:34 gpt-4.1-2025-04-14 a tool error, then resolvedR-035-0 task-35 Thu 11:34 gpt-4.1-2025-04-14 a tool error, then resolved…46 match(es) in 456 run(s)Failed calls are common. Which of them matter?
A baseline: the course’s five rules, unchanged
Section titled “A baseline: the course’s five rules, unchanged”Run them all, then check their flags against the grade:
python -m obs find all --data ../public-agent-runs/telemetry/designR-002-0 task-2 Thu 11:33 gpt-4.1-2025-04-14 loop: the model asked for get_order_details(order_id) 5 times in one run…R-004-0 task-4 Thu 11:33 gpt-4.1-2025-04-14 duplicate_write: two writes with the same idempotency key None…477 match(es) in 456 run(s)In public-agent-runs:
python -m agentruns.check --data design --rules course456 runs, 118 failed by the benchmark's grade: base rate 26%. The course's rules: rule flagged failed hit rate loop 268 74 28% duplicate_write 163 45 28% no_action 31 6 19% hidden_tool_error 15 7 47% any rule 324 88 27%Failed runs that a rule flagged: 88 of 118. Why they failed (truth.json):…What this tells us. The rules flag 324 runs, and their hit rate, 27%, is the base rate, 26%. A flag on 71% of the runs says nothing. Each rule carries an assumption about Larkfield’s workflow, and this agent breaks it:
| Rule | Larkfield’s assumption | What this agent does |
|---|---|---|
loop |
one order per ticket: the same tool with the same argument names three times is a loop | a customer has several orders, so five get_order_details(order_id) calls are normal |
duplicate_write |
every write has an idempotency key | no write has a key, so “the same key” is “no key” on every pair of writes |
no_action |
a reply is a send_reply tool call |
the reply is the model’s text answer |
hidden_tool_error |
no later call of the same tool means the facts are missing | the policy’s identity check: an email that fails, then a name and zip code that work |
followed_injection |
the practice data marks its injected texts | no text has a marker |
The rules are not wrong. They are Larkfield’s, just as Contain a runaway agent found that one agent’s limits do not fit another agent.
Part 2: adapt the rules to this agent
Section titled “Part 2: adapt the rules to this agent”agentruns/rules.py has the store’s rules. Each one reads the same Run objects as obs, and returns a sentence with span names and times, as in Agent failure patterns:
loopcounts the same tool with the same arguments: the digest, not the names.duplicate_writecompares the tool and the digest, because there is no key.hidden_tool_errorcounts the two identity lookups as one purpose.no_actionandfollowed_injectionare dropped. A run with no change is often right here, because the customer only asked a question. And no text has a marker.
-
Open
agentruns/rules.pyand findTODO (Part 2). The starter counts a tool with its argument names, as the course’s rule does. Count the digest of the values instead:return s.tool, s.get(n.ARGS_DIGEST) -
Run the tests: one still fails, the rule of Part 3.
Part 3: one rule from the store’s policy
Section titled “Part 3: one rule from the store’s policy”The shop’s policy has rules that a trace can check without words. For example, “Exchange or modify order tools can only be called once per order”. We tried several of them on the design runs. Most of them fired on no run, or flagged runs that failed no more often than the base rate. One signal stood out: the store’s database refused a change, for example “Non-delivered order cannot be exchanged”. The run went on, and no later change of that kind on that order was accepted.
-
Find
TODO (Part 3), inrefused_change. For each refused write, look for a later accepted write of the same tool on the same order:writes = _writes(run)for i, s in enumerate(writes):if s.get(n.WRITE_OUTCOME) != "refused":continuelater = [x for x in writes[i + 1:] if x.tool == s.tool and x.get(n.ORDER_ID) == s.get(n.ORDER_ID)and x.get(n.WRITE_OUTCOME) == "done"]if not later:return (f"the store refused {s.tool} {s.get(n.ORDER_ID, '')} at {at(run, s)} "f"({s.status_message[:90]}), and no later {s.tool} on it was accepted").replace(" ", " ")return None -
Run the tests: the last line says
19 passed. -
Find the runs:
Terminal window python -m agentruns.find refused_change --data designR-027-0 task-27 Thu 11:34 refused_change: the store refused exchange_delivered_order_items #W7181492 at +39.1 s (Error: Non-delivered order cannot be exchanged), and no later exchange_delivered_order_items on it was acceptedR-030-0 task-30 Thu 11:34 refused_change: the store refused exchange_delivered_order_items #W2692684 at +22.7 s (Error: New item 3788616824 not found or available), and no later exchange_delivered_order_items on it was accepted…39 match(es) in 456 run(s), 39 run(s) flagged
Check one flag against the evidence. In observable-workflow, open the first run:
python -m obs trace R-027-0 --data ../public-agent-runs/telemetry/design… 35.3 s | === | 3.85 s chat gpt-4.1-2025-04-14 6230 in / 113 out US$0.00818 tool_calls 39.1 s | = | 0.00 s execute_tool return_delivered_order_items done #W7181492 39.1 s | = | 0.00 s execute_tool exchange_delivered_order_items ERROR refused #W7181492 39.1 s | === | 5.07 s chat gpt-4.1-2025-04-14 6903 in / 138 out US$0.00973 stop…Outcome: handed_to_personOne model call asked for two changes to one order at the same time. The store accepted the return and refused the exchange. The return changed the order’s status, so the exchange was no longer possible. Only now, look at the grade, in public-agent-runs:
python -m agentruns.check --data design --run R-027-0R-027-0 (task 27, trial 0): the benchmark's grade 0, failed: different_changes changes made 1, expected 1; database as expected: no; customer told what they had to hear: yes the course's rules: no flag the store's rules: refused_change: the store refused exchange_delivered_order_items #W7181492 at +39.1 s (Error: Non-delivered order cannot be exchanged), and no later exchange_delivered_order_items on it was acceptedA true flag. Not every flag is. R-095-3 had the same refusal and passed: the agent had already exchanged that order once, and its second try changed nothing. A flag is a reason to look, not a verdict.
Part 4: a fair comparison under one measure
Section titled “Part 4: a fair comparison under one measure”First, the store’s rules against the grade:
python -m agentruns.check --data design --rules store456 runs, 118 failed by the benchmark's grade: base rate 26%. The store's rules: rule flagged failed hit rate refused_change 39 18 46% hidden_tool_error 35 15 43% duplicate_write 3 1 33% loop 2 1 50% any rule 45 21 47%Failed runs that a rule flagged: 21 of 118. Why they failed (truth.json): why failed flagged wrong_arguments 50 9 different_changes 18 9 unasked_change 11 0 missing_change 13 1 no_change 13 2 right_changes_not_told 13 0Then the measure, the same 92 reviews for every method:
python -m agentruns.compare --data designdesign: review 92 of 456 runs (one in five); 118 failed by the grade. Failed runs found, on average: how the reviewer chooses found hit rate at random 23.8 26% the course's rules first 25.0 27% (324 flagged: 92 of them at random) the store's rules first 32.1 35% (45 flagged, then 47 at random)Text description of the chart
Three bars. At random: 23.8 failed runs found in 92 reviews. The course’s rules first: 25.0. The store’s rules first: 32.1, with 45 runs flagged, 21 of them failed.
What this tells us.
- The course’s rules do not beat a random sample: 25.0 against 23.8. They flag too much.
- The store’s rules flag 45 runs, and 21 of them failed: a hit rate of 47%, nearly twice the base rate. With them first, the same 92 reviews find 32.1 failed runs instead of 23.8, about a third more.
- But they flag only 21 of the 118 failed runs. Four failed runs in five leave no mark that a rule on the trace can see.
What the traces cannot see
Section titled “What the traces cannot see”Now read truth.json for the 97 failed runs that no rule flagged:
Text description of the chart
Six bars of failed runs that no rule flagged. Wrong arguments: 41. Other change: 9. Unasked change: 11. Missing change: 12. No change: 11. Not told: 13.
The largest group is the right tool with the wrong arguments: 50 failed runs, 41 of them not flagged. In 19 of the 50, only the new item differs: the agent chose another variant of the right product, such as another colour or size. It had looked up the product, and the item it chose was a real one. Only the customer’s words say which one they wanted.
Here is a failure that looks perfect in the trace:
python -m obs trace R-051-0 --data ../public-agent-runs/telemetry/design… 9.3 s | = | 0.75 s chat gpt-4.1-2025-04-14 3932 in / 20 out US$0.00265 tool_calls 10.0 s | = | 0.00 s execute_tool get_order_details ok #W8855135… 16.7 s | = | 0.83 s chat gpt-4.1-2025-04-14 4659 in / 26 out US$0.00377 tool_calls 17.5 s | = | 0.00 s execute_tool cancel_pending_order done #W8855135 17.5 s | === | 2.45 s chat gpt-4.1-2025-04-14 5135 in / 82 out US$0.00401 stop 20.0 s | ===| 1.88 s customer_turn
Outcome: resolvedThe identity check, one lookup, one cancellation, a satisfied customer. The grade says it failed: different_changes. The task’s expected change is a return on another order, #W4689314, which had been delivered (data/runs/retail-tasks.jsonl.gz). In the original conversation, the customer asked to return a camera. The agent found a pending order instead, and the customer agreed to cancel it. The trace records every step correctly. It cannot know that the step was the wrong one.
The recommendation
Section titled “The recommendation”Priya writes it for Amira, before the shop’s agent is built:
Text description of the diagram
Failed calls, refused changes and loops: the store’s rules, and review those runs first. A change that differs from the request: record the confirmed request as IDs and compare each write with it. A customer who was not told: a person reads the words. The failure rate: a random share of reviews.
- Write the agent’s own rules before launch, and measure their hit rate. Copy the method, not the rules: unchanged, the course’s rules stayed at the base rate; adapted, they reached 47%.
- Review flagged runs first. With the same 92 reviews, the team finds about a third more failures.
- Keep a random share of the reviews. Flagged-first reviews find more failures, so they overcount the failure rate. A random sample estimates it honestly.
- Record what the customer confirmed, as IDs. 79 of the 118 failures are a change that differs from the task’s request. A rule can see that only if the run records the request it acted on: the order, the items and the payment method. Larkfield’s support workflow records its proposal and a person’s approval. See Approvals, people and waiting time and Record decisions, approvals and changes.
- Let people read the words of a few runs. Whether the customer was told the right thing is in the text. Capture content only for the sampled reviews, as in What to record, and what not to.
Part 5: new input, end to end
Section titled “Part 5: new input, end to end”Rules adapted on one model’s runs may fit only that model. Run the whole method on Claude 3.7 Sonnet’s 456 runs of the same tasks.
-
Convert them:
Terminal window python -m agentruns.convert claude-3.7-sonnetclaude-3.7-sonnet (claude-3-7-sonnet-20250219): 456 runs -> 13,486 spans in telemetry/claude-3.7-sonnet/traces.jsonl.gzthe benchmark's grade -> telemetry/claude-3.7-sonnet/truth.json (97 of 456 runs failed; obs never reads it) -
Read them with
obs, inobservable-workflow:Terminal window python -m obs runs --data ../public-agent-runs/telemetry/claude-3.7-sonnet --limit 3run ticket started outcome model tools appr cost US$ secs model flagsR-111-3 task-111 Thu 12:00 resolved 18 9 0 0.52124 80.4 claude-3-7-sonnet-20250219 duplicate_write,loopR-112-3 task-112 Thu 12:00 resolved 23 10 0 0.67582 102.9 claude-3-7-sonnet-20250219 duplicate_write,loopR-113-3 task-113 Thu 12:00 resolved 14 6 0 0.30788 48.5 claude-3-7-sonnet-20250219 duplicate_write,loop456 run(s), 315 flagged (last 3 shown) -
Check the store’s rules, unchanged:
Terminal window python -m agentruns.check --data claude-3.7-sonnet --rules store456 runs, 97 failed by the benchmark's grade: base rate 21%. The store's rules:rule flagged failed hit raterefused_change 32 20 62%hidden_tool_error 21 13 62%loop 5 5 100%any rule 34 22 65%Failed runs that a rule flagged: 22 of 97. Why they failed (truth.json):… -
Compare the methods on all four models:
Terminal window python -m agentruns.convert o4-minipython -m agentruns.convert gpt-4.1-minipython -m agentruns.compare --data design claude-3.7-sonnet o4-mini gpt-4.1-mini --json results/compare.json…claude-3.7-sonnet: review 92 of 456 runs (one in five); 97 failed by the grade. Failed runs found, on average:how the reviewer chooses found hit rateat random 19.6 21%the course's rules first 21.0 23% (315 flagged: 92 of them at random)the store's rules first 32.3 35% (34 flagged, then 58 at random)o4-mini: review 92 of 456 runs (one in five); 130 failed by the grade. Failed runs found, on average:how the reviewer chooses found hit rateat random 26.2 29%the course's rules first 24.8 27% (274 flagged: 92 of them at random)the store's rules first 27.5 30% (19 flagged, then 73 at random)gpt-4.1-mini: review 92 of 456 runs (one in five); 155 failed by the grade. Failed runs found, on average:how the reviewer chooses found hit rateat random 31.3 34%the course's rules first 34.1 37% (356 flagged: 92 of them at random)the store's rules first 41.0 45% (101 flagged: 92 of them at random)
What this tells us.
- On Claude 3.7 Sonnet, the store’s rules held. Their hit rate was 65% against a base rate of 21%. The same reviews found 32.3 failed runs instead of 19.6. Its five loops were all real failures.
- On gpt-4.1-mini, they held too: 41.0 instead of 31.3.
- On o4-mini, they nearly stopped working: 27.5 instead of 26.2. Most of its failures were wrong arguments (73 of 130). The store refused changes in only 15 of its runs. A rule finds only the failures that leave its signature.
- On every model, the course’s rules, unchanged, stayed at the base rate.
So the recommendation stands, with one addition: measure the hit rate again for each new model, as a release check.
Optional: the runs in Grafana
Section titled “Optional: the runs in Grafana”If you have Docker and 20 GB of free disk, search the runs with TraceQL, as in Search runs with TraceQL. grafana.md in the project has the steps and five queries. Start Grafana in observable-workflow (docker compose -f grafana/compose.yaml up -d), then send the runs with python -m agentruns.convert design --otlp. The converter moves the times so that the last run ends now. Stop Grafana with docker compose -f grafana/compose.yaml down -v. The course team ran these steps once. Grafana was ready after 14 seconds, and --otlp sent all 12,142 spans in about a second. Tempo then held 362 of the 456 runs: the Collector’s tail sampling keeps a trace with a write, an error or more than 30 seconds, and drops the others. The five queries of grafana.md returned 362 runs, 44 runs with a refused change, 68 runs with a failed call that ended resolved, 11 runs handed to a person, and 345 runs with a model call over 5 seconds. These are candidates, not failures: the rules in agentruns/rules.py keep fewer, because they also check what happened later in the run.
TraceQL finds the candidates, for example every refused change: { resource.service.name = "tau2-retail-agent" && span.larkfield.write.outcome = "refused" }. It cannot ask “was a later change of the same kind accepted?”. The rules in agentruns/rules.py do that.
Common mistake
The course project is not found
Symptom:
The course project is not in /Users/you/projects/observable-workflow. Put this folder next to observable-workflow, or set OBS_PROJECT to the course project's folder.Cause: public-agent-runs is not next to observable-workflow, so the case study cannot import obs.
Recover: move the folder, or set OBS_PROJECT to your course project’s folder and run the command again.
Insights and limits
Section titled “Insights and limits”- A rule carries its workflow’s assumptions. On another agent, the course’s rules flagged 71% of the runs. List each rule’s assumption before you trust its count.
- The hit rate decides whether a flag is useful. Compare it with the base rate every time. 88 of 118 failures flagged sounds good, until you see that 324 runs were flagged.
- The structure finds only the failures that leave a structure. Refused changes, failed calls and loops are visible. A wrong choice among valid options, most of this agent’s failures, is not.
- A benchmark, not a help desk. The customers are a model, and the tasks are a test set, each run 4 times. One task can fail the same way in all four trials, so the counts are not 456 independent cases.
- The grade is one answer per task. A few tasks may accept more than one good outcome, and the benchmark counts only one.
Responsible use
Section titled “Responsible use”- Name the data honestly. These are another team’s benchmark runs. Present the hit rates as a rehearsal of the method, not as a measure of Larkfield’s agent.
- Keep personal values off the spans, even when they are made up. The converter keeps argument names and digests. Do not add the values to make the search easier.
- A flag is not a judgement of a person or a model. It chooses which runs a person reads first. The person decides.
- Credit the source. The data is MIT-licensed. Keep its licence text and the attribution in your copy, as the dataset card does.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This project shows what operations teams look for. It has a converter to a standard, a baseline, rules checked against a grade, one fair measure and a test on new input.
What to put in the repository:
public-agent-runs/ README.md the question, how to run it, the results table and the recommendation data/dataset.md the source, the licence, the checksums, the changes, the limits data/runs/ the fixed sample and the τ²-bench licence text LICENSE MIT, for your code agentruns/ names, store, convert, truth, rules, find, check, compare tests/ 19 tests results/ your compare.json- Make a new Git repository from your project folder, and push it to GitHub.
.gitignorekeeps outtelemetry/andresults/: addresults/compare.jsonwithgit add -f. - Write the README’s results table from your own
python -m agentruns.compareoutput, with the baseline and the new input. - Say in the README that the project needs the course project (
OBS_PROJECT), and thatpython -m pytestpasses when it is there.
In two sentences, for an interview:
“I converted 456 open, graded runs of an AI customer-service agent to OpenTelemetry traces and checked failure rules against the grade. Rules adapted to the agent nearly doubled the hit rate and found a third more failures per review. The grade showed which failures need a record of the customer’s request instead.”
Go further: convert the airline domain’s runs of the same benchmark with python -I tools/make_sample.py, in a new empty folder, and adapt the store’s tool table. Or write a rule for “modify or exchange twice on one order”, and measure its hit rate. Or put a cost on a review and on a missed failure, and choose the budget that gives the lowest total cost.
Knowledge check
Section titled “Knowledge check”Check your understanding
The course's five rules flagged 88 of the 118 failed runs. Why does Priya say that they found nothing useful?
Show the answer
Because they flagged 324 of the 456 runs, and only 27% of the flagged runs failed: the base rate is 26%. A reviewer who reads flagged runs first finds 25.0 failed runs in 92 reviews, about the same as at random (23.8). The rules carry Larkfield’s assumptions: one order per ticket, an idempotency key on every write, a send_reply tool. This agent breaks all three.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
R-051-0 cancelled a pending order, the customer agreed, and the run ended resolved. The grade says it failed. Why can no rule on its trace find it, and what record would let a rule find it?
Show the answer
Every step in the trace is correct: the identity check, the lookup, an accepted cancellation, a satisfied customer. The failure is the choice: the customer wanted to return a camera from another, delivered order. That is in the words, not in the structure. A record of the request as IDs (the order, the items, the change) would let a rule compare each write with it. Larkfield’s proposal and approval events are such a record.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The store's rules raised the failures found per 92 reviews from 23.8 to 32.1 on gpt-4.1, and from 19.6 to 32.3 on Claude 3.7 Sonnet. On o4-mini, they found 27.5 instead of 26.2. What should Larkfield do with this result?
Show the answer
Measure the hit rate of its rules again at every change of model, as a release check. A rule finds only the failures that leave its signature. o4-mini’s failures were mostly wrong arguments, and the store refused few of its changes, so the rules had little to find. Keep a random share of the reviews, so that the failure rate stays honest, and record the confirmed request so that wrong arguments become visible.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A converter turns recorded runs from another system into traces. One run is one trace, with the course’s names and the recorded times. No personal values go on a span, and the grade has its own file.
- The course’s rules, unchanged, flagged 71% of another agent’s runs at the base rate. Adapted, they reached a hit rate of 47% and found about a third more failures in the same 92 reviews.
- Four failed runs in five left no structural mark: mostly the right tool with the wrong arguments. A record of the confirmed request, as IDs, and a few human reviews are what find them.
- The adapted rules held on two of three other models. Measure the hit rate again at every change of model.
You can now read runs that another system recorded with your own tools. You can also say, with evidence, which failures their traces can show. Next, you measure the cost of a coding agent: many more steps, and a cost that depends on the path the agent takes.