From a trace to metrics and logs
- Lesson 15
- 35 min
- Module 4: Find agent failures in Grafana
Where we are
In the last lesson, we named five failure patterns and found each one in the traces of the week. But one flagged run does not tell us whether one customer was hurt, or many. In this lesson, we move from a trace to the events of the same run and to the metrics around it, and we decide whether a failure is one run or a pattern. We start with a restaurant manager and one cold soup.
Lesson 15 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsFrom a trace to metrics and logs
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Decide whether a failure is one run or a pattern. Move from a trace to the events of the same run and to the metrics around it. Then go back to other traces through exemplars and shared IDs.
Before you startWhat to know and to have ready
What you need to know
- The five failure patterns, and the success-without-action run R-1015-023, from Agent failure patterns.
- What events and metrics are, and how they differ from a trace, from What to record, and what not to.
What you need on your computer
- The project as the last lesson left it: 93 tests pass.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, and the course did not run them. Exemplars in Grafana were not checked by the course.
One cold soup
Section titled “One cold soup”A guest at a restaurant sends back a soup: it is cold. The manager can say sorry and bring a new bowl. But first she asks herself: one bad bowl, or a problem in the kitchen?
She does two things. She reads the order slip of that table: when the soup was ordered, when it left the kitchen, who carried it. The slip has the table number. Then she looks at the count of dishes sent back, hour by hour. One cold soup at 13:10 is bad luck. Nine cold dishes between 12:30 and 15:30 mean that the warmer broke at lunch.
The slip tells the story of one order. The count tells her how often. The table number and the time join them.
The problem: the flag found four, the harm was wider
Section titled “The problem: the flag found four, the harm was wider”In the last lesson, the rule for success without action flagged 4 runs on Thursday. R-1015-023 is one of them: the reply failed, nothing was written, and the ticket closed as resolved. Amira’s first idea is to fix “those four tickets”.
But the rule asks a narrow question: resolved, and no write, and no reply. A run that made a refund and then failed to send its reply is not flagged: something was written. Was that customer hurt too? Was this one model, one ticket, one hour, or one release? The trace of R-1015-023 cannot say. It is one run.
The idea: three signals, joined by shared IDs
Section titled “The idea: three signals, joined by shared IDs”The workflow records three signals about each run, and Module 1 showed what each one answers. Here they work together:
- The trace is the story of one run, with its spans.
- Events are records with a fixed name, such as
larkfield.reply.failed, written at a moment of the run. Each event carries the trace ID, the span ID and the run ID, so it belongs to one run, like the order slip. - Metrics are counts and sums for every run, by a few labels such as the tool and its status. They are never sampled. They carry no run ID, because one series per run would make the metrics huge (Module 5 explains this). So a metric says how often, like the manager’s count.
How do you get from a count back to a run? With an exemplar: an example that a metric point carries. When the workflow adds 1 to a counter during a span, the OpenTelemetry SDK can attach that span’s trace ID to the point. The point then says, “3 replies failed in this hour, and here is one of them.”
Text description of the diagram
A metric point counts every run and has no run ID. Its exemplar carries one example trace ID, which leads to a trace. The trace has a trace ID, span IDs and a run ID. The events of the same run carry the same trace ID and run ID, and so do the audit records of Module 6.
So you can walk in both directions. From a trace to its events by the run ID. From a trace to the metrics around it by the time and the labels. From a metric point back to a trace by its exemplar.
Add the lesson’s files
Section titled “Add the lesson’s files”This lesson’s code makes your workflow write events: when a run waits for a person, when a reply fails, and when a run closes. The saved week was recorded with them already.
-
Copy the files:
Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m04cp $S/resolver/live.py resolver/cp $S/tests/test_approval_traces.py tests/cp $S/README.md .Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m04"Copy-Item $S\resolver\live.py resolver\Copy-Item $S\tests\test_approval_traces.py tests\Copy-Item -Force $S\README.md .Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m04cp $S/resolver/live.py resolver/cp $S/tests/test_approval_traces.py tests/cp $S/README.md . -
Run the tests: the last line says
93 passed, 1 skipped, 1 deselected. Your project is now the same as the snapshotend-of-m04.
Worked example: from one run to a pattern
Section titled “Worked example: from one run to a pattern”Follow R-1015-023 out of its trace, one step at a time. Each step uses a shared ID or the time.
Text description of the diagram
Step 1 is the trace of R-1015-023: the reply failed, nothing was written, and the ticket closed as resolved. Step 2 reads the events with the same run ID: a reply failure and the closing event. Step 3 looks for the same event in every run: 9 reply failures, all on Thursday between 12:32 and 15:14. All of them closed as resolved. Step 4 reads the metric of reply calls: failures from 12:00 to 16:00, and no reply sent from 13:00 to 15:00. Step 5 follows an exemplar to R-1015-020, where a refund was made and the reply failed. The conclusion: a pattern of 9 runs in one window, not one run.
-
The run’s events. Read the events with the run ID of R-1015-023:
Terminal window python -m obs events R-1015-023 --data data/telemetry/weekThu 15 Oct 14:03:34 larkfield.reply.failed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107"}Thu 15 Oct 14:03:34 larkfield.run.closed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}2 event(s)The trace ID
05e8ee9cis the trace you read in the last lesson. The closing event says what the help desk saw:helpdesk.status: resolved, withreply: failed. -
The same event in every run. Drop the run ID, and ask for the event by its name:
Terminal window python -m obs events --data data/telemetry/week --name larkfield.reply.failedThu 15 Oct 12:32:55 larkfield.reply.failed trace f62fe6a2 {"run.id": "R-1015-017", "ticket.id": "T-90209"}Thu 15 Oct 13:01:17 larkfield.reply.failed trace e1d921f7 {"run.id": "R-1015-018", "ticket.id": "T-90408"}Thu 15 Oct 13:04:37 larkfield.reply.failed trace 335d5d16 {"run.id": "R-1015-019", "ticket.id": "T-90705"}Thu 15 Oct 13:33:46 larkfield.reply.failed trace 4f92fb98 {"run.id": "R-1015-020", "ticket.id": "T-90104"}Thu 15 Oct 14:03:34 larkfield.reply.failed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107"}Thu 15 Oct 14:55:59 larkfield.reply.failed trace b720472b {"run.id": "R-1015-025", "ticket.id": "T-90701"}Thu 15 Oct 15:08:58 larkfield.reply.failed trace 8d8aa1d5 {"run.id": "R-1015-028", "ticket.id": "T-90402"}Thu 15 Oct 15:12:50 larkfield.reply.failed trace 84543e9b {"run.id": "R-1015-027", "ticket.id": "T-90101"}Thu 15 Oct 15:14:43 larkfield.reply.failed trace 0b33e67c {"run.id": "R-1015-030", "ticket.id": "T-90502"}9 event(s)Nine reply failures, all on Thursday, from 12:32 to 15:14, and none on any other day. Nine different tickets. So it is not one ticket and not one customer.
-
How each one closed. Keep the closing events with a failed reply:
Terminal window python -m obs events --data data/telemetry/week --name larkfield.run.closed | grep '"reply": "failed"'Terminal window python -m obs events --data data/telemetry/week --name larkfield.run.closed | Select-String '"reply": "failed"'Terminal window python -m obs events --data data/telemetry/week --name larkfield.run.closed | grep '"reply": "failed"'Thu 15 Oct 12:32:55 larkfield.run.closed trace f62fe6a2 {"run.id": "R-1015-017", "ticket.id": "T-90209", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}Thu 15 Oct 13:01:17 larkfield.run.closed trace e1d921f7 {"run.id": "R-1015-018", "ticket.id": "T-90408", "outcome": "resolved", "writes_done": 1, "reply": "failed", "helpdesk.status": "resolved"}…Thu 15 Oct 15:14:43 larkfield.run.closed trace 0b33e67c {"run.id": "R-1015-030", "ticket.id": "T-90502", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}All nine closed as
resolved. Four havewrites_done: 0: the four runs that the rule flagged. Five havewrites_done: 1: a refund, label or reshipment was made, and the customer was never told. No rule flagged them. -
The metric around it. Now count every reply call on Thursday, by status, hour by hour.
obs metricsprints a counter’s increase between two exported points, and the exemplar of each point:Terminal window python -m obs metrics larkfield.tool.calls --data data/telemetry/week --where gen_ai.tool.name=send_reply --day thularkfield.tool.calls: Tool calls, increase per exported interval, gen_ai.tool.name=send_reply, thuThu 09:00-10:00 +8 tool.kind=write tool.status=sent example: R-1014-051 (trace f346995f)Thu 10:00-11:00 +3 tool.kind=write tool.status=sent example: R-1015-008 (trace e2f9fffd)Thu 11:00-12:00 +6 tool.kind=write tool.status=sent example: R-1015-012 (trace 86e3553d)Thu 12:00-13:00 +1 tool.kind=write tool.status=failed example: R-1015-017 (trace f62fe6a2)Thu 12:00-13:00 +1 tool.kind=write tool.status=sent example: R-1015-015 (trace 675517b2)Thu 13:00-14:00 +3 tool.kind=write tool.status=failed example: R-1015-020 (trace 4f92fb98)Thu 14:00-15:00 +2 tool.kind=write tool.status=failed example: R-1015-023 (trace 05e8ee9c)Thu 15:00-16:00 +3 tool.kind=write tool.status=failed example: R-1015-028 (trace 8d8aa1d5)Thu 15:00-16:00 +4 tool.kind=write tool.status=sent example: R-1015-032 (trace 8fc571d6)Thu 16:00-17:00 +5 tool.kind=write tool.status=sent example: R-1015-034 (trace 24b77b9e)Thu 17:00-18:00 +5 tool.kind=write tool.status=sent example: R-1015-044 (trace 3fe78f36)Thu 18:00-19:00 +3 tool.kind=write tool.status=sent example: R-1015-048 (trace 34a77ddf)Thu 19:00-20:00 +2 tool.kind=write tool.status=sent example: R-1015-050 (trace c300cc75)13 interval(s) with an increase, 13 with an example trace
Draw the two series, and the window appears:
- sent
- failed
Text description of the chart
Two lines over Thursday, from 06:00 to 21:00. Sent replies per hour: 6:00 0, 7:00 0, 8:00 0, 9:00 8, 10:00 3, 11:00 6, 12:00 1, 13:00 0, 14:00 0, 15:00 4, 16:00 5, 17:00 5, 18:00 3, 19:00 2, 20:00 0. Failed replies per hour: 6:00 0, 7:00 0, 8:00 0, 9:00 0, 10:00 0, 11:00 0, 12:00 1, 13:00 3, 14:00 2, 15:00 3, 16:00 0, 17:00 0, 18:00 0, 19:00 0, 20:00 0. The failed line is above zero only from 12:00 to 16:00: 1, 3, 2 and 3. In those hours the sent line falls to 1, then 0, 0, and back to 4. The point at 13:00 to 14:00 carries the exemplar R-1015-020.
The metric counts every run, so it shows what the events suggested: from 13:00 to 15:00, not one reply was sent. That is the reply service’s outage, a practice incident we caused on purpose (Thursday 12:30 to 15:30). Before 12:00 and after 16:00, every reply was sent.
Now follow the exemplar of the 13:00 point back to a trace:
python -m obs trace R-1015-020 --data data/telemetry/weekRun R-1015-020 | ticket T-90104 | chat-strong (agent_structured) | release 2.1.0 | language en…Trace 2 4f92fb98fb8725c926cdcb57e19fb417 Thu 15 Oct 13:33:46 0.0 s 0.0 s |========================| 0.00 s approve support-resolution approved by usr-sam (agent, limit 100) 0.0 s |========================| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e 0.0 s | =| 0.00 s execute_tool send_reply ERROR failed
Outcome: resolvedTicket T-90104 again, the watering can. This time the refund was made once, the reply failed, and the ticket closed as resolved. It is one of the five runs that no rule flagged.
So the answer is: a pattern, not one run. Nine customers in one window, all with a failed reply, all closed as resolved. Two causes worked together. The reply service was down (the practice incident). And release 2.1.0 closed a ticket as resolved after a failed reply (its planted bug). Release 2.1.1, from Friday 11:30, sends such a ticket to a person instead. Amira’s fix is the release, not four tickets, and the support team must contact nine customers.
A second walk: the double refunds
Section titled “A second walk: the double refunds”The same path works for the double refund. Count the write calls that timed out, for the whole week:
python -m obs metrics larkfield.writes --data data/telemetry/week --where larkfield.write.outcome=timeoutlarkfield.writes: Write calls by outcome, increase per exported interval, larkfield.write.outcome=timeoutMon 11:00-12:00 +1 tool.name=create_return_label example: R-1012-012 (trace 064b08d9)Mon 11:00-12:00 +1 tool.name=request_refund example: R-1012-015 (trace 9658f20b)Tue 11:00-12:00 +1 tool.name=request_refund example: R-1013-014 (trace a50f35d4)Tue 17:00-18:00 +1 tool.name=request_refund example: R-1013-043 (trace 9dc2a00c)Wed 13:00-14:00 +1 tool.name=create_return_label example: R-1014-024 (trace 0bf792cf)Thu 09:00-10:00 +2 tool.name=request_refund example: R-1014-051 (trace f346995f)Fri 12:00-13:00 +1 tool.name=create_return_label example: R-1016-021 (trace 76e38e4d)Fri 14:00-15:00 +1 tool.name=create_return_label example: R-1016-032 (trace 3372a85a)Sat 10:00-11:00 +1 tool.name=request_refund example: R-1017-001 (trace da363bec)9 interval(s) with an increase, 9 with an example traceTimeouts happen on most days: each of these tickets has a recorded practice fault. A timeout alone is not a failure. Under releases 2.0.0 and 2.1.1, the safe retry checks with a reconcile, as R-1012-012 showed in the last lesson. The two points that matter are under release 2.1.0, with the naive retry. Wednesday 13:00 is R-1014-024, the label that the trace could not judge. Thursday 09:00 to 10:00 has +2 refunds. The payment service was slow then (a practice incident, 08:30 to 09:30), while Sam approved the night’s proposals. Its exemplar is R-1014-051, one of the two double refunds. The metric shows when and how often; the exemplar and the traces show which runs.
Guided practice: make your own pattern
Section titled “Guided practice: make your own pattern”Now cause a failed reply in your own workflow, first with the safe setting, then with release 2.1.0’s bug.
-
Start from a fresh
work/folder, and run ticket T-90107 with a reply service that is down:Terminal window rm -rf workpython -m resolver run T-90107 --fault send_reply:unavailableTerminal window Remove-Item -Recurse -Force workpython -m resolver run T-90107 --fault send_reply:unavailableTerminal window rm -rf workpython -m resolver run T-90107 --fault send_reply:unavailableThe last line:
Reply failed. Ticket: handed to person.The default setting,RESOLVER_REPLY_FAILURE=person, does the right thing.python -m obs find allprints0 match(es) in 1 run(s). -
Now run it again with release 2.1.0’s bug:
Terminal window rm -rf workRESOLVER_REPLY_FAILURE=close python -m resolver run T-90107 --fault send_reply:unavailableTerminal window Remove-Item -Recurse -Force work$env:RESOLVER_REPLY_FAILURE = "close"python -m resolver run T-90107 --fault send_reply:unavailableRemove-Item Env:RESOLVER_REPLY_FAILURETerminal window rm -rf workRESOLVER_REPLY_FAILURE=close python -m resolver run T-90107 --fault send_reply:unavailableThe last line:
Reply failed. Ticket: resolved. -
Find the pattern, read the events, and count:
Terminal window python -m obs find allpython -m obs eventspython -m obs metrics larkfield.tool.calls --where gen_ai.tool.name=send_replyT-90107-agent-chat-small-1 T-90107 … chat-small no_action: closed as resolved with no write and no reply sent (send_reply failed at +3.4 s)1 match(es) in 1 run(s)… larkfield.reply.failed trace … {"run.id": "T-90107-agent-chat-small-1", "ticket.id": "T-90107"}… larkfield.run.closed trace … {"run.id": "T-90107-agent-chat-small-1", "ticket.id": "T-90107", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}2 event(s)larkfield.tool.calls: Tool calls, increase per exported interval, gen_ai.tool.name=send_reply… +1 tool.kind=write tool.status=failed example: T-90107-agent-chat-small-1 (trace …)1 interval(s) with an increase, 1 with an example traceYour times and trace IDs differ. Check that the trace ID in the events and in the metric’s example are the same: the three signals point to one run.
Common mistake
obs events prints 0 event(s) for your own run
Symptom: after a run of your own, python -m obs events prints only 0 event(s).
Cause: your resolver/live.py is still Module 3’s version, which writes no events. The saved week has them because it was recorded with the course’s final code.
Diagnose: python -m pytest -q says 93 tests either way, so check the file: grep -c "larkfield.run.closed" resolver/live.py (Windows: Select-String larkfield.run.closed resolver\live.py) finds nothing in the old file.
Recover: copy resolver/live.py from the snapshot end-of-m04 again, delete work/, and run the ticket again.
Common mistake
obs metrics says No metric
Symptom: python -m obs metrics larkfield.tool.call --data data/telemetry/week prints No metric larkfield.tool.call in data/telemetry/week.
Cause: the metric name has a typo (call, not calls), or your own work/telemetry has no metrics yet. The workflow writes your metrics when each command ends.
Recover: use the exact names: larkfield.tool.calls, larkfield.writes, larkfield.runs, larkfield.run.stops, and the GenAI duration metrics of Module 2.
In Grafana
Section titled “In Grafana”In Grafana, the same walk uses three data sources: Tempo for the trace, Loki for the events, and Prometheus for the metrics. A time-series panel can show exemplars as small dots, when Prometheus stores them, and a click on a dot opens the example trace. The MLOps course’s lesson on logs, metrics and traces sets up these links for one service. The course did not check exemplars in its Grafana run. obs metrics prints them from the saved files, so this lesson works without Grafana.
Your turn: one loop, or a pattern?
Section titled “Your turn: one loop, or a pattern?”Change one condition: start from the loop of R-1017-007 on Saturday.
-
Predict: Saturday had three loops. Was the loop a problem of one ticket, or of something wider? Which metric would tell you?
-
Count the runs that the loop’s limits stopped, for the whole week, by reason:
Terminal window python -m obs metrics larkfield.run.stops --data data/telemetry/week -
Follow two exemplars with
obs trace, and note each run’s model.
Check your prediction
larkfield.run.stops: Runs stopped by a limit, by reason, increase per exported intervalWed 10:00-11:00 +1 stop_reason=provider_error example: R-1014-014 (trace 7f6ab892)Thu 05:00-06:00 +1 stop_reason=provider_error example: R-1015-001 (trace 5b4f935f)Sat 09:00-10:00 +1 stop_reason=provider_error example: R-1017-004 (trace d21473ff)Sat 09:00-10:00 +1 stop_reason=too_many_errors example: R-1017-005 (trace 05936b96)Sat 10:00-11:00 +2 stop_reason=repeated_call example: R-1017-006 (trace 3197f96c)Sat 11:00-12:00 +2 stop_reason=too_many_errors example: R-1017-010 (trace 7ad8163f)…Sun 17:00-18:00 +1 stop_reason=repeated_call example: R-1018-023 (trace 35d276a1)Sun 21:00-22:00 +1 stop_reason=too_many_errors example: R-1018-028 (trace 3ed419fd)14 interval(s) with an increase, 14 with an example traceThe stops by repeated_call (3) and too_many_errors (10) happen only on Saturday and Sunday, and their exemplars are gemma3:4b runs. Together they are 13 of the 16 runs on the local model. The other 3 closed as resolved, among them the hidden tool errors R-1018-010 and R-1018-022. So the loop is a pattern of Tomás’s weekend trial, not of one ticket. The three provider_error stops are different: ticket T-80002 on chat-small, the content filter recorded by the agents course, on three days. Only the repeated_call stops are flagged as loops; the too_many_errors stops are a sign to read next.
When to move from a trace to metrics, and what it costs
Section titled “When to move from a trace to metrics, and what it costs”Make this walk every time a rule or a person finds a bad run, before you fix anything. The question “one run or a pattern?” decides the fix: a ticket, a release, a model, or a service.
The walk has costs and limits. Events cost storage for every run, so keep them few and named, with IDs and no text. Metrics are cheap and complete, but they have only a few labels, so they can say “9 failed replies”, never “which customer”. An exemplar is one example per point, not the full list: for the list, go back to the events or to a search. And all of this works only because every signal carries the same IDs. A signal without the trace ID or the run ID is an island.
Knowledge check
Section titled “Knowledge check”Check your understanding
The no-action rule flagged 4 runs on Thursday. How did the events show that the harm was wider, and how wide?
Show the answer
The events named larkfield.reply.failed showed 9 failed replies, all on Thursday from 12:32 to 15:14, on 9 different tickets. The closing events of those 9 runs all say resolved. Four have writes_done 0: the flagged runs. Five have writes_done 1: a refund, label or reshipment was made, and the customer was never told. So 9 customers were affected, not 4.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
What is an exemplar, and why does a metric point need one to lead you to a run?
Show the answer
An exemplar is an example that a metric point carries: the trace ID (and span ID) of one span that added to the count. A metric has no run ID in its labels, because one series per run would make the metrics far too large. So the count alone cannot name a run. The exemplar is the bridge from “3 replies failed in this hour” to one of those runs, here R-1015-020.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Module check: for one failed run of the week, show with trace evidence which step caused the failure and why the reply looked fine.
Show the answer
Run R-1018-010, gemma3:4b on ticket T-90602, a question about a double charge. The step that caused the failure is execute_tool get_payments at +3.7 s, with ERROR service_unavailable. No later get_payments call succeeded, so the model never saw the payments. The model then made five more model calls. The run sent a reply (execute_tool send_reply, sent, at +19.7 s), and the outcome is resolved, with the proposal reply_only. The reply looked fine because it was sent and the ticket closed as resolved. Nothing in the outcome, the log line or the help desk says that the payment check failed. Only the red span shows it. The course’s labels expected a hand-off to a person for this ticket.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A trace tells the story of one run. Events with the same run ID add what happened at named moments. Metrics count every run, and an exemplar leads from a count back to an example trace.
- From R-1015-023, the events found 9 failed replies on Thursday, all closed as resolved. Four had no action, and five had a write but no reply. The metric showed the window: no reply sent from 13:00 to 15:00. A pattern, caused by the reply outage and release 2.1.0’s bug.
- The timeout counter put the two double refunds in one hour, Thursday 09:00 to 10:00, with an exemplar to R-1014-051.
Your project is now the same as the snapshot end-of-m04: your workflow writes events, obs find and obs metrics read them, and 92 tests pass. You can now find agent failures and tell one run from a pattern. But your own traces still show the cost of each model call as “-”. Nobody can say which step, model or customer made the week’s bill rise. Next, in Module 5, you put the cost on every run and every step.