Skip to content
Français

From a trace to metrics and logs

  • Lesson 15
  • 35 min
  • Module 4: Find agent failures in Grafana

Where we are

In the last lesson, we named five failure patterns and found each one in the traces of the week. But one flagged run does not tell us whether one customer was hurt, or many. In this lesson, we move from a trace to the events of the same run and to the metrics around it, and we decide whether a failure is one run or a pattern. We start with a restaurant manager and one cold soup.

What you will be able to do

Decide whether a failure is one run or a pattern. Move from a trace to the events of the same run and to the metrics around it. Then go back to other traces through exemplars and shared IDs.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project as the last lesson left it: 93 tests pass.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, and the course did not run them. Exemplars in Grafana were not checked by the course.

A guest at a restaurant sends back a soup: it is cold. The manager can say sorry and bring a new bowl. But first she asks herself: one bad bowl, or a problem in the kitchen?

She does two things. She reads the order slip of that table: when the soup was ordered, when it left the kitchen, who carried it. The slip has the table number. Then she looks at the count of dishes sent back, hour by hour. One cold soup at 13:10 is bad luck. Nine cold dishes between 12:30 and 15:30 mean that the warmer broke at lunch.

The slip tells the story of one order. The count tells her how often. The table number and the time join them.

The problem: the flag found four, the harm was wider

Section titled “The problem: the flag found four, the harm was wider”

In the last lesson, the rule for success without action flagged 4 runs on Thursday. R-1015-023 is one of them: the reply failed, nothing was written, and the ticket closed as resolved. Amira’s first idea is to fix “those four tickets”.

But the rule asks a narrow question: resolved, and no write, and no reply. A run that made a refund and then failed to send its reply is not flagged: something was written. Was that customer hurt too? Was this one model, one ticket, one hour, or one release? The trace of R-1015-023 cannot say. It is one run.

The idea: three signals, joined by shared IDs

Section titled “The idea: three signals, joined by shared IDs”

The workflow records three signals about each run, and Module 1 showed what each one answers. Here they work together:

  • The trace is the story of one run, with its spans.
  • Events are records with a fixed name, such as larkfield.reply.failed, written at a moment of the run. Each event carries the trace ID, the span ID and the run ID, so it belongs to one run, like the order slip.
  • Metrics are counts and sums for every run, by a few labels such as the tool and its status. They are never sampled. They carry no run ID, because one series per run would make the metrics huge (Module 5 explains this). So a metric says how often, like the manager’s count.

How do you get from a count back to a run? With an exemplar: an example that a metric point carries. When the workflow adds 1 to a counter during a span, the OpenTelemetry SDK can attach that span’s trace ID to the point. The point then says, “3 replies failed in this hour, and here is one of them.”

Four boxes. At the top, a metric point: a count for every run, no run ID in its labels. An arrow labelled exemplar, one example trace ID, goes down to the trace: one run's steps, with trace ID, span ID and run ID. From the trace, an arrow labelled the same trace ID and run ID goes to the events: what happened, by name, with trace ID, span ID and run ID. A dotted arrow with the same label goes to the audit record of Module 6: who decided what, with trace ID and run ID.
How the signals of one run are joined: shared IDs between a trace, its events and its audit records, and an exemplar from a metric point to an example trace.
Text description of the diagram

A metric point counts every run and has no run ID. Its exemplar carries one example trace ID, which leads to a trace. The trace has a trace ID, span IDs and a run ID. The events of the same run carry the same trace ID and run ID, and so do the audit records of Module 6.

So you can walk in both directions. From a trace to its events by the run ID. From a trace to the metrics around it by the time and the labels. From a metric point back to a trace by its exemplar.

This lesson’s code makes your workflow write events: when a run waits for a person, when a reply fails, and when a run closes. The saved week was recorded with them already.

  1. Copy the files:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m04
    cp $S/resolver/live.py resolver/
    cp $S/tests/test_approval_traces.py tests/
    cp $S/README.md .
  2. Run the tests: the last line says 93 passed, 1 skipped, 1 deselected. Your project is now the same as the snapshot end-of-m04.

Follow R-1015-023 out of its trace, one step at a time. Each step uses a shared ID or the time.

Six steps from top to bottom. 1, one trace: R-1015-023, send_reply ERROR failed, closed as resolved, no write. 2, the run's events, the same run ID: larkfield.reply.failed and larkfield.run.closed with writes_done 0 and reply failed. 3, the same event in all runs: 9 reply failures, all on Thursday from 12:32 to 15:14, all closed as resolved. 4, the metric around it: send_reply failed per hour from 12:00 to 16:00, 1, 3, 2, 3; sent from 13:00 to 15:00, 0. 5, an exemplar: 13:00 to 14:00, R-1015-020, a refund was made and the reply failed. 6, a pattern, not one run: 9 runs in one window, 4 with no action and 5 with a write but no reply.
The walk of this worked example: a trace, its events, the same event in every run, the metric around it, an exemplar, and the conclusion.
Text description of the diagram

Step 1 is the trace of R-1015-023: the reply failed, nothing was written, and the ticket closed as resolved. Step 2 reads the events with the same run ID: a reply failure and the closing event. Step 3 looks for the same event in every run: 9 reply failures, all on Thursday between 12:32 and 15:14. All of them closed as resolved. Step 4 reads the metric of reply calls: failures from 12:00 to 16:00, and no reply sent from 13:00 to 15:00. Step 5 follows an exemplar to R-1015-020, where a refund was made and the reply failed. The conclusion: a pattern of 9 runs in one window, not one run.

  1. The run’s events. Read the events with the run ID of R-1015-023:

    Terminal window
    python -m obs events R-1015-023 --data data/telemetry/week
    Thu 15 Oct 14:03:34 larkfield.reply.failed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107"}
    Thu 15 Oct 14:03:34 larkfield.run.closed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}
    2 event(s)

    The trace ID 05e8ee9c is the trace you read in the last lesson. The closing event says what the help desk saw: helpdesk.status: resolved, with reply: failed.

  2. The same event in every run. Drop the run ID, and ask for the event by its name:

    Terminal window
    python -m obs events --data data/telemetry/week --name larkfield.reply.failed
    Thu 15 Oct 12:32:55 larkfield.reply.failed trace f62fe6a2 {"run.id": "R-1015-017", "ticket.id": "T-90209"}
    Thu 15 Oct 13:01:17 larkfield.reply.failed trace e1d921f7 {"run.id": "R-1015-018", "ticket.id": "T-90408"}
    Thu 15 Oct 13:04:37 larkfield.reply.failed trace 335d5d16 {"run.id": "R-1015-019", "ticket.id": "T-90705"}
    Thu 15 Oct 13:33:46 larkfield.reply.failed trace 4f92fb98 {"run.id": "R-1015-020", "ticket.id": "T-90104"}
    Thu 15 Oct 14:03:34 larkfield.reply.failed trace 05e8ee9c {"run.id": "R-1015-023", "ticket.id": "T-90107"}
    Thu 15 Oct 14:55:59 larkfield.reply.failed trace b720472b {"run.id": "R-1015-025", "ticket.id": "T-90701"}
    Thu 15 Oct 15:08:58 larkfield.reply.failed trace 8d8aa1d5 {"run.id": "R-1015-028", "ticket.id": "T-90402"}
    Thu 15 Oct 15:12:50 larkfield.reply.failed trace 84543e9b {"run.id": "R-1015-027", "ticket.id": "T-90101"}
    Thu 15 Oct 15:14:43 larkfield.reply.failed trace 0b33e67c {"run.id": "R-1015-030", "ticket.id": "T-90502"}
    9 event(s)

    Nine reply failures, all on Thursday, from 12:32 to 15:14, and none on any other day. Nine different tickets. So it is not one ticket and not one customer.

  3. How each one closed. Keep the closing events with a failed reply:

    Terminal window
    python -m obs events --data data/telemetry/week --name larkfield.run.closed | grep '"reply": "failed"'
    Thu 15 Oct 12:32:55 larkfield.run.closed trace f62fe6a2 {"run.id": "R-1015-017", "ticket.id": "T-90209", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}
    Thu 15 Oct 13:01:17 larkfield.run.closed trace e1d921f7 {"run.id": "R-1015-018", "ticket.id": "T-90408", "outcome": "resolved", "writes_done": 1, "reply": "failed", "helpdesk.status": "resolved"}
    …
    Thu 15 Oct 15:14:43 larkfield.run.closed trace 0b33e67c {"run.id": "R-1015-030", "ticket.id": "T-90502", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}

    All nine closed as resolved. Four have writes_done: 0: the four runs that the rule flagged. Five have writes_done: 1: a refund, label or reshipment was made, and the customer was never told. No rule flagged them.

  4. The metric around it. Now count every reply call on Thursday, by status, hour by hour. obs metrics prints a counter’s increase between two exported points, and the exemplar of each point:

    Terminal window
    python -m obs metrics larkfield.tool.calls --data data/telemetry/week --where gen_ai.tool.name=send_reply --day thu
    larkfield.tool.calls: Tool calls, increase per exported interval, gen_ai.tool.name=send_reply, thu
    Thu 09:00-10:00 +8 tool.kind=write tool.status=sent example: R-1014-051 (trace f346995f)
    Thu 10:00-11:00 +3 tool.kind=write tool.status=sent example: R-1015-008 (trace e2f9fffd)
    Thu 11:00-12:00 +6 tool.kind=write tool.status=sent example: R-1015-012 (trace 86e3553d)
    Thu 12:00-13:00 +1 tool.kind=write tool.status=failed example: R-1015-017 (trace f62fe6a2)
    Thu 12:00-13:00 +1 tool.kind=write tool.status=sent example: R-1015-015 (trace 675517b2)
    Thu 13:00-14:00 +3 tool.kind=write tool.status=failed example: R-1015-020 (trace 4f92fb98)
    Thu 14:00-15:00 +2 tool.kind=write tool.status=failed example: R-1015-023 (trace 05e8ee9c)
    Thu 15:00-16:00 +3 tool.kind=write tool.status=failed example: R-1015-028 (trace 8d8aa1d5)
    Thu 15:00-16:00 +4 tool.kind=write tool.status=sent example: R-1015-032 (trace 8fc571d6)
    Thu 16:00-17:00 +5 tool.kind=write tool.status=sent example: R-1015-034 (trace 24b77b9e)
    Thu 17:00-18:00 +5 tool.kind=write tool.status=sent example: R-1015-044 (trace 3fe78f36)
    Thu 18:00-19:00 +3 tool.kind=write tool.status=sent example: R-1015-048 (trace 34a77ddf)
    Thu 19:00-20:00 +2 tool.kind=write tool.status=sent example: R-1015-050 (trace c300cc75)
    13 interval(s) with an increase, 13 with an example trace

Draw the two series, and the window appears:

  • sent
  • failed
The counter larkfield.tool.calls for send_reply, increase per hour, Thursday 15 October 06:00 to 21:00 (Toronto time), by status. Each point also carries an exemplar: an example trace. Simulated traffic and reply outage; replayed decisions.
Text description of the chart

Two lines over Thursday, from 06:00 to 21:00. Sent replies per hour: 6:00 0, 7:00 0, 8:00 0, 9:00 8, 10:00 3, 11:00 6, 12:00 1, 13:00 0, 14:00 0, 15:00 4, 16:00 5, 17:00 5, 18:00 3, 19:00 2, 20:00 0. Failed replies per hour: 6:00 0, 7:00 0, 8:00 0, 9:00 0, 10:00 0, 11:00 0, 12:00 1, 13:00 3, 14:00 2, 15:00 3, 16:00 0, 17:00 0, 18:00 0, 19:00 0, 20:00 0. The failed line is above zero only from 12:00 to 16:00: 1, 3, 2 and 3. In those hours the sent line falls to 1, then 0, 0, and back to 4. The point at 13:00 to 14:00 carries the exemplar R-1015-020.

The metric counts every run, so it shows what the events suggested: from 13:00 to 15:00, not one reply was sent. That is the reply service’s outage, a practice incident we caused on purpose (Thursday 12:30 to 15:30). Before 12:00 and after 16:00, every reply was sent.

Now follow the exemplar of the 13:00 point back to a trace:

Terminal window
python -m obs trace R-1015-020 --data data/telemetry/week
Run R-1015-020 | ticket T-90104 | chat-strong (agent_structured) | release 2.1.0 | language en
…
Trace 2 4f92fb98fb8725c926cdcb57e19fb417 Thu 15 Oct 13:33:46 0.0 s
0.0 s |========================| 0.00 s approve support-resolution approved by usr-sam (agent, limit 100)
0.0 s |========================| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e
0.0 s | =| 0.00 s execute_tool send_reply ERROR failed
Outcome: resolved

Ticket T-90104 again, the watering can. This time the refund was made once, the reply failed, and the ticket closed as resolved. It is one of the five runs that no rule flagged.

So the answer is: a pattern, not one run. Nine customers in one window, all with a failed reply, all closed as resolved. Two causes worked together. The reply service was down (the practice incident). And release 2.1.0 closed a ticket as resolved after a failed reply (its planted bug). Release 2.1.1, from Friday 11:30, sends such a ticket to a person instead. Amira’s fix is the release, not four tickets, and the support team must contact nine customers.

The same path works for the double refund. Count the write calls that timed out, for the whole week:

Terminal window
python -m obs metrics larkfield.writes --data data/telemetry/week --where larkfield.write.outcome=timeout
larkfield.writes: Write calls by outcome, increase per exported interval, larkfield.write.outcome=timeout
Mon 11:00-12:00 +1 tool.name=create_return_label example: R-1012-012 (trace 064b08d9)
Mon 11:00-12:00 +1 tool.name=request_refund example: R-1012-015 (trace 9658f20b)
Tue 11:00-12:00 +1 tool.name=request_refund example: R-1013-014 (trace a50f35d4)
Tue 17:00-18:00 +1 tool.name=request_refund example: R-1013-043 (trace 9dc2a00c)
Wed 13:00-14:00 +1 tool.name=create_return_label example: R-1014-024 (trace 0bf792cf)
Thu 09:00-10:00 +2 tool.name=request_refund example: R-1014-051 (trace f346995f)
Fri 12:00-13:00 +1 tool.name=create_return_label example: R-1016-021 (trace 76e38e4d)
Fri 14:00-15:00 +1 tool.name=create_return_label example: R-1016-032 (trace 3372a85a)
Sat 10:00-11:00 +1 tool.name=request_refund example: R-1017-001 (trace da363bec)
9 interval(s) with an increase, 9 with an example trace

Timeouts happen on most days: each of these tickets has a recorded practice fault. A timeout alone is not a failure. Under releases 2.0.0 and 2.1.1, the safe retry checks with a reconcile, as R-1012-012 showed in the last lesson. The two points that matter are under release 2.1.0, with the naive retry. Wednesday 13:00 is R-1014-024, the label that the trace could not judge. Thursday 09:00 to 10:00 has +2 refunds. The payment service was slow then (a practice incident, 08:30 to 09:30), while Sam approved the night’s proposals. Its exemplar is R-1014-051, one of the two double refunds. The metric shows when and how often; the exemplar and the traces show which runs.

Now cause a failed reply in your own workflow, first with the safe setting, then with release 2.1.0’s bug.

  1. Start from a fresh work/ folder, and run ticket T-90107 with a reply service that is down:

    Terminal window
    rm -rf work
    python -m resolver run T-90107 --fault send_reply:unavailable

    The last line: Reply failed. Ticket: handed to person. The default setting, RESOLVER_REPLY_FAILURE=person, does the right thing. python -m obs find all prints 0 match(es) in 1 run(s).

  2. Now run it again with release 2.1.0’s bug:

    Terminal window
    rm -rf work
    RESOLVER_REPLY_FAILURE=close python -m resolver run T-90107 --fault send_reply:unavailable

    The last line: Reply failed. Ticket: resolved.

  3. Find the pattern, read the events, and count:

    Terminal window
    python -m obs find all
    python -m obs events
    python -m obs metrics larkfield.tool.calls --where gen_ai.tool.name=send_reply
    T-90107-agent-chat-small-1 T-90107 … chat-small no_action: closed as resolved with no write and no reply sent (send_reply failed at +3.4 s)
    1 match(es) in 1 run(s)
    … larkfield.reply.failed trace … {"run.id": "T-90107-agent-chat-small-1", "ticket.id": "T-90107"}
    … larkfield.run.closed trace … {"run.id": "T-90107-agent-chat-small-1", "ticket.id": "T-90107", "outcome": "resolved", "writes_done": 0, "reply": "failed", "helpdesk.status": "resolved"}
    2 event(s)
    larkfield.tool.calls: Tool calls, increase per exported interval, gen_ai.tool.name=send_reply
    … +1 tool.kind=write tool.status=failed example: T-90107-agent-chat-small-1 (trace …)
    1 interval(s) with an increase, 1 with an example trace

    Your times and trace IDs differ. Check that the trace ID in the events and in the metric’s example are the same: the three signals point to one run.

Common mistake

obs events prints 0 event(s) for your own run

Symptom: after a run of your own, python -m obs events prints only 0 event(s).

Cause: your resolver/live.py is still Module 3’s version, which writes no events. The saved week has them because it was recorded with the course’s final code.

Diagnose: python -m pytest -q says 93 tests either way, so check the file: grep -c "larkfield.run.closed" resolver/live.py (Windows: Select-String larkfield.run.closed resolver\live.py) finds nothing in the old file.

Recover: copy resolver/live.py from the snapshot end-of-m04 again, delete work/, and run the ticket again.

Common mistake

obs metrics says No metric

Symptom: python -m obs metrics larkfield.tool.call --data data/telemetry/week prints No metric larkfield.tool.call in data/telemetry/week.

Cause: the metric name has a typo (call, not calls), or your own work/telemetry has no metrics yet. The workflow writes your metrics when each command ends.

Recover: use the exact names: larkfield.tool.calls, larkfield.writes, larkfield.runs, larkfield.run.stops, and the GenAI duration metrics of Module 2.

In Grafana, the same walk uses three data sources: Tempo for the trace, Loki for the events, and Prometheus for the metrics. A time-series panel can show exemplars as small dots, when Prometheus stores them, and a click on a dot opens the example trace. The MLOps course’s lesson on logs, metrics and traces sets up these links for one service. The course did not check exemplars in its Grafana run. obs metrics prints them from the saved files, so this lesson works without Grafana.

Change one condition: start from the loop of R-1017-007 on Saturday.

  1. Predict: Saturday had three loops. Was the loop a problem of one ticket, or of something wider? Which metric would tell you?

  2. Count the runs that the loop’s limits stopped, for the whole week, by reason:

    Terminal window
    python -m obs metrics larkfield.run.stops --data data/telemetry/week
  3. Follow two exemplars with obs trace, and note each run’s model.

Check your prediction
larkfield.run.stops: Runs stopped by a limit, by reason, increase per exported interval
Wed 10:00-11:00 +1 stop_reason=provider_error example: R-1014-014 (trace 7f6ab892)
Thu 05:00-06:00 +1 stop_reason=provider_error example: R-1015-001 (trace 5b4f935f)
Sat 09:00-10:00 +1 stop_reason=provider_error example: R-1017-004 (trace d21473ff)
Sat 09:00-10:00 +1 stop_reason=too_many_errors example: R-1017-005 (trace 05936b96)
Sat 10:00-11:00 +2 stop_reason=repeated_call example: R-1017-006 (trace 3197f96c)
Sat 11:00-12:00 +2 stop_reason=too_many_errors example: R-1017-010 (trace 7ad8163f)
…
Sun 17:00-18:00 +1 stop_reason=repeated_call example: R-1018-023 (trace 35d276a1)
Sun 21:00-22:00 +1 stop_reason=too_many_errors example: R-1018-028 (trace 3ed419fd)
14 interval(s) with an increase, 14 with an example trace

The stops by repeated_call (3) and too_many_errors (10) happen only on Saturday and Sunday, and their exemplars are gemma3:4b runs. Together they are 13 of the 16 runs on the local model. The other 3 closed as resolved, among them the hidden tool errors R-1018-010 and R-1018-022. So the loop is a pattern of Tomás’s weekend trial, not of one ticket. The three provider_error stops are different: ticket T-80002 on chat-small, the content filter recorded by the agents course, on three days. Only the repeated_call stops are flagged as loops; the too_many_errors stops are a sign to read next.

When to move from a trace to metrics, and what it costs

Section titled “When to move from a trace to metrics, and what it costs”

Make this walk every time a rule or a person finds a bad run, before you fix anything. The question “one run or a pattern?” decides the fix: a ticket, a release, a model, or a service.

The walk has costs and limits. Events cost storage for every run, so keep them few and named, with IDs and no text. Metrics are cheap and complete, but they have only a few labels, so they can say “9 failed replies”, never “which customer”. An exemplar is one example per point, not the full list: for the list, go back to the events or to a search. And all of this works only because every signal carries the same IDs. A signal without the trace ID or the run ID is an island.

Check your understanding

The no-action rule flagged 4 runs on Thursday. How did the events show that the harm was wider, and how wide?

Show the answer

The events named larkfield.reply.failed showed 9 failed replies, all on Thursday from 12:32 to 15:14, on 9 different tickets. The closing events of those 9 runs all say resolved. Four have writes_done 0: the flagged runs. Five have writes_done 1: a refund, label or reshipment was made, and the customer was never told. So 9 customers were affected, not 4.

Check your understanding

What is an exemplar, and why does a metric point need one to lead you to a run?

Show the answer

An exemplar is an example that a metric point carries: the trace ID (and span ID) of one span that added to the count. A metric has no run ID in its labels, because one series per run would make the metrics far too large. So the count alone cannot name a run. The exemplar is the bridge from “3 replies failed in this hour” to one of those runs, here R-1015-020.

Check your understanding

Module check: for one failed run of the week, show with trace evidence which step caused the failure and why the reply looked fine.

Show the answer

Run R-1018-010, gemma3:4b on ticket T-90602, a question about a double charge. The step that caused the failure is execute_tool get_payments at +3.7 s, with ERROR service_unavailable. No later get_payments call succeeded, so the model never saw the payments. The model then made five more model calls. The run sent a reply (execute_tool send_reply, sent, at +19.7 s), and the outcome is resolved, with the proposal reply_only. The reply looked fine because it was sent and the ticket closed as resolved. Nothing in the outcome, the log line or the help desk says that the payment check failed. Only the red span shows it. The course’s labels expected a hand-off to a person for this ticket.

Recap

  • A trace tells the story of one run. Events with the same run ID add what happened at named moments. Metrics count every run, and an exemplar leads from a count back to an example trace.
  • From R-1015-023, the events found 9 failed replies on Thursday, all closed as resolved. Four had no action, and five had a write but no reply. The metric showed the window: no reply sent from 13:00 to 15:00. A pattern, caused by the reply outage and release 2.1.0’s bug.
  • The timeout counter put the two double refunds in one hour, Thursday 09:00 to 10:00, with an exemplar to R-1014-051.

Your project is now the same as the snapshot end-of-m04: your workflow writes events, obs find and obs metrics read them, and 92 tests pass. You can now find agent failures and tell one run from a pattern. But your own traces still show the cost of each model call as “-”. Nobody can say which step, model or customer made the week’s bill rise. Next, in Module 5, you put the cost on every run and every step.

Next: Cost per run and per step