Skip to content
Français

Search runs with TraceQL

  • Lesson 13
  • 40 min
  • Module 4: Find agent failures in Grafana

Where we are

In the last lesson, we read one run's waterfall and found its slow step, its expensive step and the step that sent the second refund. But the saved week has 316 runs in 434 traces, and we cannot open them one by one. In this lesson, we search runs by their structure: many calls to one tool, a write with no approval before it, an error followed by a resolved outcome. We start with the search box of an email inbox.

What you will be able to do

Find the runs that matter in a whole week by the shape of their spans, with TraceQL in Grafana and with obs find on the saved telemetry. Read each query’s result, and check what it can miss.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • How to read a waterfall: rows, parents, errors, retries and gaps, from Read a run.
  • The tool, write and approval spans and their attributes (gen_ai.tool.name, larkfield.tool.kind, larkfield.write.outcome), from Tool calls, errors and retries.
  • Tail sampling in the Collector, from The Collector.

What you need on your computer

  • The project ~/projects/observable-workflow as the last lesson left it (the snapshot end-of-m03, 86 tests pass), and the labs repository in ~/projects/nextia-learning-labs.
  • Optional: Grafana with the replayed week sent to it, as in Read a run. Every step works without it.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, and the course did not run them. The TraceQL queries were run by the course team in grafana/otel-lgtm 0.36.0.

Your work inbox has 2,000 emails. Grace asks: “Which customers sent us a photo last week, and got no answer?” You do not open 2,000 emails. You type a search: from customers, with an attachment, in the last week, with no reply. Each condition is about one part of an email. The result is a short list of whole emails, and you open only those.

A search like this has three parts. Conditions on the parts of each item. A way to combine them (“and”, “with no”). And a result that is a list of items, not of parts.

The problem: 316 runs, and the question is about their shape

Section titled “The problem: 316 runs, and the question is about their shape”

Grace’s question for the agent is: “Which runs had an error, and were still closed as resolved?” The list of runs cannot answer it. obs runs prints one line per run: the outcome, the counts and the cost. It shows resolved for 203 runs, and it shows no error at all, because errors are in the spans.

You could open each waterfall. At one minute per run, 316 runs take more than five hours. You would also make mistakes by the fortieth run. Worse, the question is about the structure of a run: one span with an error, and another span with the outcome resolved. A total or a list never holds structure.

The idea: conditions on spans, a result of traces

Section titled “The idea: conditions on spans, a result of traces”

TraceQL, the Trace Query Language, is the search language of Tempo, the trace store in Grafana. It works like the inbox search. A query has conditions on spans, and its result is a list of traces that have spans that match.

The pieces you need:

  • A spanset filter, in braces: { span.gen_ai.tool.name = "get_order" } selects the spans whose attribute gen_ai.tool.name is get_order. span. means an attribute of the span. A few fields have no prefix: name, status (ok, error), duration and rootName (the name of the trace’s root span).
  • Comparisons: =, !=, > and <, and =~ for a pattern: =~ "done|timeout" means “done or timeout”. !~ means “does not match”.
  • && inside the braces means “the same span has both”. { A } && { B } means “the trace has a span for A and a span for B”: maybe two different spans.
  • A pipeline, after |: | count() > 1 keeps the traces whose matching spans number more than 1.

Remember one thing above all: TraceQL finds traces, not runs. A run that waited for a person has two traces. A query about the writes finds trace 2, and you follow its link to trace 1 yourself.

The course’s query sheet, grafana/traceql.md, holds one query per question. obs find asks the same questions of the saved telemetry, without Grafana.

Worked example: build the duplicate-write query

Section titled “Worked example: build the duplicate-write query”

Build one query of the sheet in steps, and watch each step’s result. The question: “Which traces have more than one refund call that may have paid?” The counts are from the saved week: the course’s script applied each step’s conditions to all 434 traces.

A flow from top to bottom. The saved week: 434 traces, 2,027 spans. Step 1, choose spans: span.gen_ai.tool.name equals request_refund. Result: 45 refund spans in 39 traces. Step 2, add a condition on the same span: and span.larkfield.write.outcome matches done or timeout. Result: 41 spans in 39 traces, refund calls that may have paid. Step 3, count the spans of each trace: count greater than 1. Result: 2 traces, trace 2 of R-1014-040 and trace 2 of R-1014-051. Step 4, open each trace: two refunds of 9.95 each; the second has a new key, -retry1 and -retry2.
One TraceQL query, built in three steps, with the result of each step on the saved week (every trace). Replayed decisions, simulated faults.
Text description of the diagram

The saved week has 434 traces and 2,027 spans. Step 1 keeps the spans whose tool is request_refund: 45 spans in 39 traces. Step 2 adds a second condition on the same span, an outcome of done or timeout: 41 spans in 39 traces. These are the refund calls that may have paid. Step 3 counts those spans in each trace, and keeps the traces with more than one. That leaves 2 traces: trace 2 of R-1014-040 and trace 2 of R-1014-051. Step 4 opens them: each has two refunds of 9.95, the second with a new key.

Read each step:

  1. { span.gen_ai.tool.name = "request_refund" } keeps every refund call: 45 spans in 39 traces. Some traces have more than one span: a refused call and a retry, for example.
  2. { span.gen_ai.tool.name = "request_refund" && span.larkfield.write.outcome =~ "done|timeout" } drops the calls that surely changed nothing, such as unavailable (refused at once). 41 spans in 39 traces are left. A timeout stays, because a timed-out call may have paid.
  3. … | count() > 1 keeps the traces with more than one such span: 2 traces, the second traces of R-1014-040 and R-1014-051.

Why step 3? Because 39 traces made one refund, and that is normal work. The failure is not “a refund”. It is two refunds in one trace: a shape. Only the count turns the first two steps into a search for that shape.

Step 4 is yours: open the two traces. Each has a refund of 9.95 that timed out and a second refund with a new key. They are the week’s two double refunds. In Grafana, the course team’s run of this query returned the same 2 traces.

Grafana Explore with Tempo and the query type TraceQL. The query keeps the traces with more than one request_refund span whose outcome is done or timeout. The table under it has 2 traces, both of the service support-workflow and named approve support-resolution, with durations 10.0 s and 10.1 s.
The duplicate-write query in Grafana, from the course team's run of the replayed week: the same 2 traces as the saved week. The Start time column is blank on purpose: it showed the day of the course team's run, not a practice day.

The snapshot end-of-m04 adds obs find and the five failure rules (obs/flags.py, for the next lesson). It also adds the TraceQL sheet and a Grafana dashboard with one panel per query.

  1. Copy the files. In ~/projects/observable-workflow:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m04
    cp $S/obs/flags.py $S/obs/model.py $S/obs/__main__.py obs/
    cp $S/grafana/traceql.md grafana/
    cp $S/grafana/dashboards/agent-traces.json grafana/dashboards/
    cp $S/tests/test_obs_cli.py tests/
  2. Run the tests:

    Terminal window
    python -m pytest -q

    The last line says 87 passed, 1 skipped, 1 deselected. The new test runs obs find on the saved week.

Each search below has three parts. First the TraceQL query from the sheet, then the same question with obs find on the saved week. Last, what you see when you open the results.

An agent that calls one tool again and again may be stuck. The sheet’s query:

{ span.gen_ai.tool.name = "get_order" } | count() > 2

In Tempo, it returned 0 traces. obs find agrees:

Terminal window
python -m obs find --data data/telemetry/week --tool get_order --min-calls 3
0 match(es) in 316 run(s)

Zero is a result, but is the threshold right? Lower it to 2:

Terminal window
python -m obs find --data data/telemetry/week --tool get_order --min-calls 2
R-1012-004 T-90601 Mon 05:42 chat-small get_order called 2 times
R-1012-031 T-90201 Mon 15:07 chat-small get_order called 2 times
…
R-1017-026 T-90601 Sat 18:17 chat-strong get_order called 2 times
13 match(es) in 316 run(s)

Now 13 runs match. Open the first one:

Terminal window
python -m obs trace R-1012-004 --data data/telemetry/week
Trace 1 98cb994f5d69bd8964562841548f2855 Mon 12 Oct 05:42:07 13.2 s
0.0 s |========================| 13.17 s invoke_agent support-resolver reply_only rule W1
0.0 s |== | 0.92 s chat chat-small 2159 in / 21 out US$0.00023 tool_calls
0.9 s | =================== | 10.00 s execute_tool get_order ERROR service_timeout LK-645001
10.9 s | === | 0.89 s chat chat-small 2222 in / 21 out US$0.00023 tool_calls
11.8 s | = | 0.00 s execute_tool get_order ok LK-645001
11.8 s | ===| 1.35 s chat chat-small 2463 in / 77 out US$0.00028 tool_calls
13.2 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved

Two calls, and the first one timed out. The model asked again, the second call worked, and it replied. That is a good recovery, not a loop. So the number N is a choice, and you make it from the data. In this week, no run called get_order three times, and every two-call run that you open looks like this one. A loop shows in another way here: the agents course’s loop limits stop the run. The loop query of the next lesson looks for that stop.

Larkfield’s rule: no refund, label or reshipment without a person’s decision. The sheet’s query:

{ rootName !~ "approve.*" && span.larkfield.tool.kind = "write" && span.gen_ai.tool.name != "send_reply" }

It looks for a write span (not the reply) in a trace whose root span is not a person’s decision. In Tempo, 0 traces. obs find agrees:

Terminal window
python -m obs find --data data/telemetry/week --write-without-approval
0 match(es) in 316 run(s)

Is 0 good news, or a query that cannot match? Check it. Drop the rootName condition, and the course’s script finds 118 traces with a write in the saved week. Every one of them has the root approve support-resolution. So the query can see writes, and every write came after a decision. In this workflow, a write can only happen in trace 2, which starts with the person’s decision. That is the design of Module 3, and the query checks that the design holds.

Now Grace’s question. The sheet’s query:

{ span.larkfield.tool.kind = "read" && status = error } && { span.larkfield.outcome = "resolved" }

Two spansets: a read that failed, and a span with the outcome resolved, in the same trace. In Tempo, 8 traces. obs find:

Terminal window
python -m obs find --data data/telemetry/week --error-then-resolved
R-1012-004 T-90601 Mon 05:42 chat-small a tool error, then resolved
R-1013-038 T-90601 Tue 15:57 chat-small a tool error, then resolved
R-1014-021 T-90601 Wed 12:41 chat-strong a tool error, then resolved
R-1016-040 T-80009 Fri 14:59 chat-strong a tool error, then resolved
R-1016-044 T-80009 Fri 15:35 chat-strong a tool error, then resolved
R-1017-026 T-90601 Sat 18:17 chat-strong a tool error, then resolved
R-1018-010 T-90602 Sun 11:15 gemma3:4b a tool error, then resolved
R-1018-022 T-90601 Sun 17:00 gemma3:4b a tool error, then resolved
8 match(es) in 316 run(s)

Like the query, obs find looks at failed reads only. A failed write is another question, the duplicate write of the next lesson.

Eight runs. Are they eight failures? Open R-1014-021:

Terminal window
python -m obs trace R-1014-021 --data data/telemetry/week
Trace 1 3492400f6a05ceaeac2f8fcf3013de34 Wed 14 Oct 12:41:26 16.1 s
0.0 s |========================| 16.15 s invoke_agent support-resolver reply_only rule W1
0.0 s |=== | 1.86 s chat chat-strong 2156 in / 49 out US$0.00480 stop
1.9 s | ================ | 10.00 s execute_tool get_order ERROR service_timeout LK-645001
11.9 s | === | 1.90 s chat chat-strong 2248 in / 58 out US$0.00508 stop
13.8 s | = | 0.00 s execute_tool get_order ok LK-645001
13.8 s | ====| 2.37 s chat chat-strong 2527 in / 107 out US$0.00612 stop
16.1 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved

The error was repaired: a later get_order call worked. Six of the eight runs look like this. In two, R-1018-010 and R-1018-022, no later call of the failed tool worked. Yet the run replied and closed as resolved. Those two are the dangerous ones.

So the query found the right place, but it is too wide. It also has a limit that you must know: && between two spansets means “both are in the trace”, not “one comes after the other”. Here it works, because the outcome is set when the run closes, after every tool call. And the query cannot say “no later call of the same tool succeeded”. That needs a rule that reads the spans in order. The next lesson writes it.

If Grafana has the replayed week, run the sheet’s queries there:

  1. Open Grafana at http://localhost:3031, then Explore. Choose the data source Tempo and the query type TraceQL.
  2. Set the time range to cover the replay. The week took about 10 minutes of real time. Choose a range that starts before you sent it, for example Last 3 hours.
  3. Copy a query from grafana/traceql.md into the editor, and run it. The result is a table of traces: trace ID, start time, service, root name and duration.
  4. Click a trace ID to open its waterfall, as in the last lesson.

The dashboard Support workflow: failure patterns in traces (grafana/dashboards/agent-traces.json) runs seven of the sheet’s queries for you, one table panel per query.

The Collector of Module 2 samples traces at the tail. It keeps every trace with an approval, a write, an error or injected input, and 10% of the others. In the course team’s check of the Collector, it kept 275 of the week’s 434 traces. A search can only find kept traces. Compare the two answers for each query of the sheet:

For each query of the sheet: the traces Tempo returned in the course team's Grafana run, as a share of the traces that match in the saved week (every trace). Tail sampling keeps every trace with an approval, a write, an error or injected input, and 10% of the others; the 10% differs from run to run.
Text description of the chart

One bar per query that matches at least one saved trace. loop: Tempo 3 of 3 saved traces; hidden tool error: Tempo 8 of 8 saved traces; duplicate write: Tempo 2 of 2 saved traces; no action: Tempo 4 of 4 saved traces; injected input: Tempo 20 of 20 saved traces; slow model calls: Tempo 14 of 17 saved traces; expensive runs: Tempo 23 of 39 saved traces; a model's runs: Tempo 15 of 16 saved traces. The five failure queries are at 100%: loop, hidden tool error, duplicate write, no action and injected input. The Collector keeps every trace with an error, a write, an approval or injected input. The three that are not failures lose traces: slow model calls, gemma3:4b runs and runs that cost more than 1 cent.

Every failure query found all of its traces, because the sampling rules keep exactly those traces. The others did not. Tempo returned 14 of the 17 traces with a slow model call, and only 23 of the 39 runs over 1 cent. A run over 1 cent with no approval, no write and no error is kept only when the 10% sample picks it. So “how many expensive runs?” has no answer in Tempo. Counts belong to the metrics, which are never sampled. The last lesson of this module uses them.

Common mistake

The query returns No rows, with no error

Symptom: you paste a query in Explore, and the table says No rows. You expected some traces.

Cause: one of three. The time range does not cover the replay. An attribute name has a typo: TraceQL does not say “unknown attribute”, it just matches nothing. Or the trace was not kept by the sampling.

Diagnose: run the widest query first, { span.larkfield.ticket.id = "T-90104" }. If it returns nothing, the time range is wrong. If it works, check your attribute names against a span’s attributes in the waterfall, or against grafana/traceql.md.

Recover: widen the time range, copy the names from the sheet, and remember that a dropped trace cannot be found. obs find reads every saved trace.

Common mistake

A query returns exactly 50 traces

Symptom: a query returns exactly 50 traces, and you expected more.

Cause: Grafana’s TraceQL search has a limit, 50 traces by default (Search Options → Limit). It shows at most that many.

Recover: raise the limit in Search Options for that query. Never read a TraceQL table as a count of a whole week: use obs find, or a metric, to count.

Your turn: another tool, the same question

Section titled “Your turn: another tool, the same question”

Change one condition: search for runs that called get_payments at least twice.

  1. Predict: the payment service had practice failures this week. Do you expect runs that called get_payments twice? Will the model have recovered, as in R-1012-004?

  2. Search:

    Terminal window
    python -m obs find --data data/telemetry/week --tool get_payments --min-calls 2
  3. Open the first result with python -m obs trace RUN --data data/telemetry/week. Then write the TraceQL query that would find the same traces in Grafana.

Check your prediction
R-1013-025 T-90602 Tue 13:35 chat-small get_payments called 2 times
R-1014-034 T-90602 Wed 15:05 chat-small get_payments called 2 times
R-1016-003 T-90602 Fri 03:02 chat-small get_payments called 2 times
3 match(es) in 316 run(s)

All three are ticket T-90602, whose recorded practice fault is a payment service that is down. Open R-1013-025:

1.0 s | == | 0.20 s execute_tool get_payments ERROR service_unavailable LK-645112
1.2 s | ======= | 0.97 s chat chat-small 2530 in / 22 out US$0.00026 tool_calls
2.2 s | = | 0.20 s execute_tool get_payments ERROR service_unavailable LK-645112
2.4 s | =========| 1.31 s chat chat-small 2596 in / 105 out US$0.00031 tool_calls
Outcome: handed_to_person

Both calls failed, and chat-small handed the ticket to a person. That is the right outcome for this ticket by the course’s labels. Compare it with R-1018-010 from the last lesson: the same ticket and the same failed tool. But gemma3:4b replied and closed it as resolved. The TraceQL query: { span.gen_ai.tool.name = "get_payments" } | count() > 1.

Use TraceQL when the question is about the shape of a run: which spans, with which attributes, how many, under which root. It is fast for an engineer who knows the attribute names, and every result is a trace that you can open.

It has limits. It finds traces, so a run with two traces needs two steps. It checks “and” between spansets, not “after”. It searches only the traces that the sampling kept, and the search limit cuts long lists. Each query also needs a careful threshold, like N in “more than N calls”. Write the queries once, in a sheet that your team shares, and test each one on a run that should match. The next lesson turns the queries into five named patterns, each with a rule that reads the spans in order.

Check your understanding

The query { span.gen_ai.tool.name = "request_refund" && span.larkfield.write.outcome =~ "done|timeout" } matched 39 traces. Why is that not a list of 39 problems, and what turns it into a search for double refunds?

Show the answer

Each of the 39 traces made at least one refund that may have paid, and one refund is normal work. The failure is a shape: two such refund calls in one trace. The pipeline | count() > 1 keeps only the traces with more than one matching span. That left 2 traces: the second traces of R-1014-040 and R-1014-051.

Check your understanding

obs find --error-then-resolved found 8 runs. How many of them are real failures, and what does the query not check?

Show the answer

Two: R-1018-010 and R-1018-022, where no later call of the failed tool succeeded and the run still closed as resolved. In the other six, a later call of the same tool worked, so the error was repaired. The query checks only that a failed read and a resolved outcome are in the same trace. It does not check the order of the spans, or whether a later call repaired the error.

Check your understanding

Practical: write a TraceQL query that finds traces where a write happened with no person's decision as the root. Then say why 0 results is good news here only after a check.

Show the answer

{ rootName !~ "approve.*" && span.larkfield.tool.kind = "write" && span.gen_ai.tool.name != "send_reply" }. Zero results can also mean a query that never matches: a typo, or the wrong attribute. Check that it can match: drop the rootName condition. The saved week then has 118 traces with a write, all with the root approve support-resolution. So the query sees writes, and every write came after a decision.

Recap

  • TraceQL puts conditions on spans and returns traces. { … && … } is one span, { A } && { B } is two spansets in one trace, and | count() > N counts the matching spans of each trace.
  • Built step by step, the duplicate-write query went from 45 refund spans in 39 traces to 2 traces: the week’s two double refunds.
  • More than N calls: 0 with N = 2, 13 with N = 1, and the two-call runs were recoveries. A write without an approval: 0, and a check showed that the query can see the 118 traces with writes. An error, then resolved: 8 runs, of which 2 were not repaired.
  • Tempo searches only the kept traces: every failure query found all its traces, but only 23 of 39 runs over 1 cent were kept.

Your project now has obs find, the TraceQL sheet and the traces dashboard: 87 tests pass. You can now search a week by the shape of its runs. But each query is still a rough net: it also catches repaired errors, and it cannot read spans in order. Next, you meet the five failure patterns of agents, each with its signature in a trace and a rule that finds it.

Next: Agent failure patterns