Spans, traces and context
- Lesson 4
- 40 min
- Module 2: OpenTelemetry for agents
Where we are
In Module 1, we read traces that the project already had, and we saw a refund paid twice behind a green log line. But we did not ask how a trace is built, or how the steps of one run stay together. In this lesson, we meet the span, the trace and the context that joins them, and we make our workflow record its first span. We start in a restaurant kitchen.
Lesson 4 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsSpans, traces and context
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Amira’s log says that the double refund was “resolved”, and nothing about the steps inside the run. Make each step of a run record its own start, end and parent, so that one run becomes one tree that you can read.
Before you startWhat to know and to have ready
What you need to know
- The live workflow and
python -m obs, from Meet the live workflow. - The double refund R-1014-040, from A refund nobody can explain.
- Python functions and
withblocks. Threads help, but you do not need to have used them.
What you need on your computer
- Your project
observable-workflowat the snapshotend-of-m01(76 tests pass), with its virtual environment active. - The labs repository next to it, as in Module 1:
../nextia-learning-labs/C31/.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A kitchen and its order slips
Section titled “A kitchen and its order slips”A restaurant gets order 42: a soup and a fish for one table. The head cook writes the order on a slip and notes the time. Then the work splits. One cook makes the soup, another cuts the fish, and a third makes the sauce. Each cook writes a small card: the task, the time it started, the time it ended, and “for order 42”. The sauce card also says “part of the fish”.
At the end of the night, the manager can lay the cards out. She sees which task waited, which one took longest, and which tasks belonged to which dish. One number, 42, keeps the cards of one order together. The “part of” line puts them in order: the order, the dishes, the tasks inside each dish.
And when the fish goes to the grill station across the room, the slip goes with it. Without the slip, the grill cook’s card would have no order number, and nobody could join it to the rest.
The problem: one green line for a whole run
Section titled “The problem: one green line for a whole run”In Module 1, Amira’s log had one line per closed ticket. This is the double refund’s line:
2026-10-15T09:17:22-04:00 INFO ticket=T-90104 run=R-1014-040 model=chat-strong status=resolved writes=1 reply=sentThe line is written once, at the end. It says that the run ended well. It cannot say what happened inside: how many model calls, which tool, what waited, and which call paid the customer a second time. The line even says writes=1, because the application counted one operation.
Tomás suggests a quick fix: “Print the time of every step.” But a print line has no link to its run. About 45 runs a day, some at the same time, give 45 mixed streams of lines. A line request_refund 10.0 s does not say which ticket, which approval or which retry it belongs to.
We need what the kitchen has: a card per step, the run’s number on every card, and a “part of” line.
The idea: spans, traces and context
Section titled “The idea: spans, traces and context”OpenTelemetry, the open standard that this course uses, gives these cards a name.
A span is one timed step. It has a name, a start time, an end time, and attributes: named values such as the ticket ID or the model. It has a status: unset, or ERROR when the step failed. It has its own span ID. And it has the ID of its parent span, the step that started it. This is the “part of the fish” line.
A trace is all the spans of one run. They share one trace ID, the order number. The first span has no parent: it is the root span. Every other span names its parent, so the spans make a tree.
Context is the slip that travels with the work. In code, it is “the span that is running now”. When a new span starts, OpenTelemetry reads the context and makes the new span a child of the current one. So a function does not need a parameter for its parent: the context gives it.
Two drawings show the same trace. A tree shows who started whom. A waterfall puts each span on a time line, one row per span, children under their parent. You read the order and the waiting time in the waterfall.
Add the module’s code
Section titled “Add the module’s code”The snapshot end-of-m02 has this module’s code. Copy the files for spans and names now, so that the tiny example below can run too. The last lesson of the module copies the Collector’s files.
-
In the terminal, go to your project folder, with the virtual environment active. Check that the tests pass:
python -m pytest -q. The last line says76 passed, 1 deselected. -
Copy the new and changed files from the snapshot:
Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m02cp $S/resolver/telemetry.py $S/resolver/conventions.py $S/resolver/observe.py resolver/cp $S/resolver/runner.py $S/resolver/live.py $S/resolver/__main__.py resolver/cp $S/obs/redaction.py $S/obs/__main__.py obs/cp $S/tests/conftest.py $S/tests/test_telemetry.py tests/cp $S/requirements.txt $S/.env.example .Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m02"Copy-Item $S\resolver\telemetry.py, $S\resolver\conventions.py, $S\resolver\observe.py resolver\Copy-Item -Force $S\resolver\runner.py, $S\resolver\live.py, $S\resolver\__main__.py resolver\Copy-Item -Force $S\obs\redaction.py, $S\obs\__main__.py obs\Copy-Item -Force $S\tests\conftest.py, $S\tests\test_telemetry.py tests\Copy-Item -Force $S\requirements.txt, $S\.env.example .Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m02cp $S/resolver/telemetry.py $S/resolver/conventions.py $S/resolver/observe.py resolver/cp $S/resolver/runner.py $S/resolver/live.py $S/resolver/__main__.py resolver/cp $S/obs/redaction.py $S/obs/__main__.py obs/cp $S/tests/conftest.py $S/tests/test_telemetry.py tests/cp $S/requirements.txt $S/.env.example . -
Install the three new packages: OpenTelemetry’s API, its SDK, and an exporter that sends telemetry over HTTP.
Terminal window python -m pip install -r requirements.txtpython -m pip show opentelemetry-sdkThe second command prints
Version: 1.45.1on its second line. -
Run the tests:
python -m pytest -q. The last line now says79 passed, 1 deselected. The three new tests are intests/test_telemetry.py: one run is one trace, no text leaves the application, and the metrics carry the runs.
What did you copy? resolver/telemetry.py sets up OpenTelemetry in one place: the providers that make spans, metrics and events, and where they go. By default they go to files in work/telemetry/. The files use OTLP JSON. OTLP is OpenTelemetry’s own protocol for sending telemetry, and the Collector writes the same format. The events use OpenTelemetry’s logs API, which Python still names opentelemetry._logs, with an underscore. It is not yet stable in the Python SDK, so its names can change. resolver/observe.py makes the run span, and resolver/runner.py opens it around every run. resolver/conventions.py holds the names, the subject of the next lesson.
A tiny example: a trace by hand
Section titled “A tiny example: a trace by hand”Before we touch the workflow, build a trace by hand. This script has five steps: a ticket, an order lookup, a reply, the same reply from a thread, and a message to another service. It keeps the finished spans in memory and prints them. Save the code below as work/tiny_trace.py in your project. The work/ folder is not part of the project’s code: delete it to start again.
import threading
from opentelemetry import context, propagate, tracefrom opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import SimpleSpanProcessorfrom opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
saved = InMemorySpanExporter()provider = TracerProvider()provider.add_span_processor(SimpleSpanProcessor(saved))tracer = provider.get_tracer("tiny")
def look_up_order(): with tracer.start_as_current_span("look up the order"): pass
def write_reply(): with tracer.start_as_current_span("write the reply"): pass
def write_reply_in(ctx): token = context.attach(ctx) # the thread takes the parent's context first try: write_reply() finally: context.detach(token)
# 1. Functions: the current span travels with the call.with tracer.start_as_current_span("handle ticket"): look_up_order() write_reply()
# 2. A thread: it starts with an empty context, unless we give it ours. t = threading.Thread(target=write_reply) t.start() t.join() t = threading.Thread(target=write_reply_in, args=(context.get_current(),)) t.start() t.join()
# 3. Another service: the context goes into the message as a traceparent header. message = {} propagate.inject(message) print("message:", message)
# The other service reads the header and continues the same trace.with tracer.start_as_current_span("send the reply", context=propagate.extract(message)): pass
names = {s.context.span_id: s.name for s in saved.get_finished_spans()}for s in saved.get_finished_spans(): parent = names.get(s.parent.span_id, "?") if s.parent else "-" print(f"trace {s.context.trace_id:032x}"[:14], f"span {s.context.span_id:016x}"[:13], f"{s.name:<18} parent: {parent}")Run it with python work/tiny_trace.py. The course’s run printed:
message: {'traceparent': '00-ccd4d26ed047112cc342d874fa714b7e-699b79a1aa4f0839-03'}trace ccd4d26e span d8717ea7 look up the order parent: handle tickettrace ccd4d26e span 48a37482 write the reply parent: handle tickettrace 2995737c span 361b4d39 write the reply parent: -trace ccd4d26e span aca0c364 write the reply parent: handle tickettrace ccd4d26e span 699b79a1 handle ticket parent: -trace ccd4d26e span 469b4b4e send the reply parent: handle ticketYour IDs will differ: they are random. Read the output line by line:
- The spans print in the order they ended.
handle ticketis fifth, because it ends after its children. A parent always lasts at least as long as its children. - Functions.
look up the orderand the firstwrite the replyhave the parenthandle ticketand the traceccd4d26e. Nobody passed the parent to them: the context did. - The first thread. Its span has a new trace,
2995737c, and no parent. A new thread starts with an empty context, so its span became the root of a second trace. This is a broken run: its steps are now in two traces. - The second thread. It attaches the parent’s context first (
write_reply_incallscontext.attach), so its span joins the trace. - Another service.
propagate.injectwrote the context into the message as a traceparent. It is a short text in the W3C Trace Context format. It holds the version00, the trace ID, the parent’s span ID and the flags. The parent here is699b79a1…, the ID ofhandle ticket.01means “sampled”; this SDK also sets the “random trace ID” flag, so you see03. The receiver reads the header withpropagate.extractand continues the same trace.
Text description of the diagram
The same five spans as the script’s output. Inside one thread, the context follows every function call, so the children find their parent. A new thread starts empty: without context.attach, its span starts a second trace with no parent. Between services, the context travels in the message as a traceparent header, and the receiver continues the same trace.
So the context crosses functions by itself. It crosses threads and services only when the code carries it. Module 3 carries it across the workflow’s two workers and across an MCP call.
A run of the workflow as a tree
Section titled “A run of the workflow as a tree”Now look at a real run. In the saved week, the double refund R-1014-040 has two traces. Its first trace is the agent’s work, from the ticket to the proposal:
python -m obs trace R-1014-040 --data data/telemetry/weekTrace 1 f0433c4403e2767e1d69ece18a947526 Wed 14 Oct 17:52:00 6.6 s 0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2 0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out US$0.00479 stop 2.4 s | = | 0.00 s execute_tool get_order ok LK-793996 2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out US$0.00703 stop 6.6 s | =| 0.00 s approval.request needs agentThe output goes on with the second trace. We read the second trace and the 15-hour wait between them in Module 3.
Text description of the diagram
The tree shows one root, invoke_agent support-resolver, with four children: two model calls, one order lookup and the approval request. The waterfall puts the same spans on one time line. The first model call takes 2.39 s and asks for the order. The lookup is instant. The second call takes 4.17 s and proposes the refund. Then the run asks for a person. Almost all of the 6.56 s is model time.
Read the waterfall the way the kitchen manager reads her cards. The root span lasts as long as the whole run. Each child sits inside it on the time line. The tree says nothing about time, and the waterfall hides nothing about order: you need both.
The model spans in the saved week last as long as the recorded calls did. A replay answers in microseconds, but a 0.1 ms span would teach nothing. So the project keeps a run clock: a time line per run that moves forward by each replayed call’s recorded latency. Keep that in mind for the next section.
Guided practice: your first span
Section titled “Guided practice: your first span”-
Run one ticket:
Terminal window python -m resolver run T-80008The last line is
Reply sent. Ticket: resolved., as in Module 1. The workflow did not change what it does. -
Read its trace:
Terminal window python -m obs trace T-80008Run T-80008-agent-chat-small-1 | ticket T-80008 | chat-small (agent) | release 2.0.0 | language en0 model call(s), 0 read(s), 0 write call(s), 0 tokens in, 0 out, US$0.00000Trace 1 a9f8cfdc1f12fd4cb4a36f4ee661b5c9 … 0.0 s0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1Outcome: resolvedOne trace, one span: the root. Your trace ID and time differ.
-
Open the raw span. The file
work/telemetry/traces.jsonlhas one export request per line. Print it in a readable form:Terminal window python -m json.tool work/telemetry/traces.jsonl | lessTerminal window python -m json.tool work\telemetry\traces.jsonl | moreTerminal window python -m json.tool work/telemetry/traces.jsonl | lessPress
qto leaveless. The span itself starts like this (shortened):{"traceId": "a9f8cfdc1f12fd4cb4a36f4ee661b5c9","spanId": "0c1c066ee1b6f4e3","name": "invoke_agent support-resolver","kind": "SPAN_KIND_INTERNAL","startTimeUnixNano": "…","endTimeUnixNano": "…","attributes": [{"key": "gen_ai.operation.name", "value": {"stringValue": "invoke_agent"}},{"key": "gen_ai.request.model", "value": {"stringValue": "chat-small"}},{"key": "larkfield.ticket.id", "value": {"stringValue": "T-80008"}},{"key": "larkfield.proposal.outcome", "value": {"stringValue": "reply_only"}},{"key": "gen_ai.usage.input_tokens", "value": {"intValue": "4580"}},{"key": "larkfield.outcome", "value": {"stringValue": "resolved"}}],"status": {}}The span has a trace ID and a span ID, and no
parentSpanId: it is a root. Its status is empty: unset, so no error. The start and end times are equal.
Why does the span last 0.00 s? The run clock moves only when a model span adds a recorded latency, and there are no model spans yet. The model’s time is not in the trace until Module 3 adds the model spans. The obs summary line says 0 model call(s) for the same reason. But the root span already has the run’s totals as attributes: 4,580 tokens in, the proposal and the outcome.
Look at one more thing. live.close() adds the outcome to the span without receiving it as a parameter. It asks for the current span: trace.get_current_span(). That is the context at work inside the workflow.
When to add a span, and what it costs
Section titled “When to add a span, and what it costs”Sam asks: “Should every function get a span?”
No. A span is worth adding when it answers a question you will ask: where the time went, which step failed, which call wrote. In this course, those steps are the run, each model call, each tool call, each write and each approval. A span around a helper that formats a date only makes the tree harder to read.
A span also costs something. The course team measured the workflow’s own code on 350 replayed runs. A run took a median of 0.53 ms with telemetry off and 2.42 ms with file export, about 1.9 ms more. Each run made about 4.3 spans and 9.3 KB of trace data. Against seconds of model time, 1.9 ms is small. But storage grows with every run, and someone must keep the names stable. Module 1’s lesson What to record and what not to decides what is worth that cost.
Common mistake
A step appears as a second trace
Symptom: a step that belongs to a run appears as its own trace, with no parent. In the tiny example, it is the line trace 2995737c … write the reply parent: -.
Cause: the step ran in a new thread, or in another process, and the context did not go with it. A new thread starts with an empty context, so its first span becomes a root.
Diagnose: compare trace IDs. All the spans of one run must have the same trace ID. A span whose parent is - but which is not the run’s first step is an orphan.
Recover: carry the context. In a thread, attach the parent’s context before you start the span (write_reply_in calls context.attach), as in the second thread of the example. Across a process or a service, inject a traceparent into the message and extract it on the other side.
Common mistake
obs: No telemetry in work/telemetry
Symptom: after python -m resolver run T-80008, python -m obs trace T-80008 prints:
No telemetry in …/work/telemetry. Run a ticket first (python -m resolver run T-80008), or read the replayed week: --data data/telemetry/weekCause: the variable OBS_EXPORT is off in your shell, so the workflow recorded nothing. Or you copied the files but did not install requirements.txt, and the run failed with ModuleNotFoundError: No module named 'opentelemetry'.
Recover: remove the variable (unset OBS_EXPORT, or in PowerShell Remove-Item Env:OBS_EXPORT), install the requirements, and run the ticket again.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'Every span has the trace ID, so we do not need the parent span ID.' Using the first trace of R-1014-040, explain what you would lose without the parent.
Show the answer
The trace ID says which spans belong to the run, but not how they fit together. Without the parent span ID, you would have five spans of trace f0433c44 and no tree. You could not say that both model calls and the order lookup were steps of the agent, invoke_agent support-resolver. In a run with two workers, you could not say which worker made which call. The parent makes the tree; the times make the waterfall.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: in the tiny example, the first thread's span had no parent. What would change if the thread called look_up_order() instead of write_reply()? And what if the code attached the context first?
Show the answer
Nothing changes in the first case. Any span that starts in a new thread without the parent’s context becomes the root of a new trace, whatever the function. The span ‘look up the order’ would get its own trace ID and the parent ‘-’. With the context attached first, as in write_reply_in, the span joins the run’s trace. Its parent is ‘handle ticket’, as for span aca0c364 in the output.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A span is one timed step: name, start, end, attributes, status, its own ID and its parent’s ID.
- A trace is the tree of one run’s spans, joined by one trace ID; the root span has no parent.
- Context is the current span. It follows function calls by itself; threads and services must carry it, or the run breaks into two traces. Between services it travels as a
traceparent. - A tree shows who started whom; a waterfall shows when. R-1014-040’s first trace: 6.56 s, almost all model time.
- Your workflow now records one span per run. It lasts 0.00 s on the run clock until the model spans arrive.
You can now make one run into one trace and read it as a tree and a waterfall. But our span’s attributes have names such as gen_ai.request.model and larkfield.ticket.id. Who chose them, and why would a tool understand them? Next, you meet the names that OpenTelemetry agreed for agent runs, model calls and tool calls, and you add a span of your own.