Many agents and handoffs
- Lesson 10
- 35 min
- Module 3: Instrument the workflow
Where we are
In the last lesson, a run that waits for a person stayed one story: two traces, joined by a link. But the workflow has a design where two workers share one ticket, and its order lookup can run in another process, through an MCP server. In this lesson, we carry the trace context across both, so that nothing breaks the story. We start at a hospital's shift change.
Lesson 10 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsMany agents and handoffs
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Keep a run one trace when its work moves to another worker or another process. Put the trace context into the handoff message and into the MCP request, and read it on the other side. Record the handoff itself: who handed over to whom, and whether the context arrived.
Before you startWhat to know and to have ready
What you need to know
- Context propagation and the
traceparentheader, from Spans, traces and context. - The agents course’s two-worker design,
workers: an investigator reads the order and writes a report, and a resolver decides from the report, on the same ticket. - For the MCP part: what an MCP server and a tool call are, from MCP: Connect AI Applications to Tools and Data. This part is optional.
What you need on your computer
- The project as the last lesson left it: 84 tests pass, and
show_spans.pyis in the project folder. - A terminal in
observable-workflow, with the virtual environment active. The MCP part installs one more package,mcp.
Tested on macOS (Apple silicon) with Python 3.12.13 and the MCP SDK 2.3.0. The Windows and Linux commands are written, not run.
A shift change in a hospital
Section titled “A shift change in a hospital”At 7 in the morning, the night nurse hands over to the day nurse. She gives a short note: what happened in the night, what to watch, and the patient’s chart number. The day nurse writes everything she does into that chart.
If the note forgets the chart number, the day nurse still does her work. But she writes it into a new, empty chart. The next morning, a doctor reads the patient’s chart and sees nothing after 7. The work happened, and the story has a hole.
The note must carry the reference to the story, not only the facts. That is the whole lesson.
The problem: two workers, two stories
Section titled “The problem: two workers, two stories”In the workers design, the investigator reads the order, then hands a report to the resolver. In production, the two are separate processes: the report is a message from one to the other. Here, the project behaves the same way: the resolver takes its parent from the message, not from the memory of the process.
The project has a switch to forget the reference, as a careless handoff would. The course team ran the same ticket twice with this lesson’s code, with the context and without it. You run the same commands after the next section:
python -m resolver run T-90103 --variant workersOBS_HANDOFF_CONTEXT=off python -m resolver run T-90103 --variant workerspython -m obs runsIn PowerShell, set the variable first, $env:OBS_HANDOFF_CONTEXT = "off", run the ticket, then remove it with Remove-Item Env:OBS_HANDOFF_CONTEXT. The list of runs looks normal:
run ticket started outcome model tools appr cost US$ secs model flagsT-90103-workers-chat-small-1 T-90103 … asked_customer 3 1 0 0.00000 4.9 chat-small -T-90103-workers-chat-small-2 T-90103 … asked_customer 3 1 0 0.00000 4.9 chat-small -2 run(s), 0 flaggedBoth runs made three model calls and one tool call. obs runs groups spans by run ID, so it finds all of them. But a trace store such as Tempo groups spans by trace. Look at the second run’s trace:
Trace 1 aed2bc3da6441e9d985419a7153381da … 4.9 s 0.0 s |========================| 4.94 s invoke_workflow support-resolution ask_customer rule W5 0.0 s |================= | 3.46 s invoke_agent investigator finished 0.0 s |====== | 1.26 s chat chat-small 694 in / 21 out - tool_calls 1.3 s | = | 0.00 s execute_tool get_order ok LK-640436 1.3 s | =========== | 2.20 s chat chat-small 942 in / 259 out - tool_calls 4.9 s | =| 0.00 s execute_tool send_reply sent
Trace 2 cd65f9ddbf0b8f6e1177ccde15517aaa … 1.5 s 0.0 s |========================| 1.49 s chat chat-small 2143 in / 92 out - stop 0.0 s |========================| 1.49 s invoke_agent resolver handoff context missingTrace 1 has a hole from 3.5 to 4.9 seconds: the run lasts 4.94 seconds, but its children end at 3.46. The decision, the call that chose to ask the customer for a photo, is in another trace, with no parent. In Grafana, searching the trace of this ticket shows the investigator and the reply, and not the decision. That is the day nurse’s empty chart.
The idea: the context travels inside the message
Section titled “The idea: the context travels inside the message”Context propagation means two steps. The sender writes the current trace context into the message. The receiver reads it first and makes it the parent of its work. The W3C standard form is one line, the traceparent:
00-7dad6f36…-ea73cacf…-01It has four parts: the version (00), the trace ID (32 hexadecimal characters), the parent span’s ID (16 characters) and the flags (01: sampled). With the trace ID and the parent’s span ID, the receiver’s span joins the same tree.
Text description of the diagram
The investigator’s work ends with a report of 17 items. The workflow puts the report into a message for the resolver, with the current trace context as a traceparent. It names the trace 7dad6f36 and the span ea73cacf, the workflow’s run span. The resolver reads the traceparent first, then starts its own span, whose parent is span ea73cacf. Without the traceparent, the resolver’s span has no parent and starts a new trace.
The handoff itself becomes evidence too. The resolver’s span records larkfield.handoff.from = investigator, larkfield.handoff.to = resolver, the size of the report (larkfield.handoff.report_items) and larkfield.handoff.context: propagated or missing. Each worker also has its own agent ID: support-workflow/investigator and support-workflow/resolver. The run’s first span is now invoke_workflow support-resolution, because code, not one agent, decides the order of the two workers.
A broken handoff should not hide the run: when the context is missing, the resolver’s span still records the run ID. That is why obs runs found both runs. A search by run ID still works; only the tree is broken.
Add the handoff and MCP to your project
Section titled “Add the handoff and MCP to your project”-
Copy the files of this lesson: the two workers (
agent.py), the MCP client and server, the MCP requirements and the two tests.Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/agent.py $S/resolver/mcp_tools.py $S/resolver/mcp_server.py resolver/cp $S/requirements-mcp.txt .cp $S/tests/test_handoff.py $S/tests/test_mcp.py tests/Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m03"Copy-Item $S\resolver\agent.py, $S\resolver\mcp_tools.py, $S\resolver\mcp_server.py resolver\Copy-Item $S\requirements-mcp.txt .Copy-Item $S\tests\test_handoff.py, $S\tests\test_mcp.py tests\Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/agent.py $S/resolver/mcp_tools.py $S/resolver/mcp_server.py resolver/cp $S/requirements-mcp.txt .cp $S/tests/test_handoff.py $S/tests/test_mcp.py tests/ -
Open
resolver/agent.pyand findhandoff_message. It builds the message with the report and, whenOBS_HANDOFF_CONTEXTison, withtraceparent. Then findobserve.agent_span("resolver", model, parent_message=message). Inresolver/observe.py,agent_spanreads the traceparent from the message and makes it the parent context. -
Run the tests:
python -m pytest -q. Without the MCP package, the last line says86 passed, 1 skipped, 1 deselected. The skipped test is the MCP one.tests/test_handoff.pychecks both cases: one trace with the context, two traces without it. -
Delete
work/, and run the two commands of the problem above. Then read the first run:python -m obs trace T-90103-workers-chat-small-1.Trace 1 7dad6f369cd6a8ffa30ff4069b8d28c7 … 5.0 s0.0 s |========================| 4.96 s invoke_workflow support-resolution ask_customer rule W50.0 s |================= | 3.46 s invoke_agent investigator finished0.0 s |====== | 1.26 s chat chat-small 694 in / 21 out - tool_calls1.3 s | = | 0.00 s execute_tool get_order ok LK-6404361.3 s | =========== | 2.20 s chat chat-small 942 in / 259 out - tool_calls3.5 s | ========| 1.49 s chat chat-small 2143 in / 92 out - stop3.5 s | ========| 1.49 s invoke_agent resolver handoff context propagated4.9 s | =| 0.01 s execute_tool send_reply sentOne trace. The resolver’s span starts at 3.5 seconds, where the investigator ended, and its model call is inside it.
obs tracelists the resolver’s model call just above its span because both start at the same time. -
Print the two worker spans:
python show_spans.py invoke_agent. The resolver of the first run saysparent ea73cacf(yours differs): the span ID ofinvoke_workflow, the span that was current when the message was written. The resolver of the second run saysparent noneandlarkfield.handoff.context = missing.
Side by side, the two runs of the same ticket:
Text description of the diagram
With the context, one trace holds the workflow, the investigator with its two calls and its read, the resolver with its call, and the reply. Without the context, trace 1 has the investigator and the reply, and a gap where the resolver worked. Trace 2 has the resolver and its call, alone, with no parent: it starts at zero on its own clock.
Across a process: the order lookup through MCP
Section titled “Across a process: the order lookup through MCP”The agents course has an optional MCP server for the order lookup. With RESOLVER_TOOLS=mcp, the workflow starts it as a second process and calls its tool get_order. This is a real process boundary, not a simulated one.
You do not write the propagation here. The MCP Python SDK, version 2.3.0, instruments itself. Its client makes a span for each request and puts the traceparent into the request’s params._meta. Its server reads it. One condition: the SDK uses OpenTelemetry’s global tracer provider, so the workflow must register its own provider as the global one. resolver/mcp_tools.py does it once per process, in _global_provider().
Text description of the diagram
Inside the span execute_tool get_order, the workflow asks the MCP client to call the tool. The SDK’s client makes a CLIENT span, MCP send tools/call get_order, and puts the traceparent into the request’s _meta. The server process reads it and makes a SERVER span, tools/call get_order, in the same trace, as a child of the client span. Both processes write their spans to the same telemetry folder, so the trace holds both sides.
-
Install the MCP package:
python -m pip install -r requirements-mcp.txt. Then run the tests again: the last line says87 passed, 1 deselected. -
Delete
work/. Run the ticket with the MCP lookup, and read its trace. Each call starts the server, so the run takes about a second longer.Terminal window RESOLVER_TOOLS=mcp python -m resolver run T-80008python -m obs trace T-80008Terminal window $env:RESOLVER_TOOLS = "mcp"python -m resolver run T-80008Remove-Item Env:RESOLVER_TOOLSpython -m obs trace T-80008Terminal window RESOLVER_TOOLS=mcp python -m resolver run T-80008python -m obs trace T-80008Trace 1 0b3cc8d8dfb66eda60e19ca6dae8bcfe … 4.5 s0.0 s |========================| 4.48 s invoke_agent support-resolver reply_only rule W10.0 s |===== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls1.0 s | ============= | 2.33 s execute_tool get_order ok LK-5811061.0 s | ===== | 1.00 s MCP send initialize1.9 s | = | 0.00 s initialize2.0 s | = | 0.01 s MCP send tools/call get_order2.0 s | = | 0.01 s tools/call get_order ok2.0 s | = | 0.00 s MCP send tools/list2.0 s | = | 0.00 s tools/list3.3 s | =======| 1.18 s chat chat-small 2411 in / 96 out - tool_calls4.5 s | =| 0.01 s execute_tool send_reply sent -
Find the server’s span with
python show_spans.py tools/call.tools/call get_orderhasparentequal to the span ID ofMCP send tools/call get_order, and the same trace. It also carriesmcp.method.name = tools/call, a name that the SDK chose, not us.
Two things in this trace look odd, and both are honest. The SDK also sends initialize and tools/list, the protocol’s start, so they have spans too. And the offsets are not exact. The SDK’s spans use the wall clock, while the workflow’s spans use the run clock, which moves by the recorded model time. Here MCP send initialize starts with the tool span, but in another run of the course team it started 0.6 seconds before it, inside the model call. Read the tree, not the exact positions, in this trace.
Your turn: which context does the resolver get?
Section titled “Your turn: which context does the resolver get?”The resolver took its parent from the span that was current when the message was written: the workflow’s run span. Change one thing in your head first: what if handoff_message were called inside the investigator’s span?
- Predict: what would the resolver’s parent be, and how would the waterfall look?
- Check your answer in the code: in
resolver/agent.py, the linemessage = handoff_message(report)comes after thewith observe.agent_span("investigator", …)block, so the investigator’s span has already ended. - Decide which tree is better, and why.
Discussion
Inside the investigator’s span, the traceparent would name the investigator. The resolver would then be the investigator’s child, one level deeper, and a waterfall would show the resolver inside a span that has already ended. Both trees are valid OpenTelemetry. The project chose the workflow as the parent, because the workflow decides the order of the two workers. The workers are siblings, and the handoff attributes say who handed over to whom. A team must choose one rule and keep it, so that every trace of every design reads the same way.
Common mistake
The MCP spans are missing
Symptom: with RESOLVER_TOOLS=mcp, the run works and execute_tool get_order has larkfield.mcp.server = larkfield-orders, but no MCP send … or tools/call … span appears. Or they appear in a trace of their own.
Cause: the MCP SDK writes its spans with OpenTelemetry’s global tracer provider. If the application never registers its provider as the global one, the SDK’s spans go to a provider that does nothing. If the server process has no exporter to the same place, its half is lost.
Diagnose: python show_spans.py MCP prints nothing. Check that _global_provider() in resolver/mcp_tools.py runs before the first call, and that the server gets OBS_TELEMETRY in its environment (_call() passes it).
Recover: register the provider once with trace.set_tracer_provider(...) in each process that uses an instrumented library, and give every process the same exporter settings. In production, each process exports to the Collector, and the Collector does not care which process a span came from.
When to propagate, and what it costs
Section titled “When to propagate, and what it costs”Propagate the context across every boundary where the same run continues: a message between workers, an HTTP or MCP call, a job in a queue. It costs one short string in each message. Without it, each part of the run is correct on its own, and nobody can see the whole.
Record the handoff, too: who handed over to whom, how much they passed, and whether the context arrived. Then a broken handoff is something you can count, as larkfield.handoff.context = missing, not a mystery.
Do not propagate when a new piece of work begins that is not part of the run. A nightly report that reads many runs is an example. Link it to them instead, as the approval did. And remember what the MCP trace showed: spans that other libraries make use their own names and their own clock. Pin those libraries, as the project pins the MCP SDK.
Knowledge check
Section titled “Knowledge check”Check your understanding
With OBS_HANDOFF_CONTEXT=off, obs runs still lists one run with three model calls. So Tomás says: 'Nothing is broken.' What is broken, and who would notice it first?
Show the answer
obs groups spans by run ID, and the resolver’s span still records the run ID, so the counts look right. But the trace is broken: the resolver’s span has no parent and starts a second trace. Grafana and every trace store group spans by trace. There, the run’s trace has a gap where the decision was made, and the decision appears as an unrelated trace. The person who opens the trace of a strange answer notices it first, because the step that chose the answer is missing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
What does a traceparent such as 00-7dad6f36…-ea73cacf…-01 carry, and why is the second ID a span ID and not the run ID?
Show the answer
It carries the version (00), the trace ID, the ID of the span that is the parent (ea73cacf…, the workflow’s run span) and the flags (01: sampled). The receiver needs the parent’s span ID to attach its span at the right place in the tree, not only in the right trace. A run ID is our own label. It groups the spans of a run, but it cannot say where in the tree a new span belongs. A trace store does not use it to build trees.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: show from your own telemetry that the MCP server's span belongs to the run's trace, in another process.
Show the answer
Run RESOLVER_TOOLS=mcp python -m resolver run T-80008, then python show_spans.py tools/call. It shows two spans with the same trace ID as the run. The first is MCP send tools/call get_order, the client in the workflow process, whose parent is execute_tool get_order. The second is tools/call get_order, the server, with mcp.method.name = tools/call and larkfield.mcp.server = larkfield-orders. Its parent is the client span. The server span was written by the server process, which got OBS_TELEMETRY from the workflow.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Context propagation: the sender writes the
traceparentinto the message, and the receiver reads it first and makes it the parent of its work. - The two workers hand over a message with the report and the context. Without the context, the resolver starts a second trace with no parent, while
obs runsstill looks normal. - The handoff is recorded: from, to, the report’s size, and whether the context arrived.
- The MCP SDK propagates the context in
_metaby itself, with the global tracer provider; its spans use the wall clock and its own names.
Your project now: the files of the last three lessons plus agent.py, mcp_tools.py, mcp_server.py, requirements-mcp.txt and two tests, with 86 tests that pass (and the MCP test with its package). Every part of a run is in one story, from the ticket to the person to the write. But every model call also passes through Larkfield’s model gateway in production, which keeps its own records of the same calls. Next, you compare what the gateway sees with what the app sees, and join the two.