Skip to content
Français

Many agents and handoffs

  • Lesson 10
  • 35 min
  • Module 3: Instrument the workflow

Where we are

In the last lesson, a run that waits for a person stayed one story: two traces, joined by a link. But the workflow has a design where two workers share one ticket, and its order lookup can run in another process, through an MCP server. In this lesson, we carry the trace context across both, so that nothing breaks the story. We start at a hospital's shift change.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Keep a run one trace when its work moves to another worker or another process. Put the trace context into the handoff message and into the MCP request, and read it on the other side. Record the handoff itself: who handed over to whom, and whether the context arrived.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Context propagation and the traceparent header, from Spans, traces and context.
  • The agents course’s two-worker design, workers: an investigator reads the order and writes a report, and a resolver decides from the report, on the same ticket.
  • For the MCP part: what an MCP server and a tool call are, from MCP: Connect AI Applications to Tools and Data. This part is optional.

What you need on your computer

  • The project as the last lesson left it: 84 tests pass, and show_spans.py is in the project folder.
  • A terminal in observable-workflow, with the virtual environment active. The MCP part installs one more package, mcp.

Tested on macOS (Apple silicon) with Python 3.12.13 and the MCP SDK 2.3.0. The Windows and Linux commands are written, not run.

At 7 in the morning, the night nurse hands over to the day nurse. She gives a short note: what happened in the night, what to watch, and the patient’s chart number. The day nurse writes everything she does into that chart.

If the note forgets the chart number, the day nurse still does her work. But she writes it into a new, empty chart. The next morning, a doctor reads the patient’s chart and sees nothing after 7. The work happened, and the story has a hole.

The note must carry the reference to the story, not only the facts. That is the whole lesson.

In the workers design, the investigator reads the order, then hands a report to the resolver. In production, the two are separate processes: the report is a message from one to the other. Here, the project behaves the same way: the resolver takes its parent from the message, not from the memory of the process.

The project has a switch to forget the reference, as a careless handoff would. The course team ran the same ticket twice with this lesson’s code, with the context and without it. You run the same commands after the next section:

Terminal window
python -m resolver run T-90103 --variant workers
OBS_HANDOFF_CONTEXT=off python -m resolver run T-90103 --variant workers
python -m obs runs

In PowerShell, set the variable first, $env:OBS_HANDOFF_CONTEXT = "off", run the ticket, then remove it with Remove-Item Env:OBS_HANDOFF_CONTEXT. The list of runs looks normal:

run ticket started outcome model tools appr cost US$ secs model flags
T-90103-workers-chat-small-1 T-90103 … asked_customer 3 1 0 0.00000 4.9 chat-small -
T-90103-workers-chat-small-2 T-90103 … asked_customer 3 1 0 0.00000 4.9 chat-small -
2 run(s), 0 flagged

Both runs made three model calls and one tool call. obs runs groups spans by run ID, so it finds all of them. But a trace store such as Tempo groups spans by trace. Look at the second run’s trace:

Trace 1 aed2bc3da6441e9d985419a7153381da … 4.9 s
0.0 s |========================| 4.94 s invoke_workflow support-resolution ask_customer rule W5
0.0 s |================= | 3.46 s invoke_agent investigator finished
0.0 s |====== | 1.26 s chat chat-small 694 in / 21 out - tool_calls
1.3 s | = | 0.00 s execute_tool get_order ok LK-640436
1.3 s | =========== | 2.20 s chat chat-small 942 in / 259 out - tool_calls
4.9 s | =| 0.00 s execute_tool send_reply sent
Trace 2 cd65f9ddbf0b8f6e1177ccde15517aaa … 1.5 s
0.0 s |========================| 1.49 s chat chat-small 2143 in / 92 out - stop
0.0 s |========================| 1.49 s invoke_agent resolver handoff context missing

Trace 1 has a hole from 3.5 to 4.9 seconds: the run lasts 4.94 seconds, but its children end at 3.46. The decision, the call that chose to ask the customer for a photo, is in another trace, with no parent. In Grafana, searching the trace of this ticket shows the investigator and the reply, and not the decision. That is the day nurse’s empty chart.

The idea: the context travels inside the message

Section titled “The idea: the context travels inside the message”

Context propagation means two steps. The sender writes the current trace context into the message. The receiver reads it first and makes it the parent of its work. The W3C standard form is one line, the traceparent:

00-7dad6f36…-ea73cacf…-01

It has four parts: the version (00), the trace ID (32 hexadecimal characters), the parent span’s ID (16 characters) and the flags (01: sampled). With the trace ID and the parent’s span ID, the receiver’s span joins the same tree.

The investigator, invoke_agent investigator, makes two model calls and one get_order, and writes a report of 17 items. An arrow, 'the workflow writes the report and the current context', goes to the handoff message: from investigator, to resolver, the report with its 17 items, and traceparent 00-7dad6f36…-ea73cacf…-01. An arrow, 'the resolver reads the context first, then starts its span', goes to invoke_agent resolver: its parent is span ea73cacf…, the workflow's run span. A dashed arrow, OBS_HANDOFF_CONTEXT=off, goes to: without traceparent, the resolver starts a new trace with no parent.
The handoff message carries the report and the trace context; the resolver's span takes its parent from the message. IDs from the course team's run.
Text description of the diagram

The investigator’s work ends with a report of 17 items. The workflow puts the report into a message for the resolver, with the current trace context as a traceparent. It names the trace 7dad6f36 and the span ea73cacf, the workflow’s run span. The resolver reads the traceparent first, then starts its own span, whose parent is span ea73cacf. Without the traceparent, the resolver’s span has no parent and starts a new trace.

The handoff itself becomes evidence too. The resolver’s span records larkfield.handoff.from = investigator, larkfield.handoff.to = resolver, the size of the report (larkfield.handoff.report_items) and larkfield.handoff.context: propagated or missing. Each worker also has its own agent ID: support-workflow/investigator and support-workflow/resolver. The run’s first span is now invoke_workflow support-resolution, because code, not one agent, decides the order of the two workers.

A broken handoff should not hide the run: when the context is missing, the resolver’s span still records the run ID. That is why obs runs found both runs. A search by run ID still works; only the tree is broken.

  1. Copy the files of this lesson: the two workers (agent.py), the MCP client and server, the MCP requirements and the two tests.

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m03
    cp $S/resolver/agent.py $S/resolver/mcp_tools.py $S/resolver/mcp_server.py resolver/
    cp $S/requirements-mcp.txt .
    cp $S/tests/test_handoff.py $S/tests/test_mcp.py tests/
  2. Open resolver/agent.py and find handoff_message. It builds the message with the report and, when OBS_HANDOFF_CONTEXT is on, with traceparent. Then find observe.agent_span("resolver", model, parent_message=message). In resolver/observe.py, agent_span reads the traceparent from the message and makes it the parent context.

  3. Run the tests: python -m pytest -q. Without the MCP package, the last line says 86 passed, 1 skipped, 1 deselected. The skipped test is the MCP one. tests/test_handoff.py checks both cases: one trace with the context, two traces without it.

  4. Delete work/, and run the two commands of the problem above. Then read the first run: python -m obs trace T-90103-workers-chat-small-1.

    Trace 1 7dad6f369cd6a8ffa30ff4069b8d28c7 … 5.0 s
    0.0 s |========================| 4.96 s invoke_workflow support-resolution ask_customer rule W5
    0.0 s |================= | 3.46 s invoke_agent investigator finished
    0.0 s |====== | 1.26 s chat chat-small 694 in / 21 out - tool_calls
    1.3 s | = | 0.00 s execute_tool get_order ok LK-640436
    1.3 s | =========== | 2.20 s chat chat-small 942 in / 259 out - tool_calls
    3.5 s | ========| 1.49 s chat chat-small 2143 in / 92 out - stop
    3.5 s | ========| 1.49 s invoke_agent resolver handoff context propagated
    4.9 s | =| 0.01 s execute_tool send_reply sent

    One trace. The resolver’s span starts at 3.5 seconds, where the investigator ended, and its model call is inside it. obs trace lists the resolver’s model call just above its span because both start at the same time.

  5. Print the two worker spans: python show_spans.py invoke_agent. The resolver of the first run says parent ea73cacf (yours differs): the span ID of invoke_workflow, the span that was current when the message was written. The resolver of the second run says parent none and larkfield.handoff.context = missing.

Side by side, the two runs of the same ticket:

Three waterfalls on a scale of 0 to 5 seconds. With the context in the message, one trace: invoke_workflow support-resolution for 4.9 s; under it invoke_agent investigator for 3.5 s with chat chat-small (694 in, 21 out, tool_calls), execute_tool get_order (ok) and chat chat-small (942 in, 259 out, tool_calls); then invoke_agent resolver (handoff context propagated) from 3.5 to 4.9 s with chat chat-small (2143 in, 92 out, stop); then execute_tool send_reply (sent). Without it, trace 1: the same, but without the resolver: a gap from 3.5 to 4.9 seconds. Without it, trace 2: invoke_agent resolver (handoff context missing) with its chat chat-small, alone, starting at 0.
T-90103 on the two-worker design, with and without the trace context in the handoff message. Replayed calls of chat-small.
Text description of the diagram

With the context, one trace holds the workflow, the investigator with its two calls and its read, the resolver with its call, and the reply. Without the context, trace 1 has the investigator and the reply, and a gap where the resolver worked. Trace 2 has the resolver and its call, alone, with no parent: it starts at zero on its own clock.

Across a process: the order lookup through MCP

Section titled “Across a process: the order lookup through MCP”

The agents course has an optional MCP server for the order lookup. With RESOLVER_TOOLS=mcp, the workflow starts it as a second process and calls its tool get_order. This is a real process boundary, not a simulated one.

You do not write the propagation here. The MCP Python SDK, version 2.3.0, instruments itself. Its client makes a span for each request and puts the traceparent into the request’s params._meta. Its server reads it. One condition: the SDK uses OpenTelemetry’s global tracer provider, so the workflow must register its own provider as the global one. resolver/mcp_tools.py does it once per process, in _global_provider().

A sequence with three participants: the workflow process, inside the span execute_tool get_order; the MCP client, the MCP SDK; and the MCP server process, larkfield-orders. The workflow calls get_order(order_id). The client makes the span 'MCP send tools/call get_order', kind CLIENT, a child of execute_tool. It sends tools/call get_order with params._meta.traceparent = 00-…-01. The server makes the span 'tools/call get_order', kind SERVER, in the same trace, whose parent is the client span. The server returns the order, and the client returns ok, LK-581106. Both processes write to the same telemetry folder: one trace.
The MCP SDK carries the trace context in the request's _meta, so the server's span joins the workflow's trace in another process.
Text description of the diagram

Inside the span execute_tool get_order, the workflow asks the MCP client to call the tool. The SDK’s client makes a CLIENT span, MCP send tools/call get_order, and puts the traceparent into the request’s _meta. The server process reads it and makes a SERVER span, tools/call get_order, in the same trace, as a child of the client span. Both processes write their spans to the same telemetry folder, so the trace holds both sides.

  1. Install the MCP package: python -m pip install -r requirements-mcp.txt. Then run the tests again: the last line says 87 passed, 1 deselected.

  2. Delete work/. Run the ticket with the MCP lookup, and read its trace. Each call starts the server, so the run takes about a second longer.

    Terminal window
    RESOLVER_TOOLS=mcp python -m resolver run T-80008
    python -m obs trace T-80008
    Trace 1 0b3cc8d8dfb66eda60e19ca6dae8bcfe … 4.5 s
    0.0 s |========================| 4.48 s invoke_agent support-resolver reply_only rule W1
    0.0 s |===== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls
    1.0 s | ============= | 2.33 s execute_tool get_order ok LK-581106
    1.0 s | ===== | 1.00 s MCP send initialize
    1.9 s | = | 0.00 s initialize
    2.0 s | = | 0.01 s MCP send tools/call get_order
    2.0 s | = | 0.01 s tools/call get_order ok
    2.0 s | = | 0.00 s MCP send tools/list
    2.0 s | = | 0.00 s tools/list
    3.3 s | =======| 1.18 s chat chat-small 2411 in / 96 out - tool_calls
    4.5 s | =| 0.01 s execute_tool send_reply sent
  3. Find the server’s span with python show_spans.py tools/call. tools/call get_order has parent equal to the span ID of MCP send tools/call get_order, and the same trace. It also carries mcp.method.name = tools/call, a name that the SDK chose, not us.

Two things in this trace look odd, and both are honest. The SDK also sends initialize and tools/list, the protocol’s start, so they have spans too. And the offsets are not exact. The SDK’s spans use the wall clock, while the workflow’s spans use the run clock, which moves by the recorded model time. Here MCP send initialize starts with the tool span, but in another run of the course team it started 0.6 seconds before it, inside the model call. Read the tree, not the exact positions, in this trace.

Your turn: which context does the resolver get?

Section titled “Your turn: which context does the resolver get?”

The resolver took its parent from the span that was current when the message was written: the workflow’s run span. Change one thing in your head first: what if handoff_message were called inside the investigator’s span?

  1. Predict: what would the resolver’s parent be, and how would the waterfall look?
  2. Check your answer in the code: in resolver/agent.py, the line message = handoff_message(report) comes after the with observe.agent_span("investigator", …) block, so the investigator’s span has already ended.
  3. Decide which tree is better, and why.
Discussion

Inside the investigator’s span, the traceparent would name the investigator. The resolver would then be the investigator’s child, one level deeper, and a waterfall would show the resolver inside a span that has already ended. Both trees are valid OpenTelemetry. The project chose the workflow as the parent, because the workflow decides the order of the two workers. The workers are siblings, and the handoff attributes say who handed over to whom. A team must choose one rule and keep it, so that every trace of every design reads the same way.

Common mistake

The MCP spans are missing

Symptom: with RESOLVER_TOOLS=mcp, the run works and execute_tool get_order has larkfield.mcp.server = larkfield-orders, but no MCP send … or tools/call … span appears. Or they appear in a trace of their own.

Cause: the MCP SDK writes its spans with OpenTelemetry’s global tracer provider. If the application never registers its provider as the global one, the SDK’s spans go to a provider that does nothing. If the server process has no exporter to the same place, its half is lost.

Diagnose: python show_spans.py MCP prints nothing. Check that _global_provider() in resolver/mcp_tools.py runs before the first call, and that the server gets OBS_TELEMETRY in its environment (_call() passes it).

Recover: register the provider once with trace.set_tracer_provider(...) in each process that uses an instrumented library, and give every process the same exporter settings. In production, each process exports to the Collector, and the Collector does not care which process a span came from.

Propagate the context across every boundary where the same run continues: a message between workers, an HTTP or MCP call, a job in a queue. It costs one short string in each message. Without it, each part of the run is correct on its own, and nobody can see the whole.

Record the handoff, too: who handed over to whom, how much they passed, and whether the context arrived. Then a broken handoff is something you can count, as larkfield.handoff.context = missing, not a mystery.

Do not propagate when a new piece of work begins that is not part of the run. A nightly report that reads many runs is an example. Link it to them instead, as the approval did. And remember what the MCP trace showed: spans that other libraries make use their own names and their own clock. Pin those libraries, as the project pins the MCP SDK.

Check your understanding

With OBS_HANDOFF_CONTEXT=off, obs runs still lists one run with three model calls. So Tomás says: 'Nothing is broken.' What is broken, and who would notice it first?

Show the answer

obs groups spans by run ID, and the resolver’s span still records the run ID, so the counts look right. But the trace is broken: the resolver’s span has no parent and starts a second trace. Grafana and every trace store group spans by trace. There, the run’s trace has a gap where the decision was made, and the decision appears as an unrelated trace. The person who opens the trace of a strange answer notices it first, because the step that chose the answer is missing.

Check your understanding

What does a traceparent such as 00-7dad6f36…-ea73cacf…-01 carry, and why is the second ID a span ID and not the run ID?

Show the answer

It carries the version (00), the trace ID, the ID of the span that is the parent (ea73cacf…, the workflow’s run span) and the flags (01: sampled). The receiver needs the parent’s span ID to attach its span at the right place in the tree, not only in the right trace. A run ID is our own label. It groups the spans of a run, but it cannot say where in the tree a new span belongs. A trace store does not use it to build trees.

Check your understanding

Practical check: show from your own telemetry that the MCP server's span belongs to the run's trace, in another process.

Show the answer

Run RESOLVER_TOOLS=mcp python -m resolver run T-80008, then python show_spans.py tools/call. It shows two spans with the same trace ID as the run. The first is MCP send tools/call get_order, the client in the workflow process, whose parent is execute_tool get_order. The second is tools/call get_order, the server, with mcp.method.name = tools/call and larkfield.mcp.server = larkfield-orders. Its parent is the client span. The server span was written by the server process, which got OBS_TELEMETRY from the workflow.

Recap

  • Context propagation: the sender writes the traceparent into the message, and the receiver reads it first and makes it the parent of its work.
  • The two workers hand over a message with the report and the context. Without the context, the resolver starts a second trace with no parent, while obs runs still looks normal.
  • The handoff is recorded: from, to, the report’s size, and whether the context arrived.
  • The MCP SDK propagates the context in _meta by itself, with the global tracer provider; its spans use the wall clock and its own names.

Your project now: the files of the last three lessons plus agent.py, mcp_tools.py, mcp_server.py, requirements-mcp.txt and two tests, with 86 tests that pass (and the MCP test with its package). Every part of a run is in one story, from the ticket to the person to the write. But every model call also passes through Larkfield’s model gateway in production, which keeps its own records of the same calls. Next, you compare what the gateway sees with what the app sees, and join the two.

Next: The gateway's view and the app's view