Skip to content
Français

The gateway's view and the app's view

  • Lesson 11
  • 40 min
  • Module 3: Instrument the workflow

Where we are

In the last four lessons, our workflow learned to record its own story, from the ticket to the person to the write, across workers and processes. But at Larkfield, every model call also passes through the model gateway of the agents governance course, which keeps its own record of each call. In this lesson, we send the workflow through that gateway, compare what each side sees, and join the two records by agent ID and trace context. We start with a company card and an expense report.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Send the workflow’s model calls through Larkfield’s model gateway, and compare the gateway’s usage records with the workflow’s traces. Join each record to its model span by the trace context, and check that tokens and cost agree. When the workflow asks only for a capability, record the model that really answered.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project as the last lesson left it: 86 tests pass (87 with the MCP package).
  • A terminal in observable-workflow, with the virtual environment active.
  • For the optional live part: the gateway’s packages (requirements-gateway.txt), a second terminal, and the ports 8320 to 8326 free.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

An employee pays a hotel with the company’s card. The bank’s statement lists the payment: whose card, which hotel, the amount and the date. It lists every payment, even one that nobody reported.

The employee’s expense report knows other things: which trip, which client, who approved the trip. The bank never sees those.

At the end of the month, someone checks that each receipt matches a line of the statement. If both carry the same reference number, the check is easy and exact. Without it, they match by date and amount, and two similar payments get confused.

In Governing AI Agents, Larkfield puts an AI gateway in front of every model call. An AI gateway is one service between the apps and the model providers. It checks which agent calls, applies its limits, sends the call to a model, and writes one usage record per call. Kwame owns it.

The agents ask the gateway for a capability, not a model: a name for what they need, such as strong-reasoning. The gateway’s configuration decides which model serves it. Then Kwame can move a capability to a new model without changing any app.

So each model call is now recorded twice: by the gateway, and by the workflow’s chat span. Kwame and Amira tried to check one against the other, and hit two problems:

  • The records did not join. The governance course’s workflow sent a traceparent with each call, but it was made from the run ID. It used one made-up trace per run, and a random span ID per call. No span had that ID. Of 135 gateway records, 0 joined a model span by trace context.

  • The app did not know what it paid. The workflow asked for strong-reasoning, and its own price table has prices for models, not for capabilities. The workflow printed unknown for the cost:

    Outcome: resolve (rule W2) | 2 model call(s), 1 tool call(s) | 4586 tokens in, 265 out | unknown

The idea: each side sees part of the truth

Section titled “The idea: each side sees part of the truth”

The gateway and the app see the same call from two places:

The gateway sees The app sees
Who the registered agent, support-workflow, and its team, service the run, the ticket, the design and the step
What the capability, the provider, the deployment and the model version that answered the model name it asked for
How much tokens and cost of each call, as charged tokens of each call
What else calls that it refused, and calls from apps with no instrumentation at all tool calls, approvals, writes, the reply and the outcome

Neither view is complete. The gateway cannot say why a call happened, or whether the run paid a refund twice. The app cannot see the calls that the gateway refused before they reached a model, or an app that records nothing.

So we join them, with three keys:

  • The trace context. The workflow sends the real traceparent of its chat span with each call. The gateway saves it in the record. The span ID inside it is the span’s ID: one record, one span.
  • The run ID. The workflow also sends X-Run-Id. It joins all the calls of a run, but not one call to one span.
  • The agent ID. The gateway’s agent_id, support-workflow, is the workflow’s gen_ai.agent.id. It joins whole agents, for cost per agent in Module 5.

And the gateway gives something back. Its answer has headers: the model that answered, its request ID and the cost it charged. resolver/gateway.py records them on the same chat span, as larkfield.gateway.model, larkfield.gateway.request_id and larkfield.gateway.cost_usd.

Two boxes. Left, the app's view, the workflow's traces: the run with its ticket, design, outcome and release; the chat span with span ID e41cc07e…, which asked for strong-reasoning; the tool calls, the approval, the writes and the reply. Right, the gateway's view, one usage record per call: agent support-workflow, team service, capability strong-reasoning; provider azure, deployment chat-strong, model gpt-6.1-sol-2026-09-29; tokens, cost, refused calls, the traceparent ending -e41cc07e…-03 and the run ID. A thick arrow from the chat span to the record: sends traceparent and X-Run-Id. A dashed arrow back from the model to the chat span: answer header x-gateway-model, the model that answered.
One model call, two records: the chat span sends its trace context to the gateway, and the gateway's answer names the model. IDs from the course team's run of T-90104.
Text description of the diagram

The app’s traces hold the run, its chat spans, the tool calls, the approval, the writes and the reply. The gateway’s records hold, per call, the agent and its team, the capability, the provider, the deployment and the model version, tokens, cost and refusals. The chat span e41cc07e sends its traceparent and the run ID with the call, so the gateway’s record of that call carries the same span ID. The gateway’s answer header x-gateway-model names the model that answered, and the app puts it on the span.

The course team ran T-90104 through the gateway. Here is the second model call, as each side recorded it:

Field The chat span The gateway’s record
ID span e41cc07e075559ae in trace 59b819fc… traceparent 00-59b819fc8cd8d52726b79a2f93f88aeb-e41cc07e075559ae-03
Model asked for strong-reasoning; larkfield.gateway.model = chat-strong (gpt-6.1-sol-2026-09-29) capability strong-reasoning, deployment chat-strong, model version gpt-6.1-sol-2026-09-29
Tokens 2,426 in, 218 out tokens_in 2426, tokens_out 218
Cost larkfield.gateway.cost_usd = 0.007032 cost_usd 0.007032
Only here the run’s step, the design, the outcome agent_id support-workflow, team service, provider azure, replica a, request_id bc2abb98aa244246

The span ID inside the record’s traceparent is the span’s own ID, so the join is exact: one record, one span. The traceparent ends with -03: these are flags that the OpenTelemetry SDK set. The join uses only the two IDs.

  1. Copy the files of this lesson: the gateway provider, the configuration that knows it and the join command. Also copy the pinned copy of the governance course’s estate, the course team’s saved run and the lab’s notes.

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m03
    cp $S/resolver/gateway.py $S/resolver/config.py resolver/
    cp $S/obs/gateway.py $S/obs/__main__.py obs/
    cp -R $S/vendor .
    cp -R $S/data/telemetry/gateway data/telemetry/
    cp $S/gateway-lab.md $S/requirements-gateway.txt $S/.env.example $S/README.md .
  2. Open resolver/gateway.py and read its docstring. Then find headers["traceparent"] = tp: the call carries the context of the current span, the chat span that observed() opened. Find c.GATEWAY_MODEL: h.get("x-gateway-model"): the answer header goes on the same span.

  3. Run the tests: python -m pytest -q. The count does not change: 86 passed (87 with the MCP package). Your project is now the same as the snapshot end-of-m03, apart from your show_spans.py.

Guided practice: join the course team’s run

Section titled “Guided practice: join the course team’s run”

The course team ran all 70 tasks of the agents course through the gateway, on strong-reasoning, with agent_structured, the governance course’s production design. The traces are in data/telemetry/gateway, and the gateway’s records in gateway-usage.json. You need no running gateway for this part.

  1. Join the two:

    Terminal window
    python -m obs gateway data/telemetry/gateway/gateway-usage.json --data data/telemetry/gateway
    Gateway records: 135 model calls (ok 134, upstream_error 1)
    Workflow traces: 70 runs, 135 model calls, 83 tool calls, 28 approval requests
    Joined by trace context (the span ID in traceparent): 135 of 135 records
    Tokens in / out: gateway 310,041 / 15,702; traces 310,041 / 15,702
    Cost: gateway US$0.77710; traces US$0.77710
    Only the gateway sees: the agent's registration and team, the capability, the provider, refused calls.
    Only the traces see: the tool calls, the approval, the outcome of the run, and which step made each call.
    capability strong-reasoning was answered by chat-strong (gpt-6.1-sol-2026-09-29)
  2. Read the third line: every record found its span. Then the tokens and the cost: they agree exactly. The saved spans carry a cost that Module 5 computes from the model the gateway named, so a capability name no longer hides the price.

  3. Read the record that failed: upstream_error. It is T-80002, whose recorded call the provider refused with its content filter. Read the trace:

    Terminal window
    python -m obs trace T-80002 --data data/telemetry/gateway
    Run G-with-context-T-80002 | ticket T-80002 | strong-reasoning (agent_structured) | release 2.0.0 | language en
    1 model call(s), 0 read(s), 0 write call(s), 0 tokens in, 0 out, US$0.00000
    Trace 1 36d45a4fb18eddea54bf7f9e7ab72aa7 … 0.4 s
    0.0 s |========================| 0.40 s chat strong-reasoning ERROR ProviderError
    0.0 s |========================| 0.40 s invoke_agent support-resolver provider_error
    Outcome: handed_to_person | stopped: provider_error

    The span says only ProviderError. The gateway’s record says why: status 400, check_name provider, and the provider’s message about its content management policy. Here, the two views together explain one failure.

The course team also ran the same 70 tasks with OBS_GATEWAY_CONTEXT=off, the governance course’s behaviour. Then the workflow sent no real trace context:

Measured: the course team's run of the 70 tasks through the pinned gateway, twice; replayed model answers from the gateway's simulator.
Text description of the chart

Three bars, out of 135 gateway records. With the real trace context: 135 joined to their model span. Without it: 0. By run ID, in both runs: 135, but the run ID joins a record to a run of two or three calls, not to one span.

The run ID joins records to runs, so it is enough for cost per run. But only the trace context answers “which step made this expensive call?” And only the traces count the 83 tool calls and the 28 approval requests of these runs: the gateway never saw them.

The project carries a pinned copy of the governance course’s estate in vendor/ai-estate/ (PINNED.md lists each file’s hash). Its gateway answers with a simulator that replays the same recordings. No account and no key are needed.

  1. Install the estate’s packages: python -m pip install -r requirements-gateway.txt.

  2. In a second terminal, in the project folder, activate the virtual environment and go to the estate: cd vendor/ai-estate (PowerShell: cd vendor\ai-estate).

  3. Prepare the estate, as gateway-lab.md says. Each command prints a few lines:

    Terminal window
    python -m estate init
    python -m estate register registrations/support-workflow.yaml
    python -m estate review support-workflow --next 2026-12-01
    python -m estate credentials issue support-workflow
    python -m estate config publish --note "the registrations"

    The registration starts pending until a review approves it, as in the governance course. The credentials are in state/credentials/; never copy them into code.

  4. Start it. PYTHONPATH=../.. lets the estate’s tool server find the workflow’s code in the project folder:

    Terminal window
    PYTHONPATH=../.. python -m estate up
    provsim up (process 87973) http://127.0.0.1:8320
    control up (process 87974) http://127.0.0.1:8321
    replica-a up (process 87975) http://127.0.0.1:8323
    replica-b up (process 87976) http://127.0.0.1:8324
    replica-c up (process 87977) http://127.0.0.1:8325
    front up (process 87980) http://127.0.0.1:8322/front
    toolgate up (process 87984) http://127.0.0.1:8326

    Your process numbers differ. toolgate is the estate’s tool gateway. Without PYTHONPATH, it prints NOT ANSWERING, and the model gateway still works. This lesson does not use it: the model gateway is front and the three replicas.

  5. In the first terminal, run T-90104 through the gateway, with the capability as the model:

    Terminal window
    RESOLVER_PROVIDER=gateway ESTATE_STATE=vendor/ai-estate/state python -m resolver run T-90104 --variant agent_structured --model strong-reasoning

    The summary ends with | unknown: this stage of the project has no price for a capability name.

  6. Print the model spans: python show_spans.py chat. Each one now has larkfield.gateway.model = chat-strong (gpt-6.1-sol-2026-09-29), a larkfield.gateway.request_id and a larkfield.gateway.cost_usd.

  7. In the second terminal, read the gateway’s last records: python -m estate usage --last 2. Compare each record’s traceparent with the span IDs of step 6, and its tokens_in, tokens_out and cost_usd with the span’s.

  8. Stop the estate: python -m estate down. Every service prints stopped.

The gateway’s strongest claim is that it sees calls even from apps with no instrumentation. Test it. With the estate running, send one ticket with the trace context turned off:

Terminal window
RESOLVER_PROVIDER=gateway ESTATE_STATE=vendor/ai-estate/state OBS_GATEWAY_CONTEXT=off python -m resolver run T-80008 --variant agent_structured --model strong-reasoning

In PowerShell, add $env:OBS_GATEWAY_CONTEXT = "off" to the variables of step 5. Before you read the gateway’s records with python -m estate usage --last 2, predict which fields they have, and which one is missing.

The course team’s result
request_id 6b83b0dde924497e
agent_id support-workflow
team service
capability strong-reasoning
provider azure
deployment chat-strong
model_version gpt-6.1-sol-2026-09-29
tokens_in 2166
tokens_out 48
cost_usd 0.004812
status 200
outcome ok
run_id T-80008-agent_structured-strong-reasoning-1
fallback 0

The lines about time, the replica and the configuration are left out here. The record has the agent, the team, the capability, the model, the tokens and the cost: the gateway’s view is complete without any help from the app. Only the traceparent line is missing. The run ID is there because the workflow still sends X-Run-Id; an app with no instrumentation at all would not send it either. So the gateway can count every call and its cost per agent, but it cannot say which step or which run made it.

Common mistake

The gateway is not running

Symptom: you run a ticket with RESOLVER_PROVIDER=gateway before python -m estate up, or after python -m estate down. The run stops at once:

Run T-90104-agent_structured-strong-reasoning-1 | single agent with one JSON step per call (for models without tool calling) | strong-reasoning
Stopped: provider_error. No proposal: the ticket goes to a person.
Last error: Cannot reach the platform's token endpoint: <urlopen error [Errno 61] Connection refused>

On Linux the number is 111; on Windows, 10061 (not tested). If you never issued the credentials, the run stops earlier, with No credentials at … and the command that issues them.

Cause: the workflow first asks the estate’s control plane for a token, on port 8321, and nothing answers. The workflow does the safe thing: no proposal, and the ticket goes to a person.

Diagnose: python -m obs trace T-90104 shows one chat strong-reasoning span, ERROR ProviderError, and the outcome handed_to_person | stopped: provider_error. python -m estate usage --last 1 shows no record of this call. This failure is in the app’s view only: the call never reached the gateway.

Recover: start the estate in the second terminal (python -m estate up) and run the ticket again. So the two views fail in different places. A call the provider refuses is explained by the gateway’s record; a call that never reaches the gateway is visible only in the app’s trace.

Use the gateway’s records for questions about agents and money: cost per registered agent and team, which model really answered, refused calls, and apps that record nothing. Use the traces for questions about runs: which step made a call, what the tools did, who approved, and what the customer got. Use both, joined, for the questions in between, such as “which step of which run made this expensive call?”

The join costs two headers per call: the traceparent of the real chat span, and the run ID. Send the real context, never one made up from another ID. Record the gateway’s answer headers on the span, so that the app knows the model behind a capability. One more control to know: the gateway limits each agent, 120 requests a minute for the workflow. In the course team’s run, the gateway refused a second pass of the 70 tasks within the same minute, with HTTP 429. So the team paused 1.2 seconds between tasks. Then check, as the course did, that the two totals agree: tokens 310,041 and 15,702 on both sides, and US$0.77710. A difference is a finding: a refused call, a retry, or a price table that is out of date.

Check your understanding

The governance course's workflow sent a traceparent with every call, and still 0 of 135 records joined a span. Why, and what does the join need?

Show the answer

Its traceparent was made from the run ID: a made-up trace ID per run and a random span ID per call. So no span in the workflow’s traces had that ID. A join needs the context of the real chat span: the trace ID and the span ID that the span really has. With it, 135 of 135 records joined their span. A traceparent is only useful if it is the real one.

Check your understanding

Kwame says: 'The gateway records every model call with tokens and cost. We do not need the traces any more.' Name two things the gateway cannot answer, with numbers from the course's run.

Show the answer

The gateway never sees what happens between model calls: the 83 tool calls and the 28 approval requests of the 70 runs, the writes, the reply and the outcome. So it cannot say that a run paid a refund twice, or who approved it. And without the trace context it cannot say which step made a call. By run ID, it joins a record only to a run of two or three calls. On the other side, the traces cannot see refused calls or apps with no instrumentation, so the two views are used together.

Check your understanding

Practical check: from your own run through the live gateway (or the saved run), join one gateway record to its span and show that the model, the tokens and the cost agree.

Show the answer

Take one record from python -m estate usage –last 2 (or from gateway-usage.json) and the span from python show_spans.py chat (or obs trace on the saved data). The span ID in the record’s traceparent equals the chat span’s span ID, for example e41cc07e075559ae in the course team’s run. The record’s deployment and model_version equal the span’s larkfield.gateway.model, chat-strong (gpt-6.1-sol-2026-09-29). tokens_in and tokens_out equal gen_ai.usage.input_tokens and output_tokens (2,426 and 218), and cost_usd equals larkfield.gateway.cost_usd (0.007032).

Recap

  • An AI gateway records every model call: the agent, its team, the capability, the model that answered, tokens, cost and refusals, even for apps that record nothing.
  • Only the app’s traces see the run: the tool calls, the approvals, the writes and the outcome.
  • Join them by the real trace context (one record, one span), by the run ID (one run) and by the agent ID (one agent). A traceparent made up from the run ID joins nothing: 0 of 135.
  • Record the gateway’s answer headers on the span, so that a capability name does not hide the model. In the course’s run, tokens and cost agreed exactly: 310,041 and 15,702 tokens, US$0.77710.

Your project now: the same as the snapshot end-of-m03, with 86 tests that pass (87 with the MCP package), plus your show_spans.py. In this module, you instrumented every model call, tool call, approval, handoff and gateway call. The double refund is now one connected story. But a story is only useful if you can read it fast, and find the runs that need reading. Next, in Module 4, you read a run in Grafana and find the agent failures it holds.

Next: Read a run