Model calls and tokens
- Lesson 7
- 35 min
- Module 3: Instrument the workflow
Where we are
In Module 2, we built the pieces: spans, the pinned GenAI names and a Collector that redacts and routes. But our trace of a ticket is still one span that lasts 0.00 seconds, because the model's time is not in it. In this lesson, we give every model call its own span, with the model, the tokens, the finish reason, the latency and the prompt version, and never the text. We start with a phone bill.
Lesson 7 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsModel calls and tokens
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Give every model call of the workflow its own span. Record which model answered, the tokens in and out, the finish reason, the time it took and the prompt version. Then a trace explains where a run’s time and tokens went. Record none of the text.
Before you startWhat to know and to have ready
What you need to know
- What a span, a trace and a span’s attributes are, from Spans, traces and context.
- The pinned GenAI names, such as
chat {model}andgen_ai.usage.input_tokens, from The GenAI conventions for agent runs. - What a token is, and that the workflow replays the recorded model calls of AI Agents and Workflow Orchestration.
What you need on your computer
- The project as Module 2 left it, in
observable-workflow: the same as the snapshotend-of-m02, with 81 tests that pass. - A terminal in
observable-workflow, with the virtual environment active. The labs repository,nextia-learning-labs, is in the same parent folder.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A phone bill
Section titled “A phone bill”A phone bill lists every call: the number, the time, how long it lasted and what it cost. It never contains what you said. Still, it answers useful questions. Why was this month expensive? Which call was the long one? Did a new plan change anything?
The bill can answer them because it records the same few facts for every call, in the same form. One line per call, and no conversation. That is the idea of this lesson.
The problem: a run that took no time
Section titled “The problem: a run that took no time”At the end of Module 2, run a ticket and read its trace:
python -m resolver run T-80008python -m obs trace T-80008The workflow prints that the run made two model calls and used 4,580 tokens:
Run T-80008-agent-chat-small-1 | single agent with tool calling: the model chooses every step | chat-smallOutcome: reply_only (rule W1) | 2 model call(s), 1 tool call(s) | 4580 tokens in, 117 out | US$0.00052Reply draft: Your garden shed order LK-581106 shipped on Monday, October 5. Its expected delivery date is Tuesday, October 13. It is currently in transit, and there is no delivery date showing yet.Reply sent. Ticket: resolved.But the trace says something else:
Run T-80008-agent-chat-small-1 | ticket T-80008 | chat-small (agent) | release 2.0.0 | language en0 model call(s), 0 read(s), 0 write call(s), 0 tokens in, 0 out, US$0.00000
Trace 1 88ec5e0f1603e609be2414b7caee63ee … 0.0 s 0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1
Outcome: resolvedThe trace has one span, and it lasts 0.00 seconds. It knows the outcome, but not the two model calls, their tokens or their time. If Omar asks “why did the bill double this week?”, this trace cannot help. The numbers exist in the program, but nothing records them per call.
Why 0.00 seconds? The workflow replays recorded model answers, and a replay answers in microseconds. From Module 2, spans use the run clock: each run’s clock moves forward by the recorded latency of each model call. But only a model span moves it, and there is none yet.
The idea: one chat span per model call
Section titled “The idea: one chat span per model call”Give each model call its own span, as a child of the run span. The GenAI conventions name it chat {model}, for example chat chat-small, with the operation name chat. Its kind is CLIENT, because the workflow calls another service. Each chat span records the same facts, like one line of the phone bill:
Text description of the diagram
A model call has a request (the model, a token limit, the prompt and the ticket) and an answer (text or a tool call, tokens, a finish reason). The span keeps the model asked for and the model that answered, the tokens in and out, and the finish reason. It also keeps the latency as its own duration, the prompt’s name and version, the step number and the names of the steps the model asked for. It leaves out every text: the prompt, the ticket, the answer and the values of tool arguments.
Five attributes need a short explanation:
- Input and output tokens (
gen_ai.usage.input_tokens,gen_ai.usage.output_tokens). The provider counts them and returns them with the answer. Prices are per token, so Module 5 computes cost from them. - The finish reason (
gen_ai.response.finish_reasons): why the model stopped.stopmeans a normal end,tool_callsmeans it asks for a tool,lengthmeans it hit the token limit, andcontent_filtermeans the provider refused. The conventions make it a list, because one request can ask for several answers. - The model that answered (
gen_ai.response.model). The workflow asks forchat-small, a deployment name. The provider answers with the model behind it,gpt-6-luna-2026-09-22. When the deployment moves to a new model, only this attribute shows it. - The prompt version (
gen_ai.prompt.version): the first 8 characters of a hash of the prompt file and the policy, as in The GenAI conventions for agent runs. So a trace can say which prompt a call used, without the text. - The latency is the span’s duration. In a replay, the span lasts as long as the recorded call did:
larkfield.recorded_latency_skeeps the recorded number.
The project also records three metrics for each call: the call’s duration (gen_ai.client.inference.duration, a histogram) and two counters for input and output tokens. Their labels are the operation, the provider and the two model names, never a ticket. Module 5 explains why labels must stay few.
Worked example: two calls, one run
Section titled “Worked example: two calls, one run”Here is what the two chat spans of T-80008 say, from the course team’s run:
| Call | Finish reason | Input tokens | Output tokens | Recorded latency |
|---|---|---|---|---|
| 1 | tool_calls: the model asks for get_order(order_id) |
2169 | 21 | 0.96 s |
| 2 | tool_calls: the model calls finish with its proposal |
2411 | 96 | 1.18 s |
| Run | 4,580 | 117 | 2.14 s |
The sums are the numbers the workflow printed: 4,580 tokens in and 117 out. The run span now lasts 0.96 + 1.18 = 2.14 seconds, because the run clock moved forward by each call’s recorded latency.
Look at the input tokens. The second call sends 242 more tokens than the first. A model has no memory between calls, so the workflow sends the whole conversation again. That is the prompt, the ticket, the first answer and the order that get_order returned. So input tokens grow with every step, and they are most of the total:
Text description of the chart
Four bars. Call 1: 2169 input tokens and 21 output tokens. Call 2: 2411 input tokens and 96 output tokens. The input bars are more than twenty times higher than the output bars. The second input bar is higher than the first, because the second call sends the whole conversation again.
Of the 4,697 tokens of this run, 97% are input. A run with more steps sends the conversation more often, so it costs more than its number of answers suggests. Only a span per call shows this.
Add the model spans to your project
Section titled “Add the model spans to your project”The change is in two files. resolver/observe.py gets a wrapper, observed(), around the provider’s complete() function. resolver/runner.py wraps the provider once, for every design of the workflow. The copy of observe.py also contains the tool and write spans of the next lessons, but nothing calls them yet.
-
Copy the two files from the snapshot
end-of-m03:Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/observe.py $S/resolver/runner.py resolver/Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m03"Copy-Item $S\resolver\observe.py, $S\resolver\runner.py resolver\Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/observe.py $S/resolver/runner.py resolver/ -
Open
resolver/runner.py. The run now passesobserve.observed(complete)instead ofcomplete. One line of the workflow changed: every design gets the same model spans. -
Open
resolver/observe.pyand finddef observed(complete). Read the two groups of attributes. The first group is set before the call: the model asked for, the prompt and the step. The second group is set after the answer: the model that answered, the tokens and the finish reason. -
Run the ticket again and read its trace:
Terminal window python -m resolver run T-80008python -m obs trace T-80008Run T-80008-agent-chat-small-2 | ticket T-80008 | chat-small (agent) | release 2.0.0 | language en2 model call(s), 0 read(s), 0 write call(s), 4580 tokens in, 117 out, US$0.00000Trace 1 488d2ec9ffa2868b9e43ebd9ea937fdc … 2.1 s0.0 s |========================| 2.14 s invoke_agent support-resolver reply_only rule W10.0 s |=========== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls1.0 s | ==============| 1.18 s chat chat-small 2411 in / 96 out - tool_callsYour trace ID and time differ. The run number ends in
-2if you ran this ticket once before. The cost column shows-: the spans have no cost yet, and Module 5 adds it.
The two traces of the same ticket, before and after:
Text description of the diagram
Before, the trace has one span with no length. After, the run span lasts 2.14 seconds and has two children, the two calls to chat-small. The first lasts 0.96 seconds and ends with tool_calls. The second starts when the first ends and lasts 1.18 seconds. The run’s time is now the sum of its model calls. The tool call between them is not in the trace yet.
The read, get_order, happened between the two calls, but it has no span yet. The next lesson adds it.
Read one span’s attributes
Section titled “Read one span’s attributes”obs trace shows a summary. To see every attribute, read the telemetry file itself. The file work/telemetry/traces.jsonl holds OTLP JSON: one export request per line, with the spans inside.
-
Create a file
show_spans.pyin the project folder, with this content:"""Print the spans in work/telemetry/traces.jsonl whose name contains a word, with every attribute."""import jsonimport sysfrom pathlib import Pathword = sys.argv[1] if len(sys.argv) > 1 else ""for line in Path("work/telemetry/traces.jsonl").read_text().splitlines():for resource in json.loads(line)["resourceSpans"]:for scope in resource["scopeSpans"]:for span in scope["spans"]:if word not in span["name"]:continueparent = span.get("parentSpanId", "")[:8] or "none"print(f'{span["name"]} trace {span["traceId"][:8]} span {span["spanId"][:8]} parent {parent}')for a in span.get("attributes", []):value = next(iter(a["value"].values()))if isinstance(value, dict): # a list, such as the finish reasonsvalue = [next(iter(v.values())) for v in value.get("values", [])]print(f' {a["key"]} = {value}')for link in span.get("links", []):print(f' link to trace {link["traceId"][:8]}, span {link["spanId"][:8]}')This helper is yours: it is not in the snapshot, and the next lessons use it.
-
Print the model spans:
python show_spans.py chat. The first one says:chat chat-small trace 488d2ec9 span 0ec2f8ab parent 34b1f358gen_ai.operation.name = chatgen_ai.provider.name = azure.ai.openaigen_ai.request.model = chat-smallgen_ai.request.max_tokens = 2000larkfield.step = 1gen_ai.prompt.name = agentgen_ai.prompt.version = d6c784d1gen_ai.output.type = textgen_ai.conversation.id = T-80008-agent-chat-small-2gen_ai.response.model = gpt-6-luna-2026-09-22gen_ai.usage.input_tokens = 2169gen_ai.usage.output_tokens = 21gen_ai.response.finish_reasons = ['tool_calls']larkfield.replay = exactlarkfield.recorded_latency_s = 0.956larkfield.model.requested = ['get_order(order_id)'] -
Find the three kinds of names.
gen_ai.*names come from the pinned conventions.larkfield.*names are ours: our prefix says so.gen_ai.conversation.idis the run ID, so a tool that knows the conventions can group the calls of one run. -
Find
larkfield.replay = exact. It says that the recorded answer was replayed for a byte-for-byte equal request. A live call would saylive.
larkfield.model.requested keeps what the model asked to do next, as a tool name and its argument names: get_order(order_id). It never keeps the order number that the model wrote. Module 4 uses it to find loops: a model that asks for the same step again and again.
The tests that pinned the old shape
Section titled “The tests that pinned the old shape”-
Run the tests:
python -m pytest -q.FAILED tests/test_telemetry.py::test_no_text_leaves_the_application_by_default1 failed, 80 passed, 1 deselected -
Read the cause above the summary:
> assert counted["spans"] == 1 and problems == []E assert (3 == 1)The test runs one ticket and checks that no text left the application. It also expected one span per run, the shape of Module 2. Now a run has three spans: the run and two model calls.
-
In
tests/test_telemetry.py, changecounted["spans"] == 1tocounted["spans"] > 1. Keepproblems == []: that half of the test is the important one. -
Run the tests again. The last line says
81 passed, 1 deselected.
A test that checks the exact shape of telemetry breaks each time you add a span. The next lesson replaces this file with the snapshot’s version, which checks the spans that matter.
Check that no text left
Section titled “Check that no text left”The model spans are the place where text could leak: the request carries the prompt and the ticket. Run the check from The Collector on your own telemetry:
python -m obs check-redaction work/telemetryChecked 4 span(s), 0 event(s), 4 metric line(s) in work/telemetry.No text attribute, no e-mail address and no card number found.Your counts depend on how many runs you made. The check passes because observed() copies counts, names and IDs into the span, and no text.
Your turn: the other design
Section titled “Your turn: the other design”The agents course’s structured design, agent_structured, asks the model for one JSON step per call, without tool calling. It runs on chat-strong. Before you run it, write down your predictions for its two model spans:
- Which finish reason do they have,
stoportool_calls? - Is the prompt version the same as the one of
agent,d6c784d1? - Are the input tokens close to those of chat-small?
Then run it, and check your answers with obs trace T-80008 and python show_spans.py chat:
python -m resolver run T-80008 --variant agent_structured --model chat-strongThe course team’s result
0.0 s |========================| 5.48 s invoke_agent support-resolver reply_only rule W1 0.0 s |========== | 2.28 s chat chat-strong 2166 in / 48 out - stop 2.3 s | ===============| 3.20 s chat chat-strong 2436 in / 125 out - stopBoth calls finish with stop: the model writes a JSON step as plain text, so for the provider every answer is a normal end. The workflow reads the step from the text. The prompt is another file, so the prompt version is another hash, 39329efa. The input tokens are almost the same, 2,166 and 2,436: the two prompts have about the same size. But the calls take 2.4 and 2.7 times longer, and Module 5 shows that each chat-strong token costs 20 times more.
So a finish reason means different things in different designs. tool_calls tells you that a model asked for a tool only in the design that uses tool calling. That is why the span also records the design and the prompt’s name.
Common mistake
The prompt text is on the spans
Symptom: Tomás wants to debug a strange answer. He runs a ticket with OBS_CAPTURE_CONTENT=on and forgets to turn it off. The redaction check then fails:
OBS_CAPTURE_CONTENT=on python -m resolver run T-80008python -m obs check-redaction work/telemetryChecked 7 span(s), 0 event(s), 6 metric line(s) in work/telemetry.2 problem(s): text that must not leave the application: span chat chat-small: attribute gen_ai.input.messages (text) span chat chat-small: attribute gen_ai.input.messages (text)In PowerShell, set the variable first: $env:OBS_CAPTURE_CONTENT = "on", then run the ticket.
Cause: with content capture on, observed() adds gen_ai.input.messages: the whole conversation, with the customer’s words, on every model span. The conventions allow it as an opt-in, for debugging. The course measured what it costs: a run’s telemetry grows from 9.3 KB to 26.7 KB, about 2.9 times. In the replayed week with content capture on, 718 model spans carried prompt text.
Diagnose: python show_spans.py chat shows a long gen_ai.input.messages value.
Recover: turn the variable off (unset OBS_CAPTURE_CONTENT; in PowerShell, Remove-Item Env:OBS_CAPTURE_CONTENT). Delete work/telemetry, and run again. In the course’s Collector run, the redaction processor removed the text from all 718 spans before any export: 0 problems after the Collector. The Collector is the second line of defence, not the first. Keep content capture off in the application.
When to record what, and what it costs
Section titled “When to record what, and what it costs”Record the model span on every call, in every environment. It costs little: in the course’s measurement, the whole instrumentation added 1.9 milliseconds per run, against seconds of model time. And it answers the most frequent questions: which step was slow, which model answered, and where the tokens went.
Keep the attributes to counts, names, IDs and codes. Kwame’s rule for a new attribute: “Could two runs have different values for it, and would I ever group by it?” The design, the step and the prompt version pass. A customer’s email address does not, and it is personal data.
Do not record the text by default. A text attribute makes every trace larger. It puts personal data in a store with other readers, for the whole retention time. When you need the text to debug one answer, the workflow’s own records are the place, as the agents course did with its recordings.
Knowledge check
Section titled “Knowledge check”Check your understanding
A run of T-80008 made two model calls: 2,169 tokens in and 21 out, then 2,411 in and 96 out. Why does the second call send more input tokens than the first, and what does that mean for a run with ten steps?
Show the answer
A model keeps no memory between calls, so each call sends the whole conversation again: the prompt, the ticket, the earlier answers and the tool results. The second call adds the first answer and the order that get_order returned: 242 more tokens. In a run with ten steps, each call is larger than the one before it. So input tokens grow faster than the number of steps, and they are most of the run’s tokens: 97% for T-80008.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'We should put the prompt and the answer on every model span. Then we can explain anything.' Give two reasons not to, and say what the span records instead.
Show the answer
First, the text contains the customer’s words and personal data. It would then go to every telemetry store and to everyone who can read traces, for the whole retention time. Second, it makes the telemetry about three times larger: 26.7 KB per run instead of 9.3 KB in the course’s measurement. Instead, the span records which prompt was used (its name and its version, a hash) and which model answered. It also records the tokens, the finish reason, the latency and the names of the steps the model asked for. That is enough to group, compare and find runs. The text stays in the workflow’s own records.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run T-90103 with --variant agent --model chat-small, and use your own telemetry. How many model calls did it make, what was the finish reason of the last one, and which prompt version did it use?
Show the answer
Run python -m resolver run T-90103 –variant agent –model chat-small, then python -m obs trace T-90103 and python show_spans.py chat. In the course team’s run, the trace has two chat chat-small spans: 2,185 tokens in and 21 out, then 2,433 in and 199 out. That is 4,618 and 220 in all, as the workflow printed. Both end with tool_calls: the last one calls finish with the proposal, so the run then waits for an approval. Both show gen_ai.prompt.version = d6c784d1, the same as for T-80008, because the agent prompt did not change. Your trace ID and time differ, but the tokens and the version must be the same.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- One
chat {model}span per model call, a child of the run span. It records the model asked for and the model that answered, tokens in and out, the finish reason, the latency and the prompt’s name and version. - In a replay, a model span lasts as long as the recorded call did, so the run’s time is the sum of its calls.
- Input tokens grow with every step, because each call sends the whole conversation again: 97% of the tokens of T-80008.
- No text: names, counts, IDs and a hash of the prompt.
obs check-redactionproves it on your own telemetry.
Your project now: observe.py and runner.py from the snapshot end-of-m03, 81 tests that pass, and your own show_spans.py. Every model call is in the trace. But the model only proposes: the tools read the orders, and they write the refunds, and none of them is in the trace yet. Next, you record each tool call, its errors and its retries.