Skip to content
Français

Model calls and tokens

  • Lesson 7
  • 35 min
  • Module 3: Instrument the workflow

Where we are

In Module 2, we built the pieces: spans, the pinned GenAI names and a Collector that redacts and routes. But our trace of a ticket is still one span that lasts 0.00 seconds, because the model's time is not in it. In this lesson, we give every model call its own span, with the model, the tokens, the finish reason, the latency and the prompt version, and never the text. We start with a phone bill.

What you will be able to do

Give every model call of the workflow its own span. Record which model answered, the tokens in and out, the finish reason, the time it took and the prompt version. Then a trace explains where a run’s time and tokens went. Record none of the text.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project as Module 2 left it, in observable-workflow: the same as the snapshot end-of-m02, with 81 tests that pass.
  • A terminal in observable-workflow, with the virtual environment active. The labs repository, nextia-learning-labs, is in the same parent folder.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

A phone bill lists every call: the number, the time, how long it lasted and what it cost. It never contains what you said. Still, it answers useful questions. Why was this month expensive? Which call was the long one? Did a new plan change anything?

The bill can answer them because it records the same few facts for every call, in the same form. One line per call, and no conversation. That is the idea of this lesson.

At the end of Module 2, run a ticket and read its trace:

Terminal window
python -m resolver run T-80008
python -m obs trace T-80008

The workflow prints that the run made two model calls and used 4,580 tokens:

Run T-80008-agent-chat-small-1 | single agent with tool calling: the model chooses every step | chat-small
Outcome: reply_only (rule W1) | 2 model call(s), 1 tool call(s) | 4580 tokens in, 117 out | US$0.00052
Reply draft: Your garden shed order LK-581106 shipped on Monday, October 5. Its expected delivery date is Tuesday, October 13. It is currently in transit, and there is no delivery date showing yet.
Reply sent. Ticket: resolved.

But the trace says something else:

Run T-80008-agent-chat-small-1 | ticket T-80008 | chat-small (agent) | release 2.0.0 | language en
0 model call(s), 0 read(s), 0 write call(s), 0 tokens in, 0 out, US$0.00000
Trace 1 88ec5e0f1603e609be2414b7caee63ee … 0.0 s
0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1
Outcome: resolved

The trace has one span, and it lasts 0.00 seconds. It knows the outcome, but not the two model calls, their tokens or their time. If Omar asks “why did the bill double this week?”, this trace cannot help. The numbers exist in the program, but nothing records them per call.

Why 0.00 seconds? The workflow replays recorded model answers, and a replay answers in microseconds. From Module 2, spans use the run clock: each run’s clock moves forward by the recorded latency of each model call. But only a model span moves it, and there is none yet.

Give each model call its own span, as a child of the run span. The GenAI conventions name it chat {model}, for example chat chat-small, with the operation name chat. Its kind is CLIENT, because the workflow calls another service. Each chat span records the same facts, like one line of the phone bill:

On the left, one model call: the request, with the model chat-small, at most 2000 tokens, the prompt and the ticket's text, then the answer: text or a tool call, tokens and a finish reason. An arrow goes to 'The chat span records': the model asked for and the model that answered; tokens in and out and the finish reason; the latency, which is the span's own duration; the prompt's name and version and the step number; the steps the model asked for, as names only, such as get_order(order_id). A dashed arrow, 'off by default', goes to 'It never records': the prompt's text and the ticket's text, the answer's text and the values of tool arguments.
What a chat span keeps from one model call, and what it leaves out.
Text description of the diagram

A model call has a request (the model, a token limit, the prompt and the ticket) and an answer (text or a tool call, tokens, a finish reason). The span keeps the model asked for and the model that answered, the tokens in and out, and the finish reason. It also keeps the latency as its own duration, the prompt’s name and version, the step number and the names of the steps the model asked for. It leaves out every text: the prompt, the ticket, the answer and the values of tool arguments.

Five attributes need a short explanation:

  • Input and output tokens (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens). The provider counts them and returns them with the answer. Prices are per token, so Module 5 computes cost from them.
  • The finish reason (gen_ai.response.finish_reasons): why the model stopped. stop means a normal end, tool_calls means it asks for a tool, length means it hit the token limit, and content_filter means the provider refused. The conventions make it a list, because one request can ask for several answers.
  • The model that answered (gen_ai.response.model). The workflow asks for chat-small, a deployment name. The provider answers with the model behind it, gpt-6-luna-2026-09-22. When the deployment moves to a new model, only this attribute shows it.
  • The prompt version (gen_ai.prompt.version): the first 8 characters of a hash of the prompt file and the policy, as in The GenAI conventions for agent runs. So a trace can say which prompt a call used, without the text.
  • The latency is the span’s duration. In a replay, the span lasts as long as the recorded call did: larkfield.recorded_latency_s keeps the recorded number.

The project also records three metrics for each call: the call’s duration (gen_ai.client.inference.duration, a histogram) and two counters for input and output tokens. Their labels are the operation, the provider and the two model names, never a ticket. Module 5 explains why labels must stay few.

Here is what the two chat spans of T-80008 say, from the course team’s run:

Call Finish reason Input tokens Output tokens Recorded latency
1 tool_calls: the model asks for get_order(order_id) 2169 21 0.96 s
2 tool_calls: the model calls finish with its proposal 2411 96 1.18 s
Run 4,580 117 2.14 s

The sums are the numbers the workflow printed: 4,580 tokens in and 117 out. The run span now lasts 0.96 + 1.18 = 2.14 seconds, because the run clock moved forward by each call’s recorded latency.

Look at the input tokens. The second call sends 242 more tokens than the first. A model has no memory between calls, so the workflow sends the whole conversation again. That is the prompt, the ticket, the first answer and the order that get_order returned. So input tokens grow with every step, and they are most of the total:

Replayed: the agents course's recorded calls of chat-small on T-80008, from the chat spans of the course team's run.
Text description of the chart

Four bars. Call 1: 2169 input tokens and 21 output tokens. Call 2: 2411 input tokens and 96 output tokens. The input bars are more than twenty times higher than the output bars. The second input bar is higher than the first, because the second call sends the whole conversation again.

Of the 4,697 tokens of this run, 97% are input. A run with more steps sends the conversation more often, so it costs more than its number of answers suggests. Only a span per call shows this.

The change is in two files. resolver/observe.py gets a wrapper, observed(), around the provider’s complete() function. resolver/runner.py wraps the provider once, for every design of the workflow. The copy of observe.py also contains the tool and write spans of the next lessons, but nothing calls them yet.

  1. Copy the two files from the snapshot end-of-m03:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m03
    cp $S/resolver/observe.py $S/resolver/runner.py resolver/
  2. Open resolver/runner.py. The run now passes observe.observed(complete) instead of complete. One line of the workflow changed: every design gets the same model spans.

  3. Open resolver/observe.py and find def observed(complete). Read the two groups of attributes. The first group is set before the call: the model asked for, the prompt and the step. The second group is set after the answer: the model that answered, the tokens and the finish reason.

  4. Run the ticket again and read its trace:

    Terminal window
    python -m resolver run T-80008
    python -m obs trace T-80008
    Run T-80008-agent-chat-small-2 | ticket T-80008 | chat-small (agent) | release 2.0.0 | language en
    2 model call(s), 0 read(s), 0 write call(s), 4580 tokens in, 117 out, US$0.00000
    Trace 1 488d2ec9ffa2868b9e43ebd9ea937fdc … 2.1 s
    0.0 s |========================| 2.14 s invoke_agent support-resolver reply_only rule W1
    0.0 s |=========== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls
    1.0 s | ==============| 1.18 s chat chat-small 2411 in / 96 out - tool_calls

    Your trace ID and time differ. The run number ends in -2 if you ran this ticket once before. The cost column shows -: the spans have no cost yet, and Module 5 adds it.

The two traces of the same ticket, before and after:

Two waterfalls on the same time scale, from 0 to 2.14 seconds. Before, Module 2's trace of T-80008: one span, invoke_agent support-resolver, with no length: 0.00 s, no model span yet. After: invoke_agent support-resolver lasts 2.14 s, with two child spans, chat chat-small from 0 to 0.96 s (2169 in, 21 out, tool_calls) and chat chat-small from 0.96 to 2.14 s (2411 in, 96 out, tool_calls).
The same ticket before and after the model spans. Replayed calls: the model spans last as long as the recorded calls did.
Text description of the diagram

Before, the trace has one span with no length. After, the run span lasts 2.14 seconds and has two children, the two calls to chat-small. The first lasts 0.96 seconds and ends with tool_calls. The second starts when the first ends and lasts 1.18 seconds. The run’s time is now the sum of its model calls. The tool call between them is not in the trace yet.

The read, get_order, happened between the two calls, but it has no span yet. The next lesson adds it.

obs trace shows a summary. To see every attribute, read the telemetry file itself. The file work/telemetry/traces.jsonl holds OTLP JSON: one export request per line, with the spans inside.

  1. Create a file show_spans.py in the project folder, with this content:

    """Print the spans in work/telemetry/traces.jsonl whose name contains a word, with every attribute."""
    import json
    import sys
    from pathlib import Path
    word = sys.argv[1] if len(sys.argv) > 1 else ""
    for line in Path("work/telemetry/traces.jsonl").read_text().splitlines():
    for resource in json.loads(line)["resourceSpans"]:
    for scope in resource["scopeSpans"]:
    for span in scope["spans"]:
    if word not in span["name"]:
    continue
    parent = span.get("parentSpanId", "")[:8] or "none"
    print(f'{span["name"]} trace {span["traceId"][:8]} span {span["spanId"][:8]} parent {parent}')
    for a in span.get("attributes", []):
    value = next(iter(a["value"].values()))
    if isinstance(value, dict): # a list, such as the finish reasons
    value = [next(iter(v.values())) for v in value.get("values", [])]
    print(f' {a["key"]} = {value}')
    for link in span.get("links", []):
    print(f' link to trace {link["traceId"][:8]}, span {link["spanId"][:8]}')

    This helper is yours: it is not in the snapshot, and the next lessons use it.

  2. Print the model spans: python show_spans.py chat. The first one says:

    chat chat-small trace 488d2ec9 span 0ec2f8ab parent 34b1f358
    gen_ai.operation.name = chat
    gen_ai.provider.name = azure.ai.openai
    gen_ai.request.model = chat-small
    gen_ai.request.max_tokens = 2000
    larkfield.step = 1
    gen_ai.prompt.name = agent
    gen_ai.prompt.version = d6c784d1
    gen_ai.output.type = text
    gen_ai.conversation.id = T-80008-agent-chat-small-2
    gen_ai.response.model = gpt-6-luna-2026-09-22
    gen_ai.usage.input_tokens = 2169
    gen_ai.usage.output_tokens = 21
    gen_ai.response.finish_reasons = ['tool_calls']
    larkfield.replay = exact
    larkfield.recorded_latency_s = 0.956
    larkfield.model.requested = ['get_order(order_id)']
  3. Find the three kinds of names. gen_ai.* names come from the pinned conventions. larkfield.* names are ours: our prefix says so. gen_ai.conversation.id is the run ID, so a tool that knows the conventions can group the calls of one run.

  4. Find larkfield.replay = exact. It says that the recorded answer was replayed for a byte-for-byte equal request. A live call would say live.

larkfield.model.requested keeps what the model asked to do next, as a tool name and its argument names: get_order(order_id). It never keeps the order number that the model wrote. Module 4 uses it to find loops: a model that asks for the same step again and again.

  1. Run the tests: python -m pytest -q.

    FAILED tests/test_telemetry.py::test_no_text_leaves_the_application_by_default
    1 failed, 80 passed, 1 deselected
  2. Read the cause above the summary:

    > assert counted["spans"] == 1 and problems == []
    E assert (3 == 1)

    The test runs one ticket and checks that no text left the application. It also expected one span per run, the shape of Module 2. Now a run has three spans: the run and two model calls.

  3. In tests/test_telemetry.py, change counted["spans"] == 1 to counted["spans"] > 1. Keep problems == []: that half of the test is the important one.

  4. Run the tests again. The last line says 81 passed, 1 deselected.

A test that checks the exact shape of telemetry breaks each time you add a span. The next lesson replaces this file with the snapshot’s version, which checks the spans that matter.

The model spans are the place where text could leak: the request carries the prompt and the ticket. Run the check from The Collector on your own telemetry:

Terminal window
python -m obs check-redaction work/telemetry
Checked 4 span(s), 0 event(s), 4 metric line(s) in work/telemetry.
No text attribute, no e-mail address and no card number found.

Your counts depend on how many runs you made. The check passes because observed() copies counts, names and IDs into the span, and no text.

The agents course’s structured design, agent_structured, asks the model for one JSON step per call, without tool calling. It runs on chat-strong. Before you run it, write down your predictions for its two model spans:

  • Which finish reason do they have, stop or tool_calls?
  • Is the prompt version the same as the one of agent, d6c784d1?
  • Are the input tokens close to those of chat-small?

Then run it, and check your answers with obs trace T-80008 and python show_spans.py chat:

Terminal window
python -m resolver run T-80008 --variant agent_structured --model chat-strong
The course team’s result
0.0 s |========================| 5.48 s invoke_agent support-resolver reply_only rule W1
0.0 s |========== | 2.28 s chat chat-strong 2166 in / 48 out - stop
2.3 s | ===============| 3.20 s chat chat-strong 2436 in / 125 out - stop

Both calls finish with stop: the model writes a JSON step as plain text, so for the provider every answer is a normal end. The workflow reads the step from the text. The prompt is another file, so the prompt version is another hash, 39329efa. The input tokens are almost the same, 2,166 and 2,436: the two prompts have about the same size. But the calls take 2.4 and 2.7 times longer, and Module 5 shows that each chat-strong token costs 20 times more.

So a finish reason means different things in different designs. tool_calls tells you that a model asked for a tool only in the design that uses tool calling. That is why the span also records the design and the prompt’s name.

Common mistake

The prompt text is on the spans

Symptom: Tomás wants to debug a strange answer. He runs a ticket with OBS_CAPTURE_CONTENT=on and forgets to turn it off. The redaction check then fails:

Terminal window
OBS_CAPTURE_CONTENT=on python -m resolver run T-80008
python -m obs check-redaction work/telemetry
Checked 7 span(s), 0 event(s), 6 metric line(s) in work/telemetry.
2 problem(s): text that must not leave the application:
span chat chat-small: attribute gen_ai.input.messages (text)
span chat chat-small: attribute gen_ai.input.messages (text)

In PowerShell, set the variable first: $env:OBS_CAPTURE_CONTENT = "on", then run the ticket.

Cause: with content capture on, observed() adds gen_ai.input.messages: the whole conversation, with the customer’s words, on every model span. The conventions allow it as an opt-in, for debugging. The course measured what it costs: a run’s telemetry grows from 9.3 KB to 26.7 KB, about 2.9 times. In the replayed week with content capture on, 718 model spans carried prompt text.

Diagnose: python show_spans.py chat shows a long gen_ai.input.messages value.

Recover: turn the variable off (unset OBS_CAPTURE_CONTENT; in PowerShell, Remove-Item Env:OBS_CAPTURE_CONTENT). Delete work/telemetry, and run again. In the course’s Collector run, the redaction processor removed the text from all 718 spans before any export: 0 problems after the Collector. The Collector is the second line of defence, not the first. Keep content capture off in the application.

Record the model span on every call, in every environment. It costs little: in the course’s measurement, the whole instrumentation added 1.9 milliseconds per run, against seconds of model time. And it answers the most frequent questions: which step was slow, which model answered, and where the tokens went.

Keep the attributes to counts, names, IDs and codes. Kwame’s rule for a new attribute: “Could two runs have different values for it, and would I ever group by it?” The design, the step and the prompt version pass. A customer’s email address does not, and it is personal data.

Do not record the text by default. A text attribute makes every trace larger. It puts personal data in a store with other readers, for the whole retention time. When you need the text to debug one answer, the workflow’s own records are the place, as the agents course did with its recordings.

Check your understanding

A run of T-80008 made two model calls: 2,169 tokens in and 21 out, then 2,411 in and 96 out. Why does the second call send more input tokens than the first, and what does that mean for a run with ten steps?

Show the answer

A model keeps no memory between calls, so each call sends the whole conversation again: the prompt, the ticket, the earlier answers and the tool results. The second call adds the first answer and the order that get_order returned: 242 more tokens. In a run with ten steps, each call is larger than the one before it. So input tokens grow faster than the number of steps, and they are most of the run’s tokens: 97% for T-80008.

Check your understanding

Tomás says: 'We should put the prompt and the answer on every model span. Then we can explain anything.' Give two reasons not to, and say what the span records instead.

Show the answer

First, the text contains the customer’s words and personal data. It would then go to every telemetry store and to everyone who can read traces, for the whole retention time. Second, it makes the telemetry about three times larger: 26.7 KB per run instead of 9.3 KB in the course’s measurement. Instead, the span records which prompt was used (its name and its version, a hash) and which model answered. It also records the tokens, the finish reason, the latency and the names of the steps the model asked for. That is enough to group, compare and find runs. The text stays in the workflow’s own records.

Check your understanding

Practical check: run T-90103 with --variant agent --model chat-small, and use your own telemetry. How many model calls did it make, what was the finish reason of the last one, and which prompt version did it use?

Show the answer

Run python -m resolver run T-90103 –variant agent –model chat-small, then python -m obs trace T-90103 and python show_spans.py chat. In the course team’s run, the trace has two chat chat-small spans: 2,185 tokens in and 21 out, then 2,433 in and 199 out. That is 4,618 and 220 in all, as the workflow printed. Both end with tool_calls: the last one calls finish with the proposal, so the run then waits for an approval. Both show gen_ai.prompt.version = d6c784d1, the same as for T-80008, because the agent prompt did not change. Your trace ID and time differ, but the tokens and the version must be the same.

Recap

  • One chat {model} span per model call, a child of the run span. It records the model asked for and the model that answered, tokens in and out, the finish reason, the latency and the prompt’s name and version.
  • In a replay, a model span lasts as long as the recorded call did, so the run’s time is the sum of its calls.
  • Input tokens grow with every step, because each call sends the whole conversation again: 97% of the tokens of T-80008.
  • No text: names, counts, IDs and a hash of the prompt. obs check-redaction proves it on your own telemetry.

Your project now: observe.py and runner.py from the snapshot end-of-m03, 81 tests that pass, and your own show_spans.py. Every model call is in the trace. But the model only proposes: the tools read the orders, and they write the refunds, and none of them is in the trace yet. Next, you record each tool call, its errors and its retries.

Next: Tool calls, errors and retries