Cost per run and per step
- Lesson 16
- 35 min
- Module 5: Cost
Where we are
In the last module, we found failures in the traces and moved from one run to the metrics around it. Every model span already carries the tokens of its call, but not what the call cost. In this lesson, we turn tokens into money with a dated price table, read the cost of each run and each step, and decide where the cost should be computed. We start at the checkout of a grocery shop.
Lesson 16 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsCost per run and per step
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Give every model call, step and run a cost in dollars, from its tokens and a dated price table. Then you can answer “what did this run cost, and which step paid?”, now and next year.
Before you startWhat to know and to have ready
What you need to know
- Model spans and their token counts, from Model calls and tokens.
- How to read a run’s waterfall with
obs trace, from Read a run. - The Collector and its tail sampling, from The Collector.
What you need on your computer
- Your project as the last module left it: the snapshot
end-of-m04, wherepython -m pytest -qends with93 passed, 1 skipped, 1 deselected. - The virtual environment active, and a terminal in the project folder,
observable-workflow.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The lesson calls no model: every decision is a replayed recording of the agents course. So every cost here is the recorded tokens times the course’s prices.
A receipt at the checkout
Section titled “A receipt at the checkout”At a grocery shop, the receipt has one line per item: 2 kg of apples at US$3.00 a kilogram, US$6.00. The shop’s price list sets the price of each item, and the receipt prints the date. Next week, apples cost US$3.50. Your old receipt still says US$6.00, because it kept the price of the day.
Now imagine a receipt that prints only “2 kg of apples”. To know what you paid, you would look at today’s price list, and you would get US$7.00: a wrong number for last week.
The problem: the trace knows the tokens, not the money
Section titled “The problem: the trace knows the tokens, not the money”Our workflow already knows what a run costs. Run the double refund’s ticket on your project as the last module left it:
python -m resolver run T-90104 --variant agent_structured --model chat-strongThe first two lines of its output are:
Run T-90104-agent_structured-chat-strong-1 | single agent with one JSON step per call (for models without tool calling) | chat-strongOutcome: resolve (rule W2) | 2 model call(s), 1 tool call(s) | 4586 tokens in, 265 out | US$0.01182The agents course’s code adds up the cost of each call for its spending limit. It prints US$0.01182. Now read the same run in its telemetry with python -m obs trace T-90104:
Run T-90104-agent_structured-chat-strong-1 | ticket T-90104 | chat-strong (agent_structured) | release 2.0.0 | language en2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.00000
Trace 1 c58e3444eabc8d5e2d40fa7232c2a4fc … 6.6 s 0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2 0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out - stop 2.4 s | = | 0.00 s execute_tool get_order ok LK-793996 2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out - stop 6.6 s | =| 0.00 s approval.request needs agentThe tokens are there. The money is not: a dash on each model span, and US$0.00000 for the run. The receipt prints “2 kg of apples” only.
Kwame owns the platform and its bill. On Monday he asks Amira two questions: “What did Wednesday cost?” and “Which step of a run costs the most?” With tokens only, Amira must find a price list and multiply. Which price list? The one of last Wednesday, or today’s? If she takes today’s, and a price changed, last week’s cost changes after the fact.
The idea: tokens times a dated price table, written on the span
Section titled “The idea: tokens times a dated price table, written on the span”A price table lists, for each model, the price of a million input tokens and of a million output tokens. Providers charge the two kinds of token at different prices. The cost of one model call is:
cost = input tokens × input price ÷ 1,000,000 + output tokens × output price ÷ 1,000,000
The table needs a version, and the simplest version is its date. When the prices change, you make a new table with a new date. You never edit the old one.
Then you write two attributes on every model span, as the receipt prints the price and the date:
larkfield.cost.usd: what this call cost, in US dollars.larkfield.price_table: the version of the table that made the number.
The run’s root span gets the sum of its calls, with the same version. A counter, larkfield.run.cost, adds each run’s cost when the ticket closes. That counter is what dashboards and alerts read.
The course’s price table is the one the agents course used for its recordings, in resolver/usage.py:
| Model | Input, US$ per 1M tokens | Output, US$ per 1M tokens | Note |
|---|---|---|---|
chat-small |
0.10 | 0.50 | gpt-6-luna on Azure |
chat-strong |
2.00 | 10.00 | gpt-6.1-sol on Azure |
gemma3:4b |
0 | 0 | local, through Ollama: an assumption |
Its version is 2026-10-08. Treat every price here as an assumption with its date: prices change, so check your provider’s current price page before you use them. The local model’s price of 0 is also an assumption: it leaves out the computer that runs it.
A tiny worked example
Section titled “A tiny worked example”The double refund’s run made two calls to chat-strong. Here is its receipt, one line per call:
| Call | Input tokens × US$2.00 ÷ 1M | Output tokens × US$10.00 ÷ 1M | Cost |
|---|---|---|---|
| step 1 | 2,160 → US$0.004320 | 47 → US$0.000470 | US$0.004790 |
| step 2 | 2,426 → US$0.004852 | 218 → US$0.002180 | US$0.007032 |
| the run | 4,586 | 265 | US$0.011822 |
So the run cost US$0.01182, the number the agents course’s code printed. Two things show already. An output token costs five times as much as an input token, for both models. And the second call cost more than the first, because it wrote the proposal and the reply: 218 output tokens instead of 47.
Python gives the same numbers from the project’s own function:
python -c "from resolver.usage import cost_usd; print(cost_usd('chat-strong', 2160, 47), cost_usd('chat-strong', 2426, 218))"0.00479 0.007032One run as a cost waterfall
Section titled “One run as a cost waterfall”A cost waterfall draws each step as a bar that starts where the last one ended, so you see how the total grows. Here is the most expensive run of the replayed week, R-1016-040: ticket T-80009 on chat-strong, after Omar’s switch.
Text description of the diagram
Read it from top to bottom. Each model call adds a bar after the last one, and each tool call adds nothing. The input tokens grow at every step, from 2,163 to 2,681, because each call sends the whole conversation so far. Step 4 is the most expensive: the most input, and the reply as output. The first step bought little: the model asked for order LK-55190, with a digit missing, and the order service refused it. Steps 2 and 3 found the real order, LK-551906. The run cost US$0.02219.
Read the waterfall as a story of decisions. The model’s first call asked for an order number with a digit missing. The next two calls paid to recover from that. A step’s cost is not only its own tokens: it is also every earlier step that the call sends again.
Add the cost to your project
Section titled “Add the cost to your project”The project’s change is small. In resolver/observe.py, each model span gets its cost and the table’s version as soon as the call returns. The run span gets the run’s sum. In resolver/live.py, the ticket’s close adds the run’s cost to the counter larkfield.run.cost, with the same labels as the run counter.
-
Copy this module’s files from the snapshot
end-of-m05:Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m05cp $S/resolver/observe.py $S/resolver/live.py resolver/cp $S/obs/__main__.py $S/obs/gateway.py obs/cp $S/tests/test_stage_m05.py $S/tests/test_obs_cli.py $S/tests/test_telemetry.py tests/cp $S/README.md .Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m05"Copy-Item -Force $S\resolver\observe.py, $S\resolver\live.py resolver\Copy-Item -Force $S\obs\__main__.py, $S\obs\gateway.py obs\Copy-Item -Force $S\tests\test_stage_m05.py, $S\tests\test_obs_cli.py, $S\tests\test_telemetry.py tests\Copy-Item -Force $S\README.md .Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m05cp $S/resolver/observe.py $S/resolver/live.py resolver/cp $S/obs/__main__.py $S/obs/gateway.py obs/cp $S/tests/test_stage_m05.py $S/tests/test_obs_cli.py $S/tests/test_telemetry.py tests/cp $S/README.md . -
Run the tests:
python -m pytest -q. The last line says96 passed, 1 skipped, 1 deselected. Your project is now the same as the snapshotend-of-m05. The new test runs one ticket. It checks that every model span has the table’s version, and that the costs add up to the run’s cost. -
Open
resolver/observe.pyand search forPRICE_TABLE. Find the two places: the model span and the run span. -
Delete the folder
work, then run the ticket again:python -m resolver run T-90104 --variant agent_structured --model chat-strong. In PowerShell, delete the folder withRemove-Item -Recurse work. -
Read its trace:
python -m obs trace T-90104. Each model span now shows its cost:2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.01182…0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out US$0.00479 stop2.4 s | = | 0.00 s execute_tool get_order ok LK-7939962.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out US$0.00703 stop -
Read the two attributes on your own spans, with the project’s reader:
Terminal window python -c "from obs.model import Telemetry; r = next(iter(Telemetry('work/telemetry').runs.values())); print([(s.name, s.get('larkfield.cost.usd'), s.get('larkfield.price_table')) for s in r.model_spans])"[('chat chat-strong', 0.00479, '2026-10-08'), ('chat chat-strong', 0.007032, '2026-10-08')] -
Split the saved week’s cost by model and step:
python -m obs cost --by step --data data/telemetry/week. The rows forchat-strongare:model step calls tokens in out cost US$chat-strong 1 40 86447 2108 0.1940chat-strong 2 39 95057 6391 0.2540chat-strong 3 8 20244 1250 0.0530chat-strong 4 2 5362 362 0.0143
The last command reads every model span of the 316 runs and adds the cost by model and step. Here is the strong model’s part as a chart:
Text description of the chart
Four bars. Step 1: 40 calls, US$0.1940. Step 2: 39 calls, US$0.2540, the highest. Step 3: 8 calls, US$0.0530. Step 4: 2 calls, US$0.0143. Most runs end after two steps, so steps 1 and 2 hold most of the cost. For chat-small, steps 1 and 2 cost US$0.0600 and US$0.0667, for 260 and 225 calls.
So Kwame’s second question has an answer. Step 2 is the most expensive step of the week, because most runs write their proposal there. Steps 1 and 2 of chat-strong together cost US$0.448, more than the 260 runs of chat-small (US$0.1513).
Where to compute the cost
Section titled “Where to compute the cost”You can turn tokens into money in three places. Each one sees different data.
Text description of the diagram
At the moment of the call, the application knows the model, the tokens and the price table. So it writes the cost on the span and adds it to a counter. The Collector could multiply too, but its tail sampling keeps only some traces. A sum over sampled traces misses money. A query at read time can multiply tokens by prices too, but with today’s prices it changes last week’s numbers.
In the application (the course’s choice). The code knows which model answered, the exact tokens and the table in force. It can also handle a special case. The workflow can call the gateway of the agents governance course and ask for a capability such as strong-reasoning. The app has no price for that name. So it prices the model that the gateway names in its answer header, and writes larkfield.priced_as. The cost: every application needs the table, and a new price means a new release.
In the Collector. A transform processor could multiply two attributes by a price that lives in the Collector’s configuration. One table serves every app. But the Collector sees only what the spans say: a capability name, no answer header, no price. And look at the Collector of Module 2: its tail sampling kept 275 of 434 traces. Cost summed from the traces that reached Tempo would miss the dropped ones. If you do it in the Collector, do it before sampling, and read totals from metrics.
At query time. Grafana can multiply the token counters by prices in PromQL. That is quick to change, and it needs no release. But the query uses the prices it knows today, for every week it reads.
Why the table needs a version
Section titled “Why the table needs a version”Try a made-up change to see why. Suppose that next month chat-strong’s prices fall by half. This change is constructed for the example: it did not happen.
- With a query at read time, last Wednesday’s
chat-strongcost of US$0.1409 becomes about US$0.0705. Wednesday’s total falls from US$0.1638 to about US$0.0934. Nobody spent less: the report changed. - With the cost on the span, Wednesday stays at US$0.1638, and every span says
larkfield.price_table = 2026-10-08. To compare, you compute the week again with the new table on purpose, and you label it: “the same week at the new prices”.
The version also explains a difference. If two reports disagree, check their tables first. In Module 8, Langfuse receives the same spans and computes no cost at all: it does not know the names chat-small and gpt-6-luna. A tool needs its own price definitions. Our table is ours, with its date.
When to use it, and what it costs
Section titled “When to use it, and what it costs”Compute the cost in the application when the app calls models directly and knows the tokens: that is most agents. Use the gateway’s records as a second, independent count, as the next lesson does. Do not compute cost from sampled traces alone.
The cost is small. Two attributes on each model span, one on the run span, one counter. The trace data grows by a few bytes a span. The real work is the table: someone must own it, check the provider’s prices and make a new version when they change. At Larkfield, that is Kwame.
Common mistake
The cost shows a dash, or the total is too low
Symptom: obs trace shows - instead of a cost on a model span, and obs cost gives a total lower than the provider’s bill.
Cause: the price table has no price for the model’s name. The function returns None, not 0, on purpose: an unknown price is not a free call. The usual names that cause it are a new model, a deployment name instead of a model name, or a gateway capability:
python -c "from resolver.usage import cost_usd; print(cost_usd('strong-reasoning', 2160, 47))"NoneDiagnose: list the model names on the spans with a dash: python -m obs runs shows the model of each run. Compare them with the names in resolver/usage.py.
Recover: add the model to the table and give the table a new version, so that the new numbers say where they came from. For a gateway capability, price the model that the gateway names in its answer, as the project does. Never fill a missing price with 0.
Your turn: one step by hand
Section titled “Your turn: one step by hand”The waterfall’s step 4 has 2,681 input tokens and 181 output tokens, on chat-strong.
- Compute its cost by hand with the table. Then check it with
cost_usd('chat-strong', 2681, 181). - Predict the cost of the same tokens on
chat-small. This is a what-if: the small model would read and write other tokens. - Run
python -m obs trace R-1016-040 --data data/telemetry/weekand check your step 4 against the trace.
The answer is in the first knowledge check.
Knowledge check
Section titled “Knowledge check”Check your understanding
Your turn: what does step 4 of R-1016-040 cost (2,681 in, 181 out, chat-strong), and what would the same tokens cost on chat-small?
Show the answer
On chat-strong: 2,681 × US$2.00 ÷ 1,000,000 = US$0.005362, plus 181 × US$10.00 ÷ 1,000,000 = US$0.001810, so US$0.007172. The trace shows US$0.00717. On chat-small, the same tokens cost 2,681 × US$0.10 ÷ 1M = US$0.000268, plus 181 × US$0.50 ÷ 1M = US$0.000091, so about US$0.00036: 20 times less. That is a what-if with the same tokens, and an assumption with the table’s date. A real chat-small run reads and writes other tokens, and it may not resolve the ticket.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'Store only the tokens. We can multiply by the prices in Grafana whenever we need the money.' What goes wrong, and what does the course do instead?
Show the answer
A query multiplies with the prices it knows when it runs. When a price changes, every old week changes with it, although nobody spent less or more. The query also needs a price for every model name, including capability names, which it cannot know. The course computes the cost in the application, at the moment of the call. It writes the cost and the table’s version (larkfield.price_table = 2026-10-08) on every model span, and it adds the run’s cost to a counter. A new price means a new table with a new version. To compare prices, compute the week again on purpose and label it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In R-1016-040, each step costs more than the one before, although the model writes almost the same short answer in steps 1 to 3. Why? And why is it a mistake to sum cost from the traces that the Collector kept?
Show the answer
Each model call sends the whole conversation so far: the instructions, the ticket and every earlier tool result. So the input tokens grow at each step, from 2,163 to 2,681, and the cost grows with them. A wasted step is paid again in every later call. The Collector’s tail sampling keeps only some traces: in Module 2’s run, 275 of 434. A sum over the kept traces misses the cost of the dropped runs. Read totals from the counter larkfield.run.cost, which counts every run, and use traces to explain one run.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The cost of a call is input tokens × input price plus output tokens × output price, per million tokens. An output token of our models costs five times an input token.
- The price table has a version, its date. Every model span carries
larkfield.cost.usdandlarkfield.price_table; the run span carries the sum; the counterlarkfield.run.costadds each run. - Every price here is an assumption with its date. Prices change: check your provider’s page. An unknown model has no price, never a price of 0.
- A cost waterfall shows how a run’s cost grows step by step. Each call sends the conversation again, so a wasted step is paid again later.
- Compute cost in the application, before any sampling; read totals from the counter, not from sampled traces.
- Your project is now the snapshot
end-of-m05: 96 tests pass.
You can now say what each run and each step cost. But Kwame’s bill is per team, and Omar’s question is per resolved ticket, not per call. Next, you split the cost by model, slice and customer without making the metrics explode, and you check your totals against the gateway’s.