Skip to content
Français

Cost per run and per step

  • Lesson 16
  • 35 min
  • Module 5: Cost

Where we are

In the last module, we found failures in the traces and moved from one run to the metrics around it. Every model span already carries the tokens of its call, but not what the call cost. In this lesson, we turn tokens into money with a dated price table, read the cost of each run and each step, and decide where the cost should be computed. We start at the checkout of a grocery shop.

What you will be able to do

Give every model call, step and run a cost in dollars, from its tokens and a dated price table. Then you can answer “what did this run cost, and which step paid?”, now and next year.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your project as the last module left it: the snapshot end-of-m04, where python -m pytest -q ends with 93 passed, 1 skipped, 1 deselected.
  • The virtual environment active, and a terminal in the project folder, observable-workflow.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The lesson calls no model: every decision is a replayed recording of the agents course. So every cost here is the recorded tokens times the course’s prices.

At a grocery shop, the receipt has one line per item: 2 kg of apples at US$3.00 a kilogram, US$6.00. The shop’s price list sets the price of each item, and the receipt prints the date. Next week, apples cost US$3.50. Your old receipt still says US$6.00, because it kept the price of the day.

Now imagine a receipt that prints only “2 kg of apples”. To know what you paid, you would look at today’s price list, and you would get US$7.00: a wrong number for last week.

The problem: the trace knows the tokens, not the money

Section titled “The problem: the trace knows the tokens, not the money”

Our workflow already knows what a run costs. Run the double refund’s ticket on your project as the last module left it:

Terminal window
python -m resolver run T-90104 --variant agent_structured --model chat-strong

The first two lines of its output are:

Run T-90104-agent_structured-chat-strong-1 | single agent with one JSON step per call (for models without tool calling) | chat-strong
Outcome: resolve (rule W2) | 2 model call(s), 1 tool call(s) | 4586 tokens in, 265 out | US$0.01182

The agents course’s code adds up the cost of each call for its spending limit. It prints US$0.01182. Now read the same run in its telemetry with python -m obs trace T-90104:

Run T-90104-agent_structured-chat-strong-1 | ticket T-90104 | chat-strong (agent_structured) | release 2.0.0 | language en
2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.00000
Trace 1 c58e3444eabc8d5e2d40fa7232c2a4fc … 6.6 s
0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2
0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out - stop
2.4 s | = | 0.00 s execute_tool get_order ok LK-793996
2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out - stop
6.6 s | =| 0.00 s approval.request needs agent

The tokens are there. The money is not: a dash on each model span, and US$0.00000 for the run. The receipt prints “2 kg of apples” only.

Kwame owns the platform and its bill. On Monday he asks Amira two questions: “What did Wednesday cost?” and “Which step of a run costs the most?” With tokens only, Amira must find a price list and multiply. Which price list? The one of last Wednesday, or today’s? If she takes today’s, and a price changed, last week’s cost changes after the fact.

The idea: tokens times a dated price table, written on the span

Section titled “The idea: tokens times a dated price table, written on the span”

A price table lists, for each model, the price of a million input tokens and of a million output tokens. Providers charge the two kinds of token at different prices. The cost of one model call is:

cost = input tokens × input price ÷ 1,000,000 + output tokens × output price ÷ 1,000,000

The table needs a version, and the simplest version is its date. When the prices change, you make a new table with a new date. You never edit the old one.

Then you write two attributes on every model span, as the receipt prints the price and the date:

  • larkfield.cost.usd: what this call cost, in US dollars.
  • larkfield.price_table: the version of the table that made the number.

The run’s root span gets the sum of its calls, with the same version. A counter, larkfield.run.cost, adds each run’s cost when the ticket closes. That counter is what dashboards and alerts read.

The course’s price table is the one the agents course used for its recordings, in resolver/usage.py:

Model Input, US$ per 1M tokens Output, US$ per 1M tokens Note
chat-small 0.10 0.50 gpt-6-luna on Azure
chat-strong 2.00 10.00 gpt-6.1-sol on Azure
gemma3:4b 0 0 local, through Ollama: an assumption

Its version is 2026-10-08. Treat every price here as an assumption with its date: prices change, so check your provider’s current price page before you use them. The local model’s price of 0 is also an assumption: it leaves out the computer that runs it.

The double refund’s run made two calls to chat-strong. Here is its receipt, one line per call:

Call Input tokens × US$2.00 ÷ 1M Output tokens × US$10.00 ÷ 1M Cost
step 1 2,160 → US$0.004320 47 → US$0.000470 US$0.004790
step 2 2,426 → US$0.004852 218 → US$0.002180 US$0.007032
the run 4,586 265 US$0.011822

So the run cost US$0.01182, the number the agents course’s code printed. Two things show already. An output token costs five times as much as an input token, for both models. And the second call cost more than the first, because it wrote the proposal and the reply: 218 output tokens instead of 47.

Python gives the same numbers from the project’s own function:

Terminal window
python -c "from resolver.usage import cost_usd; print(cost_usd('chat-strong', 2160, 47), cost_usd('chat-strong', 2426, 218))"
0.00479 0.007032

A cost waterfall draws each step as a bar that starts where the last one ended, so you see how the total grows. Here is the most expensive run of the replayed week, R-1016-040: ticket T-80009 on chat-strong, after Omar’s switch.

A cost waterfall for run R-1016-040, ticket T-80009 on chat-strong. Step 1: 2,163 input and 48 output tokens, plus US$0.00481. Then get_order fails with invalid_arguments and costs US$0. Step 2: 2,253 in, 48 out, plus US$0.00499. get_customer_orders, ok, US$0. Step 3: 2,367 in, 49 out, plus US$0.00522. get_order, ok, US$0. Step 4: 2,681 in, 181 out, plus US$0.00717. send_reply, ok, US$0. The whole run: four model calls, four tool calls, US$0.02219.
Replayed tokens × the course's price table (version 2026-10-08): the week's most expensive run, step by step. Tool calls cost no tokens; every model call sends the whole conversation again.
Text description of the diagram

Read it from top to bottom. Each model call adds a bar after the last one, and each tool call adds nothing. The input tokens grow at every step, from 2,163 to 2,681, because each call sends the whole conversation so far. Step 4 is the most expensive: the most input, and the reply as output. The first step bought little: the model asked for order LK-55190, with a digit missing, and the order service refused it. Steps 2 and 3 found the real order, LK-551906. The run cost US$0.02219.

Read the waterfall as a story of decisions. The model’s first call asked for an order number with a digit missing. The next two calls paid to recover from that. A step’s cost is not only its own tokens: it is also every earlier step that the call sends again.

The project’s change is small. In resolver/observe.py, each model span gets its cost and the table’s version as soon as the call returns. The run span gets the run’s sum. In resolver/live.py, the ticket’s close adds the run’s cost to the counter larkfield.run.cost, with the same labels as the run counter.

  1. Copy this module’s files from the snapshot end-of-m05:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m05
    cp $S/resolver/observe.py $S/resolver/live.py resolver/
    cp $S/obs/__main__.py $S/obs/gateway.py obs/
    cp $S/tests/test_stage_m05.py $S/tests/test_obs_cli.py $S/tests/test_telemetry.py tests/
    cp $S/README.md .
  2. Run the tests: python -m pytest -q. The last line says 96 passed, 1 skipped, 1 deselected. Your project is now the same as the snapshot end-of-m05. The new test runs one ticket. It checks that every model span has the table’s version, and that the costs add up to the run’s cost.

  3. Open resolver/observe.py and search for PRICE_TABLE. Find the two places: the model span and the run span.

  4. Delete the folder work, then run the ticket again: python -m resolver run T-90104 --variant agent_structured --model chat-strong. In PowerShell, delete the folder with Remove-Item -Recurse work.

  5. Read its trace: python -m obs trace T-90104. Each model span now shows its cost:

    2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.01182
    …
    0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out US$0.00479 stop
    2.4 s | = | 0.00 s execute_tool get_order ok LK-793996
    2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out US$0.00703 stop
  6. Read the two attributes on your own spans, with the project’s reader:

    Terminal window
    python -c "from obs.model import Telemetry; r = next(iter(Telemetry('work/telemetry').runs.values())); print([(s.name, s.get('larkfield.cost.usd'), s.get('larkfield.price_table')) for s in r.model_spans])"
    [('chat chat-strong', 0.00479, '2026-10-08'), ('chat chat-strong', 0.007032, '2026-10-08')]
  7. Split the saved week’s cost by model and step: python -m obs cost --by step --data data/telemetry/week. The rows for chat-strong are:

    model step calls tokens in out cost US$
    chat-strong 1 40 86447 2108 0.1940
    chat-strong 2 39 95057 6391 0.2540
    chat-strong 3 8 20244 1250 0.0530
    chat-strong 4 2 5362 362 0.0143

The last command reads every model span of the 316 runs and adds the cost by model and step. Here is the strong model’s part as a chart:

Replayed tokens × the course's price table (version 2026-10-08): the cost of every chat-strong call in the replayed week, by the step of the run that made it.
Text description of the chart

Four bars. Step 1: 40 calls, US$0.1940. Step 2: 39 calls, US$0.2540, the highest. Step 3: 8 calls, US$0.0530. Step 4: 2 calls, US$0.0143. Most runs end after two steps, so steps 1 and 2 hold most of the cost. For chat-small, steps 1 and 2 cost US$0.0600 and US$0.0667, for 260 and 225 calls.

So Kwame’s second question has an answer. Step 2 is the most expensive step of the week, because most runs write their proposal there. Steps 1 and 2 of chat-strong together cost US$0.448, more than the 260 runs of chat-small (US$0.1513).

You can turn tokens into money in three places. Each one sees different data.

A flow from left to right. A model call with 2,160 input and 47 output tokens goes to 'In the application: tokens times price table 2026-10-08 equals US$0.00479 on the span'. From there, two arrows. One goes to 'The span keeps larkfield.cost.usd = 0.00479 and larkfield.price_table = 2026-10-08', then to 'The Collector could multiply here, but tail sampling drops 159 of 434 traces' (dashed red box), then to 'At query time, tokens times today's prices changes last week's numbers' (dashed red box). The other arrow goes to 'The metric larkfield.run.cost counts every run, sampled or not', then also to the query box.
Three places to compute cost. The course computes it in the application and keeps the table's version on the span; the Collector and the query read the result.
Text description of the diagram

At the moment of the call, the application knows the model, the tokens and the price table. So it writes the cost on the span and adds it to a counter. The Collector could multiply too, but its tail sampling keeps only some traces. A sum over sampled traces misses money. A query at read time can multiply tokens by prices too, but with today’s prices it changes last week’s numbers.

In the application (the course’s choice). The code knows which model answered, the exact tokens and the table in force. It can also handle a special case. The workflow can call the gateway of the agents governance course and ask for a capability such as strong-reasoning. The app has no price for that name. So it prices the model that the gateway names in its answer header, and writes larkfield.priced_as. The cost: every application needs the table, and a new price means a new release.

In the Collector. A transform processor could multiply two attributes by a price that lives in the Collector’s configuration. One table serves every app. But the Collector sees only what the spans say: a capability name, no answer header, no price. And look at the Collector of Module 2: its tail sampling kept 275 of 434 traces. Cost summed from the traces that reached Tempo would miss the dropped ones. If you do it in the Collector, do it before sampling, and read totals from metrics.

At query time. Grafana can multiply the token counters by prices in PromQL. That is quick to change, and it needs no release. But the query uses the prices it knows today, for every week it reads.

Try a made-up change to see why. Suppose that next month chat-strong’s prices fall by half. This change is constructed for the example: it did not happen.

  • With a query at read time, last Wednesday’s chat-strong cost of US$0.1409 becomes about US$0.0705. Wednesday’s total falls from US$0.1638 to about US$0.0934. Nobody spent less: the report changed.
  • With the cost on the span, Wednesday stays at US$0.1638, and every span says larkfield.price_table = 2026-10-08. To compare, you compute the week again with the new table on purpose, and you label it: “the same week at the new prices”.

The version also explains a difference. If two reports disagree, check their tables first. In Module 8, Langfuse receives the same spans and computes no cost at all: it does not know the names chat-small and gpt-6-luna. A tool needs its own price definitions. Our table is ours, with its date.

Compute the cost in the application when the app calls models directly and knows the tokens: that is most agents. Use the gateway’s records as a second, independent count, as the next lesson does. Do not compute cost from sampled traces alone.

The cost is small. Two attributes on each model span, one on the run span, one counter. The trace data grows by a few bytes a span. The real work is the table: someone must own it, check the provider’s prices and make a new version when they change. At Larkfield, that is Kwame.

Common mistake

The cost shows a dash, or the total is too low

Symptom: obs trace shows - instead of a cost on a model span, and obs cost gives a total lower than the provider’s bill.

Cause: the price table has no price for the model’s name. The function returns None, not 0, on purpose: an unknown price is not a free call. The usual names that cause it are a new model, a deployment name instead of a model name, or a gateway capability:

Terminal window
python -c "from resolver.usage import cost_usd; print(cost_usd('strong-reasoning', 2160, 47))"
None

Diagnose: list the model names on the spans with a dash: python -m obs runs shows the model of each run. Compare them with the names in resolver/usage.py.

Recover: add the model to the table and give the table a new version, so that the new numbers say where they came from. For a gateway capability, price the model that the gateway names in its answer, as the project does. Never fill a missing price with 0.

The waterfall’s step 4 has 2,681 input tokens and 181 output tokens, on chat-strong.

  1. Compute its cost by hand with the table. Then check it with cost_usd('chat-strong', 2681, 181).
  2. Predict the cost of the same tokens on chat-small. This is a what-if: the small model would read and write other tokens.
  3. Run python -m obs trace R-1016-040 --data data/telemetry/week and check your step 4 against the trace.

The answer is in the first knowledge check.

Check your understanding

Your turn: what does step 4 of R-1016-040 cost (2,681 in, 181 out, chat-strong), and what would the same tokens cost on chat-small?

Show the answer

On chat-strong: 2,681 × US$2.00 ÷ 1,000,000 = US$0.005362, plus 181 × US$10.00 ÷ 1,000,000 = US$0.001810, so US$0.007172. The trace shows US$0.00717. On chat-small, the same tokens cost 2,681 × US$0.10 ÷ 1M = US$0.000268, plus 181 × US$0.50 ÷ 1M = US$0.000091, so about US$0.00036: 20 times less. That is a what-if with the same tokens, and an assumption with the table’s date. A real chat-small run reads and writes other tokens, and it may not resolve the ticket.

Check your understanding

Tomás says: 'Store only the tokens. We can multiply by the prices in Grafana whenever we need the money.' What goes wrong, and what does the course do instead?

Show the answer

A query multiplies with the prices it knows when it runs. When a price changes, every old week changes with it, although nobody spent less or more. The query also needs a price for every model name, including capability names, which it cannot know. The course computes the cost in the application, at the moment of the call. It writes the cost and the table’s version (larkfield.price_table = 2026-10-08) on every model span, and it adds the run’s cost to a counter. A new price means a new table with a new version. To compare prices, compute the week again on purpose and label it.

Check your understanding

In R-1016-040, each step costs more than the one before, although the model writes almost the same short answer in steps 1 to 3. Why? And why is it a mistake to sum cost from the traces that the Collector kept?

Show the answer

Each model call sends the whole conversation so far: the instructions, the ticket and every earlier tool result. So the input tokens grow at each step, from 2,163 to 2,681, and the cost grows with them. A wasted step is paid again in every later call. The Collector’s tail sampling keeps only some traces: in Module 2’s run, 275 of 434. A sum over the kept traces misses the cost of the dropped runs. Read totals from the counter larkfield.run.cost, which counts every run, and use traces to explain one run.

Recap

  • The cost of a call is input tokens × input price plus output tokens × output price, per million tokens. An output token of our models costs five times an input token.
  • The price table has a version, its date. Every model span carries larkfield.cost.usd and larkfield.price_table; the run span carries the sum; the counter larkfield.run.cost adds each run.
  • Every price here is an assumption with its date. Prices change: check your provider’s page. An unknown model has no price, never a price of 0.
  • A cost waterfall shows how a run’s cost grows step by step. Each call sends the conversation again, so a wasted step is paid again later.
  • Compute cost in the application, before any sampling; read totals from the counter, not from sampled traces.
  • Your project is now the snapshot end-of-m05: 96 tests pass.

You can now say what each run and each step cost. But Kwame’s bill is per team, and Omar’s question is per resolved ticket, not per call. Next, you split the cost by model, slice and customer without making the metrics explode, and you check your totals against the gateway’s.

Next: Cost per customer, agent, tenant and model