Skip to content
Français

Cost per customer, agent, tenant and model

  • Lesson 17
  • 40 min
  • Module 5: Cost

Where we are

In the last lesson, we gave every run and every step its cost, with the price table's date. Omar wants more: what each model, each slice and each customer costs, and what one resolved ticket costs. In this lesson, we split the cost without making the metrics explode, we check our totals against the gateway's, and we see what Omar's switch did. We start at an office printer.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Split the workflow’s cost by model, slice, customer, agent and team. Measure what one resolved ticket costs, and keep the metrics small enough to store.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • Your project as the last lesson left it (the snapshot end-of-m05, 96 tests pass), the virtual environment active.
  • Optional: Grafana from Module 1, for the cost panel.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The week’s traffic is constructed, and every model decision in it is a replayed recording. Every cost is replayed tokens × the course’s price table (version 2026-10-08).

An office has one printer and six departments. Before you print, you type your department’s code. At the end of the month, the printer shows six counters: pages per department. Finance divides the bill by those six numbers.

Then someone has an idea: “Let us type our name and the document’s title as the code, so we know everything.” The printer now keeps a counter for every name and title pair. After a month, it has ten thousand counters, most with one page. Its memory is full, and the report is useless. To know who printed one document, you look in the printer’s job log, not at its counters.

The problem: two wrong ways to split the cost

Section titled “The problem: two wrong ways to split the cost”

Kwame wants the cost per model, per slice and per customer. Omar wants to know if his switch was worth it. Tomás has a fast answer for both: “Put the customer ID and the run ID on the cost metric as labels. Then any split is one query.”

A label per customer or per run makes the metric explode. Each different set of label values is a separate series: a list of numbers that Prometheus stores and indexes on its own. Our counter larkfield.run.cost has five labels today: model, outcome, language, release and design. In the replayed week, it has 20 series. Count what Tomás’s labels would make on the same 316 runs:

Labels on larkfield.run.cost Different label sets in the week And after a year
the five labels of today 23 (of the runs; 20 with a cost) about the same: the values are a fixed list
+ the customer ID 176 (68 customers) one set more for every new customer
+ the run ID 316 (one per run) one new series for every run, forever

The first row stays small because every label has a short list of values. The last row grows with the traffic: about 45 new series a day at Larkfield, and millions at a busy service. Each series costs memory, and every query reads more of them.

Cost per call answers the wrong question. A chat-strong call cost about US$0.006 in the week. That number does not help Omar, because a ticket needs two or more calls, and some runs end without a result. He needs the cost of the outcome he pays for: a resolved ticket.

The idea: labels with few values, IDs on spans

Section titled “The idea: labels with few values, IDs on spans”

The number of different values of an attribute is its cardinality. “Model” has a cardinality of 3 here. “Customer ID” has a cardinality that grows with the business. A label with high cardinality makes the number of series explode. The series multiply: every value of one label combines with every value of the others.

So the course puts attributes in two places, as the office keeps counters and a job log:

One closed run, R-1014-040, sends two kinds of attribute. 'Few values: metric labels: model (3), outcome (3), language (3), release (3), design (2)' goes to 'larkfield.run.cost, 20 series in the week'. 'A new value for almost every run: span attributes only: customer ID, ticket ID, run ID' (a dashed red box) goes to 'Traces: obs cost --by customer, TraceQL, one run at a time'. A dotted arrow from the IDs to the metric says: as labels, 176 or 316 series.
Measured on the saved week: attributes with a short list of values become metric labels; IDs stay on the spans, where a query reads them run by run.
Text description of the diagram

Each closed run has two kinds of attribute. The model, the outcome, the language, the release and the design have short lists of values. They are labels of the cost counter, which has 20 series in the week. The customer ID, the ticket ID and the run ID have a new value for almost every run: they stay on the spans. To split by customer, read the traces. Used as labels, the customer ID would make 176 series in one week, and the run ID 316.

The rule of thumb: a metric label must have values that you could write down as a list. A tenant is a customer organisation that shares one platform, for example one shop on a help-desk service. A platform with 40 shops can label its metrics with the tenant: 40 values, a list. It cannot label them with each shop’s customers. Larkfield’s help desk has two tenants, Larkfield and Bramble Books, a bookshop, and this course looks only at Larkfield’s tenant. A customer is a person who writes a ticket.

Cost per resolved ticket is the cost of all runs in a period, divided by the runs that ended resolved. The runs that asked the customer or went to a person cost money too, and that money bought the resolved tickets. Here is the week from python -m obs cost --by outcome --data data/telemetry/week:

outcome runs resolved cost US$ per run per resolved success
asked_customer 19 0 0.0101 0.00053 - 0.0%
handed_to_person 94 0 0.0559 0.00059 - 0.0%
resolved 203 203 0.6007 0.00296 0.00296 100.0%
total 316 203 0.6667 0.00211 0.00328 64.2%

The week cost US$0.6667, and 203 tickets ended resolved: US$0.6667 ÷ 203 = US$0.00328 per resolved ticket. The row resolved alone says US$0.00296, because it leaves out the US$0.0660 of the 113 other runs. That is 10% too low. The column success here is the share of runs that ended resolved.

  1. Split the week by model:

    Terminal window
    python -m obs cost --by model --data data/telemetry/week
    model runs resolved cost US$ per run per resolved success
    chat-small 260 162 0.1513 0.00058 0.00093 62.3%
    chat-strong 40 38 0.5153 0.01288 0.01356 95.0%
    gemma3:4b 16 3 0.0000 0.00000 0.00000 18.8%
    total 316 203 0.6667 0.00211 0.00328 64.2%

    chat-strong made 40 of the 316 runs and 77% of the cost. Each resolved ticket on it cost US$0.01356, 14.6 times chat-small’s US$0.00093. The local model looks free: its price of 0 is an assumption, and it resolved 3 of 16 tickets.

  2. Split it by language, the slice that Camille watches:

    Terminal window
    python -m obs cost --by language --data data/telemetry/week
    language runs resolved cost US$ per run per resolved success
    de 6 5 0.0029 0.00048 0.00057 83.3%
    en 302 190 0.6480 0.00215 0.00341 62.9%
    fr 8 8 0.0158 0.00198 0.00198 100.0%

    German tickets look cheap and well resolved. But by the course’s labels, only 1 of the 6 German runs was right: chat-small reships the order on T-90305, where a person should decide. A low cost per resolved ticket can hide wrong answers. Read cost next to quality, always.

  3. Split it by customer, from the traces:

    Terminal window
    python -m obs cost --by customer --data data/telemetry/week

    The table has 68 rows, one per customer. This works because the customer ID is on the run’s spans, not on a metric. In PowerShell, add | Select-Object -First 6 to see the first lines; on macOS or Linux, | head -6.

  4. Ask the gateway for its split. Only the gateway knows the registered agent and its team. Its records of the gateway lesson’s run (70 tickets through the agents governance course’s gateway) are in data/telemetry/gateway/:

    Terminal window
    python -m obs gateway data/telemetry/gateway/gateway-usage.json --by team
    team calls not ok tokens in out cost US$
    service 135 1 310041 15702 0.77710
    The gateway prices each call with its own price table; the traces price the same calls with the app's.

    With --by agent, the one row is support-workflow. In a shared estate, the same command gives each team its bill. The call that was “not ok” is T-80002’s recorded content filter.

  5. Check that the two totals agree:

    Terminal window
    python -m obs gateway data/telemetry/gateway/gateway-usage.json --data data/telemetry/gateway
    Gateway records: 135 model calls (ok 134, upstream_error 1)
    Workflow traces: 70 runs, 135 model calls, 83 tool calls, 28 approval requests
    Joined by trace context (the span ID in traceparent): 135 of 135 records
    Tokens in / out: gateway 310,041 / 15,702; traces 310,041 / 15,702
    Cost: gateway US$0.77710; traces US$0.77710

The two totals agree to the last digit. They agree because both sides count the same 135 calls with the same prices. Also, the app priced the capability strong-reasoning by the model that the gateway named in its answer. A difference would be a finding, not noise. Look for a refused call that the app retried, a call that skipped the gateway, or an old price table on one side. Check the totals every month, before you send a bill.

Each side answers a different question. The gateway’s records split the cost by agent and team: that is the bill. The traces split it by outcome, step and customer: that is what the money bought.

From Wednesday 09:00, release 2.1.0 sends hard tickets to chat-strong: a late or missing delivery, two or more order numbers, or more than 60 words. In the agents course’s recordings, chat-small was right on 11 of the 15 hard tasks for US$0.0090, and chat-strong on 15 of 15 for US$0.1946: 21.6 times the money. So the switch is a real trade, not a mistake.

The weekdays before and after the switch had the same traffic: 52 tickets a weekday. Before is Monday to Wednesday 09:00; after is Wednesday 09:00 to Friday. The weekend is left out, because it had the local-model trial.

  • B: before the switch
  • A: after the switch
Replayed tokens × the course's price table (version 2026-10-08), constructed traffic, weekdays only, in US$ per 1,000 resolved tickets. Hard tickets ran on chat-small before the switch and on chat-strong after it; other tickets stayed on chat-small.
Text description of the chart

Four bars, in pairs. All tickets: US$0.00094 before (112 runs) and US$0.00478 after (148 runs), 5.1 times more. Hard tickets: US$0.00080 before (26 runs on chat-small) and US$0.01333 after (33 runs on chat-strong).

The cost per resolved ticket rose 5.1 times, from US$0.00094 to US$0.00478. On the other tickets, which stayed on chat-small, it did not rise: US$0.00099 before and US$0.00093 after. The whole rise is the hard tickets’ model.

Now the other half of the trade, from the same runs. The share of runs that ended resolved went from 64.3% to 69.6%. By the course’s labels, the right outcome went from 84.8% to 90.5%. On hard tickets alone, it went from 18 of 26 (69.2%) on chat-small to 31 of 33 (93.9%) on chat-strong. The labels are the agents course’s; in a real team, sampled reviews give this number, as Module 7 shows.

Explore the week yourself before you read on.

Explore

Split a week's cost and read it per resolved ticket

Runs
316
Resolved
203 (64.2%)
Cost
US$0.66667
Per run
US$0.00211
Per resolved ticket
US$0.00328
Against the budget
1.6 × the budget
Right (the course’s labels, standing in for Priya’s reviews)
273 (86.4%)

The week: 316 runs, US$0.00328 per resolved ticket (1.6 × the budget). The group with the highest cost per resolved ticket: chat-strong, US$0.01356 (77% of the cost).

US$ per resolved ticketbudget US$0.002chat-small · 260 runschat-small: US$0.00093 · 47% of the budget0.00093chat-strong · 40 runschat-strong: US$0.01356 · 6.8 × the budget0.01356 ▲gemma3:4b · 16 runsgemma3:4b: US$0.00000 · 0% of the budget0.00000
US$ per resolved ticketbudget US$0.002chat-small · 260 runschat-small: US$0.00093 · 47% of the budget0.00093chat-strong · 40 runschat-strong: US$0.01356 · 6.8 × the budget0.01356 ▲gemma3:4b · 16 runsgemma3:4b: US$0.00000 · 0% of the budget0.00000
Cost of the replayed week (Monday to Sunday), by model
GroupRunsResolvedCost (US$)Per runPer resolved ticketRight
chat-small260162 (62%)0.151340.000580.0009388%
chat-strong4038 (95%)0.515330.012880.0135695%
gemma3:4b163 (19%)0.000000.000000.0000038%
total316203 (64%)0.666670.002110.0032886%

Cost per resolved ticket = the cost of ALL the group's runs / its resolved tickets. replayed tokens x the agents course's prices (price table 2026-10-08); constructed traffic. The budget is an assumption (Omar's).

Replayed tokens × the course's price table (version 2026-10-08); constructed traffic. The budget line, US$0.002 per resolved ticket, is an assumption: Omar's. 'Right' is the course's labels, standing in for Priya's reviews.
Text description of the explorer

The explorer starts with the whole replayed week split by model. chat-small: 260 runs at US$0.00093 per resolved ticket. chat-strong: 40 runs at US$0.01356. gemma3:4b: 16 runs at US$0, an assumed price. In total, US$0.00328, 1.6 times the budget. Choose “Before the switch”: 112 runs, all on chat-small, US$0.00094, under the budget. Choose “After the switch”: 148 runs, US$0.00478, 2.4 times the budget; chat-strong’s 33 runs cost 87% of it. Split by “Hard ticket” to see that only the hard tickets’ cost changed. Choose “The healthy week”: the same tickets on chat-small, US$0.00091. Split by “Day”: every weekday has 52 runs, so the number of tickets did not change.

The answer to the prediction: the model. Every weekday had 52 tickets. Wednesday’s cost rose because the hard tickets from 09:00 on ran on a model whose tokens cost 20 times more.

Split by a metric label when the attribute has a short, fixed list of values: model, outcome, language, release, design, tenant, team. Dashboards and alerts read these, and they stay cheap. Split by span attributes when the attribute is an ID: customer, ticket, run. A query over traces is slower, and sampling can drop traces, so use it to explain, not to bill.

The cost of each label is the multiplication. Five labels with 3, 3, 3, 3 and 2 values allow 162 series, and the week used 20. A sixth label with 10 values allows ten times more. Before you add a label, ask: “Can I write down all its values?” If not, it belongs on the span.

Common mistake

The cost per resolved ticket looks lower than the bill

Symptom: your dashboard says US$0.00296 per resolved ticket for the week, but the week cost US$0.6667 for 203 resolved tickets, which is US$0.00328.

Cause: the query divides the cost of the resolved runs by the resolved runs. It leaves out the runs that asked the customer or went to a person, although they were paid for. In PromQL, the mistake is a filter on both sides: {larkfield_outcome="resolved"} on the cost as well as on the count.

Diagnose: run python -m obs cost --by outcome --data data/telemetry/week and compare the row resolved with the row total. The column per resolved of the total is the right number.

Recover: filter only the count, never the cost. The course’s panel divides all the cost by the resolved runs: sum(increase(larkfield_run_cost_total[90s])) / sum(increase(larkfield_runs_total{larkfield_outcome="resolved"}[90s])).

Your turn: a cost panel by model and outcome

Section titled “Your turn: a cost panel by model and outcome”

Build the panel that Kwame asks for: the cost by model and outcome. It shows what the unresolved runs cost on each model.

  • Without Docker: use the filter --model with the split by outcome, once per model:

    Terminal window
    python -m obs cost --by outcome --model chat-strong --data data/telemetry/week
    python -m obs cost --by outcome --model chat-small --data data/telemetry/week

    For chat-strong, the output starts with handed_to_person 2 0 0.0210. Two runs went to a person and cost US$0.0210 between them.

  • In Grafana (optional): if Grafana runs with the replayed week from Module 1, open the runs dashboard. Add a panel with this query, as a time series or a bar chart:

    sum by (gen_ai_request_model, larkfield_outcome) (increase(larkfield_run_cost_total[90s]))

    It follows the dashboard’s own panel “Cost per resolved ticket, by model”. The course team did not run this new panel: compare its total with obs cost.

Then answer, for each model: what share of its cost bought no resolved ticket? Our answer is in the second knowledge check.

Check your understanding

Tomás wants the customer ID as a label on larkfield.run.cost, so that Grace can see the cost per customer in Grafana. Answer him with the numbers of the week, and say where the cost per customer comes from instead.

Show the answer

Each different set of label values is a separate series. Today’s five labels give 20 series of the cost counter in the week, and the list stays the same over time. With the customer ID, the 316 runs give 176 label sets, one more for every new customer; with the run ID, one new series for every run, forever. That is high cardinality: memory and query time grow with the traffic. The customer ID stays on the run’s spans, where python -m obs cost --by customer reads it (68 customers in the week), or a TraceQL query in Tempo.

Check your understanding

Your turn: in the week, what share of each model's cost bought no resolved ticket? Use the two outputs of the independent task.

Show the answer

For chat-strong, the 2 runs handed to a person cost US$0.0210 of US$0.5153: about 4%. For chat-small, the 19 runs that asked the customer and the 79 handed to a person cost US$0.0101 + US$0.0348 = US$0.0449 of US$0.1513: about 30%. That money is part of the cost per resolved ticket: US$0.5153 ÷ 38 = US$0.01356 for chat-strong and US$0.1513 ÷ 162 = US$0.00093 for chat-small. A cheap model that often fails pays a larger share of its money for no result.

Check your understanding

The gateway's records and the traces both say US$0.77710 for the gateway run. Name what each side can split by that the other cannot, and two causes that would make the totals differ.

Show the answer

Only the gateway knows the registered agent (support-workflow), its team (service), the capability, the provider and the calls it refused: it splits the bill by agent and team. Only the traces know the tool calls, the approval, the outcome of the run and the step of each call: they split the cost by outcome, step and customer, and give the cost per resolved ticket. The totals would differ if one side had another price table, if the app retried a call that the gateway refused or counted differently, or if some calls skipped the gateway. Here they agree because both count the same 135 calls with the same prices, and the app prices strong-reasoning by the model that the gateway names.

Recap

  • Cardinality is the number of different values of an attribute. Metric labels need few values (model, outcome, language, release, design, tenant, team); IDs (customer, ticket, run) stay on spans.
  • The cost counter has 20 series in the week; with the customer ID, 176; with the run ID, one per run.
  • Cost per resolved ticket = the cost of all runs ÷ the resolved runs: US$0.00328 for the week, not the US$0.00296 of the resolved runs alone.
  • The gateway splits the bill by agent and team; the traces split it by outcome, step and customer. Their totals agree here: US$0.77710. A difference is a finding.
  • Omar’s switch raised the cost per resolved ticket 5.1 times, from US$0.00094 to US$0.00478, all from the hard tickets. The right outcome rose from 84.8% to 90.5%.
  • Read cost next to quality: the German tickets were cheap and mostly wrong.

You can now say where the money went and what it bought. But a split on Monday does not stop a run that spends too much on Sunday. Next, you set a budget, an alert that names the runs behind a spike, and the limit in code that actually stops the spending.

Next: Budgets, alerts and the expensive run