Cost per customer, agent, tenant and model
- Lesson 17
- 40 min
- Module 5: Cost
Where we are
In the last lesson, we gave every run and every step its cost, with the price table's date. Omar wants more: what each model, each slice and each customer costs, and what one resolved ticket costs. In this lesson, we split the cost without making the metrics explode, we check our totals against the gateway's, and we see what Omar's switch did. We start at an office printer.
Lesson 17 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsCost per customer, agent, tenant and model
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Split the workflow’s cost by model, slice, customer, agent and team. Measure what one resolved ticket costs, and keep the metrics small enough to store.
Before you startWhat to know and to have ready
What you need to know
- Cost on spans and the counter
larkfield.run.cost, from Cost per run and per step. - The gateway’s records and how they join the traces, from The gateway’s view and the app’s view.
- Omar’s switch: from Wednesday 09:00, hard tickets go to
chat-strong(release 2.1.0), from Meet the live workflow.
What you need on your computer
- Your project as the last lesson left it (the snapshot
end-of-m05, 96 tests pass), the virtual environment active. - Optional: Grafana from Module 1, for the cost panel.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The week’s traffic is constructed, and every model decision in it is a replayed recording. Every cost is replayed tokens × the course’s price table (version 2026-10-08).
An office printer with department codes
Section titled “An office printer with department codes”An office has one printer and six departments. Before you print, you type your department’s code. At the end of the month, the printer shows six counters: pages per department. Finance divides the bill by those six numbers.
Then someone has an idea: “Let us type our name and the document’s title as the code, so we know everything.” The printer now keeps a counter for every name and title pair. After a month, it has ten thousand counters, most with one page. Its memory is full, and the report is useless. To know who printed one document, you look in the printer’s job log, not at its counters.
The problem: two wrong ways to split the cost
Section titled “The problem: two wrong ways to split the cost”Kwame wants the cost per model, per slice and per customer. Omar wants to know if his switch was worth it. Tomás has a fast answer for both: “Put the customer ID and the run ID on the cost metric as labels. Then any split is one query.”
A label per customer or per run makes the metric explode. Each different set of label values is a separate series: a list of numbers that Prometheus stores and indexes on its own. Our counter larkfield.run.cost has five labels today: model, outcome, language, release and design. In the replayed week, it has 20 series. Count what Tomás’s labels would make on the same 316 runs:
Labels on larkfield.run.cost |
Different label sets in the week | And after a year |
|---|---|---|
| the five labels of today | 23 (of the runs; 20 with a cost) | about the same: the values are a fixed list |
| + the customer ID | 176 (68 customers) | one set more for every new customer |
| + the run ID | 316 (one per run) | one new series for every run, forever |
The first row stays small because every label has a short list of values. The last row grows with the traffic: about 45 new series a day at Larkfield, and millions at a busy service. Each series costs memory, and every query reads more of them.
Cost per call answers the wrong question. A chat-strong call cost about US$0.006 in the week. That number does not help Omar, because a ticket needs two or more calls, and some runs end without a result. He needs the cost of the outcome he pays for: a resolved ticket.
The idea: labels with few values, IDs on spans
Section titled “The idea: labels with few values, IDs on spans”The number of different values of an attribute is its cardinality. “Model” has a cardinality of 3 here. “Customer ID” has a cardinality that grows with the business. A label with high cardinality makes the number of series explode. The series multiply: every value of one label combines with every value of the others.
So the course puts attributes in two places, as the office keeps counters and a job log:
Text description of the diagram
Each closed run has two kinds of attribute. The model, the outcome, the language, the release and the design have short lists of values. They are labels of the cost counter, which has 20 series in the week. The customer ID, the ticket ID and the run ID have a new value for almost every run: they stay on the spans. To split by customer, read the traces. Used as labels, the customer ID would make 176 series in one week, and the run ID 316.
The rule of thumb: a metric label must have values that you could write down as a list. A tenant is a customer organisation that shares one platform, for example one shop on a help-desk service. A platform with 40 shops can label its metrics with the tenant: 40 values, a list. It cannot label them with each shop’s customers. Larkfield’s help desk has two tenants, Larkfield and Bramble Books, a bookshop, and this course looks only at Larkfield’s tenant. A customer is a person who writes a ticket.
Cost per resolved ticket, worked by hand
Section titled “Cost per resolved ticket, worked by hand”Cost per resolved ticket is the cost of all runs in a period, divided by the runs that ended resolved. The runs that asked the customer or went to a person cost money too, and that money bought the resolved tickets. Here is the week from python -m obs cost --by outcome --data data/telemetry/week:
outcome runs resolved cost US$ per run per resolved successasked_customer 19 0 0.0101 0.00053 - 0.0%handed_to_person 94 0 0.0559 0.00059 - 0.0%resolved 203 203 0.6007 0.00296 0.00296 100.0%total 316 203 0.6667 0.00211 0.00328 64.2%The week cost US$0.6667, and 203 tickets ended resolved: US$0.6667 ÷ 203 = US$0.00328 per resolved ticket. The row resolved alone says US$0.00296, because it leaves out the US$0.0660 of the 113 other runs. That is 10% too low. The column success here is the share of runs that ended resolved.
Split the week’s cost
Section titled “Split the week’s cost”-
Split the week by model:
Terminal window python -m obs cost --by model --data data/telemetry/weekmodel runs resolved cost US$ per run per resolved successchat-small 260 162 0.1513 0.00058 0.00093 62.3%chat-strong 40 38 0.5153 0.01288 0.01356 95.0%gemma3:4b 16 3 0.0000 0.00000 0.00000 18.8%total 316 203 0.6667 0.00211 0.00328 64.2%chat-strongmade 40 of the 316 runs and 77% of the cost. Each resolved ticket on it cost US$0.01356, 14.6 timeschat-small’s US$0.00093. The local model looks free: its price of 0 is an assumption, and it resolved 3 of 16 tickets. -
Split it by language, the slice that Camille watches:
Terminal window python -m obs cost --by language --data data/telemetry/weeklanguage runs resolved cost US$ per run per resolved successde 6 5 0.0029 0.00048 0.00057 83.3%en 302 190 0.6480 0.00215 0.00341 62.9%fr 8 8 0.0158 0.00198 0.00198 100.0%German tickets look cheap and well resolved. But by the course’s labels, only 1 of the 6 German runs was right:
chat-smallreships the order on T-90305, where a person should decide. A low cost per resolved ticket can hide wrong answers. Read cost next to quality, always. -
Split it by customer, from the traces:
Terminal window python -m obs cost --by customer --data data/telemetry/weekThe table has 68 rows, one per customer. This works because the customer ID is on the run’s spans, not on a metric. In PowerShell, add
| Select-Object -First 6to see the first lines; on macOS or Linux,| head -6. -
Ask the gateway for its split. Only the gateway knows the registered agent and its team. Its records of the gateway lesson’s run (70 tickets through the agents governance course’s gateway) are in
data/telemetry/gateway/:Terminal window python -m obs gateway data/telemetry/gateway/gateway-usage.json --by teamteam calls not ok tokens in out cost US$service 135 1 310041 15702 0.77710The gateway prices each call with its own price table; the traces price the same calls with the app's.With
--by agent, the one row issupport-workflow. In a shared estate, the same command gives each team its bill. The call that was “not ok” is T-80002’s recorded content filter. -
Check that the two totals agree:
Terminal window python -m obs gateway data/telemetry/gateway/gateway-usage.json --data data/telemetry/gatewayGateway records: 135 model calls (ok 134, upstream_error 1)Workflow traces: 70 runs, 135 model calls, 83 tool calls, 28 approval requestsJoined by trace context (the span ID in traceparent): 135 of 135 recordsTokens in / out: gateway 310,041 / 15,702; traces 310,041 / 15,702Cost: gateway US$0.77710; traces US$0.77710
The two totals agree to the last digit. They agree because both sides count the same 135 calls with the same prices. Also, the app priced the capability strong-reasoning by the model that the gateway named in its answer. A difference would be a finding, not noise. Look for a refused call that the app retried, a call that skipped the gateway, or an old price table on one side. Check the totals every month, before you send a bill.
Each side answers a different question. The gateway’s records split the cost by agent and team: that is the bill. The traces split it by outcome, step and customer: that is what the money bought.
What did Omar’s switch do?
Section titled “What did Omar’s switch do?”From Wednesday 09:00, release 2.1.0 sends hard tickets to chat-strong: a late or missing delivery, two or more order numbers, or more than 60 words. In the agents course’s recordings, chat-small was right on 11 of the 15 hard tasks for US$0.0090, and chat-strong on 15 of 15 for US$0.1946: 21.6 times the money. So the switch is a real trade, not a mistake.
The weekdays before and after the switch had the same traffic: 52 tickets a weekday. Before is Monday to Wednesday 09:00; after is Wednesday 09:00 to Friday. The weekend is left out, because it had the local-model trial.
- B: before the switch
- A: after the switch
Text description of the chart
Four bars, in pairs. All tickets: US$0.00094 before (112 runs) and US$0.00478 after (148 runs), 5.1 times more. Hard tickets: US$0.00080 before (26 runs on chat-small) and US$0.01333 after (33 runs on chat-strong).
The cost per resolved ticket rose 5.1 times, from US$0.00094 to US$0.00478. On the other tickets, which stayed on chat-small, it did not rise: US$0.00099 before and US$0.00093 after. The whole rise is the hard tickets’ model.
Now the other half of the trade, from the same runs. The share of runs that ended resolved went from 64.3% to 69.6%. By the course’s labels, the right outcome went from 84.8% to 90.5%. On hard tickets alone, it went from 18 of 26 (69.2%) on chat-small to 31 of 33 (93.9%) on chat-strong. The labels are the agents course’s; in a real team, sampled reviews give this number, as Module 7 shows.
Explore the week yourself before you read on.
Explore
Split a week's cost and read it per resolved ticket
Predict first: Wednesday cost five times Tuesday. Did the cost per resolved ticket rise because there were more tickets, or because of the model?
- Runs
- 316
- Resolved
- 203 (64.2%)
- Cost
- US$0.66667
- Per run
- US$0.00211
- Per resolved ticket
- US$0.00328
- Against the budget
- 1.6 × the budget
- Right (the course’s labels, standing in for Priya’s reviews)
- 273 (86.4%)
The week: 316 runs, US$0.00328 per resolved ticket (1.6 × the budget). The group with the highest cost per resolved ticket: chat-strong, US$0.01356 (77% of the cost).
| Group | Runs | Resolved | Cost (US$) | Per run | Per resolved ticket | Right |
|---|---|---|---|---|---|---|
| chat-small | 260 | 162 (62%) | 0.15134 | 0.00058 | 0.00093 | 88% |
| chat-strong | 40 | 38 (95%) | 0.51533 | 0.01288 | 0.01356 | 95% |
| gemma3:4b | 16 | 3 (19%) | 0.00000 | 0.00000 | 0.00000 | 38% |
| total | 316 | 203 (64%) | 0.66667 | 0.00211 | 0.00328 | 86% |
Cost per resolved ticket = the cost of ALL the group's runs / its resolved tickets. replayed tokens x the agents course's prices (price table 2026-10-08); constructed traffic. The budget is an assumption (Omar's).
Text description of the explorer
The explorer starts with the whole replayed week split by model. chat-small: 260 runs at US$0.00093 per resolved ticket. chat-strong: 40 runs at US$0.01356. gemma3:4b: 16 runs at US$0, an assumed price. In total, US$0.00328, 1.6 times the budget. Choose “Before the switch”: 112 runs, all on chat-small, US$0.00094, under the budget. Choose “After the switch”: 148 runs, US$0.00478, 2.4 times the budget; chat-strong’s 33 runs cost 87% of it. Split by “Hard ticket” to see that only the hard tickets’ cost changed. Choose “The healthy week”: the same tickets on chat-small, US$0.00091. Split by “Day”: every weekday has 52 runs, so the number of tickets did not change.
The answer to the prediction: the model. Every weekday had 52 tickets. Wednesday’s cost rose because the hard tickets from 09:00 on ran on a model whose tokens cost 20 times more.
When to split, and what it costs
Section titled “When to split, and what it costs”Split by a metric label when the attribute has a short, fixed list of values: model, outcome, language, release, design, tenant, team. Dashboards and alerts read these, and they stay cheap. Split by span attributes when the attribute is an ID: customer, ticket, run. A query over traces is slower, and sampling can drop traces, so use it to explain, not to bill.
The cost of each label is the multiplication. Five labels with 3, 3, 3, 3 and 2 values allow 162 series, and the week used 20. A sixth label with 10 values allows ten times more. Before you add a label, ask: “Can I write down all its values?” If not, it belongs on the span.
Common mistake
The cost per resolved ticket looks lower than the bill
Symptom: your dashboard says US$0.00296 per resolved ticket for the week, but the week cost US$0.6667 for 203 resolved tickets, which is US$0.00328.
Cause: the query divides the cost of the resolved runs by the resolved runs. It leaves out the runs that asked the customer or went to a person, although they were paid for. In PromQL, the mistake is a filter on both sides: {larkfield_outcome="resolved"} on the cost as well as on the count.
Diagnose: run python -m obs cost --by outcome --data data/telemetry/week and compare the row resolved with the row total. The column per resolved of the total is the right number.
Recover: filter only the count, never the cost. The course’s panel divides all the cost by the resolved runs: sum(increase(larkfield_run_cost_total[90s])) / sum(increase(larkfield_runs_total{larkfield_outcome="resolved"}[90s])).
Your turn: a cost panel by model and outcome
Section titled “Your turn: a cost panel by model and outcome”Build the panel that Kwame asks for: the cost by model and outcome. It shows what the unresolved runs cost on each model.
-
Without Docker: use the filter
--modelwith the split by outcome, once per model:Terminal window python -m obs cost --by outcome --model chat-strong --data data/telemetry/weekpython -m obs cost --by outcome --model chat-small --data data/telemetry/weekFor
chat-strong, the output starts withhanded_to_person 2 0 0.0210. Two runs went to a person and cost US$0.0210 between them. -
In Grafana (optional): if Grafana runs with the replayed week from Module 1, open the runs dashboard. Add a panel with this query, as a time series or a bar chart:
sum by (gen_ai_request_model, larkfield_outcome) (increase(larkfield_run_cost_total[90s]))It follows the dashboard’s own panel “Cost per resolved ticket, by model”. The course team did not run this new panel: compare its total with
obs cost.
Then answer, for each model: what share of its cost bought no resolved ticket? Our answer is in the second knowledge check.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás wants the customer ID as a label on larkfield.run.cost, so that Grace can see the cost per customer in Grafana. Answer him with the numbers of the week, and say where the cost per customer comes from instead.
Show the answer
Each different set of label values is a separate series. Today’s five labels give 20 series of the cost counter in the week, and the list stays the same over time. With the customer ID, the 316 runs give 176 label sets, one more for every new customer; with the run ID, one new series for every run, forever. That is high cardinality: memory and query time grow with the traffic. The customer ID stays on the run’s spans, where python -m obs cost --by customer reads it (68 customers in the week), or a TraceQL query in Tempo.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your turn: in the week, what share of each model's cost bought no resolved ticket? Use the two outputs of the independent task.
Show the answer
For chat-strong, the 2 runs handed to a person cost US$0.0210 of US$0.5153: about 4%. For chat-small, the 19 runs that asked the customer and the 79 handed to a person cost US$0.0101 + US$0.0348 = US$0.0449 of US$0.1513: about 30%. That money is part of the cost per resolved ticket: US$0.5153 ÷ 38 = US$0.01356 for chat-strong and US$0.1513 ÷ 162 = US$0.00093 for chat-small. A cheap model that often fails pays a larger share of its money for no result.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The gateway's records and the traces both say US$0.77710 for the gateway run. Name what each side can split by that the other cannot, and two causes that would make the totals differ.
Show the answer
Only the gateway knows the registered agent (support-workflow), its team (service), the capability, the provider and the calls it refused: it splits the bill by agent and team. Only the traces know the tool calls, the approval, the outcome of the run and the step of each call: they split the cost by outcome, step and customer, and give the cost per resolved ticket. The totals would differ if one side had another price table, if the app retried a call that the gateway refused or counted differently, or if some calls skipped the gateway. Here they agree because both count the same 135 calls with the same prices, and the app prices strong-reasoning by the model that the gateway names.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Cardinality is the number of different values of an attribute. Metric labels need few values (model, outcome, language, release, design, tenant, team); IDs (customer, ticket, run) stay on spans.
- The cost counter has 20 series in the week; with the customer ID, 176; with the run ID, one per run.
- Cost per resolved ticket = the cost of all runs ÷ the resolved runs: US$0.00328 for the week, not the US$0.00296 of the resolved runs alone.
- The gateway splits the bill by agent and team; the traces split it by outcome, step and customer. Their totals agree here: US$0.77710. A difference is a finding.
- Omar’s switch raised the cost per resolved ticket 5.1 times, from US$0.00094 to US$0.00478, all from the hard tickets. The right outcome rose from 84.8% to 90.5%.
- Read cost next to quality: the German tickets were cheap and mostly wrong.
You can now say where the money went and what it bought. But a split on Monday does not stop a run that spends too much on Sunday. Next, you set a budget, an alert that names the runs behind a spike, and the limit in code that actually stops the spending.