Budgets, alerts and the expensive run
- Lesson 18
- 35 min
- Module 5: Cost
Where we are
In the last two lessons, we gave every run its cost and split the week's cost by model, slice and step. We know why it rose. But a report on Monday does not stop a run that spends too much on Sunday. In this lesson, we set a budget, an alert that names the runs behind a spike, and the limit in code that actually stops the spending. We start with the electricity at home.
Lesson 18 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsBudgets, alerts and the expensive run
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Set a budget for the workflow’s cost, and get an alert that names the runs behind a rise. Then put a limit in the code that stops one run from spending too much: no dashboard can do that.
Before you startWhat to know and to have ready
What you need to know
- Cost per resolved ticket, and the switch’s effect, from Cost per customer, agent, tenant and model.
- The agents course’s run limits (steps, time, cost), from AI Agents and Workflow Orchestration.
- Optional: the gateway’s limits, from the agents governance course.
What you need on your computer
- Your project as the last lesson left it (the snapshot
end-of-m05, 96 tests pass), the virtual environment active.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The Grafana numbers come from the course team’s run of the replayed week at 1,000 times its real speed.
A bill, a message and a fuse
Section titled “A bill, a message and a fuse”At home, three things deal with the cost of electricity. The bill comes once a month and tells you what you spent. A smart meter can send a message: “You used 40% more than usual this week.” Both tell you, after the fact, and the power still flows. Only the fuse acts: when the current is too high, it cuts it, at once, without asking anyone.
A good message also says where to look: “the water heater, since Tuesday”. A message without that sends you round the house, switching things off one by one.
The problem: a report tells you too late, and an alert without runs tells you too little
Section titled “The problem: a report tells you too late, and an alert without runs tells you too little”Larkfield’s cost report is the table from obs cost --by day, read on Monday:
day runs resolved cost US$ per run per resolved successMon 12 52 33 0.0322 0.00062 0.00098 63.5%Tue 13 52 36 0.0303 0.00058 0.00084 69.2%Wed 14 52 33 0.1638 0.00315 0.00497 63.5%Thu 15 52 38 0.1614 0.00310 0.00425 73.1%Fri 16 52 35 0.1726 0.00332 0.00493 67.3%Sat 17 28 12 0.0477 0.00170 0.00397 42.9%Sun 18 28 16 0.0587 0.00210 0.00367 57.1%The cost per resolved ticket jumped on Wednesday and stayed high. A person who reads this on Monday learns it five days late. That is the bill.
An alert is faster, but an alert that says only “cost is high” starts a search: which model, which release, which runs? And neither the bill nor the alert stops anything. On Wednesday afternoon, a run that loops on a paid model keeps calling until something in the code stops it.
The idea: a budget, an alert with its runs, and a limit in code
Section titled “The idea: a budget, an alert with its runs, and a limit in code”A budget is an amount you decide to spend for a unit of work. Omar set one for the workflow: US$0.002 per resolved ticket. It is an assumption, his decision, not a fact of the data. A budget per resolved ticket follows quality and cost together. More cost for the same results breaks it, and so do fewer results for the same cost.
A budget alert checks the budget over a window, a stretch of recent work. Ours is the last 45 closed runs, one day at Larkfield. It fires when the window’s cost per resolved ticket is over the budget. It waits for 20 runs, so that the first few runs of a day cannot set it off alone. The alert also names the runs that cost the most in its window, so the first click leads to a trace, not to a search.
A limit is the fuse. It sits in the code that spends the money, and it acts before the next call. The workflow has one per run, from the agents course: RESOLVER_MAX_COST_USD, US$0.05 by default.
The window, worked by hand
Section titled “The window, worked by hand”Here is the moment the budget broke. After closed run 120 of the week, the window held 45 runs: 31 resolved, and US$0.06698 of cost.
US$0.06698 ÷ 31 resolved tickets = US$0.00216 per resolved ticket, over the budget of US$0.002.
Three of those 45 runs were on chat-strong, the first hard tickets after the switch at 09:00. They cost US$0.04225 together: 63% of the window’s money for 3 of its 45 runs. The other 42 runs cost US$0.02473.
- the replayed week
- the healthy week
- budget US$0.002
Text description of the chart
Two lines over the 316 closed runs of each week, and a dashed budget line at US$0.002. The healthy week, with the same tickets on chat-small only, stays between US$0.00076 and US$0.00117: always under the budget. The replayed week is the same until Wednesday morning. At closed run 120 (Wednesday at 10:39), it crosses the budget at US$0.00216, and it stays over it until the end of the week, with a peak of US$0.00642.
The healthy week never crosses the line: the alert would stay quiet, as it must. The replayed week crosses it 1 hour 39 minutes after the switch, and never comes back.
An alert that names its runs
Section titled “An alert that names its runs”-
Ask
obswhen the week broke the budget, and why:Terminal window python -m obs cost --budget 0.002 --data data/telemetry/weekBudget: US$0.002 per resolved ticket over the last 45 closed runs (from 20 runs on)Over budget at Wed 14 Oct 10:39 (closed run 120, R-1014-016): US$0.00216 per resolved ticket (31 resolved, US$0.06698)The 5 runs that cost most in that window (67% of its cost):run ticket started outcome model tools appr cost US$ secs model flagsR-1014-009 T-90201 Wed 09:15 resolved 3 2 1 0.01844 9.1 chat-strong -R-1014-016 T-80005 Wed 10:39 resolved 2 1 0 0.01197 6.3 chat-strong -R-1014-011 T-90203 Wed 09:29 resolved 2 1 1 0.01184 6.5 chat-strong -R-1014-004 T-65001 Wed 04:17 resolved 4 2 0 0.00124 6.8 chat-small -R-1014-010 T-65001 Wed 09:25 resolved 4 2 0 0.00124 6.8 chat-small -The first three rows are the answer: three
chat-strongruns after 09:00. The twochat-smallruns below them cost ten times less. -
Open the most expensive one:
python -m obs trace R-1014-009 --data data/telemetry/week. It is a hard ticket with two order numbers: three model calls, US$0.01844, and a refund approved by Sam after 9 minutes. -
Check the same rule on the healthy week:
Terminal window python -m obs cost --budget 0.002 --data data/telemetry/healthy-weekBudget: US$0.002 per resolved ticket over the last 45 closed runs (from 20 runs on)Never over budget. -
List every run of the cause on that day:
python -m obs runs --day wed --model chat-strong --data data/telemetry/week. It shows 11 runs, from US$0.01148 to US$0.01844. Two of them carry the flagduplicate_write: the double refunds of Module 4, on the same model.
In Grafana, the same rule is a query on the two counters, and Module 7 turns it into an alert rule file:
sum(increase(larkfield_run_cost_total[90s])) / sum(increase(larkfield_runs_total{larkfield_outcome="resolved"}[90s])) > 0.002The course team ran the week in Grafana at 1,000 times its speed. This alert fired at Wednesday 16:41 in practice time, later than obs says. Grafana’s window is 90 seconds of the fast replay, about one practice day. It checks the rule every 10 seconds, about every 2.8 practice hours. So a compressed replay is coarse; the saved runs give the exact moment.
To make the Grafana alert name its runs, put a link to Tempo in its message. Use the TraceQL query for expensive runs from the course’s sheet, grafana/traceql.md:
{ span.gen_ai.operation.name =~ "invoke_agent|invoke_workflow" && span.larkfield.cost.usd > 0.01 }In the same Grafana run, this query found 23 runs over 1 cent. The saved week has 39 such runs, all on chat-strong. Tempo has only the traces that the Collector’s tail sampling kept. So use the link to find examples, and use obs cost or the counter for the totals. A person who opens the alert sees the cause in one click.
Observability does not stop spending
Section titled “Observability does not stop spending”Look again at the order of events. The model answers, the provider counts the tokens, and the money is spent. Then the span records it, the counter adds it, and the alert, a few runs later, tells a person. Every piece of telemetry comes after the spending. It can tell you, fast and with evidence, but it cannot stop a call.
Text description of the diagram
Two checks come before the provider is called. In the run’s own code, the agents course’s limit stops a run whose cost so far is over RESOLVER_MAX_COST_USD. The ticket goes to a person. At the gateway of the agents governance course, each registered agent has a monthly budget. When the month’s money is spent, the gateway refuses the call with HTTP 429 budget_exhausted. If neither check stops the call, the provider answers and the money is spent. Only then does telemetry record it, and a budget alert tells a person, who decides later.
There are two fuses in our estate. The run’s limit is in resolver/loop.py: before each model call, it compares the run’s cost so far with RESOLVER_MAX_COST_USD. The gateway’s limit is in the registration of support-workflow: budget_usd_month: 25. The gateway refuses every call after that, and from 80% of the budget it adds a warning header to its answers. Your provider’s account may have its own spending limit too; check its settings.
See the run’s limit act
Section titled “See the run’s limit act”The week’s most expensive run, T-80009 on chat-strong, made four calls for US$0.02219. Run it again with a limit of US$0.01:
-
Delete the folder
work. Then run the ticket once without a limit, for comparison:Terminal window python -m resolver run T-80009 --variant agent_structured --model chat-strongThe second line ends with
4 model call(s), 3 tool call(s) | 9464 tokens in, 326 out | US$0.02219. -
Run it with the limit:
Terminal window RESOLVER_MAX_COST_USD=0.01 python -m resolver run T-80009 --variant agent_structured --model chat-strongTerminal window $env:RESOLVER_MAX_COST_USD = "0.01"python -m resolver run T-80009 --variant agent_structured --model chat-strongRemove-Item Env:RESOLVER_MAX_COST_USDTerminal window RESOLVER_MAX_COST_USD=0.01 python -m resolver run T-80009 --variant agent_structured --model chat-strongRun T-80009-agent_structured-chat-strong-2 | single agent with one JSON step per call (for models without tool calling) | chat-strongStopped: cost. No proposal: the ticket goes to a person. -
Read the stopped run:
python -m obs trace T-80009-agent_structured-chat-strong-2.3 model call(s), 3 read(s), 0 write call(s), 6783 tokens in, 145 out, US$0.01502…0.0 s |========================| 5.60 s invoke_agent support-resolver cost0.0 s |========== | 2.35 s chat chat-strong 2163 in / 48 out US$0.00481 stop2.4 s | = | 0.00 s execute_tool get_order ERROR invalid_arguments LK-551902.4 s | ======= | 1.66 s chat chat-strong 2253 in / 48 out US$0.00499 stop4.0 s | = | 0.00 s execute_tool get_customer_orders ok4.0 s | =======| 1.59 s chat chat-strong 2367 in / 49 out US$0.00522 stop5.6 s | =| 0.00 s execute_tool get_order ok LK-551906Outcome: handed_to_person | stopped: cost -
Compare the two runs:
python -m obs cost --by outcome.outcome runs resolved cost US$ per run per resolved successhanded_to_person 1 0 0.0150 0.01502 - 0.0%resolved 1 1 0.0222 0.02219 0.02219 100.0%total 2 1 0.0372 0.01860 0.03720 50.0%
The fuse worked: the fourth call never happened. But read the numbers. The run spent US$0.01502 against a limit of US$0.01. After two calls it had spent US$0.00979, under the limit, so it made a third call. The check comes before a call, and it cannot know what the call will cost.
And the stopped run bought nothing. It cost two thirds of a full run, resolved no ticket, and sent the ticket to a person. With it, the cost per resolved ticket of these two runs is US$0.03720 instead of US$0.02219. A limit set too low makes the cost per resolved ticket worse, not better.
Choose the limit from the data
Section titled “Choose the limit from the data”A run’s limit is for runaway runs, not for ordinary hard tickets. Set it from your own runs. In the replayed week, the most expensive run cost US$0.02219, so the default of US$0.05 leaves more than twice that before a stop. A loop on a paid model reaches it in a few calls, and then it stops. The week’s three loops ran on the local model, at a price of 0: in money, this limit would not have seen them. That is why the agents course’s step and time limits matter too.
Use the budget alert for the daily trend, the run’s limit for one runaway run, and the gateway’s monthly budget for the whole agent. Each one has a cost. An alert needs a person who acts on it. A limit stops some runs that would have succeeded. A gateway budget can stop a whole team on the last day of the month.
Common mistake
The run spent more than its limit
Symptom: with RESOLVER_MAX_COST_USD=0.01, a run ends with Stopped: cost, but obs trace shows US$0.01502.
Cause: the limit is checked before each model call, against the cost so far. The run was under the limit after two calls (US$0.00979), so the third call went out and took it over. A limit cannot refuse a call whose cost it does not know yet.
Diagnose: read the trace. The last model call before the stop is the one that crossed the limit. Its cost is the overshoot.
Recover: treat the limit as “the most that a run may spend, minus one call”. For chat-strong, one call in the week cost up to US$0.00816. Or estimate the next call before you send it: it sends at least the input tokens of the last call, so its cost is at least those tokens times the input price.
Your turn: find what made last week’s cost rise
Section titled “Your turn: find what made last week’s cost rise”Kwame asks you on Monday: “Last week cost more than three times the healthy week. Why?” Answer with commands, not guesses, and write down each output’s meaning.
- Compare the two weeks:
python -m obs cost --by model --data data/telemetry/healthy-weekand the same fordata/telemetry/week. - Find when it started:
--by day, then--by release. - Find who spent it:
--by design --day wed, thenpython -m obs runs --day wed --model chat-strong --data data/telemetry/week. - Decide between the two causes: more tickets, or dearer tickets.
Our answer is in the first knowledge check.
Knowledge check
Section titled “Knowledge check”Check your understanding
Your turn: what made last week's cost rise? Give the numbers that prove it.
Show the answer
Not the number of tickets: both weeks had the same 316 runs, 52 each weekday, and the same 203 resolved. The healthy week cost US$0.1850 (US$0.00091 per resolved ticket); the replayed week cost US$0.6667 (US$0.00328), 3.6 times more. By day, the rise starts on Wednesday (US$0.00497 per resolved ticket against US$0.00084 on Tuesday). By release, it is 2.1.0 from Wednesday 09:00 (US$0.00476) and 2.1.1 (US$0.00426), against 2.0.0 (US$0.00094). On Wednesday, the 11 agent_structured runs on chat-strong cost US$0.1409 of the day’s US$0.1638. So the cause is Omar’s switch: the hard tickets now run on a model whose tokens cost 20 times more.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'We have the budget alert now, so we cannot overspend.' What is wrong with that, and what does stop spending?
Show the answer
An alert comes after the money is spent. The provider answers, the span records the cost, and the alert reads the counter a few runs later. It tells a person, with the runs behind it, but the person decides later. What stops spending is a limit in the code that makes the call, before the call. One is the run’s limit, RESOLVER_MAX_COST_USD, checked before each model call: T-80009 stopped after three calls. The other is the gateway’s monthly budget per agent, budget_usd_month: 25, then HTTP 429. Even the run’s limit overshoots by one call: US$0.01502 against US$0.01.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Completion check for this module: Omar asks whether to keep or roll back his switch to chat-strong for hard tickets. Recommend one, with the cost and the success evidence of the weekdays before and after the switch.
Show the answer
A good answer puts cost and results side by side and names its assumptions. Cost: the cost per resolved ticket rose from US$0.00094 to US$0.00478 (5.1 times), over Omar’s budget of US$0.002 since Wednesday 10:39. The rise is all on hard tickets: US$0.00080 per resolved hard ticket on chat-small before, US$0.01333 on chat-strong after; other tickets did not change. Results: on hard tickets, the right outcome by the course’s labels rose from 18 of 26 runs (69.2%) to 31 of 33 (93.9%), and resolved from 73.1% to 97.0%. Over all weekday runs, right rose from 84.8% to 90.5%. This matches the recordings: chat-small right on 11 of 15 hard tasks, chat-strong on 15 of 15. The price of the gain: a hard ticket costs US$0.01234 more per run for 24.7 points more right answers, so about US$0.05 for each extra right answer. Recommendation: keep the switch, because a wrong answer on a late delivery (a refund or a reshipment wrongly given or refused) costs more than 5 cents of a person’s time. But change the budget, not the model: US$0.002 was set for a chat-small mix. Give hard tickets their own budget, about US$0.015 per resolved ticket, keep US$0.002 for the others, and alert on each. Roll back only if the money is a fixed limit, and then expect about 25 more wrong answers per 100 hard tickets. Limits: the week’s traffic is constructed, the labels stand in for a reviewer, and the prices are assumptions with their date.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A budget is a decision: Omar’s US$0.002 per resolved ticket is an assumption. A budget per resolved ticket follows cost and results together.
- A budget alert checks a window (the last 45 closed runs, after 20 runs) and names the runs that cost the most in it. It broke on Wednesday 10:39, run 120, at US$0.00216; three chat-strong runs held 63% of the window’s cost. The healthy week stayed under it.
- In Grafana, the same rule is a PromQL ratio of two counters, and a TraceQL link leads to the expensive runs.
- Observability reports spending after it happens. Limits in code stop it before a call: the run’s
RESOLVER_MAX_COST_USDand the gateway’s monthly budget. A limit overshoots by one call, and a stopped run resolves nothing. - Omar’s switch costs about 5 cents for each extra right answer on a hard ticket: keep it, and give hard tickets their own budget.
You can now say what the workflow costs, why, and what stops it from spending too much. But cost is only one question that Grace and the auditors ask. “Who approved the second refund of T-90104, and on what evidence?” is another, and a trace that may be sampled cannot answer it. Next, in Module 6, you see why an audit trail is not a log.