The cost of a coding agent
- Case study 2
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In the cost module, we split the cost of Larkfield's workflow by run, step, model and slice. A coding agent takes many more steps, and its cost depends even more on the path it takes. In this case study, we measure the cost of open coding-agent runs and propose a budget and a step limit.
Case study 2 of 3 · Observability for AI Agents: Traces, Costs and Audit TrailsThe cost of a coding agent
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Measure what a coding agent costs per task, per model call and per outcome, from 445 recorded runs of an open agent. Find the runs that loop or spend too much, and choose a step limit, an alert and a budget from the evidence. Then check your choice on 445 runs of the same agent with another model.
This case study is a project folder, not a notebook. It is small (about 1 MB), it uses only Python’s standard library, and it runs in the course project’s virtual environment. At one step, the course’s own obs tool reads the case study’s runs, so the folder sits next to your course project.
Before you startWhat to know and to have ready
What you need to know
This case study uses the cost module most:
- Tokens times a dated price table, from Cost per run and per step.
- Cost per resolved ticket, and why a split needs the outcome, from Cost per customer, agent, tenant and model.
- A budget alert that names its runs, and the limit in code that stops spending, from Budgets, alerts and the expensive run.
- The token attributes of a
chatspan, from Model calls and tokens. - The loop pattern, from Agent failure patterns.
The page teaches one new idea: every call of a coding agent sends the whole history again, and the provider’s cache changes its price.
What you need on your computer
- The course project at the end of the course, in
~/projects/observable-workflow, with its virtual environment. If yours is broken, download the snapshotend-of-m08and follow its README. - About 2 MB of disk. No account, no key and no money: every token count is a recording.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, not run.
The situation
Section titled “The situation”Hana (she) leads a small software team (made up for this case study). The team has a backlog of maintenance tasks: a broken build, a script to port, a crash to find. Leo (he), a platform engineer, wants a coding agent to work on them at night. The agent gets a terminal in a fresh container, a task and a time limit. It asks a model what to do, runs the commands, reads the output and asks again. In the morning, a test says whether the task is done.
Hana must give finance a number before the trial. She asks Leo three questions:
- What does a task cost, and what does a solved task cost?
- Which runs waste money, and how would we see them?
- What stops a run that goes wrong at 3 a.m., and what budget do we watch?
Each wrong answer has a price. A limit that is too loose lets a loop spend all night. A limit that is too tight stops a run that was about to pass its test. Then an engineer does that task by hand, which costs far more than any model call.
The team has no runs of its own yet. So Leo measures recorded runs of an open coding agent on public tasks, with the course’s cost tools. The team’s own runs must check the numbers after launch.
What you will build
Section titled “What you will build”You will build a cost report and a limits recommendation for a coding agent, from recorded runs. You finish the cost of a step, check it against the record, and explore where the money goes. Then you finish the limits, compare three of them under one measure, and choose a step limit, an alert and a budget. Last, you run the whole check on the same agent with another model.
Set up the project
Section titled “Set up the project”-
Download the starter project from the box at the top of this page. Unzip it into
~/projects, so that the foldercoding-agent-costis next toobservable-workflow. -
In VS Code, choose File > Open Folder, and open
coding-agent-cost. Then open a terminal: Terminal > New Terminal. -
Activate the course project’s virtual environment:
Terminal window source ../observable-workflow/.venv/bin/activateTerminal window ..\observable-workflow\.venv\Scripts\Activate.ps1Terminal window source ../observable-workflow/.venv/bin/activate -
Run the tests:
python -m pytest -q. The starter has two parts for you to finish, so six tests fail:FAILED tests/test_cost.py::test_a_step_by_hand - assert 0.0030100000000000005...FAILED tests/test_cost.py::test_our_price_table_gives_every_recorded_cost - A...FAILED tests/test_limits.py::test_a_step_limit_of_4_lets_4_calls_through - As...FAILED tests/test_limits.py::test_a_stopped_solved_run_is_a_wrong_block - Ass...FAILED tests/test_limits.py::test_a_cost_limit_is_checked_before_the_call_so_the_run_ends_over_itFAILED tests/test_limits.py::test_the_measure_counts_both_errors - assert (1 ...6 failed, 10 passed in 0.90sOne test reads the case study’s runs with the course’s
obs. If your course project is in another folder, setOBS_PROJECTto that folder, or pytest skips the test.
The finished project is in the same box, to compare with your work. Every output on this page comes from it.
The data
Section titled “The data”The data is the published runs of Terminal-Bench 2.0, an open benchmark of AI agents that work in a terminal. Its 89 tasks are engineering jobs: build a compiler, fix a crash, port a script, train a small model. A test decides if the task is done. The benchmark’s own agent, Terminus 2, ran every task 5 times with many models.
Leo needed the tokens of each model call, the commands, how each run ended, and some real loops. Open SWE-agent, SWE-smith and OpenHands runs have no token counts, or one total per run, as the agents governance course’s case study found. Terminal-Bench records the tokens of every call and the cached part. For some models, it also records the cost that the runner computed.
The design sample is 445 runs of Terminus 2 with DeepSeek-V3.2, a low-price model: we choose the limits on it. The 445 runs of the same agent with Claude Opus 4.6 are the new input for the end. One run has these fields:
| Field | What it holds |
|---|---|
id, task |
the run (task name and a trial code); one of 89 tasks, each run 5 times |
reward, exception |
how it ended: the tests passed (solved), failed (failed), or the agent hit the task’s time limit first (timed_out) |
steps[].prompt, cached, completion |
one model call: input tokens (cached ones included), the cached part, output tokens |
steps[].cost |
the cost that the runner’s software recorded for the call, when it did |
steps[].programs, digests |
each command’s first word, and a short SHA-256 digest of its exact keystrokes |
First checks
Section titled “First checks”agentcost/data.py checks each file’s SHA-256 before it reads it. After Part 1 below, python -m agentcost check prints:
terminus2-claude-opus-4.6.jsonl.gz SHA-256 okterminus2-deepseek-v3.2.jsonl.gz SHA-256 okdeepseek: 445 runs of deepseek-chat (89 tasks), 10,966 model calls, 295,559,649 tokens in (286,105,280 cached), 7,001,540 outoutcomes: solved 176, failed 183, timed_out 85, error 1model calls with 0 input tokens recorded: 0recorded cost on 10,966 calls: US$13.60; our price table: US$13.60 (10,966 calls equal to the recorded cost, +0.0% in all)price table 2026-10-23: DeepSeek-V3.2 (deepseek-chat) US$0.28 in, US$0.028 cached in, US$0.42 out per 1M tokensWhat this tells us.
- The agent solved 176 of 445 runs (39.6%) of hard tasks. One run ended in an error of the benchmark; we count it as
error. - 97% of the input tokens came from the cache. Keep that number in mind: it decides most of the bill.
- Our price table gives every recorded cost exactly, on all 10,966 calls. And a surprise: DeepSeek’s price page no longer lists this model.
prices.jsonkeeps the prices these runs paid, under its own version. A provider’s page moves on; an old report must keep its numbers. - No call records 0 input tokens. Another model’s file on the same leaderboard records 0 for every call. A recorded zero is not a free run, so we did not use that file.
Frame the problem
Section titled “Frame the problem”The unit is one run: one attempt of the agent at one task. The outcome is the test’s result, and “cost per solved task” is this case study’s cost per resolved ticket.
The price is the project’s price table, prices.json, version 2026-10-23. Per million tokens: US$0.28 input, US$0.028 cached input and US$0.42 output. These prices are assumptions with their date: check your provider’s current prices.
The two errors of a limit. A limit that stops a run which would have solved its task is a wrong block. A limit that never stops a loop lets the money go. We choose the measure in Part 3, after we see where the money goes.
The replay rule. A recording shows what the agent did when nothing stopped it. When a limit stops a run, the replay ends there: the run is not solved, and its task goes to a person.
A new idea: every call pays for the whole history
Section titled “A new idea: every call pays for the whole history”Think of a meeting where, before each new point, someone reads all the minutes aloud from the start. The first point is quick. By the twentieth, most of the meeting is the reading. A helper with a copy of the old minutes makes it faster, but the copy is still read.
A coding agent’s model calls work like that meeting. A model has no memory between calls. So each call sends the instructions, the task and everything so far: every earlier answer and every terminal output. The input of a call grows with each step, and a long run pays for its history again and again. Many providers keep a prompt cache. When the start of a request is the same as a recent one, they serve that part from the cache and charge less for it. DeepSeek charges a tenth of the input price for a cached token. A cached token is a cheaper token, not a free one.
Text description of the diagram
The first call of the run sends 830 tokens. Each later call sends all the earlier ones, their answers and the terminal’s output. At call 135, the input is 115,291 tokens, and 115,136 of them come from the cache. That call costs US$0.0034. The new input is 155 tokens at US$0.28 per million. The cached input is 115,136 tokens at US$0.028 per million. The output is 215 tokens at US$0.42 per million.
Count call 135 by hand, in millionths of a dollar. The new input costs 155 × 0.28 = 43. The cached input costs 115,136 × 0.028 = 3,224. The output costs 215 × 0.42 = 90. Almost all of the US$0.0034 is the cached history. Without the cache, the call would cost US$0.0324, ten times more.
Part 1: finish the cost of a step
Section titled “Part 1: finish the cost of a step”-
Run
python -m agentcost checkin the starter. Its sixth line shows the problem:recorded cost on 10,966 calls: US$13.60; our price table: US$85.70 (27 calls equal to the recorded cost, +530.2% in all)The starter charges every input token at the full price: the bill without a cache.
-
Open
agentcost/prices.pyand findTODO (Part 1)instep_cost. Replace itsreturnline with:cached = step.cached_tokens if cache else 0uncached = step.prompt_tokens - cachedreturn (uncached * p["input"] + cached * p["cached_input"] + step.completion_tokens * p["output"]) / 1e6prompt_tokensincludes the cached tokens, asgen_ai.usage.input_tokensdoes in the GenAI conventions. -
Run
python -m agentcost checkagain. The line now says10,966 calls equal to the recorded cost, as above. The two tests of Part 1 pass.
The course’s tool reads the same runs
Section titled “The course’s tool reads the same runs”The project can write the runs as OpenTelemetry traces, in the file format of the course’s Collector. Each run gets an invoke_agent span, a chat span per model call and an execute_tool span per command. The token names are the GenAI conventions’ (gen_ai.usage.input_tokens, gen_ai.usage.cache_read.input_tokens). The cost, the step and the outcome use the course project’s names, so obs reads them unchanged.
-
Write the traces:
python -m agentcost export.Wrote 445 traces to work/telemetry/deepseek/traces.jsonl.gz (1.4 MB, 45,095 spans), no limit, price table 2026-10-23Read them with the course's obs, in ../observable-workflow: python -m obs cost --by outcome --data ../coding-agent-cost/work/telemetry/deepseek -
In a second terminal, in
~/projects/observable-workflowwith its environment active, run the command that the export printed:outcome runs resolved cost US$ per run per resolved successerror 1 0 0.0374 0.03738 - 0.0%failed 183 0 5.8051 0.03172 - 0.0%resolved 176 176 2.4986 0.01420 0.01420 100.0%timed_out 85 0 5.2577 0.06186 - 0.0%total 445 176 13.5988 0.03056 0.07727 39.6% -
Read one short run as a waterfall:
python -m obs trace openssl-selfsigned-cert__4n5TDfS --data ../coding-agent-cost/work/telemetry/deepseek. It shows 8chatspans with their costs, and one span per command.
What this tells us. A tool written for another workflow finds the same total to the cent: US$13.5988, and US$0.07727 per solved task. Record once, with standard names, and any tool can answer. But obs trace also prints Flag no_action: closed as resolved with no write and no reply sent. That rule is Larkfield’s: a resolved ticket needs a refund or a reply. A flag rule belongs to one workflow, as a limit does.
Explore what the decision needs
Section titled “Explore what the decision needs”Where the money goes, by outcome
Section titled “Where the money goes, by outcome”python -m agentcost cost --by outcomeoutcome runs solved steps cost US$ share per run per solved successsolved 176 176 14.5 2.4986 18% 0.0142 0.0142 100.0%failed 183 0 21 5.8051 43% 0.0317 - 0.0%timed_out 85 0 30 5.2577 39% 0.0619 - 0.0%error 1 0 33 0.0374 0% 0.0374 - 0.0%total 445 176 19 13.5988 100% 0.0306 0.0773 39.6%(steps = the median number of model calls per run)What this tells us. A solved run costs US$0.0142, but a solved task costs US$0.0773: the failed and timed-out runs are part of its price. Only 18% of the money bought solved tasks. A timed-out run costs four times a solved run. With --no-cache, the total would be US$85.70 instead of US$13.60: the cache cuts the bill by 6.3 times.
Where the money goes, by run length
Section titled “Where the money goes, by run length”Text description of the chart
Seven bars, by the run’s number of model calls. Runs of 1 to 10 calls: 87 runs, 63 solved, 3% of the spending. 11 to 20 calls: 157 runs, 17%. 21 to 30: 19%. 31 to 40: 17%. 41 to 50: 24 runs, 1 solved, 11%. 51 to 100: 25 runs, 1 solved, 16%. More than 100: 5 runs, none solved, 17%.
What this tells us. Short runs are cheap and usually succeed: 63 of the 87 runs with at most 10 calls. After 40 calls, the agent almost never succeeds: 2 of 54 runs. Those 54 runs are 12% of the runs, but 44% of the spending. The run’s length decides where the money goes.
One long run, call by call
Section titled “One long run, call by call”The most expensive run fixes a crash in a language’s runtime (fix-ocaml-gc). All 5 runs of that task failed, and they cost 16% of the sample.
python -m agentcost steps fix-ocaml-gc__PB7o5ME --every 30fix-ocaml-gc__PB7o5ME: task fix-ocaml-gc, deepseek-chat, timed_out, 307 model calls, US$0.8696 step tokens in cached out cost US$ so far kind commands 1 830 768 265 0.00015 0.0002 work ls cat 30 26,334 25,216 215 0.00111 0.0205 work sed grep grep 60 62,531 62,080 421 0.00204 0.0739 work grep find 90 86,831 86,528 268 0.00262 0.1454 work grep awk grep 120 103,712 103,424 236 0.00308 0.2310 work awk cat 150 116,702 116,672 36 0.00329 0.3306 none - 180 118,540 118,464 36 0.00335 0.4303 none - 210 120,377 120,320 36 0.00340 0.5316 none - 240 122,215 122,176 36 0.00345 0.6344 none - 270 124,052 123,968 36 0.00351 0.7387 none - 300 125,890 125,824 36 0.00356 0.8447 none - 307 126,316 126,272 41 0.00357 0.8696 none -- Tokens in
- Tokens in from the cache
Text description of the chart
Two lines over the 307 model calls of one run. The input grows from 830 tokens at the first call to about 115,000 at call 135, then slowly to 126,316 at the last call. The cached tokens follow the same line, a little below it: almost all the input comes from the cache.
What this tells us. The first call cost US$0.00015; the last one US$0.00357, 24 times more, for a similar answer. From call 135, the model answered 173 times in a row without a command: the column kind says none. We read the original record of this run. There, the model writes that it cannot fix the bug. Then it repeats a summary until the task’s time limit. Those 173 calls cost US$0.59, 68% of the run. The cache kept it cheap: without it, this one run would have cost US$8.28.
Idle steps
Section titled “Idle steps”The course’s loop rule cannot see this pattern. It counts the same tool with the same arguments, and here the model calls no tool at all. The project calls a step idle when the terminal gets nothing new to do. The answer has no command, or only empty commands (the agent waits), or the same commands as the step before. One idle step is often normal, for example a wait for an install. A long streak of idle steps is a loop.
python -m agentcost idleoutcome runs longest idle streak: median maxsolved 176 2 8failed 183 2 8timed_out 85 0 173error 1 2 2The 8 runs with the longest idle streak:run outcome steps cost US$ streak from kind streak US$fix-ocaml-gc__PB7o5ME timed_out 307 0.8696 173 135 none 0.5917fix-ocaml-gc__WB7L8Nf timed_out 240 0.6388 154 83 none 0.4831custom-memory-heap-crash__exMxSPi timed_out 168 0.1823 85 33 none 0.0840fix-ocaml-gc__G9if5xy timed_out 166 0.3827 55 69 none 0.1846crack-7z-hash__Sykdxfk timed_out 63 0.0462 17 47 none 0.0144path-tracing-reverse__nEW2gSJ timed_out 61 0.1528 14 48 none 0.0442train-fasttext__bJf9npe timed_out 85 0.0886 10 32 wait 0.0088rstan-to-pystan__fDESBRy timed_out 55 0.0800 10 33 wait 0.0166What this tells us. No solved run had more than 8 idle steps in a row. Six runs had 14 or more, all timed out, most of them with no command at all. So idle streaks separate loops from work on this agent. We will see if that holds on another model.
A baseline: no limit, then another agent’s limits
Section titled “A baseline: no limit, then another agent’s limits”The baseline is the design sample with no limit: US$13.60, and US$0.0773 per solved task. Leo’s first idea is to copy the limits of Larkfield’s support workflow from the course: 8 model calls and US$0.05 a run.
python -m agentcost limits --max-steps 8 --max-cost 0.05deepseek (deepseek-chat): 445 runs | step limit 8, cost limit US$0.05 (checked before each model call)recorded outcome runs stopped cost US$ of US$solved 176 146 0.8986 2.4986failed 183 172 1.0602 5.8051timed_out 85 84 0.4587 5.2577error 1 1 0.0054 0.0374cost US$2.4229 (no limit: US$13.5988, saved US$11.1759) | solved 30 of 176 | solved runs stopped (wrong blocks): 146cost per solved task: US$0.0808 (no limit: US$0.0773)What this tells us. It “saves” US$11.18 by stopping almost everything: 146 of 176 solved runs. And the cost per solved task got worse, US$0.0808, because the money spent on stopped runs bought nothing. Larkfield’s workflow makes 1 to 5 calls a ticket; this agent’s median solved run makes 15. A limit belongs to one agent, and it comes from that agent’s runs.
Part 2: finish the limits
Section titled “Part 2: finish the limits”agentcost/limits.py replays every run under a Limits object, call by call. The check is the one of the agents course’s loop. Before the next call, stop if the run has made max_steps calls or if its cost so far is over max_cost_usd. The starter does not have these two checks.
-
Open
agentcost/limits.pyand findTODO (Part 2)inreplay. Replace the comment with:if limits.max_steps and out.steps >= limits.max_steps:out.stopped_by = "max_steps"breakif limits.max_cost_usd and out.cost > limits.max_cost_usd:out.stopped_by = "cost"breakThe check comes before the call is paid. A cost limit cannot know what the next call will cost, so a run can end a little over it.
-
Run the tests again:
python -m pytest -q. The last line says16 passed.
The tiny run in tests/test_limits.py shows the overshoot: six calls of US$0.28 each, and a cost limit of US$0.60. After two calls the run has spent US$0.56, under the limit, so the third call goes out. The check stops the fourth call, and the run ends at US$0.84.
Part 3: a fair comparison under one measure
Section titled “Part 3: a fair comparison under one measure”First, a trap. sweep replays the sample with each setting of a limit. Look at the cost per solved task, the last column:
python -m agentcost sweep --max-steps 30,40,45,50,51,55,60,75deepseek (deepseek-chat), 445 runs | no limit: US$13.5988, 176 solved, US$0.0773 per solved task step limit cost US$ saved solved wrong blocks timed out stopped per solved 30 9.0125 4.5863 161 15 42 of 85 0.0560 40 10.3158 3.2830 174 2 25 of 85 0.0593 45 10.7407 2.8581 175 1 20 of 85 0.0614 50 11.0477 2.5511 175 1 15 of 85 0.0631 51 11.1078 2.4910 176 0 13 of 85 0.0631 55 11.3085 2.2903 176 0 12 of 85 0.0643 60 11.4840 2.1148 176 0 10 of 85 0.0653 75 11.8631 1.7357 176 0 5 of 85 0.0674The cost per solved task falls as the limit gets tighter: at 30 calls, it is the lowest, US$0.0560. But that setting stops 15 runs that would have passed their tests. It also stops expensive failed runs, so the ratio looks better while the team loses work.
The ratio leaves out what a solved task is worth: an engineer’s hour or more, against about 8 cents of model calls. One wrong block costs more than any limit saves on this sample. So Leo’s measure is: the US$ that a limit saves with no wrong block. Each limit gets its tightest setting that stops no solved run of the design sample. Then we compare what each setting saves.
Run the other two limits the same way. --max-idle stops a run after that many idle steps in a row:
python -m agentcost sweep --max-cost 0.04,0.06,0.08,0.1python -m agentcost sweep --max-idle 5,8,9,10,12cost limit US$ cost US$ saved solved wrong blocks timed out stopped per solved 0.04 9.3128 4.2860 170 6 26 of 85 0.0548 0.06 10.4507 3.1481 175 1 15 of 85 0.0597 0.08 11.0902 2.5086 175 1 9 of 85 0.0634 0.1 11.4110 2.1878 176 0 7 of 85 0.0648 idle limit cost US$ saved solved wrong blocks timed out stopped per solved 5 11.9181 1.6807 174 2 11 of 85 0.0685 8 12.0311 1.5678 174 2 8 of 85 0.0691 9 12.0901 1.5087 176 0 8 of 85 0.0687 10 12.1066 1.4922 176 0 8 of 85 0.0688 12 12.2200 1.3789 176 0 6 of 85 0.0694| Limit (checked before each call) | Tightest setting with no wrong block | US$ saved of US$13.60 | Timed-out runs stopped |
|---|---|---|---|
| Step limit | 51 model calls | 2.49 (18%) | 13 of 85 |
| Cost limit | US$0.10 a run | 2.19 (16%) | 7 of 85 |
| Idle limit | 9 idle steps in a row | 1.51 (11%) | 8 of 85 |
What this tells us.
- The step limit saves the most, and its setting is easy to read: the longest solved run made 51 calls.
- The cost limit is close, but its setting is a price: the most expensive solved run, at this model’s price.
- The idle limit saves the least, because a loop must start before it acts. But it is the only one that says why a run wastes money: a good alert.
- No limit stops many of the 85 timed-out runs. Most were slow, not long, and the benchmark’s own time limit already stops them.
Part 4: the operating point and the budget
Section titled “Part 4: the operating point and the budget”A setting at exactly 51 calls is the length of one run: the longest success of one sample. The next longer good run would be a wrong block. Leo chooses 60 calls a run, about 18% above it.
python -m agentcost limits --max-steps 60deepseek (deepseek-chat): 445 runs | step limit 60 (checked before each model call)recorded outcome runs stopped cost US$ of US$solved 176 0 2.4986 2.4986failed 183 6 5.5454 5.8051timed_out 85 10 3.4026 5.2577error 1 0 0.0374 0.0374cost US$11.4840 (no limit: US$13.5988, saved US$2.1148) | solved 176 of 176 | solved runs stopped (wrong blocks): 0cost per solved task: US$0.0653 (no limit: US$0.0773)The margin costs US$0.38 of savings: US$2.11 instead of US$2.49. The 307-call run now stops after 60 calls, before its loop starts.
A budget per solved task
Section titled “A budget per solved task”A step limit stops one run. It does not tell Hana when the agent as a whole gets more expensive, after a new kind of task, model or prompt. That is the budget alert of Budgets, alerts and the expensive run. Its window here is 89 finished runs, about one run of each task. First, try a budget of US$0.09 per solved task on the runs as recorded, with no limit:
python -m agentcost budget --budget 0.09Budget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on)Cost per solved task of the windows: lowest US$0.0469, median US$0.0762, highest US$0.1019Over budget after finished run 206 (chess-best-move__XimTT7i): US$0.0905 per solved task (33 solved, US$2.9849)The 5 runs that cost most in that window (40% of its cost):run outcome steps cost US$ idle streakfix-ocaml-gc__PB7o5ME timed_out 307 0.8696 173feal-linear-cryptanalysis__e9Scv8w failed 43 0.0871 2path-tracing-reverse__vLssimd failed 45 0.0849 1schemelike-metacircular-eval__H62ET7V failed 41 0.0760 1train-fasttext__MvTjpMW timed_out 70 0.0725 1Windows over the budget: 40 of 426The alert names its runs, and the first one is the loop: 307 calls and an idle streak of 173. Now the same budget, with the step limit in place:
python -m agentcost budget --budget 0.09 --max-steps 60Budget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on, with step limit 60)Cost per solved task of the windows: lowest US$0.0469, median US$0.0626, highest US$0.0815Never over budget.What this tells us. With the step limit, the most expensive window costs US$0.0815 per solved task. A budget of US$0.09 sits about 10% above it. It stays quiet on normal work, and fires when a stretch costs more than any window of this sample. The limit stops one run; the budget watches the agent.
Write the choice in recommendation.json. Find the default entry, the team’s settings for any model, and give it your values:
"default": {"max_steps": 60, "budget_per_solved_usd": 0.09},The file also sets "idle_alert_streak": 10: a message to the agent’s owner when a run makes 10 idle calls in a row. It is an alert, not a stop. On the design sample, it flags 8 runs, all timed out.
Text description of the diagram
Before each model call, the step limit of the model decides: 60 calls for DeepSeek-V3.2, 75 for Claude Opus 4.6. When the run reaches it, the run stops and its task goes to a person. Otherwise the provider answers and the money is spent. The span records the tokens, the cost and the price table’s version. Then two alerts watch. The idle alert fires at 10 idle calls in a row. The budget per solved task, per model and over 89 finished runs, names its most expensive runs. A person decides what to change.
Part 5: new input, end to end
Section titled “Part 5: new input, end to end”Later, a vendor offers Hana a stronger model: Claude Opus 4.6, at about 18 times the price per input token. Leo runs the whole check on the same agent’s runs with that model, with the settings he has. The command report does the steps of this page in one run.
-
Run the check with the default entry only:
Terminal window python -m agentcost report --data opus== opus: 445 runs of claude-opus-4-6 | settings: recommendation.json, the default entry (no entry for claude-opus-4-6)outcomes: solved 280, failed 92, timed_out 73 | cost US$281.16, US$0.6318 a run, US$1.0041 per solved tasksolved runs: model calls median 9, longest 63 | cost median US$0.2901, highest US$5.1244== the step limit: step limit 60cost US$269.87 (saved US$11.29, 4%) | timed-out runs stopped 9 of 73 | solved runs stopped (wrong blocks): 1cost per solved task: US$0.9673 (no limit: US$1.0041)== the idle alert: 10 idle steps in a row (an alert to a person, not a stop)runs it flags: 13 (solved 1, failed 1, timed_out 11)== the budgetBudget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on, with step limit 60)Cost per solved task of the windows: lowest US$0.3179, median US$0.9237, highest US$1.2267Over budget after finished run 20 (overfull-hbox__RhLQrUa): US$0.3290 per solved task (16 solved, US$5.2640)…Windows over the budget: 426 of 426 -
Check the price table on this model:
python -m agentcost check --data opus.recorded cost on 6,663 calls: US$291.93; our price table: US$281.16 (1,410 calls equal to the recorded cost, -3.7% in all)price table 2026-10-23: Claude Opus 4.6 US$5 in, US$0.5 cached in, US$25 out per 1M tokensThese are the list prices on Anthropic’s pricing page, checked on the date of the table’s version. They are assumptions with that date, as every price in this course.
-
Replay the three limits on the new runs:
Terminal window python -m agentcost sweep --data opus --max-steps 51,55,60,63,64,75python -m agentcost sweep --data opus --max-cost 0.1,1,3,6python -m agentcost sweep --data opus --max-idle 9,10,12step limit cost US$ saved solved wrong blocks timed out stopped per solved51 263.3778 17.7833 278 2 13 of 73 0.947455 266.4335 14.7277 279 1 12 of 73 0.955060 269.8707 11.2904 279 1 9 of 73 0.967363 271.5874 9.5738 280 0 6 of 73 0.970064 271.9707 9.1905 280 0 5 of 73 0.971375 275.1832 5.9780 280 0 3 of 73 0.9828cost limit US$ cost US$ saved solved wrong blocks timed out stopped per solved0.1 55.5477 225.6135 29 251 69 of 73 1.91541 208.1909 72.9703 261 19 33 of 73 0.79773 264.8661 16.2951 275 5 9 of 73 0.96316 280.8568 0.3044 280 0 1 of 73 1.0031idle limit cost US$ saved solved wrong blocks timed out stopped per solved9 272.3962 8.7650 278 2 12 of 73 0.979810 274.1643 6.9969 279 1 10 of 73 0.982712 277.6973 3.4639 280 0 7 of 73 0.9918
- DeepSeek-V3.2 (the design sample)
- Claude Opus 4.6 (the new input)
Text description of the chart
Two lines of solved runs stopped, for step limits from 10 to 100 in steps of 5. DeepSeek-V3.2: 113 at 10, 45 at 20, 15 at 30, 2 at 40, 1 at 50, and none from 55. Claude Opus 4.6: 120 at 10, 37 at 20, 16 at 30, 4 at 40, 2 at 50, 1 at 55 and 60, and none from 65.
What this tells us.
- The step limit almost travelled. Opus’s longest solved run made 63 calls, so 60 stopped one good run. A limit of 75 stops none.
- The cost limit did not travel at all. US$0.10 was safe on DeepSeek; on Opus it stops 251 of 280 solved runs. A money limit is a price, so it belongs to one model.
- The idle limit did not travel either. Opus’s solved runs had up to 11 idle steps in a row. Its longest streaks are honest waits: 32 empty commands while a compiler builds (
python -m agentcost idle --data opus). The idle rule stays an alert. - The budget broke at once: every window was over US$0.09, because Opus’s windows cost US$0.32 to US$1.23 per solved task.
- Our price table is 3.7% under the recorded cost. The records count cached reads but not cache writes, which cost 1.25 times the input price. For Opus, treat the project’s numbers as a floor.
Leo gives the model its own entry in recommendation.json. The budget is about 10% above Opus’s highest window with the new limit (US$1.2658):
"models": { "claude-opus-4-6": {"max_steps": 75, "budget_per_solved_usd": 1.40}}Run python -m agentcost report --data opus again:
== the step limit: step limit 75cost US$275.18 (saved US$5.98, 2%) | timed-out runs stopped 3 of 73 | solved runs stopped (wrong blocks): 0cost per solved task: US$0.9828 (no limit: US$1.0041)…Budget: US$1.4 per solved task over the last 89 finished runs (from 20 runs on, with step limit 75)Cost per solved task of the windows: lowest US$0.3179, median US$0.9373, highest US$1.2658Never over budget.Is the stronger model worth it? It solved 280 tasks against 176, and cost US$281.16 against US$13.60. Each of the 104 extra solved tasks cost (281.16 − 13.60) ÷ 104 = US$2.57. At these prices, the switch pays if an engineer’s hour costs more than that. That is Hana’s decision, and now she can make it with numbers.
The recommendation
Section titled “The recommendation”Leo writes it for Hana, in her words:
- Stop a run after 60 model calls with DeepSeek-V3.2, and after 75 with Claude Opus 4.6. On the recorded runs, 60 saved 16% and stopped no solved task.
- Set a budget per solved task for each model, over the last 89 finished runs: US$0.09 for DeepSeek-V3.2, US$1.40 for Claude Opus 4.6. When it fires, start with the runs that its alert lists.
- Alert on 10 idle calls in a row; do not stop on them. On Opus, some of them were honest waits.
- Keep the price table versioned, and check it against the provider’s bill each month. For Opus, our table misses the cache writes: it is 3.7% low.
- Check every setting again at each change of model, and after our first weeks of runs. These numbers come from a public benchmark.
The limits do not make a failed run succeed. The 183 failed runs cost US$5.81, and the step limit stopped only 6 of them. Failures are a quality question: evaluate the agent on your own tasks, as in the evaluation course.
Insights and limits
Section titled “Insights and limits”- Length decides the bill. Each call resends the history, so a run’s cost grows faster than its number of calls.
- The cache is the biggest price lever. A timestamp at the top of the prompt would break it, and multiply the bill with no other sign.
- Cost per solved task needs a second number. On its own, it praised a limit that stopped 15 good runs. Count the wrong blocks next to it.
- A setting belongs to one agent and one model. Larkfield’s limits stopped 146 of 176 good runs; DeepSeek’s money limit stopped 251 of Opus’s 280.
- A benchmark is not your backlog. 89 hard tasks in fresh containers, run many at a time. The window’s order is the benchmark’s, not a working day’s.
- The records have gaps. An idle step is a guess from digests. The model’s and the terminal’s time are not separate. Cache writes are not counted.
Responsible use
Section titled “Responsible use”- Name the data honestly. These are public benchmark runs of another team’s agent, priced with our table: a starting point, not a measurement of your agent.
- A stop must hand over well. A person needs the task, the run’s trace and the reason for the stop. Test that path before the first night.
- Keep code and secrets out of the records. A digest of each command is enough to find repeats. Your agent’s commands and outputs may hold code or keys: keep them out of spans, as in What to record and what not to.
- Credit the source. Keep the Apache-2.0 licence text and the notice of our changes with your copy, as the dataset card does.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This project shows what a platform or finance partner looks for. A record checks the costs, evidence sets the limits, the page counts both errors, and new input tests the choice.
What to put in the repository:
coding-agent-cost/ README.md the question, how to run it, the results table and the recommendation data/dataset.md the source, the licence, the checksums, the changes, the limits data/runs/ the fixed sample and the Apache-2.0 licence text prices.json the versioned price table recommendation.json your settings, per model agentcost/ data, prices, loops, limits, budget, the OpenTelemetry export tests/ 16 tests LICENSE MIT, for your code- Make a new Git repository from your project folder, and push it to GitHub.
.gitignorekeeps outwork/andresults/. - Write the README’s results table from your own outputs: the baseline, the three limits, your setting and the new-input check.
- Say in the README which numbers come from the records, which from your price table, and which are assumptions.
In two sentences, for an interview:
“I measured 445 recorded runs of an open coding agent per task, per call and per outcome, and chose its limits from the evidence. A step limit of 60 calls saved 16% with no solved task lost; another model needed 75.”
Go further: price the Opus runs with cache writes as an assumption, for example every new input token at 1.25 times the price. How close do you get to the recorded US$291.93? Or give a solved task a value, such as an engineer’s hour, and choose the step limit with the highest value minus cost. Or export the runs with --max-steps 60, and compare obs cost --by outcome before and after the limit.
Knowledge check
Section titled “Knowledge check”Check your understanding
In the design sample, a solved run costs US$0.0142 on average, but a solved task costs US$0.0773. Explain the difference, and say which number Hana should give finance.
Show the answer
A solved run’s cost counts only the runs that passed. The cost per solved task divides all the spending, US$13.60, by the 176 solved tasks. So the failed and timed-out runs are part of each success’s price. Finance should get the cost per solved task, with the price table’s version: the team pays for every run. With the step limit of 60, it falls to US$0.0653.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A step limit of 30 calls gave the lowest cost per solved task, US$0.0560. Why did Leo not choose it, and what measure did he use instead?
Show the answer
At 30 calls, the limit stopped 15 runs that would have passed their tests, and many expensive failed runs. The ratio fell while the team lost work worth far more than the cents saved. So Leo measured the US$ that each limit saves with no wrong block. The step limit won (51 calls, US$2.49 saved), and he added a margin: 60.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The team moves the agent to Claude Opus 4.6 and keeps the settings that worked on DeepSeek-V3.2. What does the new input show for the step limit, the cost limit and the budget?
Show the answer
The step limit of 60 stopped one solved run: Opus’s longest solved run made 63 calls, and 75 stops none. A cost limit of US$0.10 would stop 251 of 280 solved runs, because a money limit is a price. The budget of US$0.09 was over in every window. Each model needs its own entry: here 75 calls and US$1.40 per solved task.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Each call of a coding agent sends its whole history, so length decides the cost. The cache cut this agent’s bill by 6.3 times.
- A solved task cost US$0.0773; only 18% of the money bought solved tasks. Runs over 40 calls were 12% of the runs and 44% of the cost.
- Idle steps found the loops: up to 173 calls in a row with no command.
- Cost per solved task alone favours limits that stop good work. Measure the money saved with no wrong block; then add a margin.
- On another model, the step limit needed 75, and the money limit and the budget changed completely.
You can now turn another team’s recorded runs into a cost report and a set of limits you can defend. The last case study asks a different question about records. When a company had to defend what its AI assistant did, could its records have answered?