Skip to content
Français

The cost of a coding agent

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In the cost module, we split the cost of Larkfield's workflow by run, step, model and slice. A coding agent takes many more steps, and its cost depends even more on the path it takes. In this case study, we measure the cost of open coding-agent runs and propose a budget and a step limit.

What you will be able to do

Measure what a coding agent costs per task, per model call and per outcome, from 445 recorded runs of an open agent. Find the runs that loop or spend too much, and choose a step limit, an alert and a budget from the evidence. Then check your choice on 445 runs of the same agent with another model.

Case studyPortfolio piece

Licence
Apache-2.0
Time
About 2.5 hours
Runs in
VS Code, Python 3.12+
Skills used
  • Cost per run, step and outcome
  • A versioned price table
  • Prompt caching in the cost
  • Idle steps and loops
  • Step, cost and idle limits
  • Wrong blocks
  • One measure for a fair comparison
  • Budget per solved task
  • OpenTelemetry export
  • New input

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

This case study is a project folder, not a notebook. It is small (about 1 MB), it uses only Python’s standard library, and it runs in the course project’s virtual environment. At one step, the course’s own obs tool reads the case study’s runs, so the folder sits next to your course project.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the cost module most:

The page teaches one new idea: every call of a coding agent sends the whole history again, and the provider’s cache changes its price.

What you need on your computer

  • The course project at the end of the course, in ~/projects/observable-workflow, with its virtual environment. If yours is broken, download the snapshot end-of-m08 and follow its README.
  • About 2 MB of disk. No account, no key and no money: every token count is a recording.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, not run.

Hana (she) leads a small software team (made up for this case study). The team has a backlog of maintenance tasks: a broken build, a script to port, a crash to find. Leo (he), a platform engineer, wants a coding agent to work on them at night. The agent gets a terminal in a fresh container, a task and a time limit. It asks a model what to do, runs the commands, reads the output and asks again. In the morning, a test says whether the task is done.

Hana must give finance a number before the trial. She asks Leo three questions:

  1. What does a task cost, and what does a solved task cost?
  2. Which runs waste money, and how would we see them?
  3. What stops a run that goes wrong at 3 a.m., and what budget do we watch?

Each wrong answer has a price. A limit that is too loose lets a loop spend all night. A limit that is too tight stops a run that was about to pass its test. Then an engineer does that task by hand, which costs far more than any model call.

The team has no runs of its own yet. So Leo measures recorded runs of an open coding agent on public tasks, with the course’s cost tools. The team’s own runs must check the numbers after launch.

You will build a cost report and a limits recommendation for a coding agent, from recorded runs. You finish the cost of a step, check it against the record, and explore where the money goes. Then you finish the limits, compare three of them under one measure, and choose a step limit, an alert and a budget. Last, you run the whole check on the same agent with another model.

  1. Download the starter project from the box at the top of this page. Unzip it into ~/projects, so that the folder coding-agent-cost is next to observable-workflow.

  2. In VS Code, choose File > Open Folder, and open coding-agent-cost. Then open a terminal: Terminal > New Terminal.

  3. Activate the course project’s virtual environment:

    Terminal window
    source ../observable-workflow/.venv/bin/activate
  4. Run the tests: python -m pytest -q. The starter has two parts for you to finish, so six tests fail:

    FAILED tests/test_cost.py::test_a_step_by_hand - assert 0.0030100000000000005...
    FAILED tests/test_cost.py::test_our_price_table_gives_every_recorded_cost - A...
    FAILED tests/test_limits.py::test_a_step_limit_of_4_lets_4_calls_through - As...
    FAILED tests/test_limits.py::test_a_stopped_solved_run_is_a_wrong_block - Ass...
    FAILED tests/test_limits.py::test_a_cost_limit_is_checked_before_the_call_so_the_run_ends_over_it
    FAILED tests/test_limits.py::test_the_measure_counts_both_errors - assert (1 ...
    6 failed, 10 passed in 0.90s

    One test reads the case study’s runs with the course’s obs. If your course project is in another folder, set OBS_PROJECT to that folder, or pytest skips the test.

The finished project is in the same box, to compare with your work. Every output on this page comes from it.

The data is the published runs of Terminal-Bench 2.0, an open benchmark of AI agents that work in a terminal. Its 89 tasks are engineering jobs: build a compiler, fix a crash, port a script, train a small model. A test decides if the task is done. The benchmark’s own agent, Terminus 2, ran every task 5 times with many models.

Leo needed the tokens of each model call, the commands, how each run ended, and some real loops. Open SWE-agent, SWE-smith and OpenHands runs have no token counts, or one total per run, as the agents governance course’s case study found. Terminal-Bench records the tokens of every call and the cached part. For some models, it also records the cost that the runner computed.

The design sample is 445 runs of Terminus 2 with DeepSeek-V3.2, a low-price model: we choose the limits on it. The 445 runs of the same agent with Claude Opus 4.6 are the new input for the end. One run has these fields:

Field What it holds
id, task the run (task name and a trial code); one of 89 tasks, each run 5 times
reward, exception how it ended: the tests passed (solved), failed (failed), or the agent hit the task’s time limit first (timed_out)
steps[].prompt, cached, completion one model call: input tokens (cached ones included), the cached part, output tokens
steps[].cost the cost that the runner’s software recorded for the call, when it did
steps[].programs, digests each command’s first word, and a short SHA-256 digest of its exact keystrokes

agentcost/data.py checks each file’s SHA-256 before it reads it. After Part 1 below, python -m agentcost check prints:

terminus2-claude-opus-4.6.jsonl.gz SHA-256 ok
terminus2-deepseek-v3.2.jsonl.gz SHA-256 ok
deepseek: 445 runs of deepseek-chat (89 tasks), 10,966 model calls, 295,559,649 tokens in (286,105,280 cached), 7,001,540 out
outcomes: solved 176, failed 183, timed_out 85, error 1
model calls with 0 input tokens recorded: 0
recorded cost on 10,966 calls: US$13.60; our price table: US$13.60 (10,966 calls equal to the recorded cost, +0.0% in all)
price table 2026-10-23: DeepSeek-V3.2 (deepseek-chat) US$0.28 in, US$0.028 cached in, US$0.42 out per 1M tokens

What this tells us.

  • The agent solved 176 of 445 runs (39.6%) of hard tasks. One run ended in an error of the benchmark; we count it as error.
  • 97% of the input tokens came from the cache. Keep that number in mind: it decides most of the bill.
  • Our price table gives every recorded cost exactly, on all 10,966 calls. And a surprise: DeepSeek’s price page no longer lists this model. prices.json keeps the prices these runs paid, under its own version. A provider’s page moves on; an old report must keep its numbers.
  • No call records 0 input tokens. Another model’s file on the same leaderboard records 0 for every call. A recorded zero is not a free run, so we did not use that file.

The unit is one run: one attempt of the agent at one task. The outcome is the test’s result, and “cost per solved task” is this case study’s cost per resolved ticket.

The price is the project’s price table, prices.json, version 2026-10-23. Per million tokens: US$0.28 input, US$0.028 cached input and US$0.42 output. These prices are assumptions with their date: check your provider’s current prices.

The two errors of a limit. A limit that stops a run which would have solved its task is a wrong block. A limit that never stops a loop lets the money go. We choose the measure in Part 3, after we see where the money goes.

The replay rule. A recording shows what the agent did when nothing stopped it. When a limit stops a run, the replay ends there: the run is not solved, and its task goes to a person.

A new idea: every call pays for the whole history

Section titled “A new idea: every call pays for the whole history”

Think of a meeting where, before each new point, someone reads all the minutes aloud from the start. The first point is quick. By the twentieth, most of the meeting is the reading. A helper with a copy of the old minutes makes it faster, but the copy is still read.

A coding agent’s model calls work like that meeting. A model has no memory between calls. So each call sends the instructions, the task and everything so far: every earlier answer and every terminal output. The input of a call grows with each step, and a long run pays for its history again and again. Many providers keep a prompt cache. When the start of a request is the same as a recent one, they serve that part from the cache and charge less for it. DeepSeek charges a tenth of the input price for a cached token. A cached token is a cheaper token, not a free one.

A column of boxes. Call 1 sends the instructions and the task: 830 tokens in. The terminal runs the commands, and its output joins the history. Call 2 sends call 1, its answer and the output: the history grows. Every call sends the whole history again. Call 135: 115,291 tokens in, 115,136 of them from the cache. What call 135 costs: 155 new tokens times US$0.28 per million, plus 115,136 cached tokens times US$0.028 per million, plus 215 output tokens times US$0.42 per million: together US$0.0034.
Each model call sends the whole history again. The cache serves the repeated part at a tenth of the input price. Real numbers from one run of the design sample.
Text description of the diagram

The first call of the run sends 830 tokens. Each later call sends all the earlier ones, their answers and the terminal’s output. At call 135, the input is 115,291 tokens, and 115,136 of them come from the cache. That call costs US$0.0034. The new input is 155 tokens at US$0.28 per million. The cached input is 115,136 tokens at US$0.028 per million. The output is 215 tokens at US$0.42 per million.

Count call 135 by hand, in millionths of a dollar. The new input costs 155 × 0.28 = 43. The cached input costs 115,136 × 0.028 = 3,224. The output costs 215 × 0.42 = 90. Almost all of the US$0.0034 is the cached history. Without the cache, the call would cost US$0.0324, ten times more.

  1. Run python -m agentcost check in the starter. Its sixth line shows the problem:

    recorded cost on 10,966 calls: US$13.60; our price table: US$85.70 (27 calls equal to the recorded cost, +530.2% in all)

    The starter charges every input token at the full price: the bill without a cache.

  2. Open agentcost/prices.py and find TODO (Part 1) in step_cost. Replace its return line with:

    cached = step.cached_tokens if cache else 0
    uncached = step.prompt_tokens - cached
    return (uncached * p["input"] + cached * p["cached_input"] + step.completion_tokens * p["output"]) / 1e6

    prompt_tokens includes the cached tokens, as gen_ai.usage.input_tokens does in the GenAI conventions.

  3. Run python -m agentcost check again. The line now says 10,966 calls equal to the recorded cost, as above. The two tests of Part 1 pass.

The project can write the runs as OpenTelemetry traces, in the file format of the course’s Collector. Each run gets an invoke_agent span, a chat span per model call and an execute_tool span per command. The token names are the GenAI conventions’ (gen_ai.usage.input_tokens, gen_ai.usage.cache_read.input_tokens). The cost, the step and the outcome use the course project’s names, so obs reads them unchanged.

  1. Write the traces: python -m agentcost export.

    Wrote 445 traces to work/telemetry/deepseek/traces.jsonl.gz (1.4 MB, 45,095 spans), no limit, price table 2026-10-23
    Read them with the course's obs, in ../observable-workflow: python -m obs cost --by outcome --data ../coding-agent-cost/work/telemetry/deepseek
  2. In a second terminal, in ~/projects/observable-workflow with its environment active, run the command that the export printed:

    outcome runs resolved cost US$ per run per resolved success
    error 1 0 0.0374 0.03738 - 0.0%
    failed 183 0 5.8051 0.03172 - 0.0%
    resolved 176 176 2.4986 0.01420 0.01420 100.0%
    timed_out 85 0 5.2577 0.06186 - 0.0%
    total 445 176 13.5988 0.03056 0.07727 39.6%
  3. Read one short run as a waterfall: python -m obs trace openssl-selfsigned-cert__4n5TDfS --data ../coding-agent-cost/work/telemetry/deepseek. It shows 8 chat spans with their costs, and one span per command.

What this tells us. A tool written for another workflow finds the same total to the cent: US$13.5988, and US$0.07727 per solved task. Record once, with standard names, and any tool can answer. But obs trace also prints Flag no_action: closed as resolved with no write and no reply sent. That rule is Larkfield’s: a resolved ticket needs a refund or a reply. A flag rule belongs to one workflow, as a limit does.

Terminal window
python -m agentcost cost --by outcome
outcome runs solved steps cost US$ share per run per solved success
solved 176 176 14.5 2.4986 18% 0.0142 0.0142 100.0%
failed 183 0 21 5.8051 43% 0.0317 - 0.0%
timed_out 85 0 30 5.2577 39% 0.0619 - 0.0%
error 1 0 33 0.0374 0% 0.0374 - 0.0%
total 445 176 19 13.5988 100% 0.0306 0.0773 39.6%
(steps = the median number of model calls per run)

What this tells us. A solved run costs US$0.0142, but a solved task costs US$0.0773: the failed and timed-out runs are part of its price. Only 18% of the money bought solved tasks. A timed-out run costs four times a solved run. With --no-cache, the total would be US$85.70 instead of US$13.60: the cache cuts the bill by 6.3 times.

445 recorded runs of Terminus 2 with DeepSeek-V3.2 (Terminal-Bench 2.0). Cost: recorded tokens × the project's price table, version 2026-10-23.
Text description of the chart

Seven bars, by the run’s number of model calls. Runs of 1 to 10 calls: 87 runs, 63 solved, 3% of the spending. 11 to 20 calls: 157 runs, 17%. 21 to 30: 19%. 31 to 40: 17%. 41 to 50: 24 runs, 1 solved, 11%. 51 to 100: 25 runs, 1 solved, 16%. More than 100: 5 runs, none solved, 17%.

What this tells us. Short runs are cheap and usually succeed: 63 of the 87 runs with at most 10 calls. After 40 calls, the agent almost never succeeds: 2 of 54 runs. Those 54 runs are 12% of the runs, but 44% of the spending. The run’s length decides where the money goes.

The most expensive run fixes a crash in a language’s runtime (fix-ocaml-gc). All 5 runs of that task failed, and they cost 16% of the sample.

Terminal window
python -m agentcost steps fix-ocaml-gc__PB7o5ME --every 30
fix-ocaml-gc__PB7o5ME: task fix-ocaml-gc, deepseek-chat, timed_out, 307 model calls, US$0.8696
step tokens in cached out cost US$ so far kind commands
1 830 768 265 0.00015 0.0002 work ls cat
30 26,334 25,216 215 0.00111 0.0205 work sed grep grep
60 62,531 62,080 421 0.00204 0.0739 work grep find
90 86,831 86,528 268 0.00262 0.1454 work grep awk grep
120 103,712 103,424 236 0.00308 0.2310 work awk cat
150 116,702 116,672 36 0.00329 0.3306 none -
180 118,540 118,464 36 0.00335 0.4303 none -
210 120,377 120,320 36 0.00340 0.5316 none -
240 122,215 122,176 36 0.00345 0.6344 none -
270 124,052 123,968 36 0.00351 0.7387 none -
300 125,890 125,824 36 0.00356 0.8447 none -
307 126,316 126,272 41 0.00357 0.8696 none -
  • Tokens in
  • Tokens in from the cache
fix-ocaml-gc__PB7o5ME, Terminus 2 with DeepSeek-V3.2: recorded token counts. Each call sends the whole history again; from call 135 the model sends no command.
Text description of the chart

Two lines over the 307 model calls of one run. The input grows from 830 tokens at the first call to about 115,000 at call 135, then slowly to 126,316 at the last call. The cached tokens follow the same line, a little below it: almost all the input comes from the cache.

What this tells us. The first call cost US$0.00015; the last one US$0.00357, 24 times more, for a similar answer. From call 135, the model answered 173 times in a row without a command: the column kind says none. We read the original record of this run. There, the model writes that it cannot fix the bug. Then it repeats a summary until the task’s time limit. Those 173 calls cost US$0.59, 68% of the run. The cache kept it cheap: without it, this one run would have cost US$8.28.

The course’s loop rule cannot see this pattern. It counts the same tool with the same arguments, and here the model calls no tool at all. The project calls a step idle when the terminal gets nothing new to do. The answer has no command, or only empty commands (the agent waits), or the same commands as the step before. One idle step is often normal, for example a wait for an install. A long streak of idle steps is a loop.

Terminal window
python -m agentcost idle
outcome runs longest idle streak: median max
solved 176 2 8
failed 183 2 8
timed_out 85 0 173
error 1 2 2
The 8 runs with the longest idle streak:
run outcome steps cost US$ streak from kind streak US$
fix-ocaml-gc__PB7o5ME timed_out 307 0.8696 173 135 none 0.5917
fix-ocaml-gc__WB7L8Nf timed_out 240 0.6388 154 83 none 0.4831
custom-memory-heap-crash__exMxSPi timed_out 168 0.1823 85 33 none 0.0840
fix-ocaml-gc__G9if5xy timed_out 166 0.3827 55 69 none 0.1846
crack-7z-hash__Sykdxfk timed_out 63 0.0462 17 47 none 0.0144
path-tracing-reverse__nEW2gSJ timed_out 61 0.1528 14 48 none 0.0442
train-fasttext__bJf9npe timed_out 85 0.0886 10 32 wait 0.0088
rstan-to-pystan__fDESBRy timed_out 55 0.0800 10 33 wait 0.0166

What this tells us. No solved run had more than 8 idle steps in a row. Six runs had 14 or more, all timed out, most of them with no command at all. So idle streaks separate loops from work on this agent. We will see if that holds on another model.

A baseline: no limit, then another agent’s limits

Section titled “A baseline: no limit, then another agent’s limits”

The baseline is the design sample with no limit: US$13.60, and US$0.0773 per solved task. Leo’s first idea is to copy the limits of Larkfield’s support workflow from the course: 8 model calls and US$0.05 a run.

Terminal window
python -m agentcost limits --max-steps 8 --max-cost 0.05
deepseek (deepseek-chat): 445 runs | step limit 8, cost limit US$0.05 (checked before each model call)
recorded outcome runs stopped cost US$ of US$
solved 176 146 0.8986 2.4986
failed 183 172 1.0602 5.8051
timed_out 85 84 0.4587 5.2577
error 1 1 0.0054 0.0374
cost US$2.4229 (no limit: US$13.5988, saved US$11.1759) | solved 30 of 176 | solved runs stopped (wrong blocks): 146
cost per solved task: US$0.0808 (no limit: US$0.0773)

What this tells us. It “saves” US$11.18 by stopping almost everything: 146 of 176 solved runs. And the cost per solved task got worse, US$0.0808, because the money spent on stopped runs bought nothing. Larkfield’s workflow makes 1 to 5 calls a ticket; this agent’s median solved run makes 15. A limit belongs to one agent, and it comes from that agent’s runs.

agentcost/limits.py replays every run under a Limits object, call by call. The check is the one of the agents course’s loop. Before the next call, stop if the run has made max_steps calls or if its cost so far is over max_cost_usd. The starter does not have these two checks.

  1. Open agentcost/limits.py and find TODO (Part 2) in replay. Replace the comment with:

    if limits.max_steps and out.steps >= limits.max_steps:
    out.stopped_by = "max_steps"
    break
    if limits.max_cost_usd and out.cost > limits.max_cost_usd:
    out.stopped_by = "cost"
    break

    The check comes before the call is paid. A cost limit cannot know what the next call will cost, so a run can end a little over it.

  2. Run the tests again: python -m pytest -q. The last line says 16 passed.

The tiny run in tests/test_limits.py shows the overshoot: six calls of US$0.28 each, and a cost limit of US$0.60. After two calls the run has spent US$0.56, under the limit, so the third call goes out. The check stops the fourth call, and the run ends at US$0.84.

Part 3: a fair comparison under one measure

Section titled “Part 3: a fair comparison under one measure”

First, a trap. sweep replays the sample with each setting of a limit. Look at the cost per solved task, the last column:

Terminal window
python -m agentcost sweep --max-steps 30,40,45,50,51,55,60,75
deepseek (deepseek-chat), 445 runs | no limit: US$13.5988, 176 solved, US$0.0773 per solved task
step limit cost US$ saved solved wrong blocks timed out stopped per solved
30 9.0125 4.5863 161 15 42 of 85 0.0560
40 10.3158 3.2830 174 2 25 of 85 0.0593
45 10.7407 2.8581 175 1 20 of 85 0.0614
50 11.0477 2.5511 175 1 15 of 85 0.0631
51 11.1078 2.4910 176 0 13 of 85 0.0631
55 11.3085 2.2903 176 0 12 of 85 0.0643
60 11.4840 2.1148 176 0 10 of 85 0.0653
75 11.8631 1.7357 176 0 5 of 85 0.0674

The cost per solved task falls as the limit gets tighter: at 30 calls, it is the lowest, US$0.0560. But that setting stops 15 runs that would have passed their tests. It also stops expensive failed runs, so the ratio looks better while the team loses work.

The ratio leaves out what a solved task is worth: an engineer’s hour or more, against about 8 cents of model calls. One wrong block costs more than any limit saves on this sample. So Leo’s measure is: the US$ that a limit saves with no wrong block. Each limit gets its tightest setting that stops no solved run of the design sample. Then we compare what each setting saves.

Run the other two limits the same way. --max-idle stops a run after that many idle steps in a row:

Terminal window
python -m agentcost sweep --max-cost 0.04,0.06,0.08,0.1
python -m agentcost sweep --max-idle 5,8,9,10,12
cost limit US$ cost US$ saved solved wrong blocks timed out stopped per solved
0.04 9.3128 4.2860 170 6 26 of 85 0.0548
0.06 10.4507 3.1481 175 1 15 of 85 0.0597
0.08 11.0902 2.5086 175 1 9 of 85 0.0634
0.1 11.4110 2.1878 176 0 7 of 85 0.0648
idle limit cost US$ saved solved wrong blocks timed out stopped per solved
5 11.9181 1.6807 174 2 11 of 85 0.0685
8 12.0311 1.5678 174 2 8 of 85 0.0691
9 12.0901 1.5087 176 0 8 of 85 0.0687
10 12.1066 1.4922 176 0 8 of 85 0.0688
12 12.2200 1.3789 176 0 6 of 85 0.0694
Limit (checked before each call) Tightest setting with no wrong block US$ saved of US$13.60 Timed-out runs stopped
Step limit 51 model calls 2.49 (18%) 13 of 85
Cost limit US$0.10 a run 2.19 (16%) 7 of 85
Idle limit 9 idle steps in a row 1.51 (11%) 8 of 85

What this tells us.

  • The step limit saves the most, and its setting is easy to read: the longest solved run made 51 calls.
  • The cost limit is close, but its setting is a price: the most expensive solved run, at this model’s price.
  • The idle limit saves the least, because a loop must start before it acts. But it is the only one that says why a run wastes money: a good alert.
  • No limit stops many of the 85 timed-out runs. Most were slow, not long, and the benchmark’s own time limit already stops them.

Part 4: the operating point and the budget

Section titled “Part 4: the operating point and the budget”

A setting at exactly 51 calls is the length of one run: the longest success of one sample. The next longer good run would be a wrong block. Leo chooses 60 calls a run, about 18% above it.

Terminal window
python -m agentcost limits --max-steps 60
deepseek (deepseek-chat): 445 runs | step limit 60 (checked before each model call)
recorded outcome runs stopped cost US$ of US$
solved 176 0 2.4986 2.4986
failed 183 6 5.5454 5.8051
timed_out 85 10 3.4026 5.2577
error 1 0 0.0374 0.0374
cost US$11.4840 (no limit: US$13.5988, saved US$2.1148) | solved 176 of 176 | solved runs stopped (wrong blocks): 0
cost per solved task: US$0.0653 (no limit: US$0.0773)

The margin costs US$0.38 of savings: US$2.11 instead of US$2.49. The 307-call run now stops after 60 calls, before its loop starts.

A step limit stops one run. It does not tell Hana when the agent as a whole gets more expensive, after a new kind of task, model or prompt. That is the budget alert of Budgets, alerts and the expensive run. Its window here is 89 finished runs, about one run of each task. First, try a budget of US$0.09 per solved task on the runs as recorded, with no limit:

Terminal window
python -m agentcost budget --budget 0.09
Budget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on)
Cost per solved task of the windows: lowest US$0.0469, median US$0.0762, highest US$0.1019
Over budget after finished run 206 (chess-best-move__XimTT7i): US$0.0905 per solved task (33 solved, US$2.9849)
The 5 runs that cost most in that window (40% of its cost):
run outcome steps cost US$ idle streak
fix-ocaml-gc__PB7o5ME timed_out 307 0.8696 173
feal-linear-cryptanalysis__e9Scv8w failed 43 0.0871 2
path-tracing-reverse__vLssimd failed 45 0.0849 1
schemelike-metacircular-eval__H62ET7V failed 41 0.0760 1
train-fasttext__MvTjpMW timed_out 70 0.0725 1
Windows over the budget: 40 of 426

The alert names its runs, and the first one is the loop: 307 calls and an idle streak of 173. Now the same budget, with the step limit in place:

Terminal window
python -m agentcost budget --budget 0.09 --max-steps 60
Budget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on, with step limit 60)
Cost per solved task of the windows: lowest US$0.0469, median US$0.0626, highest US$0.0815
Never over budget.

What this tells us. With the step limit, the most expensive window costs US$0.0815 per solved task. A budget of US$0.09 sits about 10% above it. It stays quiet on normal work, and fires when a stretch costs more than any window of this sample. The limit stops one run; the budget watches the agent.

Write the choice in recommendation.json. Find the default entry, the team’s settings for any model, and give it your values:

"default": {"max_steps": 60, "budget_per_solved_usd": 0.09},

The file also sets "idle_alert_streak": 10: a message to the agent’s owner when a run makes 10 idle calls in a row. It is an alert, not a stop. On the design sample, it flags 8 runs, all timed out.

A flow from the top. The agent wants its next model call. A diamond: the step limit of this model, 60 calls for DeepSeek-V3.2, 75 for Claude Opus 4.6. Reached: stop the run before the call, and the task goes to a person (green box). Not reached: the provider answers and the money is spent. The chat span records the tokens, the cost and the price table's version. From the span, two dashed orange boxes: an idle alert at 10 idle calls in a row, a message to the owner with the run; and a budget per solved task, per model, over the last 89 finished runs, US$0.09 or US$1.40, whose alert names the runs that cost the most in its window. Both lead to a person who decides: fix the agent, change a limit, or change the model.
The recommendation: one stop before the call (green), and two alerts after the money is spent (orange, dashed). Each setting belongs to one model.
Text description of the diagram

Before each model call, the step limit of the model decides: 60 calls for DeepSeek-V3.2, 75 for Claude Opus 4.6. When the run reaches it, the run stops and its task goes to a person. Otherwise the provider answers and the money is spent. The span records the tokens, the cost and the price table’s version. Then two alerts watch. The idle alert fires at 10 idle calls in a row. The budget per solved task, per model and over 89 finished runs, names its most expensive runs. A person decides what to change.

Later, a vendor offers Hana a stronger model: Claude Opus 4.6, at about 18 times the price per input token. Leo runs the whole check on the same agent’s runs with that model, with the settings he has. The command report does the steps of this page in one run.

  1. Run the check with the default entry only:

    Terminal window
    python -m agentcost report --data opus
    == opus: 445 runs of claude-opus-4-6 | settings: recommendation.json, the default entry (no entry for claude-opus-4-6)
    outcomes: solved 280, failed 92, timed_out 73 | cost US$281.16, US$0.6318 a run, US$1.0041 per solved task
    solved runs: model calls median 9, longest 63 | cost median US$0.2901, highest US$5.1244
    == the step limit: step limit 60
    cost US$269.87 (saved US$11.29, 4%) | timed-out runs stopped 9 of 73 | solved runs stopped (wrong blocks): 1
    cost per solved task: US$0.9673 (no limit: US$1.0041)
    == the idle alert: 10 idle steps in a row (an alert to a person, not a stop)
    runs it flags: 13 (solved 1, failed 1, timed_out 11)
    == the budget
    Budget: US$0.09 per solved task over the last 89 finished runs (from 20 runs on, with step limit 60)
    Cost per solved task of the windows: lowest US$0.3179, median US$0.9237, highest US$1.2267
    Over budget after finished run 20 (overfull-hbox__RhLQrUa): US$0.3290 per solved task (16 solved, US$5.2640)
    …
    Windows over the budget: 426 of 426
  2. Check the price table on this model: python -m agentcost check --data opus.

    recorded cost on 6,663 calls: US$291.93; our price table: US$281.16 (1,410 calls equal to the recorded cost, -3.7% in all)
    price table 2026-10-23: Claude Opus 4.6 US$5 in, US$0.5 cached in, US$25 out per 1M tokens

    These are the list prices on Anthropic’s pricing page, checked on the date of the table’s version. They are assumptions with that date, as every price in this course.

  3. Replay the three limits on the new runs:

    Terminal window
    python -m agentcost sweep --data opus --max-steps 51,55,60,63,64,75
    python -m agentcost sweep --data opus --max-cost 0.1,1,3,6
    python -m agentcost sweep --data opus --max-idle 9,10,12
    step limit cost US$ saved solved wrong blocks timed out stopped per solved
    51 263.3778 17.7833 278 2 13 of 73 0.9474
    55 266.4335 14.7277 279 1 12 of 73 0.9550
    60 269.8707 11.2904 279 1 9 of 73 0.9673
    63 271.5874 9.5738 280 0 6 of 73 0.9700
    64 271.9707 9.1905 280 0 5 of 73 0.9713
    75 275.1832 5.9780 280 0 3 of 73 0.9828
    cost limit US$ cost US$ saved solved wrong blocks timed out stopped per solved
    0.1 55.5477 225.6135 29 251 69 of 73 1.9154
    1 208.1909 72.9703 261 19 33 of 73 0.7977
    3 264.8661 16.2951 275 5 9 of 73 0.9631
    6 280.8568 0.3044 280 0 1 of 73 1.0031
    idle limit cost US$ saved solved wrong blocks timed out stopped per solved
    9 272.3962 8.7650 278 2 12 of 73 0.9798
    10 274.1643 6.9969 279 1 10 of 73 0.9827
    12 277.6973 3.4639 280 0 7 of 73 0.9918
  • DeepSeek-V3.2 (the design sample)
  • Claude Opus 4.6 (the new input)
Terminus 2 on the same 89 tasks, 5 runs each: 445 runs of DeepSeek-V3.2 (the design sample) and 445 runs of Claude Opus 4.6 (the new input), replayed with a step limit checked before each call.
Text description of the chart

Two lines of solved runs stopped, for step limits from 10 to 100 in steps of 5. DeepSeek-V3.2: 113 at 10, 45 at 20, 15 at 30, 2 at 40, 1 at 50, and none from 55. Claude Opus 4.6: 120 at 10, 37 at 20, 16 at 30, 4 at 40, 2 at 50, 1 at 55 and 60, and none from 65.

What this tells us.

  • The step limit almost travelled. Opus’s longest solved run made 63 calls, so 60 stopped one good run. A limit of 75 stops none.
  • The cost limit did not travel at all. US$0.10 was safe on DeepSeek; on Opus it stops 251 of 280 solved runs. A money limit is a price, so it belongs to one model.
  • The idle limit did not travel either. Opus’s solved runs had up to 11 idle steps in a row. Its longest streaks are honest waits: 32 empty commands while a compiler builds (python -m agentcost idle --data opus). The idle rule stays an alert.
  • The budget broke at once: every window was over US$0.09, because Opus’s windows cost US$0.32 to US$1.23 per solved task.
  • Our price table is 3.7% under the recorded cost. The records count cached reads but not cache writes, which cost 1.25 times the input price. For Opus, treat the project’s numbers as a floor.

Leo gives the model its own entry in recommendation.json. The budget is about 10% above Opus’s highest window with the new limit (US$1.2658):

"models": {
"claude-opus-4-6": {"max_steps": 75, "budget_per_solved_usd": 1.40}
}

Run python -m agentcost report --data opus again:

== the step limit: step limit 75
cost US$275.18 (saved US$5.98, 2%) | timed-out runs stopped 3 of 73 | solved runs stopped (wrong blocks): 0
cost per solved task: US$0.9828 (no limit: US$1.0041)
…
Budget: US$1.4 per solved task over the last 89 finished runs (from 20 runs on, with step limit 75)
Cost per solved task of the windows: lowest US$0.3179, median US$0.9373, highest US$1.2658
Never over budget.

Is the stronger model worth it? It solved 280 tasks against 176, and cost US$281.16 against US$13.60. Each of the 104 extra solved tasks cost (281.16 − 13.60) ÷ 104 = US$2.57. At these prices, the switch pays if an engineer’s hour costs more than that. That is Hana’s decision, and now she can make it with numbers.

Leo writes it for Hana, in her words:

  1. Stop a run after 60 model calls with DeepSeek-V3.2, and after 75 with Claude Opus 4.6. On the recorded runs, 60 saved 16% and stopped no solved task.
  2. Set a budget per solved task for each model, over the last 89 finished runs: US$0.09 for DeepSeek-V3.2, US$1.40 for Claude Opus 4.6. When it fires, start with the runs that its alert lists.
  3. Alert on 10 idle calls in a row; do not stop on them. On Opus, some of them were honest waits.
  4. Keep the price table versioned, and check it against the provider’s bill each month. For Opus, our table misses the cache writes: it is 3.7% low.
  5. Check every setting again at each change of model, and after our first weeks of runs. These numbers come from a public benchmark.

The limits do not make a failed run succeed. The 183 failed runs cost US$5.81, and the step limit stopped only 6 of them. Failures are a quality question: evaluate the agent on your own tasks, as in the evaluation course.

  • Length decides the bill. Each call resends the history, so a run’s cost grows faster than its number of calls.
  • The cache is the biggest price lever. A timestamp at the top of the prompt would break it, and multiply the bill with no other sign.
  • Cost per solved task needs a second number. On its own, it praised a limit that stopped 15 good runs. Count the wrong blocks next to it.
  • A setting belongs to one agent and one model. Larkfield’s limits stopped 146 of 176 good runs; DeepSeek’s money limit stopped 251 of Opus’s 280.
  • A benchmark is not your backlog. 89 hard tasks in fresh containers, run many at a time. The window’s order is the benchmark’s, not a working day’s.
  • The records have gaps. An idle step is a guess from digests. The model’s and the terminal’s time are not separate. Cache writes are not counted.
  • Name the data honestly. These are public benchmark runs of another team’s agent, priced with our table: a starting point, not a measurement of your agent.
  • A stop must hand over well. A person needs the task, the run’s trace and the reason for the stop. Test that path before the first night.
  • Keep code and secrets out of the records. A digest of each command is enough to find repeats. Your agent’s commands and outputs may hold code or keys: keep them out of spans, as in What to record and what not to.
  • Credit the source. Keep the Apache-2.0 licence text and the notice of our changes with your copy, as the dataset card does.

This project shows what a platform or finance partner looks for. A record checks the costs, evidence sets the limits, the page counts both errors, and new input tests the choice.

What to put in the repository:

coding-agent-cost/
README.md the question, how to run it, the results table and the recommendation
data/dataset.md the source, the licence, the checksums, the changes, the limits
data/runs/ the fixed sample and the Apache-2.0 licence text
prices.json the versioned price table
recommendation.json your settings, per model
agentcost/ data, prices, loops, limits, budget, the OpenTelemetry export
tests/ 16 tests
LICENSE MIT, for your code
  1. Make a new Git repository from your project folder, and push it to GitHub. .gitignore keeps out work/ and results/.
  2. Write the README’s results table from your own outputs: the baseline, the three limits, your setting and the new-input check.
  3. Say in the README which numbers come from the records, which from your price table, and which are assumptions.

In two sentences, for an interview:

“I measured 445 recorded runs of an open coding agent per task, per call and per outcome, and chose its limits from the evidence. A step limit of 60 calls saved 16% with no solved task lost; another model needed 75.”

Go further: price the Opus runs with cache writes as an assumption, for example every new input token at 1.25 times the price. How close do you get to the recorded US$291.93? Or give a solved task a value, such as an engineer’s hour, and choose the step limit with the highest value minus cost. Or export the runs with --max-steps 60, and compare obs cost --by outcome before and after the limit.

Check your understanding

In the design sample, a solved run costs US$0.0142 on average, but a solved task costs US$0.0773. Explain the difference, and say which number Hana should give finance.

Show the answer

A solved run’s cost counts only the runs that passed. The cost per solved task divides all the spending, US$13.60, by the 176 solved tasks. So the failed and timed-out runs are part of each success’s price. Finance should get the cost per solved task, with the price table’s version: the team pays for every run. With the step limit of 60, it falls to US$0.0653.

Check your understanding

A step limit of 30 calls gave the lowest cost per solved task, US$0.0560. Why did Leo not choose it, and what measure did he use instead?

Show the answer

At 30 calls, the limit stopped 15 runs that would have passed their tests, and many expensive failed runs. The ratio fell while the team lost work worth far more than the cents saved. So Leo measured the US$ that each limit saves with no wrong block. The step limit won (51 calls, US$2.49 saved), and he added a margin: 60.

Check your understanding

The team moves the agent to Claude Opus 4.6 and keeps the settings that worked on DeepSeek-V3.2. What does the new input show for the step limit, the cost limit and the budget?

Show the answer

The step limit of 60 stopped one solved run: Opus’s longest solved run made 63 calls, and 75 stops none. A cost limit of US$0.10 would stop 251 of 280 solved runs, because a money limit is a price. The budget of US$0.09 was over in every window. Each model needs its own entry: here 75 calls and US$1.40 per solved task.

Recap

  • Each call of a coding agent sends its whole history, so length decides the cost. The cache cut this agent’s bill by 6.3 times.
  • A solved task cost US$0.0773; only 18% of the money bought solved tasks. Runs over 40 calls were 12% of the runs and 44% of the cost.
  • Idle steps found the loops: up to 173 calls in a row with no command.
  • Cost per solved task alone favours limits that stop good work. Measure the money saved with no wrong block; then add a margin.
  • On another model, the step limit needed 75, and the money limit and the budget changed completely.

You can now turn another team’s recorded runs into a cost report and a set of limits you can defend. The last case study asks a different question about records. When a company had to defend what its AI assistant did, could its records have answered?

Next: Could the records have answered?