Skip to content
Français

Usage and cost per agent

  • Lesson 17
  • 40 min
  • Module 6: Usage, evidence and shadow AI

Where we are

In Module 5, we put the tool gateway, the approval policy and the kill switches between the agents and their actions. Now every model call and every tool call leaves a record that names the agent. But nobody reads those records yet, so nobody can say who spent what, or what one task cost. In this lesson, we turn the records into reports, the way an office reads its taxi account.

What you will be able to do

Turn the gateway’s records into usage and cost reports per agent, per team and per capability. Then join them with the app’s own trace by the run ID. This gives the cost of one task, and of one task that the agent resolved.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • What the gateway records for each call, from One door for model calls: the agent, its team, the capability, the model that answered, the tokens and the cost.
  • How the support workflow runs through both gateways and asks a person to approve, from Autonomy and approvals as policy.

What you need on your computer

  • Your project ~/projects/ai-estate from Module 5, the same as the snapshot end-of-m05, with its virtual environment active. python -m pytest -q says 190 passed, 1 deselected. If your project is broken, copy the snapshot end-of-m05 as the labs README explains.
  • This lesson adds no file. It reads the usage store, state/usage.db, with Python’s own sqlite3 command line, which needs Python 3.12 or newer.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them. Every answer is replayed from an earlier course’s recording, and every cost uses the course’s prices. Prices change: check the provider’s current price page for real work.

Think of an office with one account at a taxi company. Every ride gives a receipt. The receipt names the person who rode, their department, the kind of car and the price. At the end of the month, the office manager adds the receipts in different ways. Per person: who rides most? Per department: is sales inside its travel budget? Per kind of car: do people take the large cars when a small one would do?

One more question is harder: what did it cost to win the new client? The rides for that work went to the client’s office, the station and a hotel, on different days, by different people. The manager can add them only if each receipt carries the same project code. And even then, the receipts do not say whether the client signed. That is in the project’s diary.

Larkfield’s gateways now write one row for every model call and every tool call. In September, the replayed month, the model gateway answered 40,754 calls for the registered agents. Nobody can read 40,754 rows. So Kwame still gets questions that the records could answer, but do not answer yet:

  • Omar asks: “What does strong-reasoning cost us in a month?” He wants a cheaper model behind it, so he needs the number first.
  • Amira asks: “Is the service team inside its budget?” Her team owns three agents.
  • Grace asks: “What does it cost when the workflow resolves one ticket?” She wants to compare it with the time of a person.

Before the gateway, the provider’s bill was the only record. It shows keys, not apps, as you saw in Meet Larkfield’s AI estate. Now the records name the agent. They only need to be grouped.

A usage report adds the gateway’s rows in groups: per agent, per team, per capability or per model. Each group is one line, with the number of calls, how they ended, the tokens and the cost. This is the office manager’s work with the receipts.

When the platform shows each team what its agents cost, it is called showback. When the company also moves that cost into each team’s budget, it is called chargeback. Larkfield does showback: each team has a monthly budget in config/teams.yaml, and the reports show it.

The diagram shows one real row from this lesson’s run, and the four groups that it counts in.

A flow from left to right. One model call: support-workflow asks for strong-reasoning. The gateway writes one row: agent support-workflow, team service, capability strong-reasoning, model chat-strong (gpt-6.1-sol-2026-09-29), tokens 2,182 in and 55 out, cost US$0.004914, run ID T-90103-…-1. The row counts in four groups: by agent, support-workflow 5 calls, US$0.0294; by team, service 45 calls, US$0.0364; by capability, strong-reasoning 10 calls, US$0.0511; by model, chat-strong 10 calls, US$0.0511.
One call, one row, four reports. The same row is added into the agent's line, the team's line, the capability's line and the model's line.
Text description of the diagram

One model call: the support workflow asks for strong-reasoning. The gateway writes one row. It names the agent, support-workflow, and its team, service. It names the capability, strong-reasoning, and the model that answered, chat-strong, version gpt-6.1-sol-2026-09-29. It has the tokens, 2,182 in and 55 out, the cost, US$0.004914, and the run ID. The same row counts in four reports. By agent, support-workflow has 5 calls for US$0.0294. By team, service has 45 calls for US$0.0364. By capability, strong-reasoning has 10 calls for US$0.0511. By model, chat-strong has the same 10 calls.

You start a fresh estate, send a small amount of traffic from every app, and read the reports. The numbers are small, so you can check each one by hand.

  1. In ~/projects/ai-estate, stop the services and delete the old state:

    Terminal window
    python -m estate down
    python -m estate reset

    reset deletes state/: the registry, the keys, the usage records and the practice database. You start with no records, so your reports show only this lesson’s traffic.

  2. Register every app, review the three high-risk ones, issue the credentials and start the services. These are the steps of the snapshot’s README:

    Terminal window
    python -m estate init
    for f in registrations/*.yaml; do python -m estate register "$f"; done
    for a in support-workflow support-mcp orders-mcp; do python -m estate review "$a" --next 2026-12-01; done
    for a in ticket-triage support-workflow doc-assistant release-judge helpdesk-assistant; do python -m estate credentials issue "$a"; done
    python -m estate up

    The last lines (your process numbers differ):

    front up (process 82523) http://127.0.0.1:8322/front
    toolgate up (process 82528) http://127.0.0.1:8326
  3. Send real recorded requests from three apps, through the gateway:

    Terminal window
    python -m apps.replay ticket-triage --units 20 --via gateway
    python -m apps.replay doc-assistant --units 5 --via gateway
    python -m apps.replay release-judge --units 5 --via gateway
    ticket-triage: 40 requests through the gateway | status {200: 40}
    answered by: chat-small (gpt-6-luna-2026-09-22) x20, embed-small (text-embedding-3-small) x20
    tokens: 39,575 in, 6,410 out | cost: US$0.0070
    doc-assistant: 5 requests through the gateway | status {200: 5}
    answered by: chat-small (gpt-6-luna-2026-09-22) x5
    tokens: 4,509 in, 875 out | cost: US$0.0009
    release-judge: 5 requests through the gateway | status {200: 5}
    answered by: chat-strong (gpt-6.1-sol-2026-09-29) x5
    tokens: 7,690 in, 632 out | cost: US$0.0217
  4. Run three tasks of the support workflow through both gateways. Sam approves the first two:

    Terminal window
    RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver run T-90103 --variant agent_structured --model strong-reasoning
    RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver approve T-90103-agent_structured-strong-reasoning-1 --as agent --approver usr-sam
    RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver run T-90104 --variant agent_structured --model strong-reasoning
    RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver approve T-90104-agent_structured-strong-reasoning-1 --as agent --approver usr-sam
    RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver run T-90301 --variant agent_structured --model strong-reasoning

    The first line of each run names the outcome, the calls and the cost that the app saw:

    Outcome: resolve (rule W5) | 2 model call(s), 1 tool call(s) | 4647 tokens in, 266 out | US$0.01195
    Outcome: resolve (rule W2) | 2 model call(s), 1 tool call(s) | 4586 tokens in, 265 out | US$0.01182
    Outcome: hand_to_person (rule H1) | 1 model call(s), 0 tool call(s) | 2167 tokens in, 128 out | US$0.00561

    T-90103 is a cracked planter: a reship and a free return label. T-90104 is a late watering can: a refund of the 9.95 shipping fee. T-90301 is a smell of gas near a barbecue valve: a safety case, so the agent hands it to a person.

Before you add rows, look at one. The usage store is one SQLite file. Python 3.12 can query it from the command line:

Terminal window
python -m sqlite3 state/usage.db "select agent_id, team, capability, deployment, model_version, tokens_in, tokens_out, cost_usd, outcome, run_id from model_calls where agent_id = 'support-workflow' order by id limit 1"

On Windows, write state\usage.db; the rest of the command is the same.

('support-workflow', 'service', 'strong-reasoning', 'chat-strong', 'gpt-6.1-sol-2026-09-29', 2182, 55, 0.004914, 'ok', 'T-90103-agent_structured-strong-reasoning-1')

Each value answers one question. Who called: support-workflow, of the service team. What for: strong-reasoning. What answered: chat-strong, version gpt-6.1-sol-2026-09-29. How much: 2,182 tokens in and 55 out. At the course’s prices for chat-strong, US$2 per million tokens in and US$10 per million out, that is 0.004364 + 0.000550 = US$0.004914. How it ended: ok. For which task: the run ID. The row holds no prompt and no answer text: the gateway keeps none.

  1. Per agent, the default:

    Terminal window
    python -m estate usage
    agent calls ok refused failed tokens in tokens out US$
    ------------------------------------------------------------------------------------------------------
    support-workflow 5 5 0 0 11,400 659 0.0294
    release-judge 5 5 0 0 7,690 632 0.0217
    ticket-triage 40 40 0 0 39,575 6,410 0.0070
    doc-assistant 5 5 0 0 4,509 875 0.0009
    total 55 55 0 0 63,174 8,576 0.0590

    Check one line by hand: the workflow’s 5 calls are the 2 + 2 + 1 of its three tasks, and US$0.01195 + 0.01182 + 0.00561 = US$0.02938, shown rounded as 0.0294. The lines are sorted by cost, so the most expensive agent is first, although it made the fewest calls.

  2. Per team:

    Terminal window
    python -m estate usage --by team
    team calls ok refused failed tokens in tokens out US$
    ------------------------------------------------------------------------------------------------------
    service 45 45 0 0 50,975 7,069 0.0364
    evaluation 5 5 0 0 7,690 632 0.0217
    data 5 5 0 0 4,509 875 0.0009
    total 55 55 0 0 63,174 8,576 0.0590

    The service team is Amira’s: triage (40 calls) and the workflow (5 calls). The help-desk assistant is also hers, but it made no model call today.

  3. Per capability, Omar’s question:

    Terminal window
    python -m estate usage --by capability
    capability calls ok refused failed tokens in tokens out US$
    ------------------------------------------------------------------------------------------------------
    strong-reasoning 10 10 0 0 19,090 1,291 0.0511
    fast-small 25 25 0 0 42,638 7,285 0.0079
    embed 20 20 0 0 1,446 0 0.0000
    total 55 55 0 0 63,174 8,576 0.0590

    Ten strong-reasoning calls cost more than six times the 25 fast-small calls. Two consumers share strong-reasoning: the workflow (5 calls) and the release judge (5 calls).

  4. Per model:

    Terminal window
    python -m estate usage --by model
    model calls ok refused failed tokens in tokens out US$
    ------------------------------------------------------------------------------------------------------
    chat-strong (gpt-6.1-sol-2026-09-29) 10 10 0 0 19,090 1,291 0.0511
    chat-small (gpt-6-luna-2026-09-22) 25 25 0 0 42,638 7,285 0.0079
    embed-small (text-embedding-3-small) 20 20 0 0 1,446 0 0.0000
    total 55 55 0 0 63,174 8,576 0.0590

    Today each capability maps to one model, so the two reports agree. After a remap or a fallback, they differ: the capability report says what the apps asked for, and the model report says what really answered.

The same reports, on the replayed month of September, answer Kwame’s questions. The chart shows each agent’s spending in the gateway’s records against its monthly budget, from the registration forms. The budgets are the course’s assumptions.

The gateway's records for the replayed month against each agent's monthly budget. Replayed answers, the course's prices, assumed budgets.
Text description of the chart

Five bars, as a percentage of each agent’s monthly budget. The support workflow used 57% of US$25 (US$14.28). Ticket triage used 43% of US$15 (US$6.48). The release judge used 5% of US$8, and the document assistant 4% of US$3. The help-desk assistant used nothing of its US$2.

So Amira’s answer is yes: the service team spent US$20.76 of its US$40 in the gateway’s records. And Omar’s answer: strong-reasoning cost US$14.70 of the gateway’s US$21.31 in September, 2,690 of 40,754 answered calls. It is a small share of the calls and most of the money. That is why he asks for a cheaper model, and why Change a mapping safely made him pass a gate first.

But read the workflow’s line again, because a report can mislead. The next chart splits the workflow’s September money by where it went.

The replayed month: straight to the provider before the workflow moved on 10 September, through the gateway for normal work, and the runaway loop of 22 September (a constructed loop on a real recorded request).
Text description of the chart

Three bars, in US dollars. Before it moved on 10 September, the workflow spent US$4.44 straight to the provider, with the service team’s shared key. Through the gateway, its normal work cost US$10.19. The runaway loop of 22 September cost US$4.09 before the daily quota stopped it.

Two things follow. First, US$4.09 of the workflow’s US$14.28 is one bad deployment that looped. That is 29% of its month, in one hour. A cost report per agent shows that something happened. It does not say what, so you look at the day and the incident. Second, the gateway’s records start on the day each app moved. In September, the apps also spent US$7.36 straight to the provider before they moved. That is triage US$2.04, the workflow US$4.44, the judge US$0.83 on Priya’s personal key, and the documents US$0.05. Only the provider’s bill shows that money. A report that reads only the gateway’s records says nothing about calls that did not pass the gateway.

Cost per task: join with the app’s trace

Section titled “Cost per task: join with the app’s trace”

Grace’s question is about a task, not a call. One task of the workflow makes several model calls and several tool calls, and only the app knows how it ended. The gateway sees each call alone.

So the platform asks every app to send a run ID on each call, in the header X-Run-Id, and a W3C traceparent header. A run ID is the project code on the taxi receipts. The workflow sends its run ID, for example T-90103-agent_structured-strong-reasoning-1, and a traceparent whose trace part comes from the run ID. The app keeps its own record of the run, its trace: the steps, the tool results and the outcome. Joining the two by the run ID gives the cost of each task, and its outcome.

  1. Add the gateway’s rows per run of the workflow:

    Terminal window
    python -m sqlite3 state/usage.db "select run_id, count(*), sum(tokens_in), sum(tokens_out), round(sum(cost_usd), 5) from model_calls where agent_id = 'support-workflow' group by run_id"
    ('T-90103-agent_structured-strong-reasoning-1', 2, 4647, 266, 0.01195)
    ('T-90104-agent_structured-strong-reasoning-1', 2, 4586, 265, 0.01182)
    ('T-90301-agent_structured-strong-reasoning-1', 1, 2167, 128, 0.00561)

    These are the same costs that each run printed. The gateway and the app agree.

  2. Ask the app how each run ended:

    Terminal window
    python -m resolver runs
    T-90103-agent_structured-strong-reasoning-1 done …
    T-90104-agent_structured-strong-reasoning-1 done …
    T-90301-agent_structured-strong-reasoning-1 handed_to_person …

    Each line ends with the run’s time (left out here: …). Two runs are done: the agent’s proposal was approved and carried out. One was handed to a person.

  3. Look at one run’s trace, step by step:

    Terminal window
    python -m resolver trace T-90103-agent_structured-strong-reasoning-1
    Run T-90103-agent_structured-strong-reasoning-1: agent_structured, strong-reasoning, ticket T-90103, status done
    2 model call(s), 4647 tokens in, 266 out, US$0.01195, 0.031 s waiting for the model
    1 plan ["Check the order’s delivery date, planter price and stock.", "Apply the damaged-item policy and propose the …
    1 model strong-reasoning: get_order({"order_id": "LK-640436"}) (2182 in, 55 out, 0.026 s)
    1 tool get_order({"order_id": "LK-640436"}) -> ok
    2 plan []
    2 model strong-reasoning: finish({"outcome": "resolve", "rule": "W5", "a…) (2465 in, 211 out, 0.005 s)
    2 proposal resolve (W5): reship_item LK-640436, create_return_label LK-640436
    2 stop finished
    2 approval approved by agent (agent)
    2 write reship_item op 221dccf204b80a63 -> done
    2 write create_return_label op 0f8c2e98df53719d -> done
    2 stop done

    The first model step, 2,182 tokens in and 55 out, is the row you read before.

  4. Look at the tool gateway’s rows for the same runs:

    Terminal window
    python -m sqlite3 state/usage.db "select run_id, tool, kind, outcome, approver from tool_calls order by id"
    ('T-90103-agent_structured-strong-reasoning-1', 'get_order', 'read', 'ok', None)
    ('T-90103-agent_structured-strong-reasoning-1', 'reship_item', 'write', 'ok', 'usr-sam')
    ('T-90103-agent_structured-strong-reasoning-1', 'create_return_label', 'write', 'ok', 'usr-sam')
    ('T-90103-agent_structured-strong-reasoning-1', 'add_ticket_note', 'write', 'ok', None)
    ('T-90104-agent_structured-strong-reasoning-1', 'get_order', 'read', 'ok', None)
    ('T-90104-agent_structured-strong-reasoning-1', 'request_refund', 'write', 'ok', 'usr-sam')
    ('T-90104-agent_structured-strong-reasoning-1', 'add_ticket_note', 'write', 'ok', None)

    Each side knows something that the other does not. The app’s trace says approved by agent: a role. The tool gateway says usr-sam: the person whose grant it checked. The trace has the plan and the outcome; the gateway has the cost and the person.

Now join them. Three tasks cost US$0.01195 + 0.01182 + 0.00561 = US$0.02938, so US$0.00979 per task. But the agent resolved only two of them. The third went to a person, who still does the work. So the cost per resolved task is US$0.02938 / 2 = US$0.01469. That is the number Grace can compare with a person’s time, and it is 50% higher than the cost per task.

Two groups joined on the run ID. The gateway's records, what each call cost: T-90103 2 calls US$0.01195, T-90104 2 calls US$0.01182, T-90301 1 call US$0.00561. The app's trace, how each task ended: T-90103 done (reship, label), T-90104 done (refund 9.95), T-90301 handed to a person. Joined on the run ID: 3 tasks, US$0.02938; 2 resolved by the agent; US$0.01469 per resolved task.
Cost per resolved task needs both records: the gateway knows the cost of each call, and only the app knows how each task ended.
Text description of the diagram

On one side, the gateway’s records say what each call cost, grouped by run ID: T-90103 made 2 calls for US$0.01195, T-90104 2 calls for US$0.01182 and T-90301 1 call for US$0.00561. On the other side, the app’s trace says how each task ended: T-90103 done, with a reship and a label; T-90104 done, with a refund of 9.95; T-90301 handed to a person. Joined on the run ID, the three tasks cost US$0.02938. The agent resolved two of them, so one resolved task cost US$0.01469.

In this lesson the join is three lines that you add by hand. At Larkfield’s scale, it is a job for traces. Every call is a span in one trace per task, and a tracing tool adds the costs per trace. A later course on observability builds that join with real traces. The gateway’s part is already done: every row carries the run ID and the traceparent.

When to use these reports, and what they cost

Section titled “When to use these reports, and what they cost”

Use the usage reports every month, and when a budget warning arrives. They are cheap: the rows exist already, and a report is one query. Per agent and per team, they answer who spends. Per capability, they answer where a remap would save money. Per model, they show what really answered after fallbacks.

Do not use them alone to judge an agent. Cost per call says nothing about quality: the cheapest call may be the one that hands every ticket to a person. Keep three limits in mind:

  • Only what passed the gateway. Calls straight to a provider, and the document assistant’s embeddings on a local model, are not in the records. The bill and the next two lessons cover the rest.
  • Only the course’s prices. The cost in each row is tokens times the price in config/models.yaml when the call was made. A real provider’s bill can differ: discounts, cached tokens, a price change. Compare the totals with the bill each month.
  • Cost per task needs the run ID everywhere. An app that sends no run ID can be reported per agent, never per task. The triage replayer sends one per ticket; the workflow sends one per run. A new app must send it from the first day, or its history cannot be joined later.

Common mistake

The report is empty

Symptom: you ask for the month and get only a total line, with zeros:

Terminal window
python -m estate usage --month 2026-09
agent calls ok refused failed tokens in tokens out US$
------------------------------------------------------------------------------------------------------
total 0 0 0 0 0 0 0.0000

Cause: every row carries the time of the call on the estate’s clock, in UTC. Your traffic is in the month of your run, not in the replayed September. The September report is a file in the next lesson; its rows are not in your usage store.

Diagnose: python -m estate usage without --month shows every row. If it shows your traffic, the month is wrong. Late in the evening, west of UTC, your rows may already be in the next day or month.

Recover: use the month of your run, in the form YYYY-MM, or leave --month out.

Change the unit: the triage service, not the workflow. The triage replayer sends one run ID per ticket, so you can count tickets.

  1. Predict: 20 tickets, two calls each, cost US$0.0070 in the replay. What is the cost of one ticket?

  2. Check it from the gateway’s rows:

    Terminal window
    python -m sqlite3 state/usage.db "select count(distinct run_id), count(*), round(sum(cost_usd), 5) from model_calls where agent_id = 'ticket-triage'"
  3. Now predict the month. Triage moved behind the gateway on 8 September and saw about 822 tickets a day (the MLOps course’s number). How much should the gateway’s records show for triage in September? Write your number down, then compare it with the chart’s US$6.48.

Check your answer

The course’s run, replayed answers:

(20, 40, 0.00705)

20 run IDs, 40 calls, US$0.00705. One ticket costs US$0.00705 / 20 = US$0.00035, a chat-small call and an embedding.

From 8 to 30 September is 23 days, so about 822 × 23 = 18,906 tickets. At US$0.00035 each, that is about US$6.62. The gateway’s records say US$6.48, for 37,191 answered calls: about 18,600 tickets. Your estimate is close because the 20 tickets are a fair sample of the recordings. Triage moved during the 8th, so part of that day went straight to the provider. And during the outage of 17 September, 35 triage calls had no provider: those tickets went to a person.

Check your understanding

The capability report and the model report of the same month give different numbers for strong-reasoning. Name two reasons.

Show the answer

The capability report counts what the apps asked for. The model report counts the model that really answered. They differ when a call went to a fallback, for example the second region during an outage. They also differ when a remap changed the model behind the capability during the month. They also differ when a consumer pins an exact model version: its calls count under strong-reasoning but answer from the pinned version.

Check your understanding

The workflow's report says US$14.28 in September. Amira says: 'So one task costs US$14.28 divided by the number of tasks.' What is wrong with that, and what do you need instead?

Show the answer

Three things. US$4.09 of it is the runaway loop, not tasks. The gateway does not know how many tasks there were, or how they ended: only the app’s trace knows. And money spent straight to the provider before the move on 10 September is not in it. Instead, add the cost per run ID from the gateway’s rows, and join it with the app’s trace by the same run ID. Leave out the runaway’s runs, and divide by the tasks that the agent resolved.

Check your understanding

Practical: run one more workflow task through both gateways, for example T-90105. Find its cost from the gateway's rows by its run ID, and check that it is the cost that the app printed.

Show the answer

Run it with RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver run T-90105 --variant agent_structured --model strong-reasoning. In the course’s run, its second line was Outcome: reply_only (rule W6) | 2 model call(s), 1 tool call(s) | 4624 tokens in, 200 out | US$0.01125: an opened bag of soil cannot be returned, so the agent only replies. Then run python -m sqlite3 state/usage.db "select run_id, count(*), round(sum(cost_usd), 5) from model_calls where run_id = 'T-90105-agent_structured-strong-reasoning-1'". It gave ('T-90105-agent_structured-strong-reasoning-1', 2, 0.01125): the same 2 calls and the same cost. If you see no row, the run ID differs: copy it from the run’s first line.

Recap

  • The gateway writes one row per call: agent, team, capability, model, tokens, cost, outcome and run ID. It keeps no prompt and no answer.
  • python -m estate usage --by agent | team | capability | model groups the rows. Showback shows each team its cost against its budget.
  • In September, strong-reasoning cost US$14.70 of the gateway’s US$21.31, for 2,690 of 40,754 calls. US$4.09 of the workflow’s US$14.28 was one runaway hour.
  • The gateway’s records miss what did not pass the gateway: US$7.36 went straight to the provider before the apps moved.
  • Cost per task joins the gateway’s rows with the app’s trace by the run ID. Three tasks cost US$0.02938; per resolved task, US$0.01469.

Your project is unchanged: the snapshot end-of-m05, with 190 tests. Its usage store now holds this lesson’s traffic, and the next lesson uses it. You can now say who spent what, and what a resolved task cost. But the governance board asks more than cost: is every agent registered, which ones are high risk, what went wrong? Next, we answer the board’s questions from the evidence, and we find the question that the evidence cannot answer.

Next: Evidence for governance