Measure usage and cost

What you will be able to do

Answer OmarTermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s two questions with numbers: what does the assistant cost, and how long does an agent wait? You log every model call, read the totals, and set a budget that the code can check.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Cost per ticket, hidden reasoning tokens, and the median and p95 of a wait, from Measure practical performance in the language models course.
• The usage block of a response and the reasoning tokens, from Request anatomy.
• redact() in config.py, from The application boundary.
• The retry wrapper in analyse_ticket, from Handle provider limits.

What you need on your computer

• The project after Handle provider limits: retry.py and simulate.py. python -m pytest shows 75 passed, 1 deselected.
• If you skipped it, copy the snapshot end-of-m05 with the steps of Reset the project and do that lesson’s steps. Or copy end-of-m06 and read this lesson’s steps against it.

The itemised phone bill

A phone bill that says only “Total: 84 dollars” does not help you. You cannot see which call was expensive, or whether next month will cost the same. An itemised bill has one line per call: the date, the number, the minutes and the price. With it, you can find the one long call abroad and decide what to change.

The problem: the bill comes later, as one number

Omar asks two questions before LarkfieldTermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps turns the assistant on: “What will it cost per day? And how long does an agent wait for a draft?” The provider’s invoice answers the first question a month later, as one total for the whole account. It does not say which ticket, prompt or model used the money.

Some costs are hidden, too. chat-smallTermchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs is a reasoning modelTermReasoning modelA model that writes hidden reasoning tokens before its answer. chat-small is one. Such a model often accepts only its default temperature, and its hidden tokens count against the output limit.Learn more Reasoning and multimodalityLLMs: How They Work and How to Choose One: it thinks before it answers, and the provider bills those reasoning tokensTermReasoning tokensTokens that some models write before the answer, to work through the problem. You do not see them, but they count against the output limit and you pay for them.Learn more Reasoning and multimodalityLLMs: How They Work and How to Choose One as output tokens. On the evaluation setTermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One it used a medianTermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One of 215 reasoning tokens per ticket, which you never see. Once, on T-64849 (prompt v2, no structured outputTermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs), it spent all 1,000 output tokens on reasoning. It returned no text at all. Larkfield paid for an empty answer.

Time has the same problem. An average hides the slow calls, and an agent remembers the slow calls.

The idea: one log line per call

A usage log has one line for every model call, like the itemised bill. Each line says which call it was and what it used, and it says nothing that the log must not keep. The project writes the lines to logs/usage.jsonl, one JSON object per line. These are the fields, and why each one is there:

Field	Why
request_id	The provider’s ID of the call. You quote it when you ask the provider’s support about one call.
model, prompt	So that you can compare the cost of chat-small and chat-strong, or of prompt v1 and v2.
ticket_id	So that you can add up the calls of one ticket: a tool loop makes two.
input_tokens, output_tokens, reasoning_tokens	What the provider bills. Reasoning tokens are part of the output.
latency_s	How long the agent waited for this call.
cost_usd, prices_checked	The cost from a dated price table: the price was an assumption on that date.
finish_reason, status, note	What happened. The note passes through redact().

Two things are missing on purpose: the ticket text and the reply. The log is for numbers. Ticket text holds names, addresses and sometimes card numbers, and a log file is read by more people than the ticket system. The ticket_id is enough to find the ticket when you need it.

A tiny worked example: the cost of one ticket

The method is the one from Cost per ticket in the language models course: multiply each kind of token by its price, and add. The prices come from a price table: US dollars per million tokens, for input and for output. These are the Azure prices of the two deploymentsTermDeploymentA model that an account sets up at a provider, with its own name and its own limits. Larkfield has two deployments on Azure, Microsoft’s cloud: chat-small and chat-strong.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs, as we checked them for the course. Check the current price page before you use them; prices change.

Deployment	Input, per million tokens	Output, per million tokens
chat-small (gpt-6-luna)	US$0.10	US$0.50
chat-strong (gpt-6.1-sol)	US$2.00	US$10.00

T-80008, the demo ticket, used two calls on chat-small: 2,043 + 2,181 = 4,224 input tokens and 21 + 102 = 123 output tokens. So:

input   4,224 × 0.10 / 1,000,000 = 0.0004224
output    123 × 0.50 / 1,000,000 = 0.0000615
total                            = 0.0004839  ≈ US$0.00048

Input is most of the cost here, because the prompt, the policy and the tool result are long and the answer is short. That changes with the model. Here is the cost of the whole evaluation set, measured from the recordings, for the four runs with structured output:

• chat-small (gpt-6-luna)
• chat-strong (gpt-6.1-sol)

Cost per 1,000 tickets: the mean cost of the 69 recorded tickets, with the prices in the price table. No tools, structured output.

Text description of the chart

A column chart of the cost per 1,000 tickets. With chat-small: US$0.28 for prompt v1 and US$0.35 for prompt v2. With chat-strongTermchat-strongThe name that Larkfield’s code gives to a larger hosted model: gpt-6.1-sol on Azure. It is often more accurate than chat-small, and it costs about 14 to 18 times more per call.Learn more Request anatomyBuilding Reliable Applications with LLM APIs: US$3.97 for prompt v1 and US$5.03 for prompt v2. The chat-strong columns are about 14 times higher than the chat-small columns.

chat-strong costs about 14 times more per ticket. On the same 69 tickets with prompt v2, it got one more team right (68 against 67 of 68). It also got one more needs_human right (68 against 67 of 69). For Larkfield’s 45 tickets a day, chat-small with prompt v2 costs about US$0.016. chat-strong costs about US$0.23. Both are small. But the ratio stays 14 when the volume grows, and the log is how you would notice.

Prompt v2 costs about 27% more than v1 on chat-small, because it sends 609 more input tokens per ticket: the specification and two examples. A longer prompt is paid on every call.

Median and p95: how long an agent waits

You met the median and the p95 in Measure practical performance: the typical wait, and the wait that 95 of 100 calls stay under. The picture shows both for the assistant.

The recorded time of each of the 69 tickets on chat-small, prompt v2, structured output (gpt-6-luna on Azure): median 2.77 s, p95 4.34 s.

Text description of the chart

A histogram of the time for 69 tickets, in bins of half a second, from 1 to 7 seconds. Most tickets take between 2 and 3.5 seconds. A tail on the right reaches past 5 seconds. A dashed line marks the median at 2.77 seconds, and a second dashed line marks the p95Termp95The 95th percentile: the time that 95 of 100 calls stay under. It shows how long the slow calls take, which the median hides.Learn more Measure practical performanceLLMs: How They Work and How to Choose One at 4.34 seconds.

A typical agent waits under 3 seconds, but one draft in 20 takes more than 4.3 seconds. The chat-strong deployment has the same median, 2.77 s, and a longer tail: its p95 is 4.84 seconds. Retries add to the tail, not to the median. A ticket that needed two attempts after a 429 waits a little longer.

A budget per ticket and per day

A budget is a limit that the code checks, not a hope. The project has a per-ticket budget in usage.py: at most 8,000 tokens and US$0.02 for one ticket. T-80008 with its tool loop used 4,347 tokens and US$0.00048. So the budget leaves room for a longer ticket and a retry, but not for a runaway loop. The picture puts the three limits of one ticket side by side.

The request budget of one ticket: what T-80008 used, and the limit that the code checks for each part.

Text description of the diagram

The budget of one ticket, T-80008 on chat-small, has three parts.

• Tokens: it used 4,224 input and 123 output tokens, 4,347 in 2 calls. The budget is 8,000 per ticket.
• Cost: it used US$0.00048 with the prices in the price table. The budget is US$0.02 per ticket.
• Time: it used 0.99 s and 1.69 s, 2.68 s in total. The limits are 30 s per call, 20 s of retry waits and 60 s for the tool loop.

Together: 45 tickets a day like this one cost US$0.022 a day. With the budget, a day can cost at most US$0.90.

The day budget follows from the ticket budget: 45 tickets × US$0.02 = US$0.90 at most. If the daily total passes a smaller alarm value, someone should look. Something changed: a new prompt, a new model, or a loop that runs too long. over_budget() checks one ticket’s log lines and says which limit was passed.

Choose the numbers from the log, not from a guess. Set them high enough for real tickets and low enough to stop a fault. When you change the prompt or the model, look at the log again and adjust them.

Build it

1. Make assistant/usage.py:

   assistant/usage.py

   """One log line per model call: tokens, time, cost, request ID. No ticket text, no keys."""
   
   import json
   import statistics
   from dataclasses import dataclass
   from datetime import datetime, timezone
   from pathlib import Path
   
   from .config import redact
   from .providers import Completion
   
   LOG_FILE = Path(__file__).resolve().parent.parent / "logs" / "usage.jsonl"
   
   # US dollars per million tokens (input, output). Output includes reasoning tokens.
   # Azure AI Foundry, Global Standard deployments, prices checked on 2026-10-08. Check the current price page.
   PRICES_CHECKED = "2026-10-08"
   PRICES = {
       "chat-small": (0.10, 0.50),   # gpt-6-luna 2026-09-22
       "chat-strong": (2.00, 10.00),  # gpt-6.1-sol 2026-09-29
       "gemma3:4b": (0.0, 0.0),       # local model through Ollama: no price per token (your own computer)
   }
   
   
   def cost_usd(model: str, input_tokens: int, output_tokens: int) -> float | None:
       """None when the model is not in the price table: unknown cost is not zero cost."""
       if model not in PRICES:
           return None
       price_in, price_out = PRICES[model]
       return round((input_tokens * price_in + output_tokens * price_out) / 1_000_000, 8)
   
   
   @dataclass
   class Budget:
       max_tokens_per_ticket: int = 8000
       max_cost_per_ticket_usd: float = 0.02
   
   
   class UsageLog:
       def __init__(self, path: Path | None = None):
           self.path = path or LOG_FILE  # read at call time, so that the tests can point it elsewhere
           self.path.parent.mkdir(parents=True, exist_ok=True)
   
       def record(self, completion: Completion, model: str, ticket_id: str, prompt: str, status: str = "ok",
                  note: str = "") -> dict:
           entry = {
               "at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
               "request_id": completion.request_id,
               "model": model,
               "prompt": prompt,
               "ticket_id": ticket_id,
               "input_tokens": completion.input_tokens,
               "output_tokens": completion.output_tokens,
               "reasoning_tokens": completion.reasoning_tokens,
               "latency_s": completion.latency_s,
               "cost_usd": cost_usd(model, completion.input_tokens, completion.output_tokens),
               "prices_checked": PRICES_CHECKED,
               "finish_reason": completion.finish_reason,
               "status": status,
               "note": redact(note),
           }
           with self.path.open("a", encoding="utf-8") as f:
               f.write(json.dumps(entry) + "\n")
           return entry
   
       def entries(self) -> list[dict]:
           if not self.path.exists():
               return []
           return [json.loads(line) for line in self.path.read_text(encoding="utf-8").splitlines()]
   
   
   def summarise(entries: list[dict]) -> dict:
       if not entries:
           return {"calls": 0}
       latencies = sorted(e["latency_s"] for e in entries)
       costs = [e["cost_usd"] for e in entries]
       return {
           "calls": len(entries),
           "input_tokens": sum(e["input_tokens"] for e in entries),
           "output_tokens": sum(e["output_tokens"] for e in entries),
           "latency_median_s": round(statistics.median(latencies), 2),
           "latency_p95_s": round(latencies[min(len(latencies) - 1, int(0.95 * len(latencies)))], 2),
           "cost_usd": None if None in costs else round(sum(costs), 6),
       }
   
   
   def over_budget(ticket_entries: list[dict], budget: Budget) -> list[str]:
       tokens = sum(e["input_tokens"] + e["output_tokens"] for e in ticket_entries)
       cost = sum(e["cost_usd"] or 0 for e in ticket_entries)
       reasons = []
       if tokens > budget.max_tokens_per_ticket:
           reasons.append(f"{tokens} tokens > {budget.max_tokens_per_ticket}")
       if cost > budget.max_cost_per_ticket_usd:
           reasons.append(f"US${cost:.4f} > US${budget.max_cost_per_ticket_usd}")
       return reasons

2. Open assistant/analyse.py. Add the import from .usage import UsageLog below the import from .tools. Then add the parameter log after retry, and record each call in complete:

   assistant/analyse.py (changed lines)

                      retry: RetryPolicy = RetryPolicy(), log: UsageLog | None = None,
                      reasoning_effort: str = "") -> Outcome:

       def complete(req: dict) -> Completion:
           completion = call_with_retry(lambda: provider.complete(req), retry)
           if log:
               log.record(completion, model, ticket.ticket_id, prompt)
           return completion

3. Open assistant/__main__.py. Add this import below the import from .store:

   from .usage import UsageLog, cost_usd, summarise

4. In the same file, add the usage command after the eval parser and its options, just before args = parser.parse_args(argv):

       commands.add_parser("usage", help="summarise logs/usage.jsonl")

5. Handle it first, just after args = parser.parse_args(argv):

       if args.command == "usage":
           print(json.dumps(summarise(UsageLog().entries()), indent=2))
           return 0

6. Give every analysis the log. Replace the options = … line:

       log = UsageLog()
       options = dict(model=model, prompt=args.prompt, structured=not args.no_schema, log=log,
                      reasoning_effort=settings.reasoning_effort)

7. Show the cost on the last line of analyse. Replace the print(f"{outcome.status} …") line:

   assistant/__main__.py (analyse branch)

           cost = cost_usd(model, tokens_in, tokens_out)
           print(f"{outcome.status} | {model} | prompt {args.prompt} | {len(outcome.completions)} call(s) | "
                 f"{tokens_in} tokens in, {tokens_out} out | cost "
                 f"{'unknown' if cost is None else f'US${cost:.5f}'}")

   Add the line python -m assistant usage totals from logs/usage.jsonl to the docstring at the top.

8. Open tests/conftest.py. A test that runs analyse would now write to your real log. Add this fixture after use_the_mock:

   tests/conftest.py (new fixture)

   @pytest.fixture(autouse=True)
   def keep_the_usage_log_clean(tmp_path, monkeypatch):
       """A test run must not add lines to your real logs/usage.jsonl."""
       monkeypatch.setattr("assistant.usage.LOG_FILE", tmp_path / "usage.jsonl")

9. Copy tests/test_usage.py from the snapshot end-of-m06. Then run the tests:

   python -m pytest

10. Analyse the demo ticket, then read the totals:

    python -m assistant analyse T-80008
    python -m assistant usage

Step 9 prints 78 passed, 1 deselected. In step 10, the last line of analyse now ends with the cost:

valid | chat-small | prompt v2 | 2 call(s) | 4224 tokens in, 123 out | cost US$0.00048

The log has two new lines, one per model call. Open logs/usage.jsonl in your editor. The first line is the call that asked for the order lookup:

{"at": "2026-10-09T03:20:24+00:00", "request_id": "chatcmpl-EWuY0oVX3oudORY8oq8dTswM4pXCu", "model": "chat-small", "prompt": "v2", "ticket_id": "T-80008", "input_tokens": 2043, "output_tokens": 21, "reasoning_tokens": 0, "latency_s": 0.99, "cost_usd": 0.0002148, "prices_checked": "2026-10-08", "finish_reason": "tool_calls", "status": "ok", "note": ""}

The time at is when you ran it; the rest comes from the recording. usage adds up the two lines:

{
  "calls": 2,
  "input_tokens": 4224,
  "output_tokens": 123,
  "latency_median_s": 1.34,
  "latency_p95_s": 1.69,
  "cost_usd": 0.000484
}

With the mock, latency_s is the time that was, not the time on your computer. The replay is instant. That keeps the numbers the same for every learner. With a live provider, the log has your real times.

Redaction: what a log line must not hold

Logs live longer than requests and are read by more people. So the note field passes through redact(), the function from The application boundary. Try it with a made-up key and email address. Make try_log.py in the project folder:

try_log.py

"""Record one usage line with a note that holds a made-up key and email address. Print what was saved."""
import json

from assistant.context import build_request
from assistant.data import load_ticket
from assistant.providers import MockProvider
from assistant.schema import response_format
from assistant.usage import UsageLog

request = build_request(load_ticket("T-80008"), "chat-small", "v2", response_format())
completion = MockProvider().complete(request)
note = "Proxy said: Authorization: Bearer sk-proj-4f9Qx2LmTz81 rejected; reply to nadia.price@example.com"
entry = UsageLog().record(completion, "chat-small", "T-80008", "v2", note=note)
print(json.dumps(entry, indent=2))

Run python try_log.py. The end of the output:

  "finish_reason": "stop",
  "status": "ok",
  "note": "Proxy said: Authorization: [REDACTED] rejected; reply to [EMAIL REDACTED]"
}

The key and the email address never reach the file. The line is in logs/usage.jsonl too, so delete the logs folder and try_log.py afterwards. Redaction is a safety net with patterns, and patterns miss things. The first protection is the decision above: the ticket text and the reply never go to the log at all.

Your turn: the cost of the whole set

Delete the logs folder (rm -r logs on macOS or Linux, Remove-Item -Recurse logs in PowerShell). Then run the evaluation set on chat-small and read the totals:

python -m assistant eval --prompt v2
python -m assistant usage

Predict the cost of 69 tickets from the chart above. Then delete logs again and do the same with --model chat-strong.

Show the totals

chat-small, prompt v2:

{
  "calls": 69,
  "input_tokens": 131733,
  "output_tokens": 22038,
  "latency_median_s": 2.77,
  "latency_p95_s": 4.34,
  "cost_usd": 0.024192
}

chat-strong, prompt v2:

{
  "calls": 69,
  "input_tokens": 131733,
  "output_tokens": 8343,
  "latency_median_s": 2.77,
  "latency_p95_s": 4.84,
  "cost_usd": 0.346896
}

US$0.024 for 69 tickets is US$0.35 per 1,000, as in the chart. chat-strong is US$5.03 per 1,000. Both runs sent the same input tokens. chat-small wrote more than twice the output tokens, most of them hidden reasoning. But its output price is 20 times lower.

Common mistake

'Just put 0 for the models we have no price for'

Symptom: TomásTermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. tries a new model through a live provider. The model name is not in PRICES, and the analyse line says cost unknown. He suggests a default of 0, “so that the reports add up”. With that change, the daily report would say US$0 for every call of the new model, and the budget check would never stop it.

Cause: a missing price is not a free call. Zero is a real number that the budget trusts. The project returns None instead, and summarise refuses to add a total that has a gap:

print(cost_usd("chat-small", 4224, 123))
print(cost_usd("my-new-model", 4224, 123))
print(summarise([{"input_tokens": 4224, "output_tokens": 123, "latency_s": 2.68,
                  "cost_usd": cost_usd("my-new-model", 4224, 123)}]))

The output:

0.0004839
None
{'calls': 1, 'input_tokens': 4224, 'output_tokens': 123, 'latency_median_s': 2.68, 'latency_p95_s': 2.68, 'cost_usd': None}

Fix: add the model to PRICES with the price from the provider’s page, and update PRICES_CHECKED. The local model is in the table with 0 on purpose: OllamaTermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One on your own computer has no price per token. It is not free either: your computer does the work. It took 3.61 s per ticket at the median, compared with 2.77 s on chat-small.

What it means for a real AI service: an unknown value must stay visibly unknown. A report that fills gaps with zeros looks complete and is wrong.

See it end to endClassify banking requests with an LLMCase study

The banking case study compares accuracy and cost per 1,000 requests with a fine-tunedTermFine-tuningMore training of a pretrained model on your own labelled examples, so that it does one task well. Its weights change a little.Learn more Training versus inferenceDeep Learning and Transformers Explained model and word counts.

Knowledge check

Check your understanding

chat-small writes 313 output tokens per ticket (median) and chat-strong 104. Yet chat-small is much cheaper. Explain with the price table.

Show the answer

chat-small’s output price is US$0.50 per million tokens and chat-strong’s is US$10, 20 times more. Its input price is US$0.10 against US$2. So even with three times more output tokens, most of them hidden reasoning, chat-small costs about 14 times less per ticket. Count tokens and prices together; a token count alone does not tell the cost.

Check your understanding

Why does the usage log keep the ticket ID and the request ID, but not the ticket text or the reply?

Show the answer

The IDs are enough to find the ticket in the help-desk system and to ask the provider about one call. The text and the reply hold personal data: names, addresses, sometimes card numbers. A log file is copied, shipped to monitoring tools and read by more people than the ticket system. Keeping only numbers and IDs keeps the log useful and safe. redact() is a second line of defence for the free-text note.

Check your understanding

Predict: a ticket's tool loop goes wrong and makes 4 model calls of about 2,500 input tokens each. Does over_budget report it with the default Budget? What does it say?

Show the answer

Yes. Four calls send about 10,000 input tokens, plus some output, which is more than the 8,000-token budget. over_budget returns one reason, such as "10200 tokens > 8000". The cost check passes: on chat-small that is about US$0.001, far under US$0.02. The token budget catches a runaway loop long before the cost does, on a cheap model.

Recap

• A usage log has one line per model call. It holds the request ID, model, prompt version, ticket ID, tokens, latency, cost and the price date. It holds no ticket text and no reply, and redact() cleans the note.
• Cost = tokens × the price from a dated price table. An unknown price is None, never 0.
• Real recorded numbers: chat-small costs US$0.35 per 1,000 tickets with prompt v2. chat-strong costs US$5.03, about 14 times more. T-80008 with its tool loop: US$0.00048.
• Look at the median and the p95: on chat-small, 2.77 s and 4.34 s.
• A budget is a limit that the code checks: 8,000 tokens and US$0.02 per ticket. For 45 tickets a day, that is at most US$0.90.

Next, you test every failure path without a network and package the assistant for someone else.

Next: Test and package