Cost model

What you will be able to do

Build the monthly cost of the design from its parts: each driver’s quantity from the workload, times its list price, minus any free amount. Then find the driver that dominates, because that is where a saving matters.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• The workload numbers (43,950 questions, 41.6 million input tokens a month), from Workload arithmetic.
• The three kinds of numbers and the price list prices.toml, from Set up the design pack.
• The eight parts and their owners, from Service boundaries. Each part brings one or more lines of the bill.

What you need on your computer

• Your folder design-pack from the last lesson, with python -m pytest -q printing 22 passed.

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A phone bill

Look at a phone bill. It has three kinds of lines.

• A monthly fee: 20 dollars, even in a month when you never use the phone.
• Usage × price: 3 GB of extra data × 5 dollars per GB = 15 dollars.
• An allowance: the first 10 GB of data are included. You pay only for what you use above it.

To check the bill, you do not need to know how the phone company works inside. You need the quantity of each line, its price, and its allowance. Then you add the lines. To lower the bill, you look for the biggest line first. A cheaper price for a line of 50 cents does not help.

A cloud bill has exactly these three kinds of lines. This lesson builds one for the assistant.

The problem: the price you see is not the bill

Tomás has a quick estimate. “AI cost is tokens. chat-small costs US$0.10 per million input tokens and US$0.50 per million output tokens. Each answer reads 946.5 tokens and writes 152.75. So one answer costs 0.0000946 + 0.0000764 = US$0.00017. With 43,950 questions a month, that is US$7.52. The assistant costs under ten dollars a month.”

His arithmetic is right. His bill is wrong. The assistant also needs an API that waits for questions all month, a database server that runs every hour, and storage with a minimum size. None of these appear in the token price. In this lesson’s model, the same month costs US$52.67: seven times Tomás’s number. The tokens are 14% of it.

The opposite mistake is just as common. The public price list for the design’s services has 1,112 rows, and a beginner tries to price all of them. Most of them do not apply, and the ones that do apply often cost nothing, because they are inside a free amount. Without a structure, nobody can check the result.

The idea: one line per cost driver

A cost driver is something whose quantity sets one line of the bill: tokens, database hours, gigabytes stored, log data, requests. A cost model is the list of the design’s drivers, each with three parts:

• the quantity per month, from the workload arithmetic or from a design choice;
• the list price per unit, from the provider’s price list;
• the free amount, if the price page states one, taken off before the price.

Each line is then: cost = (quantity − free amount) × price. If the quantity is below the free amount, the line is 0. The monthly cost is the sum of all lines. The driver with the biggest line is the dominant driver.

The name “driver” says why the list exists. When a quantity changes, its line changes, and you see which part of the design caused it.

The anatomy of one line of the cost model: quantity × price, minus the free amount. The monthly cost is the sum of 14 lines.

Text description of the diagram

The flow starts with a quantity per month from the workload arithmetic: 41.6 million input tokens for the answers. The flow multiplies it by the price from prices.toml, US$0.10 per million tokens. Then it takes off any free amount from the price page; tokens have none. The result is one driver’s cost: US$4.16 a month. The sum of all 14 drivers is the monthly cost, US$52.67.

A tiny example: three lines of the bill

Here are three of the design’s lines, one of each kind, with every value visible. The quantities come from the last lesson; the prices from prices.toml.

A monthly fee: the database server. The design uses PostgreSQL on a small Burstable B1ms server. “Burstable” is the provider’s cheapest kind of server: it is made for light work with short busy moments. Its list price is US$0.0185 an hour, and it runs every hour of the month. Cloud price lists use 730 hours for a month (365 × 24 ÷ 12). So 0.0185 × 730 = US$13.50 a month, whether the shops ask one question or a million.

Usage × price: the answers’ tokens. 41.6 million input tokens × US$0.10 per million = US$4.16. 6.71 million output tokens × US$0.50 per million = US$3.36. Together: US$7.52 a month. This is Tomás’s number: correct, but only one line.

An allowance: the logs. Each question writes about 4 KB of logs: a request line, a trace and an audit line. That is an assumption, with a range from 1 to 10 KB. 43,950 × 4 KB = 0.18 GB a month. The price page says that the first 5 GB a month are free. 0.18 GB is below 5 GB, so the line is US$0.00.

The three lines show three different behaviours. The fee does not move with usage. The tokens grow with every question. The logs cost nothing until the volume is about 28 times bigger, and then they start to grow.

Larkfield’s cost-driver table

Kwame writes every driver of the design in one table. It is the module’s main picture: each line shows where its number comes from.

Driver	Quantity per month (base)	List price	Free amount	US$ a month
Compute: API	1 replica, 0.5 vCPU, 1 GiB, all month; busy 12 h a day (assumed)	vCPU active US$0.000034/s, idle 0.000004/s; memory 0.000004/GiB-s	180,000 vCPU-s, 360,000 GiB-s	27.43
Database server	730 hours of B1ms	US$0.0185/h	none	13.50
Model tokens: answers	41.6M in, 6.71M out (chat-small)	US$0.10 / 0.50 per 1M	none	7.52
Database storage	32 GB provisioned (the smallest size; 18.3 MB used)	US$0.1265/GB-month	none	4.05
Retries	2% of model calls sent again (assumed)	the same token prices	none	0.15
Model tokens: question embeddings	0.68M (embed-small)	US$0.02 per 1M	none	0.01
Model tokens: ingestion, summaries	11 changed documents	the same token prices	none	0.00
Compute: ingestion worker	25 s of work	as the API	shares the grant	0.00
Database backups	18.3 MB × 2 (assumed)	US$0.105/GB-month	up to 32 GB (100% of storage)	0.00
File storage	0.3 MB of documents	US$0.02/GB-month	none	0.00
Requests	43,950	US$0.40 per 1M	2M a month	0.00
Logs	0.18 GB	US$2.76/GB	5 GB a month	0.00
Network out	0.04 GB of answers	US$0.08/GB	100 GB a month	0.00
Total				52.67

The prices are list prices from the public Azure price list. The models are in East US 2; the infrastructure is in Canada Central, where the two shops are. Prices change: check the current page before you use a price in a real proposal, and write the date you checked it.

The first line needs a word. The API replica runs all month, so that the first question of the day does not wait for a start. When it works on requests, the provider charges the active vCPU price; when it waits, a much lower idle price. The design assumes 12 busy hours a day. The provider charges the memory all the time. After the free grant, that gives 15.91 + 2.59 + 8.93 = US$27.43. The idle hours are cheap, but the always-on memory is not.

The monthly cost by driver, base case: the API replica, the database server, the answers' tokens, the database storage, the retries and the nine other drivers. The two always-on lines make 78% of the total.

Text description of the chart

A column chart with six columns. The API replica costs US$27.43 a month and the database server US$13.50. The answers’ tokens cost US$7.52, the database storage US$4.05 and the retries US$0.15. The other nine drivers together cost US$0.02 (each one rounds to 0.01 or 0.00 in the table). The total is US$52.67. The first two columns are much taller than the rest.

The answer to “which driver dominates?” is Compute: API, at 52%. Together with the database server, the two always-on lines make 78% of the bill. The tokens, which most people call “the AI cost”, are the third line.

Guided practice: add the cost model to your pack

1. In your folder design-pack, with the environment active, copy the cost code and its tests from the snapshot end-of-m03.

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m03
   cp "$S/costmodel/costs.py" costmodel/
   cp "$S/tests/test_costs.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m03"
   Copy-Item "$S\costmodel\costs.py" costmodel\
   Copy-Item "$S\tests\test_costs.py" tests\

   Open costmodel/costs.py. Each driver is a few lines: a quantity from workload.py, a price from prices.toml, and _after_free where the price page has a free amount.

2. Print the monthly cost:

   python -m costmodel costs

   Monthly cost, scenario base, month 0 (2 tenants, 43,950 questions)
     Driver                                US$ per month   share
     Compute: API                                  27.43   52.1%
     Database server                               13.50   25.6%
     Model tokens: answers                          7.52   14.3%
     Database storage                               4.05    7.7%
     Retries                                        0.15    0.3%
     Model tokens: question embeddings              0.01    0.0%
     Model tokens: summaries                        0.00    0.0%
     Compute: ingestion worker                      0.00    0.0%
     File storage                                   0.00    0.0%
     Model tokens: ingestion                        0.00    0.0%
     Database backups                               0.00    0.0%
     Requests                                       0.00    0.0%
     Logs                                           0.00    0.0%
     Network out                                    0.00    0.0%
     Total                                         52.67
   Biggest driver: Compute: API (52% of the total)

   Compare it with Kwame’s table: the same 14 drivers and the same total.

3. Print the low and the high case:

   python -m costmodel costs --scenario low
   python -m costmodel costs --scenario high

   The last lines of each:

   Monthly cost, scenario low, month 0 (2 tenants, 11,850 questions)
   …
     Total                                         40.54
   Biggest driver: Compute: API (52% of the total)
   Monthly cost, scenario high, month 0 (2 tenants, 167,400 questions)
     Driver                                US$ per month   share
     Compute: API                                  37.15   43.8%
     Model tokens: answers                         28.63   33.8%
     Database server                               13.50   15.9%
     Database storage                               4.05    4.8%
     Retries                                        1.43    1.7%
   …
     Total                                         84.83
   Biggest driver: Compute: API (44% of the total)

4. Run the tests:

   python -m pytest -q

   ............................                                             [100%]
   28 passed in 0.04s

   One new test, test_free_amounts_make_small_drivers_zero, checks that logs, network and requests stay inside their free amounts in the base case.

Check the high case by hand. It has 3.8 times the questions of the base case: 167,400 ÷ 43,950. The answers’ tokens grow by the same factor: 7.52 × 3.81 = 28.63. The database server stays at 13.50. The API line grows from 27.43 to 37.15 for a different reason. The high case assumes 18 busy hours a day instead of 12, so more hours cost the active price.

The dominant driver changes with volume

The dominant driver is not a fixed fact about the design. It depends on the volume. The chart compares the base case today with the high case in month 12, when four shops ask 416,220 questions a month:

• Base, now
• High, month 12

Three groups of drivers in the base case now and in the high case in month 12. The tokens grow with the questions and become the biggest line; the database (DB) does not move.

Text description of the chart

A column chart with three pairs of columns. API replica: US$27.43 in the base case now, US$37.15 in the high case in month 12. Model tokens with retries: US$7.68 now, US$74.89 in month 12. Database server, storage and backups: US$17.55 in both. In the high case in month 12, the tokens are the tallest column, about twice the API.

In the base case, a saving on the API replica or the database matters most. In the high case of month 12, the tokens are 58% of the US$129.60 total. A cheaper model, or fewer tokens per answer, would then matter most. Module 4 tests both with real measurements.

Look at one more line in that high case: the logs. 416,220 questions × 10 KB = 4.2 GB, just under the free 5 GB. A little more traffic, or longer log lines, and the logs start to cost US$2.76 for every further GB. Allowances hide a driver until the day it appears on the bill.

When to build a cost model

Build it as soon as you have a workload and a list of parts, before anyone buys a service. It answers “how much?”, and it also answers “where would a change help?”. Rebuild it when a price, a design choice or a measured value changes. The calculator makes that a one-line change.

Do not treat it as the bill. A cost model is a forecast with list prices. Real bills add taxes and discounts, and services that you forgot. Some prices also depend on details that the price list does not show. For example, the price list has “long context” token prices, but it does not say where “long” starts. The first real bill is the test of the model. Compare them every month, line by line, and fix the model where they differ.

The cost of the model is small: one table and one script. Without it, a team can spend weeks to save cents on the third line, while the first line stays.

Common mistakes

Common mistake

A size below the price page's minimum

Symptom: you set storage_gb = 10 in design.toml, because the data is only 18 MB. The total falls to US$49.88, and a test fails:

FAILED tests/test_costs.py::test_the_database_server_is_its_hourly_price_for_a_month
1 failed, 27 passed in 0.05s

Cause: the managed database’s smallest storage size is 32 GB (pg_min_storage_gb in prices.toml). You cannot buy 10 GB, so the model now shows a price that you cannot get. The calculator multiplies whatever design.toml says; the test knows the minimum.

Fix: set storage_gb = 32 again. If a price page states a minimum size, use at least that minimum in the model, even if you need less.

Common mistake

A price name with a typo

Symptom: you change the database to price = "pg_b1m_hour", and the command ends with a Python error:

    server = price[design.database_price] * HOURS_PER_MONTH
             ~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'pg_b1m_hour'

Cause: the value must be the name of a table in prices.toml. pg_b1m_hour does not exist; the correct name is pg_b1ms_hour. The calculator checks the ranges and the evidence, but not this name. So you see Python’s error instead of an “Input problem”.

Fix: copy the name from prices.toml. The last line of the error, KeyError: '…', shows the name that was not found.

Reset. If your pack breaks, copy the snapshot end-of-m03 as the snapshots README shows.

Your turn: a bigger database server

Mei worries that the B1ms server is too small. She asks what a Burstable B2s would cost. Its list price is US$0.074 an hour, in prices.toml as pg_b2s_hour.

Before you run anything, predict the new database server line, the new total, and the new dominant driver. Then change price under [database] in design.toml to "pg_b2s_hour", run python -m costmodel costs and python -m pytest -q, and change the value back.

Show the answer

Database server: 0.074 × 730 = US$54.02 a month. The total: 52.67 − 13.50 + 54.02 = US$93.18 (the calculator rounds each line, so your hand result can differ by a cent). Nothing else changes: the other lines do not depend on the server size.

Monthly cost, scenario base, month 0 (2 tenants, 43,950 questions)
  Driver                                US$ per month   share
  Database server                               54.02   58.0%
  Compute: API                                  27.43   29.4%
  Model tokens: answers                          7.52    8.1%
  Database storage                               4.05    4.3%
…
  Total                                         93.18
Biggest driver: Database server (58% of the total)

The dominant driver changes to the database server. One choice made the bill 77% bigger, and the workload did not change at all. That is why a design choice needs a reason. At 0.58 answers in flight and 18 MB of data, no measurement asks for a bigger server yet.

The tests print 1 failed, 27 passed: test_the_database_server_is_its_hourly_price_for_a_month checks the pack’s chosen server, B1ms. Change the value back to "pg_b1ms_hour", and the tests pass again.

Knowledge check

Check your understanding

Tomás says: 'The model costs US$7.52 a month, so the assistant costs about US$7.52 a month.' Name two drivers that he left out, say why each one costs money even when nobody asks a question, and give the model's total.

Show the answer

Any two of these three. The API replica (US$27.43) runs all month, so that the first question does not wait. The database server (US$13.50) runs every hour at US$0.0185. The database storage (US$4.05) has a minimum of 32 GB, even when the data is 18 MB. All three are paid per hour or per month, not per question. The model’s total is US$52.67 a month in the base case; the answers’ tokens are 14% of it.

Check your understanding

Predict: the log volume per question doubles from 4 KB to 8 KB in the base case. What happens to the Logs line, and why? At what volume of questions would the logs start to cost money with 8 KB per question?

Show the answer

The Logs line stays at US$0.00. 43,950 × 8 KB = 0.35 GB a month, still below the free 5 GB. The logs start to cost money above 5 GB ÷ 8 KB = 625,000 questions a month, about 14 times today’s volume. Above that, each further GB costs US$2.76. A free amount makes a driver look free until the volume passes it.

Recap

• A cost driver is a quantity that sets one line of the bill. Each line is (quantity − free amount) × list price. The cost model is the sum of all lines.
• Three kinds of lines: a fee (the database server, 0.0185 × 730 = US$13.50), usage × price (the answers’ tokens, US$7.52) and usage above an allowance. The logs, network and requests are US$0 at this volume.
• Larkfield’s design costs US$52.67 a month in the base case (low 40.54, high 84.83). The dominant driver is the always-on API replica (52%); with the database server, the two always-on lines make 78%.
• The dominant driver changes with volume: in the high case of month 12, the tokens are 58% of US$129.60.
• Prices change: check the current page. The first real bill tests the model.
• python -m pytest -q prints 28 passed.

Next: Unit economics