Workload arithmetic

What you will be able to do

Turn the demand of Module 1 into four numbers that cost money and set limits. They are questions a month, answers in flight at the busiest moment, tokens a month and storage growth. Every step is a multiplication that you can check by hand.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The demand ranges and the busiest minute (0.268 questions per second in the base case), from Demand assumptions.
• The steps of an answer (embed the question, search, write the answer), from Service boundaries.
• Throughput, latency and the rule that links them, from Latency and capacity in Scaling APIs and AI Workloads. This lesson uses the same rule to size a design that does not exist yet.

What you need on your computer

• Your folder design-pack with its active virtual environment, and python -m pytest -q printing 16 passed: the snapshot end-of-m02.
• The labs folder nextia-learning-labs next to it. You copy five new files of the calculator from it.

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A café at lunch time

A small café gets busy at lunch. At the busiest time, two customers come in every minute. Each one stays about 15 minutes: they order, eat and pay. How many seats does the café need?

The owner does not need the day’s total for this. She needs two numbers: how fast people arrive, and how long each one stays. Two people a minute × 15 minutes = 30 people inside at the same time. If she has 20 seats, ten people stand. If each person stays only 5 minutes, 10 seats are enough.

The café answers other questions with simple multiplication too. How much coffee for a month? Cups per day × 30. How much room for receipts in a year? Receipts per day × the size of one × 365. None of this is clever. It needs the right numbers, in the right units, in the right order.

The problem: a day is not a size

Amira must choose the size of the assistant’s API, and Kwame must check the model provider’s limit. The demand file says 1,465 questions a day. Three people turn that into three different sizes:

• Tomás: “1,465 questions could all come at once. Size for 1,465 answers at the same time.”
• Omar: “A day has 86,400 seconds. 1,465 ÷ 86,400 = 0.017 questions per second. Each one takes 2.18 s, so 0.037 answers at once. One tiny server, and no limit to worry about.”
• Kwame: “Neither. Questions do not spread evenly over the day, and they do not all come at once.”

Omar’s number fails at the worst moment. The scaling course saw this failure. Its provider allowed a fixed number of model calls per minute. When the load went above that quota, the provider refused part of the calls, and throughput stopped growing. An average over the day never shows this, because the quota is per minute, not per day.

So a size needs the busiest moment, not the day. And the design needs more than one size: the bill depends on volumes per month, and the database depends on growth.

The idea: four numbers from demand

Workload arithmetic turns the demand into the work that the system must do. Four numbers matter for a document assistant:

1. Volume: questions per month = questions per day × 30. Most prices are per month or per unit, so the cost model needs volumes per month.
2. Concurrency: answers in flight at the busiest moment = arrival rate × time in the system. “In flight” means started and not yet finished. These are the café’s seats.
3. Tokens per month: questions per month × tokens per answer. Model providers charge per token, and they limit tokens per minute.
4. Storage growth: new records per month × bytes per record. Storage costs money, and backups and restores take longer as the data grows.

The rule for concurrency has a name: Little’s law, after John Little, who proved it in 1961. The scaling course used it to predict throughput from a load test. Here you use it the other way round: from the expected arrivals to the capacity that the design must have.

From the base demand to the four workload numbers. Each arrow is one multiplication with a measured or assumed value.

Text description of the diagram

The diagram starts with the base demand, 1,465 questions a day for both shops. Times 30 days gives the volume, 43,950 questions a month. From the volume, one arrow leads to the tokens: 946.5 in and 152.75 out per answer give 41.6 million in and 6.7 million out a month. Another arrow leads to the storage growth: 372 bytes per answer give 16.4 MB a month. A separate path goes from the demand to the busiest minute, 0.268 questions per second. Times 2.18 seconds in the system, that gives the concurrency: 0.58 answers in flight. Thick frames mark the four results.

Each arrow multiplies by one number, and each number is one of the three kinds. The 30 days and the 1,465 questions are assumptions. The 2.18 s, the tokens and the 372 bytes are measured. No price appears yet: prices come in the next lesson.

A tiny example: answers in flight

Here is Little’s law for the assistant, with every value visible.

Arrival rate. Module 1 computed the busiest minute of the base case. The busiest hour has 1,465 × 0.123 = 180 questions. The busiest minute has 180 ÷ 3,600 × 5.35 = 0.268 questions per second.

Time in the system. One answer passes three measured steps, the same steps as the request flow of Module 2:

Step	Median time	Kind
Embed the question (embed-small)	0.34 s	measured, the course team’s calls
Search and rerank 20 passages	0.32 s	measured, the RAG course, on a laptop CPU
Write the answer (chat-small)	1.52 s	measured, the course team’s calls
Time in the system	2.18 s	the sum of the three medians

In flight. 0.268 per second × 2.18 s = 0.58 answers at the same time.

Little's law for the assistant at the busiest minute of the base case: arrival rate × time in the system = answers in flight.

Text description of the diagram

On the left, questions arrive at 0.268 per second, the busiest minute of the base case. In the middle, each question spends 2.18 seconds in the system. It takes 0.34 s to embed the question, 0.32 s to search and rerank, and 1.52 s to write the answer. On the right is the product: 0.58 answers in flight at the same time.

Less than one answer is in flight at the busiest moment. Most of the time, the API waits for the next question. Compare the three guesses from the problem: Tomás’s 1,465, Omar’s 0.037 and the real 0.58. Tomás is about 2,500 times too high; Omar is 16 times too low.

The 2.18 s has two honest limits. First, it adds three medians from separate runs; nobody measured one whole answer from end to end. Second, the search time comes from a laptop, not a cloud server. Both are good enough for a first size. The latency target Q1, measured in production, replaces them later.

Larkfield’s workload, step by step

The other three numbers follow the same pattern. Kwame writes each one with its inputs, so Omar can check it:

• Volume. 1,465 questions a day × 30 days = 43,950 questions a month.
• Tokens. Each chat-small answer reads 946.5 tokens (the question and five passages) and writes 152.75 tokens, both measured means. So 43,950 × 946.5 = 41.6 million input tokens and 43,950 × 152.75 = 6.7 million output tokens a month. Each question is also embedded: 43,950 × 15.51 tokens = 0.68 million embedding tokens. The calculator shows 0.69 million, because it also embeds the changed documents.
• Storage now. Each saved answer, with its question, takes 372 bytes (measured; a real row adds a few IDs). One month of answers: 43,950 × 372 bytes = 16.3 MB. The documents’ chunks and vectors take about 2.0 MB. That is 14,016 words × 28.37 chunks per 1,000 words × 4,800 bytes per vector, plus the chunk text.
• Storage growth. Every month adds the answers again, plus 11 changed documents of about 1,515 bytes each: +16.4 MB a month.

Two things stand out. The answers, not the documents, fill the database. And all of it is small: after a year, a few hundred megabytes, about 1% of the database’s smallest size of 32 GB.

Guided practice: add workload arithmetic to your pack

You add the calculator’s workload code and the design file, run it, and check its numbers by hand.

1. In the terminal, in your folder design-pack, with the environment active, copy two code files and two test files from the snapshot end-of-m03:

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m03
   cp "$S/costmodel/design.py" "$S/costmodel/workload.py" costmodel/
   cp "$S/tests/helpers.py" "$S/tests/test_workload.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m03"
   Copy-Item "$S\costmodel\design.py", "$S\costmodel\workload.py" costmodel\
   Copy-Item "$S\tests\helpers.py", "$S\tests\test_workload.py" tests\

   design.py reads the design file. workload.py does this lesson’s arithmetic. The tests check each rule.

2. Run the workload command. The design file is still missing, so the command stops with a clear message:

   python -m costmodel workload

   Input problem: design.toml is missing (expected at …/design-pack/design.toml)

3. Copy the design file. If you opened a new terminal, set S again as in step 1.

   macOS / Linux

   cp "$S/design.toml" .

   Windows (PowerShell)

   Copy-Item "$S\design.toml" .

4. Open design.toml and read it. The top tables hold the choices of Module 2: the model, the database size, the API and the worker. One API replica always runs; the worker runs only when there is work. The [assumptions] table holds ranges with evidence, as in demand.toml. Its first four lines:

   [assumptions]
   busy_hours_per_day = [8, 12, 18]
   log_kb_per_question = [1, 4, 10]
   retry_share = [0.0, 0.02, 0.05]
   cache_hit_share = [0.0, 0.0, 0.0]

   cache_hit_share is 0 in all three cases. The request flow of Module 2 has an answer cache, but nobody knows yet how often a question repeats. Module 4 measures what a cache hit saves. Until then, the pack assumes no hits.

5. Run the workload command again:

   python -m costmodel workload

   Workload, scenario base, month 0 (2 tenants)
     questions per month               43,950
     busiest hour                       180.2 questions
     busiest minute                     0.268 questions per second
     time in the system                  2.18 s (embed the question + search + answer, medians)
     in flight at the peak               0.58 = 0.268/s x 2.18 s
     chat calls, busiest min.            16.1 (provider quota: 100 per minute)
     documents                             61 (11 changed per month)
     tokens per month          chat-small 41.60M in / 6.72M out, embed-small 0.69M in / 0.00M out
     storage now               files 0.3 MB, chunks and vectors 2.0 MB, answers and logs in the database 16.3 MB
     storage growth                      16.4 MB per month
     ingestion worker                      25 s busy per month

   Find the four numbers of the tiny example: 43,950 questions, 0.58 in flight, 41.60M and 6.72M tokens, and 16.4 MB a month. The output is 6.72M tokens, not 6.71M, because the 11 document summaries write about 2,000 tokens too.

6. Run the tests:

   python -m pytest -q

   ......................                                                   [100%]
   22 passed in 0.03s

   Six new tests pass. One of them, test_concurrency_is_arrival_rate_times_time_in_the_system, is Little’s law written as code.

Check one more line yourself: the ingestion worker. There are 11 document changes a month. Each one takes 0.39 s to embed and 1.89 s to summarise, both measured medians. So 11 × (0.39 + 1.89) = 25 seconds of work a month. The worker sleeps for the rest of the month, which is why Module 2 made it scale to zero.

Check the provider’s limit

The line chat calls, busiest min. compares the design with the first limit that it can reach. Each question needs one chat call. In the busiest minute: 0.268 per second × 60 = 16.1 chat calls. The provider’s quota for chat-small is 100 calls a minute and 100,000 tokens a minute. The scaling course read both from the provider’s rate-limit headers, for the course team’s account. Your account’s quota can be different.

Calls are not the only limit. Each call also uses tokens: 946.5 in + 152.75 out = 1,099 tokens. In the busiest minute: 16.1 × 1,099 = about 17,700 tokens, or 18% of the token quota. So the base case uses 16% to 18% of the two quotas.

Now ask for the high case, today and in a year:

python -m costmodel workload --scenario high
python -m costmodel workload --scenario high --month 12

Workload, scenario high, month 0 (2 tenants)
  questions per month              167,400
  busiest hour                     1,116.0 questions
  busiest minute                     2.480 questions per second
  time in the system                  2.18 s (embed the question + search + answer, medians)
  in flight at the peak               5.41 = 2.480/s x 2.18 s
  chat calls, busiest min.           148.8 (provider quota: 100 per minute)
  …
Workload, scenario high, month 12 (4 tenants)
  questions per month              416,220
  busiest hour                     2,774.8 questions
  busiest minute                     6.166 questions per second
  time in the system                  2.18 s (embed the question + search + answer, medians)
  in flight at the peak              13.45 = 6.166/s x 2.18 s
  chat calls, busiest min.           370.0 (provider quota: 100 per minute)
  …

The … marks lines left out here; your terminal shows them. The chart puts the chat calls of all cases next to the quota:

• Low
• Base
• High
• Quota

Chat calls in the busiest minute, now and in month 12 (M12), against the provider's quota of 100 a minute. The high case passes the quota already today.

Text description of the chart

A column chart with six columns. Now: low 2.0, base 16.1 and high 148.8 chat calls in the busiest minute. In month 12: base 25.3 and high 370.0. The last, grey column is the provider’s quota: 100 calls a minute. The high case is about one and a half times the quota now, and almost four times the quota in month 12. The base case stays at about a quarter of the quota or less.

This is the honest answer to Tomás’s “design for a million users”. In the high case, 5 to 13 answers are in flight, and one small API replica waits for the provider most of that time. The servers are not the problem. The provider quota is the first limit, and the high case passes it already in month 0. Kwame writes it into the proposal as a risk with an action: ask the provider for a higher quota before a home-page launch. Module 5 turns it into a growth signal.

When to use workload arithmetic

Use it before any cost or size, and again whenever the demand or a measured time changes. It takes minutes, and it shows which limit comes first.

Do not use it as a promise. It gives averages over the busiest minute, not what happens inside that minute. Bursts of a few seconds, slow provider calls and retries can push the real number in flight higher. The p95 times give a cautious second estimate: 0.268 × (0.48 + 0.32 + 3.79) = about 1.2 in flight. A load test, as in the scaling course, replaces the estimate when the system exists.

The cost is small: a page of multiplications that everyone can check. The risk is a wrong input, such as a time in milliseconds read as seconds. Then the answer is far too big, as the next section shows.

Common mistakes

Common mistake

Rates and times in different units

Symptom: Little’s law gives a number that is far too big. For example, 16.1 questions × 2.18 = 35.1 answers in flight, sixty times more than the calculator’s 0.58.

Cause: 16.1 is per minute, but 2.18 is in seconds. The units do not cancel, so the result is not a count of answers. The same mistake happens with milliseconds: 0.268 × 2,180 ms = 584.

Fix: write the unit next to each number and convert first. 16.1 per minute ÷ 60 = 0.268 per second; 0.268 per second × 2.18 seconds = 0.58. The seconds cancel, and the result is a count.

Common mistake

A share written as a percentage in design.toml

Symptom: you write retry_share = [0, 2, 5], meaning 2% and 5%. The calculator stops:

Input problem: design.toml: cache_hit_share and retry_share are shares below 1

Cause: a share of 2 would mean that the pack sends every model call twice more. The calculator refuses a share of 1 or more.

Fix: write shares: [0.0, 0.02, 0.05]. If the evidence of an assumption does not start with verified: or assumed:, the calculator stops in the same way and names the line.

Reset. If your pack breaks, copy the snapshot end-of-m03 as the snapshots README shows. It has everything from this module.

Your turn: the home page again

Omar’s open question from Module 1 comes back. If the assistant goes on Larkfield’s home page, he expects 1,600 asking customers a day in the base case, not 800. In Module 1, you computed the new demand: 2,665 questions a day for both shops.

Before you run anything, predict three numbers: questions a month, answers in flight at the busiest minute, and chat calls in the busiest minute. Then change the base value of Larkfield’s customers_per_day in demand.toml to 1600, run python -m costmodel workload, and change the value back.

Show the answer

• Volume: 2,665 × 30 = 79,950 questions a month.
• Busiest minute: 2,665 × 0.123 ÷ 3,600 × 5.35 = 0.487 per second. In flight: 0.487 × 2.18 = 1.06.
• Chat calls: 0.487 × 60 = 29.2 in the busiest minute, under a third of the quota.

Workload, scenario base, month 0 (2 tenants)
  questions per month               79,950
  busiest hour                       327.8 questions
  busiest minute                     0.487 questions per second
  time in the system                  2.18 s (embed the question + search + answer, medians)
  in flight at the peak               1.06 = 0.487/s x 2.18 s
  chat calls, busiest min.            29.2 (provider quota: 100 per minute)
  documents                             61 (11 changed per month)
  tokens per month          chat-small 75.68M in / 12.21M out, embed-small 1.24M in / 0.00M out
  storage now               files 0.3 MB, chunks and vectors 2.0 MB, answers and logs in the database 29.7 MB
  storage growth                      29.8 MB per month
  ingestion worker                      25 s busy per month

Every number that depends on questions grows by 82%, as the demand did. The documents and the worker do not change: the home page brings questions, not documents. Even so, about one answer is in flight at the busiest minute, and the quota is still far away in the base case. Change the value back to 800.

See it end to endCost and capacity plan from real demand dataCase study

The case study runs the same arithmetic on a city’s real demand. It checks three peak shapes on a year that it kept aside: only the shape measured from the city’s own data covers most months.

Knowledge check

Check your understanding

Predict with Little's law: in the busiest minute, questions arrive at 0.5 per second, and each answer spends 3 seconds in the system. How many answers are in flight? Then say which of the two numbers a stronger, slower model would change, and in which direction.

Show the answer

0.5 per second × 3 s = 1.5 answers in flight. A slower model changes the time in the system, not the arrival rate: questions come at the same speed whatever model answers them. With chat-strong’s measured median of 2.48 s instead of 1.52 s, the time in the system grows from 2.18 s to about 3.15 s. So more answers are in flight at the same time, and each one holds its connection longer.

Check your understanding

Omar reads the workload output and says: 'Our database grows 16.4 MB a month, so storage is our biggest risk.' Use the numbers of this lesson to agree or disagree, and name the limit that comes first.

Show the answer

Disagree. 16.4 MB a month is a few hundred megabytes after a year, about 1% of the database’s smallest size of 32 GB. In the base case, nothing is near a limit. The first limit is the provider quota. The high case needs 148.8 chat calls in the busiest minute already today, against a quota of 100. Storage matters later for backups and restore times (Module 5), but it is not the first risk.

Recap

• Workload arithmetic turns demand into four numbers: volume per month, concurrency at the busiest moment, tokens per month and storage growth. Each one is a multiplication with visible inputs.
• Little’s law: answers in flight = arrival rate × time in the system. Base case: 0.268 per second × 2.18 s = 0.58. Keep the units the same.
• The time in the system, 2.18 s, is the sum of three measured medians: embed 0.34 s, search 0.32 s, answer 1.52 s.
• Base case per month: 43,950 questions, 41.6 million input and 6.7 million output tokens, and +16.4 MB of storage.
• The first limit is the provider quota of 100 chat calls a minute. The base case uses about 16%, the high case 149% today and 370% in month 12.
• python -m pytest -q prints 22 passed.

Next: Cost model