Capacity recommendation

What you will be able to do

Turn your load tests into a recommendation that a manager can act on. It names one change that the tests showed to help, the limit that comes next, and the measurements needed before the next change. Say nothing that the tests cannot support.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The four parts of a load test, from Load-test design.
• The scorecard, passes and ranges, from Compare configurations.
• The overload results of the queue, from Backpressure, and the outage results, from Failure containment.

What you need on your computer

• The project after Compare configurations: python -m pytest prints 176 passed, 1 deselected.
• Your own results from the last two lessons: the burst, and the comparison with its --json file.

After this lesson, your project matches the snapshot end-of-m06: the end of the course project. This lesson runs no new load test. The numbers in it come from the course’s measurements on one quiet computer, with the simulated provider.

The number on the fuel label

A new car has a label with its fuel use, for example “5.2 litres per 100 km”. The number is honest. A laboratory measured it, with the car on rollers, on a fixed speed plan, at a fixed temperature. The small print says so.

Then a family drives the car up a mountain, with a roof box, in winter. It uses 8 litres. The label was not a lie. It was a measurement under stated conditions, and the family read it as a promise for their own road.

A load test on your laptop is the fuel label of your service: honest under its conditions. This lesson writes a recommendation that keeps the conditions next to the numbers, so that nobody reads it as a promise for production.

The problem: a laptop number that travelled

Tomás read the course’s measurements and wrote one line for Kwame:

“The async design did 27 tickets per second on a laptop. Production has 10 servers, so 270 per second. Black Friday’s busiest minute is 1.5 per second. We have 180 times the capacity that we need.”

Every number in it was measured. But each step of the reasoning breaks a condition of the test:

• The 27 per second ran with the quota off. The course turned the quota off on purpose, as if the account had a much larger one. With the recorded quota, the provider takes 100 chat calls per minute: at most 50 tickets per minute, or 0.83 per second.
• The quota is per account, not per server. Ten servers share the same 100 calls per minute. So ten servers still get 0.83 tickets per second of AI work.
• The provider was simulated. The simulator does not get slower with load. The real provider did. A reply took about 1.10 seconds alone, about 1.17 seconds at 16 calls in flight, and about 1.42 seconds at 40.
• The database has limits that ten servers reach first. In Dependency limits, 12 processes with pools of 10 asked for 120 connections, and PostgreSQL allows 100.

So “180 times” is not a little too optimistic. It is a claim that no test made. On Black Friday, the real limit is about half of the busiest minute’s demand.

Demand from the workload model (ten times the busiest hour and minute of the large data) and the most that the recorded quota allows. The busiest minute needs almost twice the quota.

Text description of the chart

Three bars. The busiest Black Friday hour brings 1,010 tickets, about 16.8 per minute on average. The provider’s quota of 100 chat calls per minute allows at most 50 tickets per minute, because each ticket needs two chat calls. The busiest Black Friday minute brings 90 tickets. The hour fits under the quota; the busiest minute does not.

What this tells us. On average, the quota is large enough: 17 tickets per minute against 50. In the busiest minutes, it is not: 90 against 50. So Black Friday is a question of peaks, and of what the service does with the tickets that the quota cannot take at once. The queue and its bound answer that question.

The idea: every claim rests on assumptions

A capacity recommendation is a short decision document. It says what to change, why, and what is still unknown. Its numbers come from load tests, and every load test has assumptions. They are the workload, the provider’s quota, the cost of a ticket, and where the test ran. A claim is only as strong as the assumptions under it.

So for each claim, ask two questions. Which assumptions does it rest on? And did the test hold those assumptions, or only assume them? A claim about the shape of the system usually survives the move from laptop to production. “The quota comes first”, “without a queue, a burst loses tickets” and “with a bound, refusals are explicit” are such claims. A claim about an absolute number in production does not survive, because production changes the machine, the network and the provider.

The performance scorecard tied to workload assumptions. Each claim names the assumptions it rests on. Green claims are about shapes and ratios that the tests showed; red claims need conditions that no test here had.

Text description of the diagram

The left column lists the four assumptions, A to D, with their numbers. The right part lists five claims. Each claim has its evidence and the letters of the assumptions it rests on. The course’s tests support the three green claims. The bounded queue loses no ticket in a burst. The quota is the next limit. The AI cost is about US$0.00006 per ticket. The two red claims are not supported. One turns a laptop number into a production number. The other assumes that the real provider is as fast under load as the simulator.

Look at the letter D. Every test ran on one laptop, so every measured claim rests on it. The green claims survive D, because they compare two designs under the same conditions, or follow from the quota alone. The red claims need D to be production, and it is not.

Larkfield: the capacity report

Amira wrote the recommendation for Omar and Kwame in docs/capacity-report.md. It is the course’s worked example, and every number in it comes from the course’s measurements. Its parts are the parts of every capacity recommendation:

1. The question, in one sentence: can the service accept Black Friday’s tickets, and give each one its AI work, without losing tickets?
2. The workload, from docs/workload.md: the assumptions A, B and C of the scorecard.
3. What we measured: four configurations under stated workloads. The Module 1 design lost the tickets that the provider refused. Without a queue, a burst of 10 tickets per second lost about 245 of 300. A queue without a bound accepted all 300 and needed minutes to empty. A bound of 100 accepted 172 or 173 and refused the rest with 503 queue_full and a Retry-After of 88 to 128 seconds.
4. The recommendation: the queue with a bound, and the circuit breaker on. This is the demonstrated improvement: the same burst, measured the same way, went from about 245 lost tickets to none.
5. The cost: about US$0.00006 per ticket of AI work, about US$0.06 for the busiest hour.
6. The next bottleneck: the provider’s quota, with what to measure before the next change.
7. What this test does not say.

People often skip the last parts, but they make the report trustworthy. Here they are:

docs/capacity-report.md (the end)

## The next bottleneck

The provider's quota. Before the next change, measure: the real provider's times under our load
(the simulator does not slow down with load; the real provider was about 30% slower with 40 calls
in flight), and the quota after an increase is granted.

## What a real run must measure before a production claim

A run on the real platform, in a disposable copy of production, with a budget and a deletion
afterwards: the replicas over time, their cold starts and the scale-down delay; the managed
database's connections and CPU; the real provider's times and refusals at our arrival rate; and
the cost per hour of the machines. This course did not make that run.

## What this test does not say

It ran on one laptop with a simulated provider and the small data. It does **not** say how many
tickets production hardware can handle, or how a real provider behaves on Black Friday. It shows
the shape: where tickets are lost, and which limit comes first.

Kwame asks, “Why name a bottleneck that we are not fixing?” Because the next decision depends on it. Larkfield can ask the provider for a larger quota, or accept that peak tickets wait a few minutes for their AI reply. Either way, the report tells them what to measure first.

What a real cloud run would need to measure

The course planned one live run on a cloud platform, and did not make it. So this course has no number for replicas, cold starts or cooldown in the cloud, and the report says so. Scaling signals showed autoscaling only as a simulation on your computer.

A real run is a load test like those in Load-test design, with the same four parts. Its environment is a disposable copy of production, with the same platform, sizes and settings. It has a budget, tags on every resource, and a deletion at the end that you check. Its readings are the ones that a laptop cannot give:

Reading	Why the laptop cannot give it
Replicas over time, the cold start of a new replica, the delay before scale-down	The laptop’s workers are local processes that started in about 0.28 s; a container on a platform starts differently
The managed database’s connections, CPU and query times	The laptop’s PostgreSQL ran in Docker, with no network between it and the API
The real provider’s times and 429s at your arrival rate	The simulator does not slow down with load, and its quota is a copy of the recorded one
The cost per hour of the machines and the database	The laptop has no bill
The network time between the services	Everything on the laptop ran on one machine

Even this run answers only for its own workload and its own day. A real soak of several hours, and a second run on another day, show whether the numbers are stable.

Guided practice: read the report, then write your own

1. Copy the report from the snapshot: cp ../nextia-learning-labs/C21/snapshots/end-of-m06/docs/capacity-report.md docs/. On Windows: Copy-Item ..\nextia-learning-labs\C21\snapshots\end-of-m06\docs\capacity-report.md docs\.
2. Run the tests: python -m pytest. Expect 176 passed, 1 deselected: the report is not code, so nothing changes.
3. Read docs/capacity-report.md. For each number in it, write down the letters (A to D) of the assumptions that it rests on.
4. Copy docs/benchmark-template.md to docs/my-capacity-report.md.
5. Fill it in with your own results: the burst from Load-test design and the comparison from Compare configurations. Include the output of scripts.machine and the load average of each run.
6. Under “What it means”, write exactly one demonstrated improvement, with the numbers before and after. Then write the next bottleneck, with its evidence.
7. Add a section “What this test does not say”. Name at least two things that only a real run could measure.

Show a short example of steps 6 and 7

This example uses the course’s measurements and the author’s comparison from the last lesson. Your numbers will differ.

• Demonstrated improvement: the bounded queue. The burst was 10 tickets per second for 30 seconds (quota on, 2 workers, 3 passes). Without a queue, about 245 of 300 tickets were lost; with a bound of 100, none.
• Next bottleneck: the provider’s quota. At most 50 tickets per minute get their AI work, and the busiest Black Friday minute brings 90. Evidence: with the quota on, a second worker did not raise the finished work per second beyond the range of the passes.
• What this test does not say: how many tickets per second production can accept. How fast the real provider answers at 2 tickets per second. How long a new replica takes to start on the platform. Each needs a real run in a disposable copy of production.

Completion check: claims that a test cannot support

This is the module’s completion check. Tomás wrote a first draft of the summary for Omar. Mark each sentence as supported by the course’s tests, or not supported, and say why. Then rewrite the sentences that are not supported, so that they say what the tests showed.

1. “With the queue and a bound of 100, no ticket is lost in a burst: the extra customers get a 503 with a Retry-After.”
2. “Production will handle 270 tickets per second.”
3. “More workers would not finish more AI work while the quota is 100 chat calls per minute.”
4. “The p95 of the AI reply on Black Friday will be 3 seconds.”
5. “The AI work costs about US$0.06 per 1,000 tickets at the prices in prices.py.”

Show the answer

1. Supported, for the measured burst: 10 tickets per second for 30 seconds, quota on, 2 workers, 3 passes. 172 or 173 were accepted, 127 or 128 got 503 queue_full with a Retry-After, and none was lost. Keep the conditions next to it.
2. Not supported. No test ran on production, and the quota limits the AI work to 0.83 tickets per second for the whole account. Rewrite: “The provider’s quota limits the AI work to 50 tickets per minute, whatever the number of servers. We have not measured production hardware.”
3. Supported as a shape. The quota is per account, and with the quota on, the comparison showed no gain from a second worker. It is not a measurement of production workers. Rewrite to add: “measured on one laptop with the simulated provider”.
4. Not supported. About 3 seconds was the p95 of a request on the laptop with Module 1’s design, no queue and few users. On Black Friday, a ticket of the busiest minute waits for the quota, and the real provider was slower under load. Rewrite: “In the busiest minute, a ticket’s AI reply can wait minutes in the queue. We have not measured the real provider at this rate.”
5. Supported as an estimate. It comes from the token counts and the dated prices, and the simulator counts tokens like the recordings. Rewrite to add: “prices change: check the provider’s current page; machines and database not included”.

Common mistake

The number without its conditions

Symptom: a slide or a chat message quotes one of your numbers, for example “27 tickets per second”, and someone plans with it. Nobody remembers that the quota was off.

Cause: the number left the document, and its conditions stayed behind. A load-test number means something only with its workload, its configuration and its machine.

Diagnose: for any number that you see quoted, find its source run. Check four things: the workload, the quota mode, the machine, and the number of passes. If one of them is missing, the number cannot support a decision.

Fix: write each number in one sentence with its conditions. For example: “27 tickets per second, async design, 64 users, quota off, simulated provider, one laptop, 3 passes”. Put “What this test does not say” into every report. If someone needs a production number, recommend the real run instead of a guess.

When to write a recommendation, and what it costs

Write one before every capacity decision: a new instance size, a quota request, a queue bound, an event such as Black Friday. Keep it short: one page, one demonstrated improvement, one next bottleneck. A long report with ten recommendations hides the one that matters.

Do not write a recommendation from one pass, from a busy computer, or from a test without a stated workload. Say “we do not know yet” instead. That is a valid result, and it tells the reader which measurement to pay for.

The cost is honesty about limits. A report that says “this test does not say how production behaves” can feel weak. It is the opposite: the reader can trust every sentence that remains.

See it end to endCapacity plan from a public traffic traceCase study

The case study writes a capacity plan from a real public traffic trace. It models the bursts, the queue that a given number of workers builds, and the limits of the plan.

Knowledge check

Check your understanding

Why is 'with a bound, a burst loses no ticket' a stronger claim than 'production handles 27 tickets per second', when both come from tests on the same laptop?

Show the answer

The first claim compares two designs under the same conditions and describes a behaviour: what the service does with tickets it cannot take. That behaviour comes from the code and the quota, not from the speed of the laptop. The second claim is an absolute number. It depends on the machine, the quota mode (off in that test), the simulated provider and the network, and production changes all of them.

Check your understanding

Larkfield gets a quota of 300 chat calls per minute. Which claims of the scorecard must you measure again, and which still hold?

Show the answer

The quota is assumption B, so every claim that rests on B must be measured again. The quota then allows 150 tickets per minute, more than the busiest minute’s 90. So the next limit may be the workers, the database or the real provider’s speed. “More workers do not help” may stop being true. The cost per ticket still holds: it rests on A and C, not on the quota. Run the comparison again with the new quota before you change the number of workers.

Check your understanding

Practical check: give the one demonstrated improvement and the next bottleneck of your own report, each in one sentence with its conditions.

Show the answer

For example: “With a queue bound of 100, a burst of 10 tickets per second for 30 seconds lost no ticket. Without a queue, it lost about 245 of 300 (quota on, 2 workers, simulated provider, one laptop, 3 passes).” And: “The next bottleneck is the provider’s quota: 50 tickets per minute against 90 in the busiest Black Friday minute. A second worker did not raise the finished work per second.” Your numbers differ; the conditions must be there.

Recap

• A capacity recommendation gives one demonstrated improvement, the next bottleneck, the measurements needed before the next change, and what the test does not say.
• Every claim rests on assumptions: the workload, the quota, the ticket’s cost and time, and where the test ran. Name them next to the claim.
• Claims about shapes and comparisons survive the move from laptop to production. Absolute production numbers do not.
• The provider’s quota limits Larkfield’s AI work to 50 tickets per minute, whatever the number of servers. The busiest Black Friday minute brings 90.
• A production claim needs a real run in a disposable copy of production. This course did not make one, and its report says so.

Next: Capacity plan from a public traffic trace