Latency and capacity

What you will be able to do

Learn to read a load test’s times with the right numbers: averages, percentiles, concurrency and queues. Then record Larkfield’s first baseline, the throughput and p95 latency under a stated workload, so every later change has something to beat.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Latency, throughput and your first Locust run, from Run the AI workload on your computer.
• The workload model and the closed loop, from Workload model.

What you need on your computer

• Your project from the last lesson, with the small data loaded (python -m scripts.load --reset). python -m pytest prints 123 passed.
• PostgreSQL and Azurite running. You start the simulator and the API in the guided practice.

How late is the bus?

A bus company says that its buses are “on average 3 minutes late”. Most days, your bus is 1 minute late. But once or twice a month, it breaks down, and it is 40 minutes late.

The average is true, but it is not useful. On a normal day it makes you wait for nothing. And it says nothing about the day of your job interview, when you cannot be 40 minutes late. For that day, you want another kind of number: “On 19 days of 20, the bus is at most 4 minutes late.”

Request times work in the same way. Most are close together, and a few are very slow. The few slow ones are the ones that customers remember.

The problem: the average hides the slow requests

Your first Locust table said Avg 2388. Omar reads it and asks, “So every customer waits about 2.4 seconds?” No. Here is a tiny example that shows why. It is made up, so every value is visible.

Twenty requests: nineteen take 100 ms, and one takes 5,000 ms.

Request	1 to 19	20
Time	100 ms each	5,000 ms

The mean is (19 × 100 + 5,000) ÷ 20 = 6,900 ÷ 20 = 345 ms. Nobody waited 345 ms. Nineteen customers waited 100 ms, and one waited 5 seconds. The mean is pulled up by the one slow request, and it describes no real request. One of the project’s tests, test_the_average_hides_the_slow_requests, checks this same shape with 100 requests.

The idea: percentiles

Sort the times from fast to slow. A percentile is the time at a fixed position in that sorted list:

• p50, the median: half of the requests took this long or less.
• p95: 95 of 100 requests took this long or less. One request in twenty was slower.
• p99: 99 of 100 requests took this long or less.

The name comes from “per cent”: p95 is the 95th of every 100. The slow end of the list is the tail, so p95 and p99 are called tail latency.

How to find p95 in a list of n times: sort them, compute 0.95 × n, round up, and take the time at that position. This is the nearest-rank method, and scripts.percentiles uses it. In the tiny example, n is 20:

Number	Position	Value
p50	0.50 × 20 = 10 → the 10th	100 ms
p95	0.95 × 20 = 19 → the 19th	100 ms
p99	0.99 × 20 = 19.8 → the 20th	5,000 ms
max	the 20th	5,000 ms

Two lessons hide in this table. First, p50 and p95 say “100 ms”, which is true for 19 of 20 customers. Second, with only 20 requests, p99 is the slowest request. A tail percentile needs many requests before it means much.

Which number would you promise? Try it on real times.

Explore

Which number describes a slow, uneven time?

classify

958 ms (35 of 100 slower)

draft a reply

1,140 ms (37 of 100 slower)

embed

200 ms (47 of 100 slower)

classify, mean: 958 ms; 35 of 100 recorded calls took longer. The mean adds all the times and divides by their number. A few slow requests pull it up, so it is above the median, and it belongs to no real request. For classify, the mean 958 ms is 45 ms above the median 913 ms.

100 real calls per task, one at a time

Task	mean	p50 (median)	p95	p99	max
classify	958 ms	913 ms	1,397 ms	1,985 ms	2,143 ms
draft a reply	1,140 ms	1,097 ms	1,514 ms	2,141 ms	2,172 ms
embed	200 ms	198 ms	300 ms	333 ms	341 ms

Each dot is one measured time. The calls are 100 real recorded calls per task, one at a time. The tickets are every request of a 1-user load test with the simulated provider (the author's run, 5 passes; your numbers will differ).

Text description of the explorer

The explorer shows dot strips of measured times, one dot per call. In the first view, three strips show the 100 real recorded calls of each AI task, one call at a time. Classify: mean 958 ms, p50 913 ms, p95 1,397 ms, p99 1,985 ms, max 2,143 ms. Draft a reply: mean 1,140 ms, p50 1,097 ms, p95 1,514 ms, p99 2,141 ms, max 2,172 ms. Embed: mean 200 ms, p50 198 ms, p95 300 ms, p99 333 ms, max 341 ms. In the second view, one strip shows 106 whole tickets from a 1-user load test with the simulated provider (the author’s run): mean 2,265 ms, p50 2,230 ms, p95 2,796 ms, p99 3,088 ms, max 3,358 ms. Choosing a statistic moves a marker on every strip, and the readout says how many times were slower than it.

What this tells us. The times have a long right tail: most calls are close together, and a few are much slower. So the mean sits above the median: 958 ms against 913 ms for classify. The maximum is real, but it is one event, and the next test has another one. p95 is the usual choice for a promise. It is kept for 19 customers of 20, and it moves only a little between runs. Switch to the whole tickets: the tail is still there, because each ticket adds three uneven calls.

Omar now writes his promise in a better form: “95% of new tickets are saved within 5 seconds.” That is a percentile and a limit together. A target of this kind is often called a service level objective.

Concurrency and throughput

Concurrency is the number of requests in progress at the same time. In a closed-loop test with no pause, it equals the number of Locust users.

There is a simple rule that connects the three words:

throughput = concurrency ÷ latency

Check it with one user. One ticket is in flight, and it takes about 2.3 seconds, so throughput is 1 ÷ 2.3 = 0.43 tickets per second. With two users, two tickets wait at the same time: 2 ÷ 2.3 = 0.87 per second. This rule is Little’s law, after John Little, who proved it in 1961. It holds for any system that is stable over time.

The course measured the same test with 1, 2 and 4 users, five passes of 60 seconds each, on a quiet computer:

Successful tickets per second, medians of 5 passes, Module 1's design, simulated provider with the recorded quota. One and two users follow Little's law. Four users did not double again: the provider refused part of the calls.

Text description of the chart

Successful tickets per second, medians of 5 passes: 1 user 0.43 (0.41 to 0.45), 2 users 0.85 (0.83 to 0.88), 4 users 1.54 (1.46 to 1.57). In every pass with 4 users, the provider refused between 41 and 115 calls, and those tickets got an error.

One and two users follow the rule: 0.43, then 0.85. Four users should give about 1.7, but they gave 1.54. Something refused work. Find the bottleneck finds out what.

Queueing: why waiting grows

A closed loop keeps the concurrency fixed. Real customers do not, as the last lesson showed. When requests arrive faster than the service finishes them, they wait in a queue: a line of work that has arrived but has not started.

Here is a tiny example. One clerk needs 2 seconds per customer. Four customers arrive one second apart:

Customer	Arrives at	Starts at	Finishes at	Waited in line	Total time
1	0 s	0 s	2 s	0 s	2 s
2	1 s	2 s	4 s	1 s	3 s
3	2 s	4 s	6 s	2 s	4 s
4	3 s	6 s	8 s	3 s	5 s

The work per customer never changes: 2 seconds. But the total time grows by one second for each new customer, because customers arrive every second and the clerk needs two. If they keep coming, the line grows without end. Now the arrival rate is higher than the throughput that the clerk can give. That is a queue that never empties.

This leads to capacity: the highest arrival rate that a service can handle while it keeps its promise. Above it, a queue grows, and so does the latency.

For Larkfield, one capacity limit is already clear from the numbers. The quota allows 100 chat calls per minute, and each ticket needs two. So the provider can take at most 50 tickets per minute: 0.83 tickets per second. The busiest Black Friday minute brings 1.5 per second. Whatever the API does, the extra tickets must wait, or fail.

Guided practice: record the baseline

A baseline is a measurement of the current design under a stated workload. You keep it, and you compare every later change with it. Without a baseline, “it is faster now” means nothing.

You need two scripts and a template. scripts.percentiles computes exact percentiles from every request. scripts.machine prints the computer. docs/benchmark-template.md is the form for the result.

1. In ~/projects/ticket-api, with the virtual environment active, copy the four files of this lesson from the snapshot end-of-m01:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m01
   for f in scripts/percentiles.py scripts/machine.py docs/benchmark-template.md tests/test_measure_tools.py; do
     cp "$S/$f" "$f"
   done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m01"
   foreach ($f in 'scripts\percentiles.py', 'scripts\machine.py', 'docs\benchmark-template.md', 'tests\test_measure_tools.py') { Copy-Item "$S\$f" $f }

2. Run the tests: python -m pytest. Expect 128 passed, 1 deselected.

3. Close other heavy programs: a video call, a build, a large download. They take CPU time from your test. Then print your computer:

   python -m scripts.machine

   Computer: Apple M4, 10 cores, 16.0 GB memory; Darwin 25.6.0 (arm64)
   Python 3.12.13; Docker: Engine 29.6.1, 10 CPUs, 7.8 GB
   Load average (1 minute): 5.04 (other programs that use the CPU make the numbers worse)

   This is the author’s computer, shared with other work at that moment. The load average counts the programs that want the CPU. The course’s own measurements waited until it was under 3. On Windows, the script prints no load average.

4. Start the simulator in terminal 1 (python -m simulator), as before. In terminal 2, start the API with fastapi run instead of fastapi dev. The dev mode watches your files for changes, which is extra work that a measurement does not need.

   fastapi run

5. In terminal 3, run the 1-user workload for 60 seconds, and save every request to run.csv:

   macOS / Linux

   SAMPLES_CSV=run.csv locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 1 -r 1 -t 60s

   Windows (PowerShell)

   $env:SAMPLES_CSV = "run.csv"
   locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 1 -r 1 -t 60s
   Remove-Item Env:SAMPLES_CSV

6. Compute the exact percentiles twice: with all requests, and without the first 10 seconds:

   python -m scripts.percentiles run.csv
   python -m scripts.percentiles run.csv --skip 10

The author’s run printed this. Your numbers will differ:

24 requests in 58.0 s; status codes {'201': 24}
Successful: 24 = 0.41 per second
Latency of the successful requests: mean 2414.9 ms | p50 2298.6 ms | p95 3231.2 ms | p99 3306.4 ms | max 3306.4 ms
19 requests in 46.0 s; status codes {'201': 19}
Successful: 19 = 0.41 per second
Latency of the successful requests: mean 2423.4 ms | p50 2352.8 ms | p95 3306.4 ms | p99 3306.4 ms | max 3306.4 ms

Locust’s own table for the same run said Med 2400 and 95%: 3200. Locust rounds each time before it counts, and its table says “approximated”. scripts.percentiles uses every exact time: p50 2,298.6 ms and p95 3,231.2 ms.

--skip 10 leaves out the warm-up: the first seconds, when connections open and code loads for the first time. Here it changed little. In the second line, p95, p99 and the maximum are the same request. With 19 requests, the 95th position is already the last one. One short run gives you a shape, not a stable p95.

That is why the course measured five passes and used the median of each number. On a quiet computer, with the same workload, the result was:

	1 user (median of 5 passes)	lowest and highest pass
Throughput	0.43 tickets per second	0.41 to 0.45
p50	2,200 ms	2,168 to 2,318
p95	2,995 ms	2,676 to 3,340
p99	3,098 ms	2,772 to 3,616
API CPU	0.6%	0.4 to 0.6%

The same 60-second test, five times, on a quiet computer: p95 moved from 2,676 to 3,340 ms. One pass alone could tell you either number.

Text description of the chart

The p95 of each of the five passes with 1 user: 3,340, 3,138, 2,676, 2,817 and 2,995 ms. The median of the five is 2,995 ms.

What this tells us. Nothing changed between the passes: the same code, the same workload, the same computer. Still, p95 moved by 664 ms, about 22%. With about 20 requests per pass, one slow provider call changes p95. So a difference of a few hundred milliseconds between two single runs proves nothing. Repeat, report the median, and show the lowest and highest pass.

Write the result. Copy docs/benchmark-template.md to docs/baseline.md, and fill it in for your own run: the question, the output of scripts.machine, the workload table and your numbers. Write the workload so exactly that another person can repeat it.

Show a filled example

The question	How many new tickets per second does Module 1’s design finish with one user, and how long do they take?
The machine	Apple M4, 10 cores, 16 GB, macOS; Docker 10 CPUs, 7.8 GB; Python 3.12.13; load average under 3, nothing else heavy running
Endpoint and request	POST /v1/tickets, the 200 practice tickets, picked with the fixed seed 21
Load generator and command	SAMPLES_CSV=run.csv locust -f loadtest/locustfile.py --headless -u 1 -r 1 -t 60s
Shape	steady, closed loop, 1 user, no pause; 60 s, the first 10 s left out
Dependencies	PostgreSQL 18.6, small data; AI provider simulated (normal mode, recorded quota of 100 chat requests per minute)
Configuration	fastapi run, one process, the defaults of .env.example
Passes	5; the median of each number, with the lowest and highest pass
Results	0.43 tickets per second (0.41 to 0.45); p50 2,200 ms; p95 2,995 ms (2,676 to 3,340); p99 3,098 ms; no errors; API CPU 0.6%

When to use which number

Use p95 for promises and for comparisons between designs. Use p99 only when you have thousands of requests, so the 99th position rests on many requests, not one or two. Use the median to describe the typical request. Report the maximum to show that it exists, but do not compare designs on it.

Do not use the mean of latencies for a promise. The mean is still useful for one thing: totals. The mean time multiplied by the number of requests is the total waiting time, for example to estimate a bill.

Report throughput as successful requests per second. A request that failed in 5 ms is not useful work. scripts.percentiles counts only 2xx answers for this reason.

Percentiles cost something. You must keep every request’s time, as SAMPLES_CSV does, or use a tool that estimates them, as Locust does. And you need enough requests: about 20 per pass, as here, gives a rough p95 and no real p99.

Common mistakes

Common mistake

A busy computer makes the numbers worse

Symptom: python -m scripts.machine prints a high load, for example Load average (1 minute): 19.8, and your p95 is higher than in the last run.

Cause: other programs compete for the CPU and the disk. A measurement then measures them too. The author’s computer showed 19.8 while other heavy work ran on it.

Fix: close the other programs, wait one or two minutes, and run scripts.machine again. Measure when the load average is low, and write it into your result. If you cannot make the computer quiet, say so next to the numbers.

Common mistake

Comparing one run with another

Symptom: you change nothing, run the test again, and p95 moves by several hundred milliseconds. Or you change one setting, and a single run looks 10% faster.

Cause: short runs have few requests, so the tail rests on one or two slow calls. The five baseline passes moved p95 between 2,676 and 3,340 ms.

Fix: run at least three passes, preferably five, of each configuration. Compare the medians, and show the range. If the ranges overlap, the test has not shown a difference.

Your turn: two users

Change one thing: run the same test with 2 users (-u 2 -r 2), with SAMPLES_CSV. Before you run it, predict the throughput and the p95 with Little’s law. Then compare with your 1-user result, and add a row to docs/baseline.md.

Show the answer

Little’s law predicts twice the throughput: 2 ÷ 2.3 s = about 0.87 tickets per second. The latency should stay about the same, because each ticket still waits for its own three calls. The course measured 0.85 tickets per second (0.83 to 0.88) and a p95 of 2,988 ms (2,652 to 3,139) with 2 users, over five passes. That is no change in p95: the ranges of 1 and 2 users overlap. Your single run will differ, but it should show about twice the throughput.

See it end to endCapacity plan from a public traffic traceCase study

The traffic-trace case study starts with the arithmetic of the average. It says that one job slot is plenty, but the real bursts need three, and the plan recommends four.

Knowledge check

Check your understanding

A test of 20 requests gives: mean 900 ms, p50 400 ms, p95 3,800 ms, max 4,000 ms. Omar wants to write '95% of tickets in under 1 second'. Is that true? Which number tells you?

Show the answer

No. p95 is 3,800 ms: 95% of the requests took up to 3.8 seconds, and one in twenty was slower than that. The mean (900 ms) is under 1 second only because many requests were fast; it hides the slow ones. Omar could write “half of the tickets in under 0.4 seconds” (p50) or “95% in under 4 seconds” (p95). With only 20 requests, p95 is the 19th time, so he should also run more requests before he promises anything.

Check your understanding

Predict with Little's law: 8 Locust users, no pause, and each ticket takes about 2.3 seconds. What throughput do you expect? Name one thing that could make the real result lower.

Show the answer

8 ÷ 2.3 = about 3.5 tickets per second. That needs about 7 chat calls per second, or 420 per minute, against a quota of 100 per minute. So the provider would refuse most calls after the first burst, and the successful throughput would be much lower. Other limits could also stop it, such as the database connection pool of 10. The next lesson and Module 2 measure these limits.

Check your understanding

Practical check: give your baseline in one sentence, with the workload and the computer, as another person could repeat it.

Show the answer

For example: “On an Apple M4 with 16 GB (load average under 3), Module 1’s design with the simulated provider (recorded quota) finished 0.43 tickets per second, p95 2,995 ms (2,676 to 3,340 over 5 passes). The workload was loadtest/locustfile.py: 1 user, no pause, 60 s, the first 10 s left out, the practice tickets with seed 21, small data.” Your numbers differ; the parts must all be there: the result, the range, the workload and the computer.

Recap

• The mean of latencies hides the slow requests. Percentiles do not: p50 is the typical request, p95 the usual number for a promise, p99 the far tail.
• Concurrency is the number of requests in progress. Little’s law: throughput = concurrency ÷ latency. One user gave 0.43 tickets per second, two gave 0.85.
• A queue grows when work arrives faster than it finishes. Capacity is the highest rate that keeps the promise. The quota caps Larkfield at 0.83 tickets per second.
• The baseline of Module 1’s design, 1 user, quiet computer: 0.43 tickets per second, p95 2,995 ms (2,676 to 3,340 over 5 passes).
• Every result states its workload and its computer. Short runs vary: repeat, use the median, and show the range.

Next, you split a ticket’s 2.2 seconds into its parts, and you find what stopped four users from doubling the throughput again.

Next: Find the bottleneck