Benchmark the service

What you will be able to do

Measure the service the same way every time, so that you can compare two configurations. Use a fixed workload and a warm-up, and measure p50, p95 and throughput at a known number of requests at once. Then explain where the limit is, and say only what the measurements show.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Cold start and warm-up (the self-check is the warm-up), from Load and initialize: Cold start and warm-up.
• The latency budget, p95 under 50 ms with the network, from Online inference: The latency budget.
• CPU limits, throttling and workers, from Container resource settings: CPU limits and throttling.

What you need on your computer

• The image escalation-service:1.0.0, compose.yaml and .env with your digest, from Container resource settings: Guided practice.
• docs/infrastructure.md, from CPU, GPU and managed endpoints.
• The .venv of the project, active, and data/july_tickets.csv from Module 2.
• If something is missing or broken, copy the snapshot end-of-m03, then do the guided practice of the two lessons before this one. Use a snapshot has the steps.
• Close programs that use a lot of CPU, such as other containers or a video call. The FailureCase shows why.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested. Your times will be different: compare your own runs with each other, not with the course’s.

A café at the busy hour

One barista makes a coffee in one minute. At a quiet moment you order and get your coffee in one minute. At the busy hour, eight people are in front of you, and you wait nine minutes. The barista is not slower: she still makes one coffee a minute, sixty an hour. Your wait grew because of the queue.

So a café has two different numbers. How long one customer waits is its latency, from Online inference. How many coffees it makes in an hour is its throughput, from Batch inference. They are not the same, and more customers at once change the first one, not the second.

The problem: one number hides the slow requests

Tomás has an answer ready: “One request takes 3 ms. So the service does 333 requests per second, and every user waits 3 ms.” Both parts are guesses. Amira sends 20 tickets to the service, one after the other, and looks at every time. Sorted, in milliseconds:

3.7  3.7  3.9  3.9  3.9  3.9  4.0  4.0  4.0  4.1
4.1  4.2  4.2  4.3  4.6  5.0  5.6  6.0  6.5  16.4

The mean (the average) is 5.0 ms. But 15 of the 20 requests were faster than 5.0 ms, and one took 16.4 ms. The mean is not what a typical user waits, and it does not show the slow one. The help desk feels the slow ones.

Percentiles: p50 and p95

Sort the times. A percentile says how many requests are at or below a time. p50 (the 50th percentile, also called the median) is the time that half the requests stay under: the typical request. p95 is the time that 95% of the requests stay under: only one request in twenty is slower. Grace’s budget uses p95 because the agent who waits the longest still counts.

Now the worked example, with every number visible. With 20 sorted times, half is 10, so p50 is between the 10th and 11th time. Both are 4.1, so p50 = 4.1 ms. 95% of 20 is 19, so p95 is between the 19th time (6.5) and the 20th (16.4), very close to the 19th: bench.py gives 7.0 ms.

Twenty real request times, sorted. p50 is in the middle; p95 is near the 19th of 20. The one slow request moves the mean, but not p50.

Text description of the diagram

Twenty columns, one for each request, sorted from the fastest (3.7 ms) to the slowest (16.4 ms). Most columns are between 3.7 and 4.6 ms; then 5.0, 5.6, 6.0 and 6.5 ms; the last is 16.4 ms. A dashed line between the 10th and the 11th columns marks p50 = 4.1 ms. A dashed line between the 19th and the 20th columns marks p95 = 7.0 ms.

With 1,000 requests the idea is the same: p50 is about the 500th time, p95 about the 950th. Twenty requests are too few for a p95 you can trust, because one request decides it. That is one reason why a benchmark sends many.

A benchmark: the same workload every time

A benchmark is a measurement that you can repeat. It sends the same requests in the same way, so that two runs differ only in what you changed. bench/bench.py does this for the service:

• A fixed workload: the July tickets in file order, one ticket per request. Every run sends the same tickets.
• A warm-up: the script sends the first 50 requests but does not measure them. The first-request costs (connections, caches) then do not count as typical.
• Concurrency: --concurrency threads send requests at the same time. Each thread sends its next request as soon as its answer arrives.
• The results: p50, p95, p99, the slowest request (max) and the throughput, printed and added as one line to bench/results.jsonl, with the settings.

Make the folder bench and the file:

bench/bench.py

"""Send tickets to a running service and measure how long each answer takes.

    python bench/bench.py http://127.0.0.1:8000 --requests 1000 --concurrency 4

The workload is fixed, so two runs can be compared: the July tickets from
data/july_tickets.csv, in file order, one ticket per request. The first
--warmup requests are sent but not measured. Prints p50, p95, p99 and the
throughput, and adds one line to bench/results.jsonl with the settings.
"""

import argparse
import json
import os
import statistics
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from pathlib import Path

import pandas as pd

FEATURES = ["channel", "team", "segment", "region", "priority", "order_value", "word_count",
            "customer_tenure_days", "prior_tickets_90d", "created_hour", "prior_escalations_90d"]
INTS = ["priority", "word_count", "customer_tenure_days", "prior_tickets_90d", "created_hour", "prior_escalations_90d"]


def load_tickets(path: Path) -> list[bytes]:
    df = pd.read_csv(path, keep_default_na=False, na_values=[""], dtype={c: "Int64" for c in INTS})
    bodies = []
    for row in df[["ticket_id"] + FEATURES].to_dict("records"):
        ticket = {k: (None if pd.isna(v) else (v.item() if hasattr(v, "item") else v)) for k, v in row.items()}
        bodies.append(json.dumps(ticket).encode())
    return bodies


def percentile(values: list[float], p: float) -> float:
    return statistics.quantiles(values, n=100, method="inclusive")[p - 1]


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("url")
    parser.add_argument("--requests", type=int, default=1000)
    parser.add_argument("--concurrency", type=int, default=1)
    parser.add_argument("--warmup", type=int, default=50)
    parser.add_argument("--label", default="", help="a name for this configuration, for the results file")
    parser.add_argument("--tickets", type=Path, default=Path("data/july_tickets.csv"))
    args = parser.parse_args()

    bodies = load_tickets(args.tickets)
    headers = {"Content-Type": "application/json"}
    if os.environ.get("API_KEY"):
        headers["X-API-Key"] = os.environ["API_KEY"]

    def send(i: int) -> tuple[float, int]:
        request = urllib.request.Request(f"{args.url}/v1/score", data=bodies[i % len(bodies)], headers=headers)
        started = time.perf_counter()
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                response.read()
                status = response.status
        except urllib.error.HTTPError as error:
            status = error.code
        except OSError:
            status = 0
        return (time.perf_counter() - started) * 1000, status

    for i in range(args.warmup):
        send(i)
    started = time.perf_counter()
    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        results = list(pool.map(send, range(args.warmup, args.warmup + args.requests)))
    seconds = time.perf_counter() - started

    ms = [r[0] for r in results if r[1] == 200]
    errors = sum(1 for r in results if r[1] != 200)
    summary = {
        "label": args.label,
        "at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "requests": args.requests,
        "concurrency": args.concurrency,
        "errors": errors,
        "p50_ms": round(percentile(ms, 50), 1),
        "p95_ms": round(percentile(ms, 95), 1),
        "p99_ms": round(percentile(ms, 99), 1),
        "max_ms": round(max(ms), 1),
        "per_second": round(args.requests / seconds, 1),
    }
    print(f"{args.requests} requests, concurrency {args.concurrency}: p50 {summary['p50_ms']} ms, "
          f"p95 {summary['p95_ms']} ms, p99 {summary['p99_ms']} ms, max {summary['max_ms']} ms, "
          f"{summary['per_second']} requests/s, {errors} errors.")
    with open(Path(__file__).parent / "results.jsonl", "a", encoding="utf-8") as out:
        out.write(json.dumps(summary) + "\n")


if __name__ == "__main__":
    main()

percentile uses Python’s statistics.quantiles with the inclusive method: the same p50 and p95 as the worked example. Only requests with the status 200 count as times; the others count as errors, because a fast error is not a fast answer. results.jsonl is git-ignored: it is your lab notebook, not code.

Worked example: one request at a time, then eight

Amira starts the service with one worker and one CPU (WORKERS=1, CPUS=1.0 in .env). Then she runs the benchmark, one request at a time:

Terminal, in ~/projects/escalation-service

docker compose up -d --wait
python bench/bench.py http://127.0.0.1:8000 --requests 1000 --concurrency 1 --label "workers=1 cpus=1.0"

The output:

1000 requests, concurrency 1: p50 4.1 ms, p95 6.1 ms, p99 8.7 ms, max 52.3 ms, 221.6 requests/s, 0 errors.

Predict: now eight requests at a time. What happens to p50, and to the requests per second?

Terminal

python bench/bench.py http://127.0.0.1:8000 --requests 1000 --concurrency 8 --label "workers=1 cpus=1.0"

The output:

1000 requests, concurrency 8: p50 31.6 ms, p95 72.5 ms, p99 105.7 ms, max 136.1 ms, 216.5 requests/s, 0 errors.

Throughput did not grow: 221.6 and 216.5 requests per second. p50 grew about eight times, from 4.1 to 31.6 ms. This is the café: one worker is one barista. With eight requests at once, a new request waits for about seven others, so it waits about eight request times. Check it: 8 requests ÷ 216.5 per second = 37 ms, close to the measured p50. The p95 of 72.5 ms misses Grace’s budget of 50 ms, before any network.

Explore: concurrency and the queue

Amira measured every number in the explorer: 1,000 requests for each setting, the middle of five runs. The time axis is a log scale. Each step of the same length multiplies the time by ten (1, 10, 100 ms), so 4 ms and 60 ms both fit on one axis.

Explore

Latency: what happens when more requests wait at once?

p50 (median)

4.1 ms

p95

6.1 ms

Requests per second

222

1 worker, 1 CPU, concurrency 1: one request at a time, so no request waits for another. Half of the requests take 4.1 ms or less. Most requests take about the same time, so the bars stand close together.

1 worker, 1 CPU: 1,000 requests at each concurrency

Concurrency	p50	p95	Requests per second
1 (shown)	4.1 ms	6.1 ms	222
4	14.9 ms	27.1 ms	232
8	31.6 ms	72.5 ms	216
16	60.1 ms	112 ms	243

1,000 measured request times, one worker with one CPU. Pick the concurrency: the number of requests sent at the same time.

Text description of the explorer

A histogram of 1,000 measured request times for one worker with one CPU, with dashed lines at p50 and p95, and a table of the four runs. Concurrency 1: p50 4.1 ms, p95 6.1 ms, 222 requests per second; almost every request is in the bars from 2 to 8 ms. Concurrency 4: p50 14.9 ms, p95 27.1 ms, 232 requests per second. Concurrency 8: p50 31.6 ms, p95 72.5 ms, 216 requests per second. Concurrency 16: p50 60.1 ms, p95 111.7 ms, 243 requests per second. As the concurrency grows, the whole histogram moves to the right and spreads out. The throughput stays between about 215 and 245 requests per second.

The throughput stays near 220 requests per second at every concurrency. One worker is the limit: it uses one CPU, and every extra request only waits longer. A part of a system that limits the whole is the bottleneck.

Explore: four configurations

Now the second control: the configuration. Two workers with two CPUs, two workers that share one CPU, and one worker with half a CPU.

Explore

Latency: workers and CPUs

p50 (median)

16.2 ms

p95

37.3 ms

Requests per second

423

2 workers, 2 CPUs, concurrency 8: 8 requests are sent at the same time. p50 is 16.2 ms, 3.9 times the 4.2 ms at concurrency 1. Throughput is 2 times the throughput at concurrency 1 (423 per second). More work is done at once, because more than one CPU works, but each request still waits for a turn.

2 workers, 2 CPUs: 1,000 requests at each concurrency

Concurrency	p50	p95	Requests per second
1	4.2 ms	6.9 ms	213
4	9.8 ms	20.2 ms	357
8 (shown)	16.2 ms	37.3 ms	423
16	37.4 ms	88.4 ms	366

The same workload with four configurations of workers and CPUs. Pick the configuration and the concurrency.

Text description of the explorer

The explorer starts with two workers and two CPUs at concurrency 8: p50 16.2 ms, p95 37.3 ms, 423 requests per second.

The four configurations at concurrency 1 and 8, as p50, p95 and requests per second. 1 worker, 1 CPU: 4.1 ms, 6.1 ms, 222; and 31.6 ms, 72.5 ms, 216. 2 workers, 2 CPUs: 4.2 ms, 6.9 ms, 213; and 16.2 ms, 37.3 ms, 423. 2 workers, 1 CPU: 4.7 ms, 8.0 ms, 190; and 21.4 ms, 87.2 ms, 209. 1 worker, 0.5 CPU: 4.1 ms, 44.1 ms, 112; and 88.6 ms, 196.2 ms, 82. Memory: about 155 MiB with one worker and about 350 MiB with two.

The same numbers as a table, for concurrency 1 and 8:

Configuration	Concurrency 1: p50 / p95 / req/s	Concurrency 8: p50 / p95 / req/s	Memory
1 worker, 1 CPU	4.1 / 6.1 ms / 222	31.6 / 72.5 ms / 216	155 MiB
2 workers, 2 CPUs	4.2 / 6.9 ms / 213	16.2 / 37.3 ms / 423	347 MiB
2 workers, 1 CPU	4.7 / 8.0 ms / 190	21.4 / 87.2 ms / 209	349 MiB
1 worker, 0.5 CPU	4.1 / 44.1 ms / 112	88.6 / 196.2 ms / 82	155 MiB

Two workers with two CPUs almost double the throughput under load: there are two baristas. Two workers on one CPU do not: they take turns, and p95 gets worse. Half a CPU halves the throughput, and throttling makes p95 44.1 ms even with one request at a time.

Cold and warm, and where the time goes

A benchmark measures a warm service: started, loaded, past its warm-up. The first request after a start is cold for the platform, but not for the model. The self-check scored the 14 parity cases at start-up: that is the warm-up. On one CPU, the first score after /ready took 4.5 to 5.7 ms. Later ones took a median of 3.1 to 5.5 ms. The real cold cost is the start itself.

The latency breakdown, measured on the laptop with 1 CPU. Start-up is almost all imports; one request is mostly preprocessing and HTTP. The model is the smallest part of both.

Text description of the diagram

Two stacked bars. The start-up bar, once per worker with 1 CPU, is about 1.6 s: container start 0.15 s; importing FastAPI, uvicorn and pydantic 0.41 s; importing NumPy, pandas and joblib 0.31 s; importing scikit-learn 0.52 s; loading the bundle 16 ms in total (the hashes and versions, joblib.load and the self-check); and the server start 0.21 s.

The request bar, the median of 900 warm requests, is about 3.1 ms: HTTP, the network and the client 0.68 ms; the contract checks, JSON and the log line 0.41 ms; frame 0.57 ms; prepare, the preprocessing, 1.38 ms; and model 0.06 ms.

Start-up is about 500 times one request, and almost all of it is importing libraries. Loading and checking the 5 kB bundle is 16 ms. In each request, the model is 0.06 ms; preprocessing and HTTP are most of the time.

The bottleneck, and what the numbers do not say

Amira writes what the measurements show, and only that:

On this laptop, with the July tickets, one worker with one CPU stops at about 220 requests per second. It is CPU-bound: more requests at once only wait longer. Two workers with two CPUs reached about 420. At half a CPU, p95 was 44 ms even with one request at a time, while the benchmark kept the service busy.

Tomás: “44 ms is almost the whole budget. Let us pay for 1 vCPU.” Amira checks the conditions first. The benchmark sends the next ticket as soon as the last answer arrives. So the service is busy all the time, and the CPU limit throttles a busy container. Larkfield sends about 45 tickets a day. The service is idle almost all the time, and a single request never uses up the 50 ms of a period. The benchmark did not measure that load, so it cannot decide the size. Amira keeps 0.5 vCPU and writes the condition for a change in the decision record.

Omar asks: “So in the cloud we can serve 220 requests a second?” Amira does not say yes either. The benchmark ran on a laptop, with the client and the service on the same computer, and no network. The cloud has other CPUs, and a network between the help desk and the service.

The course team measured the service on Azure Container Apps (live). The client was in Toronto and made a new HTTPS connection for each of 25 warm requests:

Size	First request from zero replicas	Warm request, total p50	Waiting for the server, p50	Server-Timing, warm (frame / prepare / model)
0.5 vCPU, 1 GiB	26.4 s	96 ms	38 ms	2.00 / 4.38 / 0.17 ms
1 vCPU, 2 GiB	23.9 s	99 ms	41 ms	2.05 / 5.31 / 0.18 ms

Twice the CPU saved about 2.5 s of the cold start, and it did not make one warm request faster. Of the 96 ms, about 22 ms went to connecting and about 32 ms more to the TLS handshake (the set-up of HTTPS). Only then did the server see the request. The cold start is mostly the platform finding a machine and downloading the 166 MB image. And the cloud’s CPU did the preprocessing about three times slower than the laptop. Nobody could predict these numbers from the laptop.

So the honest summary has two parts. Measured: the bottleneck on the laptop, and the throttling of a busy container at half a CPU. Not measured by the benchmark: the cloud’s throughput and the network. For the 50 ms budget, the network decides. A new HTTPS connection for each score costs more than the budget, so the help desk page must keep its connection to the service open. Larkfield’s load, about 45 tickets a day, is far below 220 a second anyway.

Common mistake

Two runs of the same benchmark disagree

Symptom: Amira runs the same configuration five times: one worker, one CPU, concurrency 8, the same 1,000 tickets. Nothing changed between the runs. The throughput:

202, 209, 216, 237, 283 requests/s

The fastest run is 40% faster than the slowest. With only one run, she could have “proved” that any change makes the service faster or slower.

Cause: the benchmark and the service share the laptop with every other program. During these runs, other containers and programs used several CPUs (the load average was 5 to 9 on 10 CPUs). The benchmark’s client also uses CPU. When something else takes CPU time, both the client and the service wait.

Fix: close other programs. Run each configuration at least three times, and keep the middle one (the median), not the best. Run the configurations you compare one after the other, at the same time of day. Report the spread with the numbers. The course’s explorer uses the middle of five runs for each setting.

For a real AI service: in the cloud, other customers share the machines too. Never decide from one run, and never compare a number from your laptop with a number from the cloud. Write the conditions next to the result: computer, workload, concurrency, date.

Module practice

Guided practice → independent task. The worked example compared concurrency 1 and 8 for one configuration. Now compare two configurations, and record the workload, so that someone else can repeat it.

1. Close other programs that use a lot of CPU.
2. In .env, set WORKERS=1 and CPUS=1.0. Start with docker compose up -d --wait.
3. Run bench/bench.py with --requests 1000 at concurrency 1 and 8, with a --label that names the configuration. Run each three times.
4. In .env, set CPUS=0.5. Start again with docker compose up -d --wait. Check with docker inspect that NanoCpus is 500000000.
5. Run the same six benchmarks with the new label.
6. Run docker stats --no-stream for the memory.
7. Open bench/results.jsonl. For each configuration and concurrency, take the middle run.
8. In docs/infrastructure.md, add the section Measured on a laptop: the workload, the computer, the table and the bottleneck. Say what is not measured.
9. Compare p95 with the 50 ms budget, under the load that the service really gets. If the decision must change, change it, and write the date and the reason.

Show the model answer: the new section of docs/infrastructure.md

Amira’s numbers. Yours are different; the pattern is the same.

docs/infrastructure.md

## Measured on a laptop

Workload: `bench/bench.py`, 1,000 July tickets in file order, one ticket per request, 50 warm-up requests.
Computer: Apple silicon laptop, Docker Desktop, other programs running. Each number is the middle of five runs.

| Configuration | Concurrency 1: p50 / p95 / req/s | Concurrency 8: p50 / p95 / req/s | Memory |
|---|---|---|---|
| 1 worker, 1 CPU | 4.1 / 6.1 ms / 222 | 31.6 / 72.5 ms / 216 | 155 MiB |
| 1 worker, 0.5 CPU | 4.1 / 44.1 ms / 112 | 88.6 / 196.2 ms / 82 | 155 MiB |

- Cold start (docker run to /ready, image on the computer): 1 CPU 2.0 s, 0.5 CPU 3.5 s (median of 8 runs).
- Bottleneck: one worker uses one CPU. More requests at once only wait longer; throughput stays near 220 requests per second.
- At 0.5 CPU, p95 was 44 ms even with one request at a time, because the CPU limit pauses a busy container (throttling). The benchmark kept the service busy all the time. Larkfield's 45 tickets a day leave it idle almost all the time, so this is not the load we serve. We keep 0.5 vCPU, and we watch p95 in production (Module 5). If the load grows to keep the CPU busy, we move to 1 vCPU (US$28.94 to US$101.30 for 30 days).
- For the 50 ms budget, the open question is the network between the help desk and the service, which this benchmark does not include.
- Not measured here: the cloud network, the cloud's CPUs and the image download. Do not use these numbers as cloud numbers.

In The decision, she adds a line under the size, so that a reader sees the check:

docs/infrastructure.md

  Checked after the benchmark: no change (see below).

The boxes of the first three checks become [x]. The last one, the cloud, stays open until the first deployment.

The independent task: choose two configurations of your own, for example WORKERS=2 CPUS=2.0 against WORKERS=1 CPUS=1.0 at concurrency 16. Predict the result, measure each three times, and write the result and the conditions in the same section.

Module check

Check your understanding

Retrieval: what do p50 and p95 mean, and why does Grace's budget use p95?

Show the answer

Sort the request times. p50 is the time that half the requests stay under: the typical request. p95 is the time that 95% stay under, so only one request in twenty is slower. A budget on p95 protects the agents who wait the longest; a budget on the mean or p50 would hide them. In the worked example, the mean was 5.0 ms because of one request of 16.4 ms, while p50 was 4.1 ms.

Check your understanding

Predict: with one worker and one CPU, concurrency 16 instead of 8. What happens to p50 and to requests per second, and why?

Show the answer

p50 about doubles, and throughput stays about the same. One worker on one CPU does a fixed number of requests per second, about 220 here; it is the bottleneck. Twice as many requests at once means each one waits for about twice as many others. The measurement: p50 31.6 ms at 8, 60.1 ms at 16; 216 and 243 requests per second.

Check your understanding

Completion check: explain the bottleneck of the escalation service from your measurements, without claiming anything that you did not measure.

Show the answer

A good answer names the conditions, the limit and the evidence, and says what is not known. For example: “On my laptop (Apple silicon, Docker Desktop), I sent 1,000 July tickets, one ticket per request. One worker with one CPU stopped at about 220 requests per second at concurrency 1, 8 and 16, while p50 grew with the concurrency. So the worker’s one CPU is the bottleneck. With 0.5 CPU, p95 was 44 ms even at concurrency 1, because of throttling. I did not measure the cloud: its CPUs, its network and the image download are different, so these numbers are not cloud numbers.” An answer that says “the service handles 220 requests per second in production” or “the cloud will be as fast” extrapolates beyond the measured conditions.

Recap

Module 4 chose where the model runs and measured it:

• CPU, GPU and managed endpoints: the model step is 0.05 ms of about 3 ms, so a CPU is enough. A container on Azure Container Apps costs far less than a managed endpoint for 45 tickets a day. docs/infrastructure.md records the decision.
• Container resource settings: the image contains the bundle; compose.yaml sets CPUs, memory without swap and workers. Throttling, workers = CPUs, and OOMKilled when the workers do not fit.
• Benchmark the service: a fixed workload, a warm-up, p50 and p95, throughput and concurrency. One worker on one CPU is the bottleneck at about 220 requests per second on the laptop. Half a CPU throttles a busy service, but Larkfield’s load keeps the service idle, so the size stays. In the cloud (live run), the network and the image download dominate.

Reset. To start Module 5 from a clean state:

1. Stop the service: docker compose down.
2. If your project is broken, copy the snapshot end-of-m04 to ~/projects/escalation-service, as in Use a snapshot. Then copy .env.example to .env and paste the digest of the snapshot’s bundle.
3. To remove the image as well: docker image rm escalation-service:1.0.0. The next docker compose up -d --build --wait builds it again.

Next, a new model version arrives. You release it next to the old one, compare them, and keep the way back open.

Next: Version and rollout