Load-test design

What you will be able to do

Plan a load test so that it answers one question about Black Friday. Give it a stated workload and a shape over time. Run it in a disposable environment on your own computer, on a quiet machine.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The workload model, the closed loop and the open stream, from Workload model.
• Percentiles, the warm-up and passes, from Latency and capacity.
• The queue, the worker and scripts.queue_watch, from Job lifecycle and Backpressure.

What you need on your computer

• The project after Graceful shutdown: python -m pytest prints 172 passed, 1 deselected.
• PostgreSQL, Azurite and Valkey running: docker compose up -d db azurite cache.

After this lesson, your project matches the snapshot end-of-m06, except for the files of the next two lessons. Tested on macOS (Apple silicon); the Windows commands are not tested. The AI provider is the simulated one. The numbers of this module come from an Apple M4 computer (10 cores, 16 GB). The course’s measurements ran with nothing else busy. The author’s runs, labelled each time, ran on the same computer with other work, and each one gives its load average. Your numbers will differ.

A new bridge, before it opens

Before a new bridge opens, engineers test it with heavy lorries. They close the bridge to traffic first. They drive one lorry on, then two, then four, and measure how far the bridge bends at each step. They park a line of lorries for an hour. They send a sudden convoy. Some lorries stay overnight, to see whether anything slowly gives way.

The engineers never test a bridge that is open to the public, and they write down every load and every reading. Their report says “with 12 lorries of 40 tonnes, the deck bent 18 mm”. It does not say “the bridge is strong”.

A load test is the same idea for a service. You put a planned load on a service that you own and may break, and you write down every reading. This lesson plans load tests for Larkfield’s Black Friday.

The problem: a fast answer that hid a long wait

Tomás wanted proof that the queue design is ready. He ran the burst shape of this lesson with its strongest setting. It sends 2 tickets per second, then 20 per second for 15 seconds, then 2 per second again. scripts.percentiles printed this (the author’s run, your numbers will differ):

518 requests in 80.0 s; status codes {'202': 474, '429': 44}
Successful: 474 = 5.92 per second
Latency of the successful requests: mean 16.2 ms | p50 14.9 ms | p95 31.2 ms | p99 37.9 ms | max 38.6 ms

“Every saved ticket was answered in under 39 ms”, said Tomás. “The 44 refusals are the rate limit for each customer, from Backpressure. It works. We are ready for Black Friday.” Then Omar asked the question that the test did not measure: when did the customers get their AI reply?

The 202 answer only says “your ticket is saved, and its AI work waits in the queue”. The AI work itself is limited by the provider’s quota: 100 chat calls per minute, so at most 50 tickets per minute. The queue grew to 377 jobs. The last job finished more than 7 minutes after the test stopped sending. Half of the customers waited more than 4 minutes for their reply, and the slowest waited 8 minutes.

• sent (saved, 202)
• AI work done

The strong burst (the author's run, your numbers will differ): tickets sent by Locust and tickets whose AI work had finished, counted from the start. The vertical gap is the queue; the horizontal gap is how long a ticket waited.

Text description of the chart

Two lines that count tickets. The “sent” line climbs slowly for 20 seconds, then steeply for 15 seconds, then slowly again until it stops at 476 saved tickets after 80 seconds. (Locust counted 474: its count stops when the test ends, and 2 requests were still in flight.) The “AI work done” line climbs far more slowly, at the provider’s pace. It reaches 476 only after about 525 seconds. The gap between the two lines is the queue.

Tomás’s test measured the API’s answer time correctly. But it answered a question that nobody asked, with a load much stronger than Black Friday.

The idea: a load test is a planned experiment

A useful load test has four parts. Write all four down before you start, as the bridge engineers do:

1. The question. One sentence about something a person cares about. “Does every Black Friday ticket get its AI reply within 5 minutes?” is a question. “How fast is the API?” is not.
2. The workload. What arrives, how often, and with which data. It comes from the workload model of Module 1, not from a guess.
3. The shape. How the load changes over time: a steady level, steps, a burst, or a long run.
4. The environment and the readings. Where the test runs, how it starts from the same state each time, and what you record. Readings can be request times, errors, the queue, the provider’s refusals, and the machine.

Load-test design is the work of choosing these four parts. Each choice follows from the question. Omar’s question was about the AI reply. So the queue’s oldest job and the time from submit to result are the readings that matter, not the 202 time.

From users to an arrival rate

The Module 1 tests were a closed loop: each Locust user waits for the answer before it sends again. With the queue design, an answer takes about 20 ms, so one closed-loop user sends about 50 tickets per second. No customer does that.

Real customers are an open stream: they arrive at their own rate. The locustfile has a setting for this, PACE_SECONDS. With PACE_SECONDS=10, each user sends one ticket every 10 seconds, if the answer comes in time. So the arrival rate is the number of users divided by the pace:

Users	PACE_SECONDS	Arrival rate	Compare with the workload
2	10	2 ÷ 10 = 0.2 tickets per second	Black Friday’s busiest hour: 0.28 per second
20	10	20 ÷ 10 = 2 tickets per second	Black Friday’s busiest minute: 90 tickets, 1.5 per second
20	1	20 ÷ 1 = 20 tickets per second	13 times the busiest minute

The last row was Tomás’s test. The burst file is the same in all rows; only the pace changes the rate. So a shape is a pattern, and the pace ties it to your workload. The middle rows are close to Omar’s Black Friday, so this lesson’s practice uses PACE_SECONDS=10.

Four shapes, four questions

Each shape answers a different question. The project has one file for each, in loadtest/. Each file is a locustfile with the same customers and one load shape. A load shape is a list of stages, each with an end time and a number of users.

The four load shapes of the project, drawn from their files. With PACE_SECONDS=1, users equal tickets per second; a larger pace makes the same shape slower.

Text description of the diagram

Each panel shows the number of users over time for one file. The ramp-up climbs in five steps of 20 seconds. The steady shape holds 4 users for 70 seconds, and the first 10 seconds are a warm-up that you leave out. The burst holds 2 users, jumps to 20 for 15 seconds, and falls back to 2. The soak holds 3 users for 10 minutes. Under each panel is the question that the shape answers.

• Ramp-up raises the load in steps. You look for the step where p95 or the errors start to grow: the knee of the curve. Above the knee, more load buys more waiting, not more work. Use it to find a limit that you do not know yet. The project’s ramp goes past 10 users, where Module 1’s design found its knee in Dependency limits: the pool of 10 database connections.
• Steady holds one load long enough to measure it well. It is the shape for a baseline and for comparing two configurations, as in the next lesson. Leave out the warm-up.
• Burst jumps far above the normal load for a short time, then falls back. It answers two questions. What do users see during the burst? And how long does the service need to recover, for example to empty its queue?
• Soak holds a normal load for a long time. It finds slow problems that a short test never shows. Examples are memory that grows, connections that never go back to the pool, and a queue that grows by one job per minute. The project’s soak lasts 10 minutes, so it fits on a laptop. A real soak test runs for hours.

Here is the burst file:

loadtest/burst.py

"""Burst: 2 users, then 20 users for 15 s (ten times more), then 2 users again for 45 s.
What do users see during the burst, and how long until the queue is empty again?"""

from locustfile import Customer  # noqa: F401  (the user)
from shapes import Stages


class Burst(Stages):
    stages = [(20, 2), (35, 20), (80, 2)]

Stages in loadtest/shapes.py reads the list once per second. Until second 20, it asks for 2 users; until second 35, for 20; until second 80, for 2; then the test stops. You run a shape without -u and -t, because the shape decides both.

Where to run it: your own, disposable, quiet

Your own services only. A load test sends a lot of traffic on purpose. Against another company’s service, it can harm their real users and break their terms of use. Against a real AI provider, it costs money and uses up a quota that your colleagues need. So every test in this course runs on your computer, against your own API, database and cache, and the simulated provider.

Disposable. Before each test, put everything back into the same state. Load the small data again, empty the cache, and restart the simulated provider, which fills its quota again. Then every test starts from the same place, and you may break anything.

Quiet. The load generator, the API, the worker, PostgreSQL and Valkey share your computer with every other program. A video call or a build takes CPU time from your test. Run python -m scripts.machine before and after the test, and write the load average into the result. The course’s own measurements started only when the 1-minute load was under 3.

The load generator is part of the machine. The course chose Locust (MIT licence) for three reasons. It installs with pip on every system, its workloads and shapes are Python code, and it writes CSV files. The course team compared it with oha (MIT licence, written in Rust). Both ran in Docker against /health with 64 connections. oha sent 12,201 requests per second; Locust sent 9,067, and used a whole CPU core. Locust is slower, but this course needs fewer than 30 requests per second. k6, another popular tool, was not considered: its AGPL-3.0 licence does not fit the course’s rules for tools.

Guided practice: a Black Friday burst

You run the burst at the Black Friday rate and watch both the answers and the queue. You need five terminals, each in ~/projects/ticket-api with the virtual environment active.

1. Copy the load-test files from the snapshot. locustfile.py gets the PACE_SECONDS setting:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m06
   for f in locustfile.py shapes.py ramp.py steady.py burst.py soak.py; do
     cp "$S/loadtest/$f" "loadtest/$f"
   done
   cp "$S/README.md" README.md

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m06"
   foreach ($f in 'locustfile.py', 'shapes.py', 'ramp.py', 'steady.py', 'burst.py', 'soak.py') {
     Copy-Item "$S\loadtest\$f" "loadtest\$f"
   }
   Copy-Item "$S\README.md" README.md

2. Close other heavy programs. Then print the machine: python -m scripts.machine. If the load average is high, wait a minute and print it again.

3. Put the data and the cache back into the same state:

   python -m scripts.load --reset
   python -m scripts.cache flush

4. In terminal 1, start the simulated provider: python -m simulator. If it still runs from an earlier test, stop it with Ctrl+C and start it again, so that its quota is full.

5. In terminal 2, start the API without the file watcher: fastapi run.

6. In terminal 3, start one worker: python -m ticket_api.worker.

7. In terminal 4, watch the queue for 3 minutes: python -m scripts.queue_watch --seconds 180.

8. In terminal 5, at once, run the burst at a pace of 10 seconds:

   macOS / Linux

   PACE_SECONDS=10 SAMPLES_CSV=burst.csv locust -f loadtest/burst.py --host http://127.0.0.1:8000 --headless

   Windows (PowerShell)

   $env:PACE_SECONDS = "10"; $env:SAMPLES_CSV = "burst.csv"
   locust -f loadtest/burst.py --host http://127.0.0.1:8000 --headless
   Remove-Item Env:PACE_SECONDS, Env:SAMPLES_CSV

9. When Locust stops after 80 seconds, compute the exact percentiles: python -m scripts.percentiles burst.csv.

10. Wait until terminal 4 shows 0 queued and 0 running. Then print the machine again: python -m scripts.machine.

11. Ask the database how long each ticket waited for its AI work, from submit (created_at) to result (finished_at):

    docker compose exec db psql -U tickets -d tickets -c "SELECT count(*) AS jobs, round(percentile_disc(0.5) WITHIN GROUP (ORDER BY s)::numeric, 1) AS p50_s, round(percentile_disc(0.95) WITHIN GROUP (ORDER BY s)::numeric, 1) AS p95_s, round(max(s)::numeric, 1) AS max_s FROM (SELECT extract(epoch FROM finished_at - created_at) AS s FROM jobs WHERE state = 'succeeded') AS t;"

Show the expected result

The author’s run, on a computer with a 1-minute load of 3.69 before and 2.8 after. Your numbers will differ. Step 9:

62 requests in 80.1 s; status codes {'202': 62}
Successful: 62 = 0.77 per second
Latency of the successful requests: mean 18.1 ms | p50 19.7 ms | p95 26.0 ms | p99 26.2 ms | max 26.2 ms

Terminal 4, every fifth line (time is seconds since the watch started):

  time  queued  ready  running  oldest s  done/min  dead
     0       0      0        0       0.0         0     0
     5       0      0        0       0.0         2     0
    10       0      0        0       0.0         2     0
    15       0      0        0       0.0         4     0
    20       0      0        0       0.0         4     0
    25      10     10        4       4.9        10     0
    30       1      1        4       9.9        19     0
    35      13     13        4       5.0        27     0
    40       6      6        3      10.0        36     0
    45       5      5        4       5.0        45     0
    50       2      2        0       0.0        54     0
    55       0      0        0       0.0        56     0
    60       2      2        0       0.1        56     0

The burst started at second 20. In the full output, the queue peaked at 20 jobs at second 31. The oldest job never waited more than 11 seconds. The queue was empty again about 20 seconds after the burst ended. Step 11 gave the time from submit to result for all 62 jobs: p50 6.5 s, p95 12.7 s, and at most 14.1 s.

Why did the queue stay short, when Tomás’s burst built a queue of 377? The simulated provider starts with a full quota, as the real one did: one minute’s worth of calls, 100. The Black Friday burst sent about 30 tickets in 15 seconds, so about 60 chat calls, and the full quota took them almost at once. Tomás’s burst sent about 300 tickets in the same 15 seconds: 600 chat calls, six times the quota.

• sent (saved, 202)
• AI work done

The burst at PACE_SECONDS=10 (the author's run, your numbers will differ). The AI work follows the sent tickets a few seconds behind: the queue stayed short.

Text description of the chart

Two lines that count tickets over about 83 seconds. Both climb slowly for 20 seconds, then faster for 15 seconds, then slowly again. The “AI work done” line stays a few seconds to the right of the “sent” line all the time, and both end at 62 tickets.

What this tells us. At the Black Friday rate, with a full quota at the start, the queue design gave every ticket its AI reply within 14.1 seconds. That answer is about this computer, the simulated provider and an 80-second test. A real Black Friday lasts hours, so the quota may not be full when the busiest minute comes. The soak in “Your turn” looks at a longer run.

Your turn: a short soak just under the quota

Change one thing: the shape. Run loadtest/soak.py (3 users for 10 minutes) with PACE_SECONDS=4. Before you run it, write down the arrival rate, and compare it with the 50 tickets per minute that the quota allows. Predict: does the queue stay short, or does it grow? What else could grow in 10 minutes?

Put the environment back into the same state first (steps 3 and 4). While the soak runs, watch the queue with python -m scripts.queue_watch --seconds 660 --every 10.

Show the answer

3 users ÷ 4 seconds = 0.75 tickets per second, or 45 per minute. The quota allows 50 per minute, so the provider can keep up, but only just: 90% of the quota.

The author’s run (the 1-minute load was 11.9 at the start and 4.2 at the end; your numbers will differ). scripts.percentiles printed:

453 requests in 600.0 s; status codes {'202': 453}
Successful: 453 = 0.75 per second
Latency of the successful requests: mean 10.9 ms | p50 9.2 ms | p95 19.6 ms | p99 22.9 ms | max 25.1 ms

The queue watch, every 60th line:

  time  queued  ready  running  oldest s  done/min  dead
     0       0      0        0       0.0         0     0
    60       3      3        0       0.2        45     0
   121       0      0        3       0.0        45     0
   242       0      0        3       0.0        45     0
   362       0      0        3       0.0        45     0
   483       0      0        2       0.0        45     0
   604       0      0        0       0.0        45     0

The queue never held more than 3 jobs, and done/min stayed at 45, the arrival rate. The provider refused no call. All 453 jobs succeeded. Their time from submit to result was p50 2.4 s, p95 3.2 s and at most 3.7 s. The author also read the memory of the API and the worker every 30 seconds with ps, and the database connections from pg_stat_activity. From the second minute to the end, the API used about 46 MB and the worker 37 to 43 MB. The database had 9 connections. Nothing grew.

What this tells us. Ten minutes at 90% of the quota were stable on this computer. A real soak runs for hours, because a leak of 1 MB per hour is invisible in 10 minutes.

Common mistake

The shape test stops at once

Symptom: you run a shape file, and Locust stops after a fraction of a second, with no requests:

INFO/locust.runners: Shape test starting.
INFO/locust.runners: Shape worker starting
INFO/locust.runners: Shape test stopping
INFO/locust.main: Shape test completed, shutting down

The summary table shows 0 requests.

Cause: Locust ran the wrong shape. A shape file imports the base class Stages, so the file holds two shape classes: Stages and, for example, Burst. Locust takes the first one that it finds. Stages has no stages, so its first answer is “stop”. The course’s first version of shapes.py had exactly this fault.

Diagnose: read loadtest/shapes.py, and look for the line abstract = True in the class Stages. Without it, Locust treats Stages as a shape of its own.

Fix: mark the base class as abstract: a class that is not used on its own. Locust then skips it:

loadtest/shapes.py (the fix)

class Stages(LoadTestShape):
    """A list of (until second, users). The test stops after the last stage."""

    abstract = True
    stages: list[tuple[int, int]] = []

The snapshot end-of-m06 has this fix, and tests/test_load_tests.py checks that Locust takes each file’s own shape. Tests for the test tools are worth their time: a broken load test reports nothing, and “nothing” looks fine.

When to use each shape, and what it costs

Use a ramp-up when you do not know where the limit is. Use steady for every number that you will compare or promise. Use a burst when the workload has peaks, such as Black Friday’s busiest minute, and measure the recovery as well as the answers. Use a soak before a long event, and after a change to anything that holds resources: pools, caches, workers.

Do not run a soak on a computer while you work on it: ten minutes of your other programs become part of the result.

Each test costs time. A burst takes 80 seconds, plus the time the queue needs to empty, plus the reset. A number that you compare needs several passes. Write the four parts down first, so that you run only the tests that answer a question.

Knowledge check

Check your understanding

Tomás's closed-loop test with 50 users shows a p95 of 30 ms and no errors against the queue design. Why does that not show that the service is ready for Black Friday?

Show the answer

The test measured only the 202 answer, which says “saved and queued”. The AI work waits in the queue and runs at the provider’s pace, at most 50 tickets per minute. A closed loop with 50 users also sends far more than Black Friday, so the queue grows without limit. Omar’s question needs other readings: the queue’s oldest job and the time from submit to result. And it needs a workload from the workload model.

Check your understanding

Predict: you run burst.py with PACE_SECONDS=5. What is the arrival rate before and during the burst? Do you expect a long queue?

Show the answer

Before the burst: 2 ÷ 5 = 0.4 tickets per second. During it: 20 ÷ 5 = 4 tickets per second for 15 seconds, so about 60 tickets and 120 chat calls. A full quota holds 100 calls, so about 50 tickets pass at once. The rest wait while the quota refills at 100 calls per minute. Expect a short queue of some tens of jobs, which empties within a minute or two after the burst.

Check your understanding

Practical check: write the four parts of your Black Friday burst test, as you would put them at the top of a result.

Show the answer

For example. Question: does every ticket of the busiest Black Friday minute get its AI reply within 5 minutes? Workload: new tickets from the practice data (seed 21), small data. The rate is 0.2 tickets per second, with a burst of 2 per second for 15 seconds (burst.py, PACE_SECONDS=10). Environment: one API process (fastapi run), one worker, PostgreSQL and Valkey in Docker, and the simulated provider with the recorded quota. Data and cache reset, simulator restarted. The computer and its load average from scripts.machine. Readings: the 202 time (p50, p95) and errors. The queue’s length and oldest job each second. The time from submit to result.

Recap

• A load test is a planned experiment: a question, a workload from the workload model, a shape, and an environment with readings. Write the four parts first.
• PACE_SECONDS turns users into an arrival rate: users ÷ pace. The same shape can be Black Friday, or thirteen times Black Friday.
• Ramp-up finds the knee, steady gives a baseline, burst shows the peak and the recovery, soak finds what grows slowly.
• Run only against your own services, reset them before each test, and log the machine’s load. Locust is slower than oha, but fast enough for this course’s tens of requests per second.
• A fast 202 can hide a long wait: measure the queue and the time from submit to result.

Next: Compare configurations