Compare configurations

What you will be able to do

Find out whether a change really helped. Run each configuration the same way several times, and compare a scorecard: throughput, p95 and p99, errors, resource use and cost per ticket.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The four parts of a load test and PACE_SECONDS, from Load-test design.
• Passes, medians and ranges, from Latency and capacity.
• Several processes and workers, from Horizontal scaling and Job lifecycle.

What you need on your computer

• The project after Load-test design: the shape files in loadtest/, and python -m pytest prints 172 passed, 1 deselected.
• PostgreSQL, Azurite and Valkey running: docker compose up -d db azurite cache.

Tested on macOS (Apple silicon); the Windows commands are not tested. The AI provider is the simulated one. The outputs are from the author’s computer: your numbers will differ.

Two routes to work

You find a new route to work. On Monday, it takes 24 minutes; the old route took 29 last Friday. Is the new route faster? You cannot tell. Monday may have had less traffic, or a green light at every crossing.

So you drive each route on five ordinary mornings, at the same time. The new route takes 23 to 27 minutes, the old one 26 to 31. You compare the typical mornings, and you also look at the worst ones. You notice that the new route has a toll and uses more fuel. Now you can decide.

Comparing two configurations of a service is the same. One run each tells you little. Several runs under the same conditions, and a look at every cost, tell you what changed.

The problem: one pass, one number

One pass can mislead. The course measured Module 1’s design with 4 users, five passes of 60 seconds, on a quiet computer. Nothing changed between the passes. Still, the provider refused 41 calls in one pass and 115 in another. If Amira had run one pass before a change and one after, a “drop from 115 to 41 refusals” could be pure chance.

One number can mislead. The course also compared one API process with four, with 32 users and Module 1’s design. Four processes finished almost three times as many tickets per second. That is true, but it is not the whole story. Each process has its own pool of 10 database connections, so four processes can hold 40 connections. In Dependency limits, twelve such processes asked for more connections than PostgreSQL allows. A comparison that shows only throughput hides that cost.

The idea: a fair comparison and a scorecard

A configuration is one set of settings of the same code. Examples are the number of processes or workers, the intake mode, the pool size and the queue bound. To compare two configurations fairly, keep everything else the same:

1. One change. If you change two settings, you cannot tell which one made the difference.
2. The same workload and shape. The same locustfile, users, pace, duration and warm-up.
3. The same starting state. Load the data again, empty the cache, and reset the provider before every pass.
4. Several passes. At least three, preferably five, for each configuration. Report the median of each number, with the lowest and highest pass. If the ranges of two configurations overlap, the test has not shown a difference.
5. A quiet machine. Note the load average before and after. Both configurations must run under the same conditions.

Then compare a scorecard: one row per configuration, and the same columns every time. docs/benchmark-template.md already has them:

Column	What it measures	Why it is there
Throughput	Finished work per second: for the queue, jobs whose AI work succeeded, not 202 answers	The work that customers get
p50, p95, p99	The typical, slow and slowest request times; for the queue, also the time from submit to result	What customers wait
Errors	Status codes other than success, by code	What customers lose
Resources	CPU of the API, database connections, provider calls and refusals	What the change uses up
Cost per task	Money per ticket	What the change costs

Here is the course’s comparison of one and four processes as a scorecard. Each value is the median of 5 passes on a quiet computer, with the range in brackets:

Configuration (32 users, Module 1 design, quota off)	Throughput (tickets/s)	p50 ms	p95 ms	p99 ms	Errors per pass	API CPU	DB connections allowed (pools)	AI cost per ticket
1 process	3.37 (3.30–3.41)	6,831	7,480 (7,406–7,717)	7,879	10–14 × 503 database_busy	1.1%	10	US$0.0000614
4 processes	9.59 (8.96–10.74)	2,621	4,410 (3,757–4,822)	4,944	none	3.0% (2.6–10.0)	40	US$0.0000611

Each dot is one 40-second pass on a quiet computer (32 users, Module 1's design, simulated provider with the quota off). The two groups do not overlap, so the difference is real.

Text description of the chart

Two rows of five dots. One process: 3.37, 3.3, 3.41, 3.41, 3.36 tickets per second, all close together near 3.4. Four processes: 8.96, 9.59, 10.74, 9.36, 9.76 tickets per second, spread between about 9 and 11. The lowest pass of four processes is far above the highest pass of one process.

What this tells us. The gain in throughput and p95 is real: the ranges are far apart. The errors went away. With one process, the pool of 10 was full, and 10 to 14 requests per pass waited too long for a connection. The AI cost per ticket did not change, because every ticket still makes the same three calls. The price of the gain is in the resources column: the pools allow up to 40 database connections instead of 10. Notice also the CPU range of four processes, 2.6% to 10.0%. One pass had a busier moment, and the median protects the comparison from it.

Cost per task: tokens times prices

A hosted model charges per token, a piece of text of about four characters. The price differs for the tokens that you send (input) and the tokens that the model writes (output). The project keeps the prices in ticket_api/prices.py. Chat-small costs US$0.10 per million input tokens and US$0.50 per million output tokens. Embed-small costs US$0.02 per million input tokens. Prices change: check the provider’s current price page before you use them.

A tiny worked example, with the median tokens of one ticket from the real recorded calls:

Task	Tokens in × price	Tokens out × price	Cost
classify	199 × 0.10 ÷ 1,000,000 = US$0.0000199	15 × 0.50 ÷ 1,000,000 = US$0.0000075	US$0.0000274
draft_reply	130 × 0.10 ÷ 1,000,000 = US$0.0000130	41.5 × 0.50 ÷ 1,000,000 = US$0.0000208	US$0.0000338
embed	33.5 × 0.02 ÷ 1,000,000 = US$0.0000007	none	US$0.0000007
One ticket			about US$0.0000618

So one ticket’s AI work costs about US$0.00006, or 6 US cents per 1,000 tickets. The recordings gave a median of US$0.0000617 per ticket. The course’s 30-ticket run with the simulated provider gave US$0.0000615. The reply is the most expensive task, because output tokens cost five times as much as input tokens.

From the course's run of 30 tickets with the simulated provider, whose token counts follow the recorded calls. The prices are the dated ones in ticket_api/prices.py.

Text description of the chart

Three bars. Classify costs US$0.0273 per 1,000 tickets, draft_reply US$0.0335, and embed US$0.0006. Together, about US$0.06 per 1,000 tickets.

python -m scripts.cost computes the same from the ai_runs table, for the tickets that came through the API. With --compute-usd-per-hour and --hours, it also shares the cost of your machines between the tickets of that time. On a laptop, that column has no real value; on a platform, it often costs more than the AI work.

Larkfield: does a second worker help on Black Friday?

Tomás proposes the obvious fix for a long queue: “add more workers”. Amira measures it. Her question: with the recorded quota, does a second worker finish more Black Friday tickets per second? Her workload is a little above the busiest minute, as a margin. It is 4 users with PACE_SECONDS=2, so 2 tickets per second, for 70 seconds, with the first 10 left out. The two configurations differ in one setting: WORKERS=1 or WORKERS=2.

scripts.compare does the fair part for her. For each pass, it loads the data again, empties the cache and resets the simulated provider. Then it starts the API on port 8790 (another port with --port) and the workers, and runs Locust. It waits up to 120 seconds for the queue to empty, and stops everything. It reports the median and the range of each column.

Here is her result: the author’s run on a quiet computer, with a 1-minute load of 1.9 before and 2.2 after. Your numbers will differ. Medians of 3 passes, with the range:

Configuration (2 tickets/s, 70 s, quota on)	Finished tickets per second	Request p95	Submit to result, p50	Submit to result, p95	Errors	API CPU	Provider 429s	AI cost per ticket
queue, 1 worker	1.45 (1.42–1.50)	20.2 ms	7.6 s	55.8 s (53.0–58.4)	none	0.7%	90 (78–94)	US$0.0000614
queue, 2 workers	1.43 (1.40–1.47)	19.5 ms	2.9 s	53.9 s (49.8–54.8)	none	0.7%	108 (76–119)	US$0.0000614

Each dot is one pass of 70 seconds at 2 tickets per second (the author's runs, your numbers will differ). With the recorded quota, the two groups overlap. With the quota off, they separate.

Text description of the chart

Four rows of three dots. With the recorded quota, one worker finished 1.5, 1.42, 1.45 tickets per second, and two workers 1.47, 1.43, 1.4: the rows overlap. With the quota off, one worker finished 1.75, 1.82, 1.75, and two workers 2, 2, 2.02: the rows are apart. “Your turn” below explains the second pair.

What this tells us. The second worker did not finish more tickets per second: the ranges overlap, and both stay near 1.45. The quota sets the pace: a full minute’s worth of calls at the start, then 100 calls per minute. So the slowest tickets waited about 55 seconds with one or two workers. The second worker did change two things. The typical ticket got its reply sooner, 2.9 instead of 7.6 seconds, because more jobs ran while the quota was still full. And the provider refused more calls, 108 instead of 90 per pass: more workers asked more often. The answer to Tomás is “not while the quota is the limit”. The scorecard shows why, and what it costs.

Guided practice: compare one and two workers

You run the same comparison. It takes about 15 minutes. During that time, do not use the computer for other heavy work.

1. Copy the files of this lesson from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m06
   for f in scripts/compare.py scripts/cost.py tests/test_load_tests.py; do cp "$S/$f" "$f"; done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m06"
   foreach ($f in 'scripts\compare.py', 'scripts\cost.py', 'tests\test_load_tests.py') { Copy-Item "$S\$f" $f }

2. Run the tests: python -m pytest. Expect 176 passed, 1 deselected.

3. Stop the API and every worker that still runs from the last lesson (Ctrl+C in their terminals). scripts.compare starts its own. A worker that you forget would take the jobs of the comparison.

4. In terminal 1, start the simulated provider with the recorded quota: python -m simulator.

5. In terminal 2, print the machine: python -m scripts.machine.

6. Run the comparison:

   python -m scripts.compare --config "queue, 1 worker:INTAKE_MODE=queue,WORKERS=1" --config "queue, 2 workers:INTAKE_MODE=queue,WORKERS=2" --users 4 --pace 2 --seconds 70 --passes 3 --json compare.json

7. Print the machine again: python -m scripts.machine.

8. Print the AI cost of the tickets of the last pass: python -m scripts.cost.

9. Copy docs/benchmark-template.md to docs/workers.md, and fill in the scorecard with your medians and ranges.

Show the expected result

The author’s run (your numbers will differ). The API’s warning about API_KEY appears once per pass; it is left out here:

queue, 1 worker pass 1: done 1.5/s, p95 20.2 ms, codes {'202': 120}
queue, 1 worker pass 2: done 1.42/s, p95 21.2 ms, codes {'202': 120}
queue, 1 worker pass 3: done 1.45/s, p95 18.2 ms, codes {'202': 120}
queue, 2 workers pass 1: done 1.47/s, p95 19.8 ms, codes {'202': 120}
queue, 2 workers pass 2: done 1.43/s, p95 18.7 ms, codes {'202': 120}
queue, 2 workers pass 3: done 1.4/s, p95 19.5 ms, codes {'202': 120}

Median of 3 passes (lowest-highest); 4 users, pace 2.0 s, 70 s, first 10 s left out
- queue, 1 worker: done per s 1.45 (1.42-1.5) | p50 15.4 (15.1-15.8) ms | p95 20.2 (18.2-21.2) ms | p99 22.3 (18.9-22.4) ms | submit to result p95 55771.8 (53002.1-58357.1) ms | API CPU 0.7 (0.7-0.7)% | 429 from provider 90 (78-94)
- queue, 2 workers: done per s 1.43 (1.4-1.47) | p50 15.9 (15.8-16.1) ms | p95 19.5 (18.7-19.8) ms | p99 21.2 (20.8-23.6) ms | submit to result p95 53869.4 (49826.1-54836.9) ms | API CPU 0.7 (0.7-0.7)% | 429 from provider 108 (76-119)

Step 8. The first line, the date of the prices in prices.py, is left out:

  classify     chat-small      140 calls     27799 in     2100 out  US$0.003830
  draft_reply  chat-small      140 calls     18139 in     5772 out  US$0.004700
  embed        embed-small     140 calls      3166 in        0 out  US$0.000063
AI cost per ticket: US$0.0000614 (US$0.0614 per 1,000 tickets)

scripts.cost reads the tickets in the database, so it counts the last pass only: 140 tickets. That is 70 seconds at 2 tickets per second: the 120 of the measured time, and 20 from the warm-up. Compare the cost with the worked example above: US$0.0000614 against about US$0.0000618 from the recordings.

Your turn: the same comparison without the quota

Change one condition: the quota. Stop the simulator, and start it again with the quota off: SIM_QUOTA=off python -m simulator (PowerShell: $env:SIM_QUOTA = "off"; python -m simulator). This is as if Larkfield had a much larger quota. Run step 6 again with --json compare-no-quota.json.

Predict first. One worker runs 4 jobs at a time, and a job takes about 2.2 seconds. How many tickets per second can one worker finish? Is that enough for 2 tickets per second? What do you expect from the second worker now?

Show the answer

One worker runs 4 jobs at a time, and a job takes about 2.2 seconds: 4 ÷ 2.2 = about 1.8 tickets per second. That is less than the 2 per second that arrive, so with one worker the queue grows slowly. Two workers can finish about 3.6 per second, more than enough. Now the workers are the limit, so the second worker should help.

The author’s run, quota off (the 1-minute load was 2.0 before and 4.3 after; your numbers will differ). Medians of 3 passes, with the range:

Configuration	Finished tickets per second	Request p95	Submit to result, p95	Provider 429s
queue, 1 worker	1.75 (1.75–1.82)	20.3 ms	11.5 s (10.6–11.9)	0
queue, 2 workers	2.0 (2.0–2.02)	18.5 ms	3.2 s (3.0–3.2)	0

The ranges do not overlap. The second worker finished all the work as it arrived. It cut the wait from submit to result from about 11.5 to 3.2 seconds. So “add more workers” is right when the workers are the limit, and wrong when the quota is. The comparison tells you which case you are in.

Common mistake

Locust's req/s counted promises, not work

Symptom: a comparison of two queue configurations shows the same high “req/s” in Locust’s table for both. In the strong burst of Load-test design, Locust printed 5.92 successful requests per second, while the AI work finished at a few tickets per minute.

Cause: with the queue design, a request ends with 202 Accepted as soon as the ticket is saved. Locust counts that answer. The work that the customer waits for happens later, in the worker, and Locust never sees it.

Diagnose: compare Locust’s request count with the jobs that succeeded in the same time. python -m scripts.queue_watch shows done/min; scripts.compare reports done per s from the jobs table.

Fix: for any design that answers before the work is done, measure throughput as finished work, and latency as the time from submit to result. Keep the request’s own p95 too: it tells you how fast customers get their 202.

When to compare, and what it costs

Compare before you keep a change that has a cost: more processes, more workers, a larger pool, a cache. Compare again when an assumption changes, such as a new quota or a new provider. Do not compare to choose between settings that change nothing for the customer.

A fair comparison costs time: three passes of two configurations, with resets and the queue’s drain, is about a quarter of an hour. That is cheap next to a Black Friday decision taken on one lucky run. It also costs care: one change, the same workload, a quiet computer. scripts.compare runs all passes of the first configuration before the second one. If your computer gets busier during the run, the second configuration pays for it, so check the load average before and after.

See it end to endQueue and cache for an open-data APICase study

The open-data API case study compares three versions of one API: the starter, a cache, and a cache with a queue. Each runs the same three workloads three times.

Knowledge check

Check your understanding

Configuration A has a p95 of 2,410 ms (2,200–2,630) over 5 passes, and configuration B 2,350 ms (2,150–2,700). Is B faster?

Show the answer

The test has not shown it. The ranges overlap almost completely: some passes of A were faster than some passes of B. The 60 ms between the medians is smaller than the spread between passes of the same configuration. Report “no difference shown”, or run more passes or a longer test if the difference matters.

Check your understanding

Four API processes finished 9.59 tickets per second instead of 3.37, with no errors. Name one thing in the scorecard that got worse, and why it matters.

Show the answer

The database connections: up to 40 instead of 10, because each process has its own pool. PostgreSQL allows 100, so the number of processes cannot grow much more. The API CPU also rose, from 1.1% to 3.0%, but that is still small. The AI cost per ticket did not change.

Check your understanding

Practical check: from your own comparison, give the scorecard row of each configuration, and say in one sentence whether the second worker helped, with the evidence.

Show the answer

Your row has the median and range of finished tickets per second, and p50, p95 and p99 of the request. It also has p95 from submit to result, the errors, the API CPU, the provider’s 429s and the AI cost per ticket. Above the table, write the workload and the machine with its load average. With the recorded quota, the answer should be “no difference shown”. The finished work per second of one and two workers overlaps, because the quota sets the pace. Your numbers will differ.

Recap

• A fair comparison changes one setting and keeps the workload, the shape, the starting state and the machine the same.
• Run at least three passes of each configuration, and report the median with the lowest and highest pass. Overlapping ranges mean “no difference shown”.
• The scorecard has the same columns every time: throughput as finished work, p50, p95 and p99, errors, resources and cost per task.
• One ticket’s AI work costs about US$0.00006: tokens times the dated prices. The reply costs most, because output tokens are expensive.
• scripts.compare resets everything before each pass and reports medians and ranges. More workers help only when the workers are the limit.

Next: Capacity recommendation