Horizontal scaling

What you will be able to do

Serve more tickets at once by running several copies of the API behind one address. Keep everything that requests share outside those copies, and compare two configurations with the same load test.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Threads, async I/O, processes and the event loop, from I/O versus CPU work.
• Locust users, SAMPLES_CSV and scripts.percentiles, from Latency and capacity.
• The connection pool (at most 10 connections, a wait of at most 5 seconds, then 503 database_busy), from Connection management.

What you need on your computer

• Your project after I/O versus CPU work: python -m pytest prints 137 passed.
• PostgreSQL running, with the small data loaded. This lesson adds no file.

The AI provider is the simulated one. This lesson runs it with its quota off, as if Larkfield’s account allowed many more calls per minute. With the quota on, the quota would be the limit, and you could not see the limit of the API. The course team measured the numbers on an Apple M4 (10 cores, 16 GB) with nothing else busy. Your times will differ, but the shape stays the same. Tested on macOS (Apple silicon); the Windows commands are not tested.

Four checkouts and one line

A supermarket has one checkout. On a busy Saturday, the line reaches the back of the shop. The manager does not buy a faster cash register. She opens three more checkouts. A member of staff stands at the front of the line and sends each shopper to the next free checkout.

This works for one reason: any cashier can serve any shopper. The prices, the stock and the loyalty points are in the shop’s central system, not in one cashier’s head. If cashier 2 wrote loyalty points in her own notebook, a shopper who went to checkout 3 next week would lose them.

Four checkouts are four copies of the API. The member of staff at the front is the address that hands out the work. The central system is the database.

The problem: one process and 32 users

In I/O versus CPU work, 8 users were easy. Black Friday brings more. The course team ran 32 Locust users against the Module 1 design (INTAKE_MODE=thread) in one API process:

One process, 32 users	Value (median of 5 passes)
Successful tickets per second	3.37 (3.3–3.41)
p50	6,831 ms
p95	7,480 ms
503 database_busy	10 to 14 per pass

Four times the users of the last lesson gave about the same tickets per second as 8 users (3.22). The extra users only waited: half of the answers took more than 6.8 seconds, and some requests got 503. The API’s CPU stayed near 1%, so the computer was not busy. What was busy?

The author looked at the database during such a run (the author’s run; pg_stat_activity lists every connection to PostgreSQL):

        state        | count
---------------------+-------
 idle in transaction |    10

Ten connections, the whole pool, and all ten are idle in transaction. Each one belongs to a ticket that started its transaction and now waits for the AI provider. The other 22 tickets wait in line for a connection. After 5 seconds, some of them give up with 503. One process has one pool, and the pool is full.

The idea: more copies, one address

Horizontal scaling means running more copies of the same service and spreading the requests over them. Each copy is an instance. The other way, vertical scaling, gives one instance a bigger computer. Vertical scaling would not help here: the computer is almost idle.

Two things make horizontal scaling work:

1. One address in front. Clients do not choose an instance. They send every request to one address, and something there hands each request or connection to an instance. In the cloud, that is a load balancer. On your computer, it is the server program itself. fastapi run --workers 4 starts a parent process that opens port 8000 and starts four worker processes. The operating system gives each new connection to one of them.
2. Stateless instances. A stateless instance keeps nothing between requests that a later request needs. Any instance can serve any request, like any cashier. Everything shared lives outside, in a service that all instances use: here, PostgreSQL.

Why processes, and not more threads? In Python, only one thread of a process runs Python code at a time. Separate processes run side by side on separate CPU cores. Here the reason is different again: each process has its own connection pool. Four processes have four pools of 10 connections.

A tiny example: what four processes should give

Use round numbers. One ticket holds its connection for about 2.2 seconds in the Module 1 design.

• One process: at most 10 tickets in progress, one per connection. That is at most 10 ÷ 2.2 s ≈ 4.5 tickets per second. The other 22 users wait. Measured: 3.37.
• Four processes: at most 40 tickets in progress. Only 32 users send, so at most 32 tickets are in progress. If each process gets 8 users, that gives 32 ÷ 2.2 s ≈ 14.5 tickets per second.

So four processes should give about three times the throughput, and the waiting should almost disappear. Here is the setup, with the numbers of each part:

Four worker processes behind one address. Each has its own pool; the database and the provider are shared by all of them.

Text description of the diagram

From left to right: 32 Locust users, one address on port 8000, four worker processes, and two shared services. Each worker process has a pool of up to 10 connections and a client for the provider. All four connect to one PostgreSQL. It accepts 100 connections, 3 of them for administrators, and holds the tickets and the sequence that numbers them. All four call one simulated provider, whose quota counts for the whole account. Four processes can open up to 40 connections in total. Nothing that a later request needs stays inside a process.

Measured: one process against four

The course team ran the same test with --workers 4, five passes each:

Successful tickets per second with 32 Locust users, the Module 1 design (INTAKE_MODE=thread), the median of 5 passes of 40 s. Simulated provider, quota off. Four processes served almost three times as many.

Text description of the chart

With 32 users and the Module 1 design, one API process served 3.37 tickets per second (passes from 3.3 to 3.41). Four processes served 9.59 (from 8.96 to 10.74).

32 users, Module 1 design	1 process	4 processes
Successful tickets per second	3.37	9.59 (8.96–10.74)
p50	6,831 ms	2,621 ms
p95	7,480 ms	4,410 ms
503 database_busy per pass	10 to 14	none
Database connections at most	10	40

What this tells us. Four processes served 9.59 tickets per second, 2.8 times as many, and p50 fell from 6,831 to 2,621 ms. The 503 errors disappeared. The gain came from the pools, not from the CPU: one process had used about 1% of a core.

Two numbers fall short of the tiny example. Throughput reached 9.59, not 14.5, and p95 is 4,410 ms, not about 3,000 ms. The passes also varied a lot: 8.96 to 10.74. The likely reason is the spread of the users. Each Locust user keeps one connection open, and a connection stays with the process that accepted it. The author counted the connections of each process during one run with lsof:

Locust connections per API process:   20   6   4   2
database connections per API process: 10   6   4   2

One process got 20 of the 32 users. Its pool of 10 was full, and 10 users waited in line. The other three processes had connections to spare. The operating system does not balance by work in progress. A real load balancer can do better: it can send each request, not each connection, to the least busy instance.

Keep shared state outside the processes

Four processes are four separate programs. They share no Python variable. So for each piece of data, ask: does a later request need it? If yes, it must live outside the processes.

What	Where it lives	Shared by all processes?
Tickets, AI runs, embeddings	PostgreSQL	yes
The next ticket number (T-500001, …)	PostgreSQL sequence ticket_number_seq	yes
The connection pool	each process	no: 4 pools of up to 10
The provider client and PROVIDER_MAX_CONCURRENCY	each process	no
The provider’s quota	the provider, per account	yes: more processes do not make it bigger

The ticket number shows why. Migration 013_ai_work.sql numbers new tickets with a sequence in the database: 'T-' || nextval('ticket_number_seq'). Imagine instead a Python counter in intake.py, next_number += 1 (a constructed example, not in the project). Each of the four processes would start at 500001, so four different tickets would get T-500001. The database sequence gives each number exactly once, whatever the number of processes.

The pool and the provider client are different. They are not data that a request needs later. They are resources: each process needs its own. But the number of processes multiplies each one. Four processes open up to 40 connections, and twelve would open up to 120. Dependency limits adds them up.

Larkfield: several processes, no new code

The project needs no change. Only the start command changes:

• fastapi dev starts one process that restarts when you save a file. Use it while you write code.
• fastapi run starts the API without the restarts. With --workers 4, it starts a parent process and four worker processes. uvicorn ticket_api.main:app --workers 4 does the same.

The log of fastapi run --workers 4 shows the processes (the author’s run, with the warning lines left out):

INFO:     Started parent process [92211]
INFO:     Started server process [92223]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [92225]
...

Each worker process runs the app’s start-up on its own: it opens its own pool, with 2 connections when it is quiet. So the author saw 8 idle connections before the test, 2 for each process. Each worker process also used about 87 MB of memory after start-up on the author’s Mac, plus about 52 MB for the parent.

Guided practice: compare two configurations

This is the module’s practice: the same workload against two configurations, measured the same way. Close other busy programs first.

1. In the first terminal, start the simulated provider with its quota off:

   macOS / Linux

   SIM_QUOTA=off python -m simulator

   Windows (PowerShell)

   $env:SIM_QUOTA = "off"; python -m simulator

2. In the second terminal, start one process of the Module 1 design:

   macOS / Linux

   INTAKE_MODE=thread fastapi run

   Windows (PowerShell)

   $env:INTAKE_MODE = "thread"; fastapi run

3. In the third terminal, run 32 users for 40 seconds. Locust writes every request to one.csv:

   macOS / Linux

   SAMPLES_CSV=one.csv locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 32 -r 32 -t 40s

   Windows (PowerShell)

   $env:SAMPLES_CSV = "one.csv"; locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 32 -r 32 -t 40s

4. Summarize the run without its first 10 seconds (the warm-up): python -m scripts.percentiles one.csv --skip 10.

5. Stop the API with Ctrl+C. Start four processes: INTAKE_MODE=thread fastapi run --workers 4 (Windows: fastapi run --workers 4, the variable is still set).

6. Run step 3 again with four.csv instead of one.csv, then step 4 with four.csv.

7. Copy docs/benchmark-template.md to docs/results-processes.md. Fill in the machine (python -m scripts.machine), the workload and the two rows of results.

Show the expected result

Locust prints a table every few seconds and a summary at the end. For the one-process run, it ends with exit code 1, because some requests failed with 503. The author’s two runs, on a busy shared computer:

106 requests in 28.3 s; status codes {'201': 95, '503': 11}
Successful: 95 = 3.36 per second
Latency of the successful requests: mean 6906.5 ms | p50 6867.0 ms | p95 7629.2 ms | p99 8254.4 ms | max 8254.4 ms

302 requests in 29.8 s; status codes {'201': 302}
Successful: 302 = 10.14 per second
Latency of the successful requests: mean 2840.1 ms | p50 2627.0 ms | p95 4020.6 ms | p99 4250.1 ms | max 5093.0 ms

Your numbers will differ. Expect the same shape. One process serves about 3.4 per second, with some 503 and a p50 above 5 seconds. Four processes serve about three times that, with no 503. One pass is not a result. The course team’s numbers are the median of five passes. If you have the time, run each configuration at least three times, and write down the lowest and the highest.

Your turn: one async process

Change one condition: keep one process, but use the async design. Predict first: with 32 users, does one async process serve more or fewer tickets per second than four processes of the Module 1 design? Then start the API with fastapi run (with INTAKE_MODE unset, or INTAKE_MODE=async) and run steps 3 and 4 with async.csv. On Windows, remove the variable first: Remove-Item Env:INTAKE_MODE.

Show the answer

More. The course team measured 13.39 tickets per second for one async process at 32 users, the median of 3 passes of 30 seconds. Its p95 was 2,680 ms, with no 503. The author’s run on a busy computer: 12.72 per second, p95 2,997 ms. That is more than four processes of the Module 1 design (9.59).

The async design waits for the provider without a connection, so the pool of 10 is no longer the limit. All 32 tickets are in progress at once: 32 ÷ 2.2 s ≈ 14.5 at most. Four processes multiplied the pools; async removed the need for them. Both are ways to do more at once. They do not exclude each other: in production, you would run several async processes, for safety and for the CPU work. The next lesson shows what then limits them.

When to scale out, and what it costs

Scale out when one instance is the limit and the shared services still have room. Horizontal scaling also gives you safety: if one instance crashes, the others keep serving. In the cloud, this is how a service grows from one container to many. Module 5 decides on the number of instances automatically.

Do not scale out when the limit is behind the instances. More processes do not make the provider’s quota bigger, and they do not make a slow database faster. They send it more work. For CPU-heavy work, more processes than CPU cores only take turns on the same cores.

What it costs:

• Memory: each process loads its own copy of Python, the libraries and the app. Here, about 87 MB each.
• Connections: each process has its own pool. Four processes can open 40 connections; the database’s limit is for all of them together.
• Uneven work: the spread over instances is not perfect, as the 20–6–4–2 split showed.
• No shared memory: everything that requests share must move to a shared service. Module 3 moves the cache to Valkey and Module 4 moves the rate limit there for exactly this reason.

State inside a process

Common mistake

A limit kept in a Python dictionary

Symptom (constructed, not in the project): a developer limits each customer to 30 new tickets per minute. The test passes with fastapi dev. In production, with four processes, a customer sends 100 tickets in one minute and the API accepts about 100. The log shows no 429 for that customer.

Cause: the counter is a dictionary in the module, for example sent = defaultdict(int), and sent[customer_id] += 1 for each ticket. Each process has its own dictionary. Four processes allow 4 × 30 = 120 tickets per minute, and a restart sets every counter back to 0.

Diagnose:

1. Look for module-level variables that change while the API runs: dictionaries, lists, counters and global statements. A setting read once at start-up is fine.
2. Run the test against fastapi run --workers 2, not only fastapi dev. A limit or a count that is right with one process and wrong with two lives inside the process.

Fix: keep the counter in a shared service that every process uses. Module 4 does exactly this: RATE_LIMIT_PER_MINUTE counts in Valkey, so all processes see one count. Keep inside a process only resources (a pool, a client) and data that never changes while the API runs.

Knowledge check

Check your understanding

Four processes of the Module 1 design served 9.59 tickets per second at 32 users, against 3.37 for one process. The API used about 1% of a CPU core. What did the extra processes add, if not CPU?

Show the answer

Connection pools. In the Module 1 design, each ticket keeps a database connection for about 2.2 seconds while it waits for the provider. One process has one pool of 10, so at most 10 tickets are in progress: at most about 4.5 per second. The other 22 users waited for a connection, and some got 503. Four processes have four pools, up to 40 connections, so all 32 users could be in progress.

Check your understanding

Which of these can stay inside one API process when you run four: (a) the connection pool, (b) the next ticket number, (c) a dictionary that counts each customer's tickets this minute, (d) the API key read from a file at start-up? Explain.

Show the answer

(a) and (d) can stay. The pool is a resource that each process needs for itself, and the API key does not change while the API runs. (b) and (c) cannot. A later request needs them, and every process must see the same value. The ticket number comes from the database sequence ticket_number_seq; the counter belongs in a shared service such as Valkey. Inside the process, four processes would give out T-500001 four times and allow four times the limit.

Check your understanding

Omar asks: 'If 4 processes gave almost 3 times the throughput, will 40 processes give 30 times?' What do you answer?

Show the answer

No. The test had only 32 users, and the async version showed the ceiling for them: about 32 ÷ 2.2 s ≈ 14.5 tickets per second. More processes also multiply what each process uses from the shared services. 40 processes with pools of 10 could open 400 connections, and PostgreSQL allows 100. And the provider’s quota is per account: with the quota on, it allows about 50 tickets a minute, whatever the number of processes. The next lesson adds these limits up.

Recap

• Horizontal scaling runs more instances of the same service behind one address; vertical scaling uses a bigger computer.
• An instance must be stateless: what a later request needs lives in a shared service (here PostgreSQL), never in a Python variable.
• fastapi run --workers 4 starts four worker processes, each with its own pool of up to 10 connections.
• Measured with 32 users and the Module 1 design: one process 3.37 tickets per second (p50 6,831 ms, some 503); four processes 9.59 (p50 2,621 ms, no 503). One async process: 13.39.
• Connections are not spread evenly: 20, 6, 4 and 2 users in the author’s run.
• Each process multiplies its resources: memory, connections. The provider’s quota does not grow.

Next: Dependency limits