Dependency limits

What you will be able to do

Find every limit behind the API (the connection pool, the database, the provider’s quota and the settings of each process) and add them up. Measure why more requests at once eventually give less useful work, not more.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Threads, async I/O and the INTAKE_MODE setting, from I/O versus CPU work.
• Several processes behind one address, and the comparison of two configurations, from Horizontal scaling.
• The bottleneck and the provider’s quota (100 chat requests per minute), from Find the bottleneck.

What you need on your computer

• Your project after Horizontal scaling: python -m pytest prints 137 passed.
• PostgreSQL running, with the small data loaded.

After this lesson, your project matches the snapshot end-of-m02. The AI provider is the simulated one, with its quota off in the load tests, as in the last lesson. The case with 12 processes is constructed: a wrong setting, made on purpose. The course team measured the numbers on an Apple M4 (10 cores, 16 GB) with nothing else busy. Your times will differ, but the shape stays the same. Tested on macOS (Apple silicon); the Windows commands are not tested.

A lift with ten places

An office building has one lift. It takes 10 people, and one trip up and back takes one minute. Everyone in the building has a meeting on the top floor, and the meetings start in five minutes.

With 10 people at the lift, everyone is on time: one trip. With 30 people, the last ones wait for three trips, and they are still on time. Now 60 people arrive. The lift still carries 10 people a minute: it works as hard as before. But the line is six minutes long, so the people at the end are late. If people keep arriving as fast as the lift carries them away, the line never gets shorter. Soon everyone who gets out at the top is late. The lift is busy all the time, and none of its trips helps anyone. Some people give up and take the stairs.

More people at the door did not make the lift carry more. They only made the line longer. The ticket API has a lift too.

The problem: more users, less useful work

In Horizontal scaling, one process of the Module 1 design served only about 3.4 tickets per second at 32 users. The course team then measured the same design from 2 to 64 users. Each point is the median of 3 passes of 30 seconds.

First, a word for what counts. In this module, an answer is useful if it is a success (201) within 5 seconds. A person waits in the help desk’s form while the API saves the ticket. After a few seconds, people give up or send the ticket again. Useful throughput is the number of useful answers per second: the throughput you can promise. The benchmark template’s example question uses the same 5 seconds.

Users (requests in flight)	Useful tickets per second	p50	503 per pass
2	0.85	2,310 ms	none
8	3.12	2,299 ms	none
10	3.86	2,233 ms	none
12	3.88	2,687 ms	none
16	3.68	3,608 ms	none
32	0 (3.25 finished)	6,820 ms	9 to 11
64	0 (2.53 finished)	10,842 ms	32 to 34

Up to 10 users, more users gave more useful work. From 10 to 16, nothing changed: about 3.9 tickets per second. At 32 users, the API still finished 3.25 tickets per second, but none of them within 5 seconds. Useful throughput fell to 0. At 64 users, the median answer took 10.8 seconds.

The idea: a constrained dependency

A dependency is a service that your API needs to answer: the database, the AI provider, later the cache. Each one has limits. When requests need a dependency that can serve only a few of them at a time, it is a constrained dependency. Requests line up in front of it, like people at the lift.

Here, the constraint is the connection pool: at most 10 connections, a wait of at most 5 seconds, then 503 database_busy. The databases course set these rules in Connection management. The Module 1 design keeps a connection for the whole 2.2 seconds of AI work. So 10 connections mean at most 10 tickets in progress.

Little’s law, from Latency and capacity, explains the numbers. Turn it around: time in the system = requests in flight ÷ throughput. Use the measured throughput of about 3.9 tickets per second:

• 16 users: 16 ÷ 3.9 ≈ 4.1 seconds per ticket. Under the promise of 5 seconds, so still useful. Measured p50: 3.6 s.
• 32 users: 32 ÷ 3.9 ≈ 8.2 seconds. Over the promise. Measured p50: 6.8 s; the 503 answers after 5 seconds shorten the line a little.
• 64 users: 64 ÷ 3.9 ≈ 16 seconds, with more of them turned away. Measured p50: 10.8 s.

The throughput cannot grow past the pool, so every extra user only adds waiting. From some point, each ticket waits longer than the promise. Then the work still happens, but none of it is useful. That is why more concurrency eventually reduces useful throughput. Finished tickets per second also fell a little, from 3.85 to 3.25 and 2.53. The course did not measure why; waiting requests are not free for a server either.

The async design waits for the provider without a connection. It borrows one only for the save at the end, for milliseconds. Here are both designs with 32 requests in flight:

The same 32 requests in flight, two designs. The Module 1 design turns the pool of 10 into a line; the async design does not need the pool while it waits.

Text description of the diagram

The top grid is the Module 1 design with 32 requests in flight. 10 requests hold the 10 connections of the pool while they wait for the provider. The other 22 wait for a connection, and after 5 seconds the API answers them with 503 database_busy. PostgreSQL sees 10 connections that do almost nothing. At most 10 tickets are in progress, so at most about 4.5 tickets per second; the course team measured 3.86 at 10 users.

The bottom grid is the async design with the same 32 requests. None of them holds a connection while it waits. All 32 provider calls are in flight together, and each ticket borrows a connection only for its save. At most about 14.5 tickets per second; the course team measured 13.39.

Explore: from 2 to 64 users

Move the number of users and watch both lines. The explorer shows the measured medians, not a model:

Explore

More requests in flight than the pool has connections

Users

10

Module 1 design: useful

3.86/s

Module 1 design: finished

3.85/s

Module 1 design: p95

3.2 s

Module 1 design: 503

none

async: useful

4.05/s

async: p95

2.9 s

10 requests in flight. Every request gets one of the 10 connections at once, so both designs serve about the same: 3.86/s and 4.05/s, with p95 under 3.2 s. At 10 users, the Module 1 design uses the whole pool. This is its best.

• Module 1 design (keeps a connection)
• async (a connection only to save)

The dashed line: what the Module 1 design finished per second, also the answers that came too late.

Measured medians of 3 passes of 30 s per point: Locust users without pause, pool of 10, DB_POOL_TIMEOUT 5 s, simulated provider with its quota off. Useful = 201 within 5 s.

Text description of the explorer

The explorer has one control, a slider for the number of users: 2, 4, 8, 10, 12, 16, 32 or 64. It starts at 10. Above two charts, it shows four numbers of the Module 1 design: useful throughput, finished tickets per second, p95 and the 503 count. It also shows the useful throughput and p95 of the async design.

The first chart shows useful tickets per second against users. On its scale, 2, 4, 8, 16, 32 and 64 are evenly spaced, and a dashed line marks the pool size, 10. The Module 1 design rises from 0.85 at 2 users to 3.86 at 10 and 3.88 at 12. It stays at 3.68 at 16, then falls to 0 at 32 and 64. A dashed line shows what it finished, also too late: 3.25 at 32 users and 2.53 at 64. The async design rises all the way: 4.05 at 10, 6.69 at 16, 13.39 at 32 and 27.18 at 64.

The second chart shows the p95 time of the 201 answers, with a dashed line at the promise of 5 seconds. The Module 1 design stays near 3 seconds up to 10 users, then grows: 4.2 s at 16, 7.5 s at 32 and 11.8 s at 64. The async design stays under 3.2 seconds at every point.

What this tells us. The pool decides the Module 1 design’s best: about 3.9 tickets per second, reached at 10 to 12 users. Past it, extra users add only waiting, and from 32 users no answer is on time. The async design has no such line at the pool. At 64 users it served 27.18 tickets per second with p95 2.8 s. That curve also ends somewhere, at the next limit. With the quota off, the course did not reach it.

Larkfield: count every limit

Tomás has a quick fix for the Module 1 design: “Add more workers.” Horizontal scaling showed that four processes help. But every setting with a number in it is per process, and the limits behind them are for everyone. Mei, who owns the database, asks Amira to add them up first.

Limit	Set where	Counts for	Value here
Connections in the pool (DB_POOL_MAX)	the API’s .env	each process	10
Wait for a connection (DB_POOL_TIMEOUT)	the API’s .env	each request	5 s, then 503
Threads for normal functions	FastAPI (anyio)	each process	40
Provider calls in flight (PROVIDER_MAX_CONCURRENCY)	the API’s .env	each process	0 = no limit
Connections to PostgreSQL (max_connections)	the database	all processes together	100, 3 kept for administrators
The provider’s quota	the provider	the whole account	100 chat requests per minute

The script scripts.limits reads the settings and the database’s limit, and multiplies:

Database: max_connections 100 (3 kept for administrators); 1 in use now
API: 12 process(es) x DB_POOL_MAX 10 = up to 120 connections  <- more than the database allows
Provider: PROVIDER_MAX_CONCURRENCY no limit per process; the quota is 100 chat requests per minute for the whole account, whatever the number of processes

Twelve processes could ask for 120 connections, but the database accepts 97 for the app. The quota does not grow at all. With two chat calls per ticket, it allows 50 tickets per minute for the whole account. That is 0.83 tickets per second. With the quota on, both designs stop there, with one process or with twelve. Omar’s Black Friday hour needs 0.28 tickets per second on average, so the quota is enough for the average. A busy minute needs 1.5 tickets per second: almost twice the quota. Module 4 handles those bursts with a queue.

Two more facts belong in the count. First, the real provider got slower with load. In the recordings, a reply took about 7% longer with 16 calls in flight, and about 30% longer with 40. The simulated provider does not model this. So the async curve above is better than a real provider would give. Second, a limit can protect a dependency: PROVIDER_MAX_CONCURRENCY caps the calls in flight from one process. The others wait for a free place inside the API, not at the provider. A tiny example from tests/test_concurrency.py: 6 calls, 2 places, 0.1 seconds per call. The calls go in 3 rounds of 2, so all 6 take about 0.3 seconds. Like the pool, the value is per process: four processes with a limit of 8 send up to 32 calls at once.

Guided practice: measure the fall

This is the module’s completion check: measure the useful throughput of the Module 1 design below and above the pool, and explain the difference.

1. Copy the last script of this module from the snapshot end-of-m02:

   macOS / Linux

   cp ../nextia-learning-labs/C21/snapshots/end-of-m02/scripts/limits.py scripts/limits.py

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C21\snapshots\end-of-m02\scripts\limits.py scripts\limits.py

2. Run python -m scripts.limits --processes 12, then python -m scripts.limits --processes 4. Compare the second line of each.

3. Start the simulated provider with its quota off in the first terminal, as in the last lesson (SIM_QUOTA=off python -m simulator).

4. Start one process of the Module 1 design in the second terminal: INTAKE_MODE=thread fastapi run (Windows: $env:INTAKE_MODE = "thread"; fastapi run).

5. In the third terminal, run 16 users for 40 seconds, then summarize:

   macOS / Linux

   SAMPLES_CSV=t16.csv locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 16 -r 16 -t 40s
   python -m scripts.percentiles t16.csv --skip 10

   Windows (PowerShell)

   $env:SAMPLES_CSV = "t16.csv"; locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 16 -r 16 -t 40s
   python -m scripts.percentiles t16.csv --skip 10

6. Count the useful answers: the 201 answers within 5 seconds, without the first 10 seconds. One line of Python does it; it works in both shells:

   python -c "import csv; r=list(csv.DictReader(open('t16.csv'))); t0=float(r[0]['start']); r=[x for x in r if float(x['start'])-t0>=10]; print(sum(x['status']=='201' and float(x['ms'])<=5000 for x in r), 'of', len(r), 'requests: 201 within 5 s')"

7. Repeat steps 5 and 6 with 32 users and the file t32.csv.

8. Stop the API. Start it in async mode (fastapi run with INTAKE_MODE unset; on Windows Remove-Item Env:INTAKE_MODE first). Repeat steps 5 and 6 with 32 users and a32.csv.

Show the expected result

Step 2 prints the three lines above for 12 processes. For 4 processes, the second line is API: 4 process(es) x DB_POOL_MAX 10 = up to 40 connections, with no warning. The author’s runs of steps 5 to 8, on a busy shared computer:

16 users, Module 1 design
112 requests in 29.8 s; status codes {'201': 112}
Successful: 112 = 3.76 per second
Latency of the successful requests: mean 3735.0 ms | p50 3766.0 ms | p95 4448.1 ms | p99 4816.2 ms | max 5031.4 ms
111 of 112 requests: 201 within 5 s

32 users, Module 1 design
110 requests in 29.9 s; status codes {'201': 101, '503': 9}
Successful: 101 = 3.38 per second
Latency of the successful requests: mean 6874.9 ms | p50 6878.2 ms | p95 7399.9 ms | p99 7493.9 ms | max 8042.7 ms
0 of 110 requests: 201 within 5 s

32 users, async
384 requests in 29.8 s; status codes {'201': 384}
Successful: 384 = 12.9 per second
Latency of the successful requests: mean 2299.2 ms | p50 2248.4 ms | p95 3031.2 ms | p99 3347.6 ms | max 3862.7 ms
384 of 384 requests: 201 within 5 s

This page adds the titles above each block; the commands print only the lines below them. Your numbers will differ. Expect the same shape. At 16 users, almost every answer is useful. At 32 users, the Module 1 design still finishes about 3.4 tickets per second, but about none within 5 seconds, and some requests get 503. Async at 32 users answers all of them in time.

Now answer the completion check in your own words. Write it in docs/results-processes.md from the last lesson, or in a new copy of the template. Why did doubling the users from 16 to 32 take the useful throughput from about 3.7 to about 0?

Your turn: a limit on provider calls

Change one setting. Keep the async design and 32 users, but limit each process to 8 provider calls in flight: PROVIDER_MAX_CONCURRENCY=8 fastapi run (Windows: $env:PROVIDER_MAX_CONCURRENCY = "8"; fastapi run). Predict first: what is the most tickets per second that this can give? Think about the slots as the pool of a new lift. Then run steps 5 and 6 with max8.csv.

Show the answer

The limit makes a new lift. A ticket uses a place three times: for about 0.91, 1.10 and 0.20 seconds. That is about 2.2 seconds of places per ticket. 8 places give at most 8 ÷ 2.2 ≈ 3.6 tickets per second, like the pool of 10 gave about 4.5. The other 24 tickets wait inside the API, and the time of each call now includes that wait. The author’s run, on a busy computer, was lower still:

62 requests in 29.2 s; status codes {'201': 62}
Successful: 62 = 2.12 per second
Latency of the successful requests: mean 9474.6 ms | p50 9383.7 ms | p95 10463.5 ms | p99 11077.6 ms | max 11077.6 ms

So a limit is not free: it moves the line from the provider into your API. It is still worth having when the dependency gets worse with load, as the real provider did with 40 calls in flight. Set it from the dependency’s measured behaviour, multiply it by the number of processes, and remove the variable when you are done (Remove-Item Env:PROVIDER_MAX_CONCURRENCY on Windows).

When to set a limit, and what it costs

Count first. Before you add processes, multiply every per-process setting by the number of processes, and compare the sum with the shared limit: python -m scripts.limits --processes N. Do the same for each service that more than one program uses. A worker of Module 4, a second API or a report script all count against the same 100 connections.

Hold a constrained resource only while you use it. The best fix in this lesson was not a bigger pool. It was the async design, which does not hold a connection while it waits. A bigger pool only moves the line to PostgreSQL, where each connection is a server process with its own memory.

Limit at the front, not only at the back. A line inside the API still makes every answer late. Module 4 refuses extra work at the door with a clear answer and a Retry-After header, so that the work that is accepted stays useful.

What it costs. Each limit turns some requests into waits or errors on purpose: a 503 after 5 seconds is better than an answer after 11. Set the waits shorter than the promise, so that a request that cannot be on time fails early instead of using resources for nothing.

See it end to endCapacity plan from a public traffic traceCase study

The case study uses the same arithmetic on real arrivals: how many requests are in flight, and when the line grows faster than it shrinks.

More processes than the database allows

Common mistake

Twelve processes, one database: too many clients

Symptom (constructed): Amira starts twelve processes with a large minimum pool, DB_POOL_MIN=10 fastapi run --workers 12. /health still answers 200. But the log fills with warnings, and even psql cannot connect to look. The author’s run, with the warnings counted and the port shortened to …:

 100 WARNING psycopg.pool: error connecting in 'ticket-api': connection failed: connection to server at "127.0.0.1", port … failed: FATAL:  sorry, too many clients already

psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  sorry, too many clients already

Cause: each process opens its own pool, and twelve pools of 10 are 120 connections. PostgreSQL accepts 100 in total. The first pools fill the database, and the others keep trying. The processes that got no connections answer 503 database_busy to every ticket, and a worker, a migration or an administrator cannot connect either. With the default DB_POOL_MIN of 2, the same mistake appears later, under load, when the pools grow.

Diagnose:

1. Run python -m scripts.limits --processes 12. The warning <- more than the database allows shows the problem before any test.
2. Search the log for too many clients.
3. Count the connections when you can still connect: SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';.

Fix: stop the API. Then make the sum fit. Use fewer processes, or a smaller DB_POOL_MAX per process, so that processes × pool, plus every other program, stays under max_connections with room for administrators. Better still, hold connections for less time, as the async design does. Raising max_connections is the last choice: each connection costs the database memory.

Knowledge check

Check your understanding

Explain why increasing concurrency eventually reduces useful throughput. Use the Module 1 design's numbers at 10, 16 and 32 users.

Show the answer

The Module 1 design keeps a database connection for the whole 2.2 seconds of AI work. So the pool of 10 allows at most 10 tickets in progress. Its throughput stops at about 3.9 tickets per second: 3.86 at 10 users, 3.68 at 16. More users cannot raise it; they only wait in line for a connection. By Little’s law, time = users ÷ throughput. That is about 4.1 seconds at 16 users, within the promise of 5 seconds. At 32 users, it is about 8.2 seconds, past the promise. So at 32 users, the API still finished 3.25 tickets per second, but none within 5 seconds: useful throughput 0. Some requests waited longer than 5 seconds for a connection and got 503. The extra concurrency added only waiting, and the waiting made every answer too late.

Check your understanding

Six API processes with DB_POOL_MAX 10 and two worker processes (Module 4) with pools of 10 share one PostgreSQL with max_connections 100. Is the sum safe? What about with the quota on: how many tickets per second can they serve together?

Show the answer

Six API processes can open 60 connections, and two workers 20: 80 in total. That is under the 97 that the app may use, with some room for a migration or psql. With the quota on, the limit is not the connections. The quota allows 100 chat requests per minute for the whole account, two per ticket. That is 50 tickets per minute, about 0.83 per second, whatever the number of processes.

Check your understanding

The async design served 27.18 tickets per second at 64 users with the simulated provider and its quota off. Kwame asks whether one process can take 27 tickets per second on Black Friday. What do you tell him?

Show the answer

Not from this test. The quota was off, and the real quota allows about 0.83 tickets per second for the whole account. The simulated provider does not get slower with load, but the real one did: about 30% slower with 40 calls in flight. And the test ran on one laptop with the small data. The 27 per second only shows that the API and the pool are no longer the limit. The next limit is the provider, and a load test against your own services cannot measure its capacity. Module 6 turns such results into a careful recommendation.

Recap

• A constrained dependency serves only a few requests at a time; the others line up. Here it was the pool of 10, because the Module 1 design held a connection for 2.2 s.
• Useful throughput counts successes within the promise (201 within 5 s). Module 1 design: 3.86 at 10 users, 3.68 at 16, 0 at 32 and 64, although it still finished 3.25 and 2.53 per second.
• Little’s law: time in the system = requests in flight ÷ throughput. When something caps the throughput, more concurrency only adds waiting, until every answer is late.
• The async design holds no connection while it waits: 13.39 per second at 32 users, 27.18 at 64 (quota off, simulated provider).
• Per-process settings add up: 12 processes × 10 = 120 connections, more than PostgreSQL’s 100. The quota is per account: 0.83 tickets per second for everyone.
• Your project matches end-of-m02: 137 passed.

Next: Choose cacheable work