I/O versus CPU work

What you will be able to do

Let one API process serve many tickets at the same time while each ticket waits seconds for the AI provider. Learn the difference between waiting work and computing work, and see how one wrong call makes the whole server wait.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Where a ticket’s time goes: about 2.2 seconds, almost all of it in the AI provider, from Find the bottleneck.
• Throughput, p95 and a Locust run, from Latency and capacity.
• The three terminals (simulator, API, commands), from Run the AI workload on your computer.

What you need on your computer

• Your project at the end of Module 1: python -m pytest prints 128 passed. PostgreSQL runs, with the small data loaded.
• Your copy of the labs repository next to the project, up to date (git pull in nextia-learning-labs).

After this lesson, your project matches the snapshot end-of-m02, except for one script of the last lesson of this module. The AI provider is the simulated one on your computer: its times come from real recorded calls. The blocking mode in this lesson is a mistake made on purpose. The course team measured the numbers on an Apple M4 (10 cores, 16 GB) with nothing else busy. Your times will differ, but the shape stays the same. Tested on macOS (Apple silicon); the Windows commands are not tested.

A cook and a pot of water

A small restaurant has one cook. An order for pasta comes in. The cook puts a pot of water on the stove. The water needs eight minutes to boil.

A good cook does not stand and look at the pot for eight minutes. The cook starts the next order, cuts vegetables and plates a salad. When the water boils, the cook comes back to the pasta. One cook serves many orders, because most of an order is waiting: for water, for the oven, for the fridge to cool.

Now imagine a cook who stands in front of the pot until it boils. The water boils just as fast. But for eight minutes, nobody takes an order and nobody plates a salad. A guest who only wants a glass of water waits eight minutes too.

Your API is the cook, and the AI provider is the pot of water.

The problem: a server that looks at the pot

In Find the bottleneck, one ticket took about 2,147 ms. Of that, the database took 9.5 ms and the other code 7.6 ms. The three calls to the AI provider took about 98%. During the baseline runs with 1 to 4 users, the API process used 0.6–2.2% of one CPU core. The API does not compute. It waits.

Waiting is cheap, but only if the server can do something else while it waits. The project of this lesson has a setting that shows what happens when it cannot. INTAKE_MODE=blocking is a mistake made on purpose: the endpoint waits for the provider in a way that holds the whole server. The script scripts.stall sends 8 tickets, 4 at a time. At the same time, it asks GET /health every 0.1 seconds. /health does no work at all, so it should answer in a few milliseconds. The course team’s run, with the simulated provider:

8 tickets, 4 at a time: 18.8 s, status codes [201]
GET /health during that time: 3 answers, median 8737.2 ms, slowest 9824.7 ms

An endpoint that does nothing waited 8.7 seconds. In 18.8 seconds, the health check got 3 answers. In async mode, it gets about 50 answers in 5 seconds. Think of a load balancer that checks /health with a timeout of a few seconds. It would decide that this server is dead and stop sending it traffic.

The idea: I/O work and CPU work

Every request does two kinds of work.

• CPU work is computing: parsing JSON, building a prompt, hashing a password, running a model on your own computer. The processor is busy. In the kitchen, this is cutting vegetables.
• I/O work is waiting for something outside the program: an answer from the network, a disk, a database or an AI provider. I/O means input/output. The processor has nothing to do until the answer comes. In the kitchen, this is the pot of water.

A ticket in this course is almost all I/O work. It waits about 2.1 seconds for the provider and computes for a few milliseconds. So computing faster would not help. The important question is what the server does while it waits. There are three answers.

1. Threads: several cooks in one kitchen. A thread is a line of work inside a process. Each thread can wait on its own. Module 1’s endpoint was a normal function (def). FastAPI runs such a function in a thread pool, by default up to 40 threads at a time. Four tickets wait in four threads, and /health runs in a fifth.
2. Async I/O: one cook, many pots. An async def endpoint runs on the server’s event loop: one thread with a list of tasks. When a task reaches await (the cook puts the pot on the stove), the loop runs another task. When the answer arrives, the loop continues the first task. One thread can wait for hundreds of answers at the same time.
3. Processes: several kitchens. A process is a whole running program with its own memory. Several processes of the API can serve requests side by side. That is the next lesson, Horizontal scaling.

The mistake of this lesson belongs to the second answer. Inside an async def endpoint, a normal waiting call does not say await. So the event loop never gets control back, and every other task waits. That is a blocking call: the cook who stares at the pot.

A tiny example: four tickets, two ways

Take four tickets that arrive at the same moment. Use the medians of the real recorded calls: classify 0.91 s, draft a reply 1.10 s, embed 0.20 s. One ticket’s AI work is 0.91 + 1.10 + 0.20 = 2.21 s. A GET /health arrives 0.1 s later.

• Blocking. The event loop does ticket 1’s three calls without a break, from 0 to 2.21 seconds. Then it does ticket 2, from 2.21 to 4.42 seconds. Ticket 3 ends at 6.63 seconds, and ticket 4 at 8.84 seconds. So /health waits for all of it. It answers at about 8.8 seconds, after 8.7 seconds of waiting: the 8,737 ms of the measurement.
• Async. All four tickets send classify at once and wait together. They finish at about 2.21 s. /health answers in milliseconds whenever it asks.

The difference is a factor of four for the tickets, and a factor of thousands for /health:

Four tickets and a health check on one server, drawn to scale from the medians of the recorded provider calls (a constructed timeline). Blocking does one ticket's waiting at a time; async waits for all four together.

Text description of the diagram

Top, blocking: the tickets’ bars come one after another. Hatched bars are time spent waiting for the server, not for the provider. All four tickets answer together at about 8.8 seconds, because each ticket’s last step, the save, lets the server start the next ticket’s blocking work. GET /health waits the whole time. Bottom, async: the four tickets’ bars lie on top of each other, all four answer at 2.2 seconds, and GET /health answers in a few milliseconds each time.

The author’s API log of a blocking run shows the same order. The twelve provider calls come strictly one ticket after the other, and the answers leave together. This is the author’s run on a busy computer, with the host shortened to …; your times will differ:

INFO httpx: HTTP Request: POST …/v1/chat/completions "HTTP/1.1 200 OK"
INFO httpx: HTTP Request: POST …/v1/chat/completions "HTTP/1.1 200 OK"
INFO httpx: HTTP Request: POST …/v1/embeddings "HTTP/1.1 200 OK"
... the same three lines for tickets 2, 3 and 4 ...
INFO ticket_api: GET /health 200 4143ms queries=0 id=bb7f214414ea
INFO ticket_api: POST /v1/tickets 201 8479ms queries=7 id=c5b2dd1547ab
INFO ticket_api: POST /v1/tickets 201 8480ms queries=7 id=cc7f8d112c36
INFO ticket_api: POST /v1/tickets 201 8480ms queries=7 id=6db109487a91

Measured: the stall in numbers

The course team ran scripts.stall five times in each mode, on a quiet computer, with the provider’s quota off. The simulator then acts as if Larkfield’s account allowed many more calls per minute, so the quota cannot hide the stall. The next lesson shows how to switch it off. The chart shows the median time of /health:

Median time of GET /health during scripts.stall (8 tickets, 4 at a time), the median of 5 passes per mode. Simulated provider, quota off. The async and thread bars are there: 3.4 ms is too small to see next to 8.7 seconds.

Text description of the chart

Median time of GET /health while the API worked on 8 new tickets, 4 at a time, over 5 passes: blocking 8,737.2 ms (the slowest answer of a typical pass 9,824.7 ms), async 3.4 ms, thread (the Module 1 version) 3.4 ms. The 8 tickets took 18.8 s, 5.4 s and 4.8 s.

What this tells us. A blocking call does not only slow its own request. It slows every request of the process, including the ones that do no work. The tickets suffer too: 8 tickets took 18.8 s instead of about 5.4 s.

The same three modes under a steady load of 8 Locust users for 40 seconds, 5 passes:

INTAKE_MODE	Successful tickets per second	p50	p95
thread (Module 1)	3.22	2,237.3 ms	3,030.2 ms
blocking (the mistake)	0.33	18,288.2 ms	18,292.6 ms
async	3.47	2,146.6 ms	2,734.2 ms

With 8 users, thread and async each serve about 3.3 tickets per second: 8 tickets in flight, about 2.3 s each. Blocking serves one ticket at a time, so the 8 users share it. Each waits for about 8 × 2.2 s of other tickets’ work, and the median answer took 18.3 seconds.

Larkfield: INTAKE_MODE in ticket_api/intake.py

Amira keeps all three designs in one codebase, as the databases course kept DB_POOL=false. One setting chooses the design, so every measurement compares the same code with one change. The default is async. The endpoint is now async def, and the three modes differ in a few lines:

ticket_api/intake.py (create_ticket, the part that waits)

    mode = request.app.state.settings.intake_mode
    if mode == "thread":
        ticket, work = await run_in_threadpool(create_in_one_transaction, db, provider, new, timer)
    else:
        with timer("db"):
            await run_in_threadpool(check_customer, db, new.customer_id)
        if mode == "blocking":
            # A mistake, on purpose: the waiting client inside an async endpoint.
            work = do_ai_work(provider, new.subject, new.body, timer)
        else:
            work = await do_ai_work_async(provider, new.subject, new.body, timer)
        with timer("db"):
            ticket = await run_in_threadpool(save_new_ticket, db, new, work)

Read it line by line:

• thread runs Module 1’s code, unchanged, in the thread pool. run_in_threadpool is what FastAPI did for you when the endpoint was a def.
• blocking calls do_ai_work, which uses provider.call(). That is httpx.Client.post: a normal call that waits. Nothing in it says await, so the event loop is stuck until the provider answers.
• async calls do_ai_work_async, which uses await provider.acall(...). That is httpx.AsyncClient.post. At each await, the loop serves other requests.

The database calls are normal functions too, because the project uses psycopg’s normal (synchronous) pool. So the async endpoint sends them to a thread with run_in_threadpool. They take milliseconds, and the event loop stays free.

The async mode also changes when the ticket is saved. Module 1 opened one transaction, inserted the ticket, waited 2.2 s for the AI work and then saved the results. The connection stayed borrowed the whole time. The async mode first checks that the customer exists. Then it does the AI work without a connection. At the end, it saves the ticket and its results in one short transaction. That is the rule from Connection management: no slow work inside a transaction. Dependency limits measures what that rule is worth.

The new provider method is short:

ticket_api/provider.py (acall)

    async def acall(self, path: str, body: dict) -> tuple[dict, float]:
        """The same as call(), awaited. The time includes any wait for a free slot."""
        started = time.perf_counter()
        if self._slots is None:
            response = await self._apost(path, body)
        else:
            async with self._slots:
                response = await self._apost(path, body)
        return _check(response), (time.perf_counter() - started) * 1000

_slots is an optional limit on calls in flight, PROVIDER_MAX_CONCURRENCY. Dependency limits explains it; for now it is off.

Guided practice: make the server stall, then fix it

1. In your project folder, with the virtual environment active, copy the files of this lesson from the snapshot end-of-m02:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m02
   for f in ticket_api/config.py ticket_api/intake.py ticket_api/main.py ticket_api/provider.py ticket_api/repository.py tests/fakes.py tests/test_concurrency.py scripts/stall.py README.md; do cp "$S/$f" "$f"; done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m02"
   foreach ($f in 'ticket_api\config.py', 'ticket_api\intake.py', 'ticket_api\main.py', 'ticket_api\provider.py', 'ticket_api\repository.py', 'tests\fakes.py', 'tests\test_concurrency.py', 'scripts\stall.py', 'README.md') { Copy-Item "$S\$f" $f }

2. Run the tests: python -m pytest. Expect 137 passed. The new file tests/test_concurrency.py sends 4 tickets at once to each mode, with a fake provider that takes 0.2 s per call.

3. In the first terminal, start the simulated provider: python -m simulator.

4. In the second terminal, start the API in blocking mode. A variable set in the terminal wins over .env, so .env does not change:

   macOS / Linux

   INTAKE_MODE=blocking fastapi dev

   Windows (PowerShell)

   $env:INTAKE_MODE = "blocking"; fastapi dev

5. In the third terminal, run the stall test. It takes about 20 seconds: python -m scripts.stall.

6. Look at the API’s log in the second terminal. Find the provider calls and the GET /health lines with their times.

7. Stop the API with Ctrl+C. Start it again in the default mode: fastapi dev on macOS and Linux. On Windows, first remove the variable: Remove-Item Env:INTAKE_MODE; fastapi dev.

8. Run python -m scripts.stall again.

Show the expected result

Step 2 ends with 137 passed. Step 5 prints two lines like the ones at the top of this lesson. Expect about 18–20 seconds, status codes [201], and 3 or 4 answers of /health with a median of several seconds. Step 8, in the default async mode, prints about this (the course team’s median pass, on a quiet computer):

8 tickets, 4 at a time: 5.4 s, status codes [201]
GET /health during that time: 51 answers, median 3.4 ms, slowest 6.0 ms

Your times differ. On a busy computer, the author saw a /health median of 1.7 ms and 6.1 s for the tickets. What matters is the shape: dozens of answers in a few milliseconds each, not three answers in seconds.

The simulated provider has a quota of 100 chat requests per minute, and each run of scripts.stall uses 16. If you run it more than about five times in one minute, some tickets get 503 with ai_busy. Wait a minute, then run it again.

Your turn: the Module 1 version

Module 1’s endpoint was a normal function, and nobody complained about /health. Predict first: with INTAKE_MODE=thread, how long does /health take while the 8 tickets wait? And how long do the 8 tickets take? Then start the API with INTAKE_MODE=thread fastapi dev (Windows: $env:INTAKE_MODE = "thread"; fastapi dev) and run python -m scripts.stall.

Show the answer

/health stays fast, and the tickets take about as long as in async mode. The course team’s median pass:

8 tickets, 4 at a time: 4.8 s, status codes [201]
GET /health during that time: 45 answers, median 3.4 ms, slowest 5.4 ms

Each ticket waits in its own thread of the thread pool, and /health runs on the event loop, which stays free. So threads also solve the waiting problem. They have two costs. First, the pool has 40 threads by default, and each thread needs memory; an event loop waits for hundreds of answers in one thread. Second, and more important here: Module 1’s code keeps a database connection during the whole wait. With 4 tickets, that does not matter. Dependency limits shows what happens with 32. Remove the variable again before you continue (Remove-Item Env:INTAKE_MODE on Windows).

When to use which, and what it costs

Use async for endpoints that mostly wait for I/O, when you can await every library in the path. Examples are httpx.AsyncClient for HTTP, an async database driver, and the async cache client of Module 3. One process then waits for many slow calls at once. The cost: inside an async def, you must await every call that waits, or send it to a thread. One forgotten normal call, such as time.sleep(1), requests.get(...) or a normal database query, stalls every request of the process, and the code still looks correct.

Use threads (a normal def endpoint, or run_in_threadpool) when a library has no async version, such as the psycopg pool in this project. The cost: a fixed number of threads (40 by default in FastAPI), and memory for each thread.

Use processes for CPU work. In Python, only one thread of a process runs Python code at a time, so threads do not make computing faster. The next lesson runs several processes. CPU work inside an async def blocks the event loop exactly like a waiting call. If an endpoint must compute for a long time, run that part in another process or a background worker, as Module 4 does.

Do not convert code to async only because it is newer. Module 1’s thread version already served 3.22 tickets per second at 8 users, against 3.47 for async. Convert when a measurement shows that waiting threads or held connections are the limit.

A waiting call inside an async endpoint

Common mistake

One normal call in an async def stalls every request

Symptom: under load, /health and other fast endpoints take seconds. The API’s CPU is almost 0%. In the log, the provider calls of different tickets never overlap: three calls of one ticket, then three of the next. All tickets answer at about the same moment.

Cause: an async def endpoint calls a function that waits without await. In this project, it is do_ai_work, which uses the normal httpx.Client. Other common ones are time.sleep(), requests.get(), a normal database driver, and reading a large file.

Diagnose:

1. Run python -m scripts.stall against the API. A median of seconds for /health means that the event loop is blocked.
2. Read the log. Calls that never overlap, although several requests are in flight, point to the same cause.
3. Look at every async def in the path of the slow request. Each call that waits must have await in front of it, or go through run_in_threadpool.

Fix: use the async client and await it (provider.acall). If a library has no async version, wrap the call: await run_in_threadpool(function, arguments). Or make the endpoint a normal def, and FastAPI runs it in a thread. Then add a test like test_a_blocking_call_stalls_the_whole_server: four tickets at once must take about one ticket’s time, not four.

Knowledge check

Check your understanding

In blocking mode, GET /health waited 8.7 seconds although it does no work. Explain why, with the numbers of the tiny example.

Show the answer

The endpoint and /health share one event loop. In blocking mode, the endpoint calls the provider without await, so the loop cannot switch to another task while it waits. Four tickets are in flight, and each needs about 2.21 s of provider calls (0.91 + 1.10 + 0.20). The loop does them one after another: 4 × 2.21 = 8.84 seconds. /health arrived at 0.1 seconds and had to wait until the loop was free, at about 8.8 seconds.

Check your understanding

A new endpoint computes a report for 3 seconds on the CPU, with no I/O. Amira writes it as async def. Predict what /health does while one report runs, and suggest a fix.

Show the answer

/health waits about 3 seconds. CPU work in an async def never reaches an await, so it blocks the event loop exactly like a waiting call. async helps only with waiting, not with computing. The best fix is to run the computation in another process, such as a worker or a process pool. At least, run it in a thread with run_in_threadpool, so that the loop stays free. More processes are the way to get more CPU work done at the same time in Python.

Check your understanding

Module 1's thread version and the async version both kept /health at 3.4 ms and served about the same at 8 users (3.22 and 3.47 tickets per second). So why does the project's default become async?

Show the answer

Both wait well with 8 tickets. The difference is what each request holds while it waits. A thread is one of 40 by default, and Module 1’s code also keeps a database connection for the whole 2.2 seconds. The async version holds neither: one thread waits for all the tickets, and it borrows a connection only for the short save at the end. That matters when more tickets are in flight than the pool has connections, which the last lesson of this module measures.

Recap

• CPU work computes; I/O work waits for something outside. A new ticket is about 98% waiting for the AI provider.
• Threads wait in parallel, one thread per request. Async I/O waits for many requests in one thread: at each await, the event loop runs another task.
• A blocking call (a waiting call without await in an async def) stops the whole event loop. Measured: /health 8,737.2 ms in blocking mode, 3.4 ms in async mode; 8 tickets took 18.8 s instead of 5.4 s.
• INTAKE_MODE keeps three designs in one codebase: thread (Module 1), blocking (the mistake) and async (the default from now on).
• The async mode does the AI work without a database connection and saves in one short transaction.
• CPU work needs processes, not async. Your project has 137 passed.

Next: Horizontal scaling