Graceful shutdown

What you will be able to do

Replace a worker or an API process without losing a ticket’s AI work. Stop on purpose in a way that finishes or hands back the work, and recover when a process dies without warning.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The job’s lease (locked_until), attempts, and why each step’s result is saved with the job’s ID, from Delivery and retries.
• The circuit breaker and how a worker gives a job back without spending an attempt, from Failure containment.
• Why an autoscaler stops workers, from Scaling signals.

What you need on your computer

• The project after Scaling signals: python -m pytest prints 172 passed, 1 deselected.
• PostgreSQL and Valkey running, and the simulated provider (python -m simulator) in normal mode. No worker running.

The kill in this lesson is a constructed case: the course stops a worker with SIGKILL on purpose. The provider is the simulated one.

Closing time, and a power cut

A café closes at six. At ten to six, the staff lock the door: no new guests come in. The guests inside finish their coffee, pay and leave. A guest who ordered a cake at five to six still gets it. At six, the room is empty, every bill is paid, and nobody lost their order.

Now imagine a power cut at five to six. The lights go out, the coffee machine stops, and the staff send everyone out. Some guests paid; some did not; one cake is half made. The next morning, the manager must find out from the order book which orders were finished and which were not.

The first is a graceful shutdown: stop taking new work, finish the work in progress, then stop. The second is an abrupt stop, and the order book is what makes recovery possible.

The problem: every scale-down stops a worker

In the last lesson, the autoscaler stopped workers after each burst. A platform stops processes for many more reasons: a new version is deployed, a machine is replaced, a replica is moved. Azure Container Apps, for example, documents this. When an app scales in, or a revision or the app goes away, the platform sends the signal SIGTERM. If the container has not stopped after 30 seconds, SIGKILL ends it (from the documentation, not run in this course).

A signal is a short message from the operating system to a process. SIGTERM means “please stop”: the process can catch it and clean up. SIGKILL means “stop now”: the process cannot catch it, and it ends at once. Ctrl+C in a terminal sends SIGINT, which most programs treat like SIGTERM.

What did the Module 4 worker do with SIGTERM? The author queued 12 tickets, started that worker, and sent it SIGTERM after 3 seconds. Its process ended at once with exit code 143 (SIGTERM ends a program that does not catch it). The jobs right after:

   state   | count | min | max
-----------+-------+-----+-----
 queued    |     4 |   0 |   0
 running   |     4 |   1 |   1
 succeeded |     4 |   1 |   1

Four jobs were running, but no process worked on them any more. Their lease kept them locked for 30 seconds (JOB_LEASE_SECONDS), so no other worker could take them. The Module 4 worker treated a polite SIGTERM like a power cut. Every scale-down and every deploy would delay up to 4 jobs per worker by half a minute, and spend one of their attempts.

The idea: drain, then stop

Draining means: stop taking new work, give the work in progress a limited time to finish, and hand back what cannot finish. The Module 5 worker does this when it gets SIGTERM or Ctrl+C:

1. It takes no new job, and prints how many jobs are still running.
2. It waits up to WORKER_DRAIN_SECONDS (20) for the running jobs.
3. If a job is still running after that, it gives the job back to the queue (jobs.release), and the attempt does not count. Its saved steps stay saved.
4. It closes its database pool and its provider client, and ends with exit code 0.

The drain time must be shorter than the platform’s grace period, or the platform kills the worker in the middle of its drain. Azure Container Apps waits 30 seconds, so 20 fits. docker stop waits only 10 seconds by default on Linux, so a worker in Docker Compose needs a longer stop_grace_period. The project’s compose.yaml gives the worker 30 seconds. The Compose worker was not run in this course.

What about the abrupt stop, the power cut? A process that gets SIGKILL, crashes or loses its machine cannot drain. For that case, the order book is already in place since Module 4:

• The lease: a job is locked only until locked_until. When the lease ends, another worker may take the job again.
• Saved steps: each step’s result is saved at once with the job’s ID, UNIQUE (job_id, task). The next worker does not call a finished step again.

So graceful shutdown makes the normal case fast, and the lease makes the rare case safe.

The same worker stopped two ways. A graceful stop finishes its running jobs; a kill leaves them locked until their lease ends.

Text description of the diagram

The top row is the graceful stop. A worker runs 4 jobs at a time, with 12 jobs queued. After 3 seconds it gets SIGTERM and takes no new job. Its running jobs finish; it may wait up to 20 seconds, but the process ended 2.2 to 3.4 seconds later. Then 8 jobs had succeeded, and 4 had never started, so they were still queued. A new worker finished them in about 3 seconds, all at attempt 1. The bottom row is the kill, a constructed case. The same worker gets SIGKILL after 3 seconds and ends at once. Four jobs stay running, but nobody works on them. When their lease ends (10 seconds in this demo, 30 by default), another worker may take them. A new worker finished the queue in 12.7 to 13.2 seconds. The 4 jobs ended at attempt 2, with 3 AI results each: no duplicate.

Larkfield: replace a worker while it works

scripts/shutdown_demo.py does both rounds of the diagram. It queues 12 tickets and starts a worker process with 4 jobs at a time. After 3 seconds, it stops the worker. Then it starts a new worker and times how long it needs to finish the queue. The lease is 10 seconds in the demo, not 30, so that the kill round takes less time. One measured pass:

Round 'graceful': stopped after 3 s with SIGTERM; the process ended 3.0 s later (exit code 0)
  worker said: Stopping: no new jobs; 4 running job(s) get up to 20 s to finish.
  worker said: Done in 5.7 s: 8 taken, 8 succeeded, 0 waiting for the quota, 0 to retry later, 0 failed, 0 dead letters, 0 given back to the queue.
  right after: states {'queued': 4, 'succeeded': 8}; attempts per job [0, 1]; AI results per job [0, 3]
  a new worker finished the queue in 3.6 s: states {'succeeded': 12}; attempts per job [1]; AI results per job [3]
Round 'killed': stopped after 3 s with SIGKILL; the process ended 0.0 s later (exit code -9)
  right after: states {'queued': 5, 'running': 4, 'succeeded': 3}; attempts per job [0, 1]; AI results per job [0, 2, 3]
  a new worker finished the queue in 12.7 s: states {'succeeded': 12}; attempts per job [1, 2]; AI results per job [3]

Read it line by line. In the graceful round, the worker said what it was doing, finished its 4 running jobs, and ended with exit code 0. Nothing was left running. The 4 jobs it never started were still queued, at attempt 0, and the new worker did them in about 3.5 seconds. The worker’s last line also counts the jobs that waited for the quota: none here.

In the killed round, exit code -9 means SIGKILL. Four jobs were left running. The new worker could not take them until their lease ended, about 10 seconds after the first worker took them. Then it did them as attempt 2. Every job still has exactly 3 AI results: classify, draft and embed, once each. Right after the kill, one job had 2 AI results (AI results per job [0, 2, 3]): it had saved its classification and its draft before the worker died. The new worker did not do those two steps again, because the first worker saved each step with the job’s ID.

Median of 3 passes: after a graceful stop, a new worker finished the remaining jobs in 3.1 s; after a kill, it needed 12.7 s, because 4 jobs stayed locked until their 10-second lease ended.

Text description of the chart

A column chart with two columns. After the graceful stop (SIGTERM), a new worker finished the remaining jobs in a median of 3.1 seconds (3.1 to 3.6 over 3 passes). After the kill (SIGKILL), it needed a median of 12.7 seconds (12.7 to 13.2). The difference is the lease: 4 jobs stayed locked until it ended. In both rounds, all 12 jobs succeeded, and no job got its AI work twice.

What this tells us. Both stops lost no work, and neither made a duplicate. The difference is time. With a graceful stop, the replacement is almost invisible: the customer’s draft arrives a few seconds late at most. With a kill, up to 4 drafts per worker wait for the lease. With the default lease of 30 seconds, they wait up to half a minute, and each spends an attempt.

The API drains too

The API is a process too, and it is replaced at every deploy. fastapi run uses the Uvicorn server, which already drains on SIGTERM. It stops accepting new connections and waits for the requests in progress. Then it runs the app’s shutdown code, which closes the database pool. The author tested it with the simulated provider in slow mode. A question was sent, and one second later the API got SIGTERM:

INFO:     Shutting down
INFO:     Waiting for connections to close. (CTRL+C to force quit)
INFO ticket_api: POST /v1/answers 200 7539ms queries=4 id=3b174f14eca5
INFO:     127.0.0.1:54613 - "POST /v1/answers HTTP/1.1" 200 OK
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.

The slow question still got its 200 after 7.5 seconds. A new request sent after the signal was refused: the connection failed, so a load balancer would send it to another replica. Uvicorn waits for open requests without a limit by default; uvicorn --timeout-graceful-shutdown 20 sets one. In the queue design, a new ticket takes about 12 ms, so the API drains almost at once. The slow work is in the worker, which is why the worker needs its own drain.

When to use it, and what it costs

Graceful shutdown is always worth having. Platforms stop processes all the time, and a drain turns most stops into nothing that a customer notices. Its costs are small but real:

• A stop takes longer. A deploy or a scale-down waits up to the drain time per process.
• The drain time is a guess. Too short, and long jobs are handed back and done again from their last saved step. Too long, and the platform’s SIGKILL arrives first.
• It cannot help with a crash. Only the lease and the saved steps do that.

The lease has its own trade-off. A short lease makes recovery after a kill fast. But if a job takes longer than its lease, a second worker takes it while the first still works on it. A normal job takes about 2.2 seconds, far below 30. With the 10-second timeout, the slowest possible job is about three calls of 10 seconds: 30 seconds, right at the lease. A job that can run longer than its lease needs a heartbeat: the worker renews locked_until every few seconds while it works. The project has no heartbeat, because its jobs are short.

Guided practice: replace a worker

1. Copy the script of this lesson from the snapshot end-of-m05:

   macOS / Linux

   cp ../nextia-learning-labs/C21/snapshots/end-of-m05/scripts/shutdown_demo.py scripts/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C21\snapshots\end-of-m05\scripts\shutdown_demo.py scripts\

2. Make sure that no worker runs, and that the simulator runs in normal mode. Load the data again: python -m scripts.load --reset.

3. Run the demo. It takes about 30 seconds and starts its own workers. On Windows, Python cannot send SIGTERM or SIGKILL to another process. Both rounds stop the worker at once with terminate(), so the graceful round looks like the kill (written, not tested). On Windows, see the drain with Ctrl+C in the worker’s own terminal, as in Your turn below.

   python -m scripts.shutdown_demo

4. Compare your output with the pass above. Check three things: the graceful round ends with exit code 0; the killed round leaves 4 jobs running; both rounds end with AI results per job [3].

5. Run the module’s tests of this lesson: python -m pytest tests/test_resilience.py -k "graceful or cannot_finish" -v. Then run all tests: python -m pytest. Expect 172 passed, 1 deselected.

Show the expected result

The author’s check, on a busy computer:

Round 'graceful': stopped after 3 s with SIGTERM; the process ended 1.8 s later (exit code 0)
  worker said: Stopping: no new jobs; 4 running job(s) get up to 20 s to finish.
  worker said: Done in 4.4 s: 6 taken, 6 succeeded, 0 waiting for the quota, 0 to retry later, 0 failed, 0 dead letters, 0 given back to the queue.
  right after: states {'queued': 6, 'succeeded': 6}; attempts per job [0, 1]; AI results per job [0, 3]
  a new worker finished the queue in 5.7 s: states {'succeeded': 12}; attempts per job [1]; AI results per job [3]
Round 'killed': stopped after 3 s with SIGKILL; the process ended 0.0 s later (exit code -9)
  right after: states {'queued': 5, 'running': 4, 'succeeded': 3}; attempts per job [0, 1]; AI results per job [0, 2, 3]
  a new worker finished the queue in 12.8 s: states {'succeeded': 12}; attempts per job [1, 2]; AI results per job [3]

On a slower computer, the first worker takes fewer jobs in its 3 seconds, so the counts move: here 6 succeeded instead of 8. Some things do not change. The graceful round leaves nothing running, and the killed round leaves 4 (one per slot). After a kill, the new worker needs about the lease plus one job. Every job ends with 3 AI results. Step 5 runs 2 tests: test_graceful_shutdown_finishes_the_running_jobs and test_jobs_that_cannot_finish_in_time_go_back_to_the_queue.

Your turn: a slow provider and a short drain

Change two settings, because this is the case that a drain cannot finish. Start the simulator with python -m simulator --mode slow: a job now takes about 18 seconds. Load the data again (python -m scripts.load --reset), so that the new jobs are 1 to 4. Send 4 tickets (fastapi dev must run): python -m scripts.send_tickets --rate 4 --seconds 1. Then start a worker with a drain time of 2 seconds, and stop it with Ctrl+C after about 4 seconds:

macOS / Linux

WORKER_DRAIN_SECONDS=2 python -m ticket_api.worker

Windows (PowerShell)

$env:WORKER_DRAIN_SECONDS = "2"; python -m ticket_api.worker

Predict first: what state and what attempt count do the 4 jobs have afterwards? Check with GET /v1/jobs/1 or python -m scripts.queue_watch --seconds 1.

Show the answer

The author’s run:

Stopping: no new jobs; 4 running job(s) get up to 2 s to finish.
... job 1: given back to the queue (the worker is stopping)
... job 2: given back to the queue (the worker is stopping)
... job 4: given back to the queue (the worker is stopping)
... job 3: given back to the queue (the worker is stopping)
Done in 5.7 s: 4 taken, 0 succeeded, 0 waiting for the quota, 0 to retry later, 0 failed, 0 dead letters, 4 given back to the queue.

All 4 jobs were queued again with 0 attempts: none was lost, and none paid an attempt for the stop. The next worker takes them at once, without waiting for a lease. The cost: their slow calls in progress were thrown away, and the provider still finished them. With a real provider, you pay for that work. So the drain time should cover a normal job with room: 20 seconds covers 2.2-second jobs easily, but not these 18-second ones. Start the simulator again in normal mode when you finish.

Common mistake

A worker that ignores SIGTERM

Symptom: after every deploy or scale-down, a few tickets get their draft reply about half a minute late. Their jobs show one extra attempt. python -m scripts.queue_watch shows jobs running for 30 seconds while no worker runs them. In a container platform, the log shows that the container was killed after its grace period.

Cause: the worker does not handle SIGTERM. Perhaps it has no handler, as the Module 4 worker (exit code 143 at once). Or its drain is longer than the platform’s grace period, so SIGKILL arrives during the drain. Its running jobs stay locked until their lease ends.

Diagnose:

1. Stop a worker on purpose with SIGTERM (Ctrl+C, or kill -TERM <pid> on macOS and Linux). A worker that drains prints Stopping: no new jobs … and ends with exit code 0.
2. Look for running jobs whose locked_by names a worker that no longer runs.
3. Compare WORKER_DRAIN_SECONDS with the platform’s grace period: 30 seconds on Azure Container Apps, 10 seconds for docker stop (both from their documentation).

Fix: handle SIGTERM and SIGINT as the Module 5 worker does. Keep the drain shorter than the grace period, or raise the grace period (Compose: stop_grace_period). Keep the lease and the saved steps for the crashes that no handler can catch.

For an AI service: AI jobs are slow and paid per call. A killed job repeats its unfinished calls, and a job given back too early does the same. A good drain and saved steps keep the bill at one answer per step.

Module check: overload stays bounded

The completion check of this module asks you to keep overload bounded and to explain what users observe. You now have every part of it. Remember the overload test of Backpressure. The queue had a bound of 100, and 10 tickets per second arrived for 30 seconds, with 2 workers and the quota on. 172 to 173 tickets were accepted, and 127 to 128 got 503 queue_full with a Retry-After of 88 to 128 seconds. The queue peaked at 107, because the bound check is not atomic. Each limit of the service shows the user something different:

What happens	What the service does	What the user observes
A burst bigger than the workers can handle	The queue grows; the autoscaler adds workers up to its maximum	202 at once; the draft reply comes later
The queue is full (QUEUE_MAX)	Refuses new tickets; the ticket is not saved	503 queue_full with Retry-After: send it again later
One caller sends too much	The rate limit refuses	429 rate_limited with Retry-After
The provider is down	The breaker opens; jobs wait without spending attempts	Drafts arrive late; POST /v1/answers gives 503 ai_unavailable with Retry-After
The provider’s quota is used up	Jobs wait for the quota; more workers cannot help	Drafts arrive late; the oldest job’s age grows
A worker is replaced	It drains, or its jobs wait for the lease	Nothing, or a few drafts up to a lease late

Nothing in this table grows without a limit. The queue has a bound and the workers a maximum. The retries have a budget, and the jobs have attempts and a lease.

Knowledge check

Check your understanding

A worker gets SIGTERM while 4 jobs run. Describe, step by step, what the Module 5 worker does, and what happens to a job that is still running after 20 seconds.

Show the answer

It takes no new job and prints how many jobs are still running. It waits up to WORKER_DRAIN_SECONDS (20) for them to finish. A job that finishes is saved as usual. The worker cancels a job that still runs after 20 seconds, and gives it back with jobs.release. Its state is queued again, its attempt does not count, and its saved steps stay saved. Then the worker closes its pool and its provider client and ends with exit code 0. The next worker takes the given-back job at once, without waiting for a lease.

Check your understanding

After a SIGKILL, a new worker needed 12.7 to 13.2 seconds instead of about 3. Explain the difference, and why no job got its AI work twice.

Show the answer

SIGKILL cannot be caught, so the killed worker left 4 jobs running. Their lease (10 seconds in the demo) kept them locked. The new worker had to wait for it to end, then took them as attempt 2. That wait, plus about one job’s time, gives the 12.7 to 13.2 seconds. No job got its AI work twice because each step’s result is saved at once with the job’s ID, under UNIQUE (job_id, task). The new worker skips the steps that are already saved, so every job ended with exactly 3 AI results.

Check your understanding

Module completion: during Black Friday, 10 tickets per second arrive, the queue is bounded at 100, and the provider has a 20-second outage. Explain what a customer and the help desk observe, and why nothing grows without a limit.

Show the answer

While the queue has room, customers get 202 at once. When it is full, new tickets get 503 queue_full with a Retry-After (88 to 128 seconds in the measured overload). The API does not save those tickets, so the website must say “try again later”. During the outage, the breaker opens: jobs wait without spending attempts, and the help desk sees draft replies arrive late instead of dead letters. Questions on POST /v1/answers get 503 ai_unavailable with Retry-After. The limits: the queue bound (about 100), the workers’ maximum, the retry budget, 5 attempts per job, and the lease. Each one turns more load into a clear refusal or a wait, not into unlimited growth.

Recap

• Platforms stop processes all the time: at every scale-down, deploy and machine change. They send SIGTERM first and SIGKILL after a grace period: 30 seconds on Azure Container Apps, 10 seconds for docker stop (from their documentation).
• A graceful shutdown drains: no new jobs, up to WORKER_DRAIN_SECONDS (20) for the running ones, the rest given back without spending an attempt. The Module 4 worker had none: it died at once and left 4 jobs running.
• Measured with 12 jobs: after SIGTERM, a new worker finished the rest in about 3 seconds. After a SIGKILL (constructed), it needed 12.7 to 13.2 seconds, because of the 10-second lease. All jobs succeeded, each with exactly 3 AI results.
• The API drains on SIGTERM too: a slow request still got its 200, and new connections were refused.
• The lease must be longer than the longest job; a long job needs a heartbeat.
• Overload stays bounded: a queue bound, a maximum of workers, a retry budget, a limit of attempts and a lease. Users see 202, 503 or 429 with Retry-After, or a late draft, never a lost ticket without an answer.

Next, in Module 6, you design load tests that show whether these changes really help, without claims that the test cannot support.

Next: Load-test design