Scaling signals

What you will be able to do

Run more workers when the work piles up and fewer when it is gone, without a person watching. Choose the number that drives this decision, and set the limits that keep it safe.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Timeouts and the circuit breaker, from Failure containment.
• The queue’s numbers (queued, ready, running, the oldest job’s age), from Job lifecycle.
• Per-process limits that add up (the connection pool, the provider quota), from Dependency limits.

What you need on your computer

• The project after Failure containment: python -m pytest prints 172 passed, 1 deselected.
• PostgreSQL and Valkey running. No worker running: the autoscaler starts its own.

Everything in this lesson that scales is a simulation on your computer. scripts/autoscale.py starts and stops real worker processes from the queue’s numbers, as a platform autoscaler does with containers. The course did not run an autoscaler in the cloud. What this lesson says about Azure Container Apps comes from its documentation, not from a run, and it is labelled each time.

Opening more checkouts

A supermarket on a Saturday morning has one checkout open. Then a crowd arrives. A good manager opens more checkouts, but what does she watch to decide? She does not watch how fast the cashier’s hands move: the cashier works at the same speed with 2 or 20 people waiting. She watches the line: how many people wait, and how long the person at the back has waited.

Opening a checkout is not instant. A cashier must walk over and log in: a minute of delay. When the line is gone, she does not close the extra checkouts the same second. Another wave of shoppers could arrive in a moment, and calling the cashier back would cost another minute. So she waits a few minutes first. And she always keeps at least one checkout open, and never more than the shop has.

These are the parts of this lesson: a signal to watch, a delay to start, a wait before you stop, and two limits.

The problem: a fixed number of workers

Black Friday brings about ten times the usual tickets for a few hours. In Workload model, the busiest hour at ten times became 1,010 tickets. With a fixed number of workers, Amira must choose between two bad options:

• Few workers (one, as on a normal day). When a burst arrives, the queue grows, and every customer waits longer for a draft reply.
• Many workers all day (six, sized for the busiest minute). Most of the year, five of them do nothing. Each one still holds database connections and costs money where it runs.

A tiny example, without the quota. One worker does 4 jobs at a time, and a job takes about 2.2 seconds. So one worker does about 1.8 tickets per second. A burst brings 6 tickets per second for 30 seconds: 180 tickets. One worker needs about 180 ÷ 1.8 = 100 seconds for them. The last ticket waits about 70 seconds after it arrives. Four workers do about 7 tickets per second, faster than the burst, so the queue barely grows.

Autoscaling is the answer: a program changes the number of workers (or replicas of the API) from a measured signal, within limits. The question is which signal.

Four signals, and what each one sees

Signal	What it measures	For ticket AI work
CPU	How busy the processors are	A worker waits for the provider; it does not compute. In Module 1, the API used 0.6 to 2.2% CPU while it waited. CPU stays low while the queue grows: a bad signal here
Concurrency (requests in flight)	How many requests a replica serves at once	Good for the API: Azure Container Apps scales on it by default (from the documentation). But the API now answers a new ticket in about 12 ms, so the work is not there
Queue length (the backlog)	Jobs that can run now, plus jobs that run: ready + running	Grows exactly when the workers are too few: a good signal for workers
Queue age	How long the oldest queued job has waited	What customers feel. It reacts late: the age must grow first

CPU is a good signal for work that computes, such as image resizing or a model that runs on the same machine. For work that waits on another service, it tells you nothing. This is the same lesson as I/O versus CPU work: know what your process spends its time on before you choose what to watch.

The backlog signal, with every number

The project’s autoscaler (scripts/autoscale.py) reads the queue every 2 seconds. With the backlog signal, it computes:

desired workers = ceil(backlog ÷ target per worker)

ceil rounds up to the next whole number. The target is 8 jobs per worker: 4 that it runs, and 4 that wait for it. Then it clamps the result between a minimum and a maximum. At 18.1 seconds into the measured run, the queue had 15 ready jobs and 12 running ones:

The autoscaling feedback loop: the queue's backlog sets the desired number of workers, the limits clamp it, and the workers change the queue. One moment of the local simulation.

Text description of the diagram

The loop has six steps. New tickets arrive in the queue. At this moment the queue holds 15 ready and 12 running jobs, a backlog of 27. The autoscaler reads this signal every 2 seconds. It computes the desired workers: 27 divided by 8 per worker is 3.4, rounded up to 4. It clamps 4 between the minimum 1 and the maximum 6, so the answer stays 4. Then it changes the workers. It starts new ones at once; a new worker needs about 0.28 seconds before it can take jobs (its cold start). It stops one only after the wish for fewer workers has lasted 20 seconds (the cooldown). Workers take jobs, and the queue shrinks. That closes the loop: the next reading sees a smaller backlog.

It is called a feedback loop because its own action changes the next reading. More workers empty the queue, the backlog falls, and the autoscaler wants fewer workers again. A loop like this needs two brakes, or it swings up and down:

• A cold start is the time from “start a worker” to “it takes its first job”. Locally, a new worker process needed 0.28 seconds (median). In the cloud, a new replica must also be placed, its image pulled and its connections opened. The course did not measure that time, and it quotes none.
• A cooldown is how long the wish for fewer workers must last before a worker stops. Locally, it is 20 seconds, and the autoscaler stops one worker at a time. Starting is immediate; stopping is slow on purpose. A worker stopped too early may be needed again a moment later, and then you pay its cold start again.

The minimum keeps a worker ready, so the first ticket of the morning does not wait for a cold start. The maximum protects what the workers share. Each worker has its own connection pool of up to 5 connections, so 6 workers can hold 30. Add the API’s pool, and the sum must stay well below PostgreSQL’s limit of 100, as in Dependency limits. The provider quota is shared too, as the next sections show.

Larkfield: the local autoscaler under a burst

The course ran the autoscaler with the minimum at 1, the maximum at 6, 8 jobs per worker and a cooldown of 20 seconds. Ten seconds later, scripts/send_tickets.py sent 6 tickets per second for 30 seconds: 180 tickets. The sender does not wait for answers, like real customers. The API answered every ticket with 202 in about 12 ms (median). The quota was off: the simulated provider behaved as if the account had a much larger quota.

• Backlog signal
• Age signal

A simulation on the course team's computer, quota off: the number of workers over time with each signal. Both go up within seconds and come down one worker per 20-second cooldown.

Text description of the chart

A line chart of the number of workers from 0 to 150 seconds. The tickets arrive from about 10 to 40 seconds. With the backlog signal, the workers go from 1 to 2 at 12 seconds, to 3 at 14 and to 4 at 18 seconds. The last job finished at about 44 seconds. Then the workers step down one at a time: 3 at 44 seconds, 2 at 64 and 1 at 84. With the age signal, the first extra worker starts later, at 18 seconds, and the count rises to 5 at 24 seconds. It steps down from 64 seconds and reaches 1 at 124 seconds.

• Backlog signal
• Age signal

The same two runs: jobs that are ready or running. The backlog signal kept it under 30; the age signal waited for jobs to age, so the queue grew to 51.

Text description of the chart

A line chart of the jobs that are ready or running, from 0 to 60 seconds. With the backlog signal, the count rises from 0 at 10 seconds to a peak of 28 at about 20 seconds. It falls to 0 at about 44 seconds. The oldest job never waited more than 2.3 seconds. With the age signal, the count rises to a peak of 51 at about 22 seconds. The first extra worker started only when the oldest job was older than 5 seconds. The oldest job waited up to 6.4 seconds. It also falls to 0 at about 44 seconds.

What this tells us. Both signals worked, in different ways. The backlog signal reacted to the amount of work: one more worker as soon as 8 more jobs waited. So the queue never held more than 28 jobs, and no job waited more than 2.3 seconds. The age signal reacted to the wait: nothing happened until the oldest job was 5 seconds old. By then more work had piled up, so it needed 5 workers, and jobs waited up to 6.4 seconds. In both runs, every job was done at about 44 seconds. And in both runs, the extra workers stayed for a while after that: that is the price of the cooldown.

The cold start of a worker process was 0.28 seconds (median, slowest 0.31). Compared with jobs of 2.2 seconds, it hardly mattered here. It matters when the cold start is long compared with the work, or when the minimum is 0.

The quota changes the answer

Now the same run with the quota on: 100 chat requests per minute, as the real account had. The quota allows about 50 tickets per minute, 0.83 per second, whatever the number of workers.

• All queued jobs
• Ready + running (the backlog signal)
• Workers × 10

A simulation on the course team's computer, quota on, backlog signal. Up to 116 jobs waited for the quota, but they were not ready, so the signal saw almost nothing and the workers went back down to 1.

Text description of the chart

A line chart from 0 to 150 seconds. The workers are drawn ten times larger, so that 4 workers show as 40. Until 22 seconds, the run looks like the one without the quota. The workers rise to 4, and the ready and running jobs peak at 28. Then the provider refuses calls with 429, and the jobs wait for the quota. The line of all queued jobs climbs to 116 at 40 seconds and falls slowly, to 25 at the end. The line of ready and running jobs drops to between 1 and 8, because a waiting job is not ready. So the backlog signal wants 1 worker. The workers go down one per cooldown: 3 at 44 seconds, 2 at 64 and 1 at 84. The oldest job waited more than two minutes (126.8 seconds) when the run ended.

What this tells us. The autoscaler removed workers while up to 116 jobs waited. Is that a bug? No. The jobs waited for the quota, and a fifth or sixth worker would only get more 429 answers. The backlog signal counts only jobs that can run now, so it did the right thing almost by accident. The customers still waited: the oldest ticket waited more than two minutes for its draft. More workers cannot fix that; a larger quota, or less AI work per ticket, can. Before the fix of Module 4, when a 429 still spent an attempt, the same burst turned 99 of 180 jobs into dead letters.

So choose a signal that measures what more workers can fix. And watch the age separately, as an alarm: it tells you what customers feel, even when scaling cannot help.

What a platform does: Azure Container Apps

This part is from the documentation, not run in this course. Azure Container Apps scales containers with KEDA (Kubernetes Event-driven Autoscaling), an open-source autoscaler. Its rules match the parts of the local simulation:

Part	Local simulation	Azure Container Apps (documentation)
Limits	--min 1 --max 6	minReplicas (default 0) and maxReplicas (default 10, at most 1,000)
HTTP signal	not used	HTTP rule concurrentRequests (default 10), counted every 15 s
Queue signal	backlog from the jobs table	a custom rule with a KEDA scaler, for example postgresql with a query and a targetQueryValue
How often it looks	every 2 s	the polling interval: 30 s (not for HTTP rules)
Desired count	ceil(backlog ÷ 8)	desiredReplicas = ceil(currentMetricValue / targetMetricValue)
Going up	at once, as many as needed	stabilization window 0 s; steps of 1, 4, 8, 16, 32 … up to the maximum
Going down	one worker per 20 s cooldown	after 300 s of “fewer” (the stabilization window); the cooldown period of 300 s applies only from the last replica to 0

A scale rule for the worker could look like this. It is a sketch for reading, written from the KEDA PostgreSQL scaler’s documented settings, and not tested:

# From the documentation, not run in this course.
scale:
  minReplicas: 1
  maxReplicas: 6
  rules:
    - name: ticket-jobs
      custom:
        type: postgresql
        metadata:
          query: "SELECT count(*) FROM jobs WHERE state = 'running' OR (state = 'queued' AND run_after <= now())"
          targetQueryValue: "8"
        auth:
          - secretRef: database-url
            triggerParameter: connection

The query is the autoscaler’s backlog: ready plus running jobs. targetQueryValue is the 8 jobs per worker. Two differences from the local run matter. The platform looks every 30 seconds, not every 2, so it reacts more slowly to a 30-second burst. And with minReplicas: 0, the first job after a quiet period waits for a whole cold start. Measure both before you trust them; the course has no cloud numbers for either.

Sources: Scaling in Azure Container Apps (Microsoft Learn) and the KEDA PostgreSQL scaler (KEDA documentation).

Guided practice: run the local autoscaler

You need four terminals, each in ~/projects/ticket-api with the virtual environment active.

1. Copy the script of this lesson from the snapshot end-of-m05. The sender, scripts/send_tickets.py, is already in your project from Backpressure.

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m05
   cp "$S/scripts/autoscale.py" scripts/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m05"
   Copy-Item "$S\scripts\autoscale.py" scripts\

2. Stop every worker that runs. Then load the data again: python -m scripts.load --reset.

3. In terminal 1, stop the simulated provider if it runs (Ctrl+C). Start it again with the quota off: SIM_QUOTA=off python -m simulator. In PowerShell: $env:SIM_QUOTA = "off"; python -m simulator.

4. In terminal 2, start the API without the per-caller rate limit: RATE_LIMIT_PER_MINUTE=0 fastapi dev. In PowerShell: $env:RATE_LIMIT_PER_MINUTE = "0"; fastapi dev. The measured run had it off too, so that only the workers shape the queue.

5. In terminal 3, start the autoscaler. It runs for 150 seconds and prints one line every 2 seconds:

   python -m scripts.autoscale --min 1 --max 6 --per-worker 8 --cooldown 20 --seconds 150 --log autoscale.csv

6. Within about 10 seconds, in terminal 4, send the burst:

   python -m scripts.send_tickets --rate 6 --seconds 30

7. Watch terminal 3. When it ends, read its last line: the cold start of a worker.

Show the expected result

Step 6 prints Sent 180 tickets at 6 per second in 29.9 s: status codes {202: 180}. In terminal 3, the author’s check (on a busy computer) looked like the measured run:

 time  ready  running  oldest  workers  wanted
   10      0        0     0.0        1       1
   12      8        4     1.3        2       2 +1
   14     13        8     2.0        3       3 +1
   16     13       11     2.0        3       3
   18     15       12     2.3        4       4 +1
   20     11       16     1.7        4       4
   22      8       16     1.2        4       3
   ...
   42      0        0     0.0        3       1 -1
   ...
   62      0        0     0.0        2       1 -1
...
Cold start of a worker (process start until it can take jobs): median 0.32 s, slowest 0.34 s (4 starts)

+1 marks a started worker and -1 a stopped one. wanted drops as soon as the backlog falls, but workers follows only after 20 seconds. Your times and the exact counts will differ a little; the shape is the same. autoscale.csv has one row per check, also the queued column, which the screen does not show.

Your turn: the age signal with the quota on

Change two settings at once on purpose, because the question is about both: the quota on, and the age signal. Start the simulator without SIM_QUOTA=off, and run the autoscaler with --signal age --target-age 5. Predict first: how many workers will it start, and when will it remove them?

Show the answer

In the author’s run, the age signal reached the maximum, 6 workers, at about 26 seconds. It kept all 6 until the end of the run. The oldest queued job kept getting older, because the jobs waited for the quota, so the age was always above 5 seconds. But only 1 to 6 jobs were running at any moment, because the quota allowed no more. Six workers can run 24 jobs at a time, so most of them sat idle, each with its own database connections.

The age signal measures what customers feel, and that is why it is a bad scaling signal here: workers cannot fix a quota. With the backlog signal, the same burst went back to 1 worker. Keep the age for an alarm (“tickets wait more than 2 minutes”). Send it to a person who can raise the quota or turn off part of the AI work.

Common mistake

Scaling on CPU

Symptom: Kwame proposes the usual rule: “Add a worker when CPU is above 70%.” During the burst, the queue grows to dozens of jobs and the oldest waits more and more, but the autoscaler never adds a worker.

Cause: the worker spends its time waiting for the provider, not computing. In Module 1, the API used 0.6 to 2.2% CPU while it waited for the provider. A worker is the same kind of program. CPU stays low however long the queue gets, so a CPU rule never sees the problem.

Diagnose: put the signal next to the symptom. python -m scripts.queue_watch shows ready and oldest s rising. A process monitor shows the workers almost idle. When the queue grows and CPU does not, the work waits on something else.

Fix: scale workers on the backlog (ready + running), with a target per worker from a measurement. Keep CPU rules for work that computes. Keep the age as an alarm.

For an AI service: almost every call to a hosted model is waiting, not computing. A local model that runs on the same machine is the opposite case: there, CPU or GPU use is a sensible signal.

See it end to endCapacity plan from a public traffic traceCase study

The case study computes the queue that a given number of workers would build under real, bursty arrivals. It uses the same backlog arithmetic on a public trace.

Knowledge check

Check your understanding

The queue holds 21 ready and 16 running jobs. The target is 8 jobs per worker, the minimum 1 and the maximum 4. How many workers does the backlog rule want, and how many run?

Show the answer

The backlog is 21 + 16 = 37 jobs. 37 ÷ 8 = 4.6, rounded up to 5 workers. The maximum is 4, so the clamp gives 4 workers. The queue will drain more slowly than the rule wanted. That is the job of the maximum: it protects the database connections and the provider quota that all workers share.

Check your understanding

Explain why the workers went down from 4 to 1 while up to 116 jobs waited, in the run with the quota on. Was that wrong?

Show the answer

The jobs waited for the provider’s quota, so their run_after was in the future and they were not ready. The backlog signal counts only ready and running jobs, 1 to 8 at a time, so it wanted 1 worker. It was not wrong. The quota allows about 0.83 tickets per second whatever the number of workers, and more workers would only get more 429 answers. Customers still waited more than two minutes. To fix that, you need a larger quota or less AI work per ticket, not more workers. Watch the age as an alarm.

Check your understanding

What do a cold start and a cooldown cost, and why is starting fast but stopping slowly a good default?

Show the answer

A cold start is the delay before a new worker can take jobs. It was 0.28 seconds for a local process; a cloud replica needs longer (not measured in this course). A cooldown keeps extra workers after the work is gone. In the local run, the last extra worker stopped about 40 seconds after the queue was empty. Starting fast keeps customers from waiting when a burst begins. Stopping slowly avoids paying a cold start again when the next wave comes a moment later. The cost is some idle workers after each burst.

Recap

• Autoscaling changes the number of workers or replicas from a measured signal, between a minimum and a maximum.
• CPU is a poor signal for AI work that waits on a provider. Concurrency fits the API. The backlog (ready + running jobs) fits workers: desired = ceil(backlog ÷ target per worker). The age is what customers feel, but it reacts late.
• In the local simulation (quota off, 180 tickets in 30 seconds), the backlog signal went to 4 workers. No job waited more than 2.3 seconds. The age signal went to 5, and jobs waited up to 6.4 seconds.
• With the quota on, workers went back to 1 while up to 116 jobs waited for the quota. More workers cannot beat a quota.
• Cold start: 0.28 seconds for a local worker process. Cooldown: remove workers slowly. Azure Container Apps documents a 30-second polling interval and a 300-second scale-down window. No cloud number was measured in this course.

Next, you make sure that stopping a worker, which every scale-down does, loses no work.

Next: Graceful shutdown