Container resource settings

Lesson 11
45 min
Module 4: Choose infrastructure and measure

Lesson 11 of 15 · Deploying AI Models for Real UsersContainer resource settings

Course overview

1. Define the inference contract

2. Choose a serving pattern

3. Build the prediction service

4. Choose infrastructure and measure

5. Release, monitor and recover

6. Case studies

What you will be able to do

Put the escalation service in a container and give it a size: how much CPU, how much memory, and how many workers. Then see what each setting does to the start, to each request and to memory, before a cloud platform enforces the same limits.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

Dockerfiles, the layer cache and .dockerignore, from Build efficiently in Docker and Cloud Essentials for AI Developers. A non-root user, from Runtime safety in the Docker course.
Compose and .env, from Runtime configuration in the Docker course. cpus and mem_limit in the cloud simulation, from Operational configuration in the Docker course.
One worker loads its own copy of the bundle, and the start-up error with a wrong digest, from Load and initialize: One copy for each worker.
The decision record, from CPU, GPU and managed endpoints.

What you need on your computer

The project ~/projects/escalation-service with its .venv active and docs/infrastructure.md, from CPU, GPU and managed endpoints. If it is missing or broken, copy the snapshot end-of-m03, as in Use a snapshot.
Docker Desktop, running, from Install and verify in the Docker course. The first build downloads about 90 MB.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested. Times depend on your computer.

A fuse and a booking sheet

A flat has a fuse. If the flat uses too much power at once, the fuse cuts the power. The whole flat goes dark, not only one lamp. A shared meeting room has a booking sheet. Suppose that you may use it for 30 minutes of each hour. When you have used your 30 minutes, you wait outside until the next hour, even when the room is empty.

A container gets the same two kinds of limit. A memory limit is the fuse: if the processes in the container use more, the operating system stops them. A CPU limit is the booking sheet: the container may use the CPU for only part of each short period, and then it waits.

The problem: same image, half the CPU

Amira runs the same image twice, with one setting changed: --cpus 1 and then --cpus 0.5. Nothing in the code or the model changes. She measures with the benchmark of the next lesson, one request at a time:

CPU limit	Ready after (median of 8 starts)	p50, one request at a time	p95, one request at a time
1	2.0 s	4.1 ms	6.1 ms
0.5	3.5 s	4.1 ms	44.1 ms

Half the CPU, and the start takes almost twice as long. Amira expected that. The surprise is p95, the time that 95% of the requests stay under (the next lesson explains it). Most requests are as fast as before, but one in twenty takes ten times longer, with no other request waiting. Grace’s budget is 50 ms for p95, network included. In this measurement, half a CPU used almost all of it.

To understand why, Amira first needs the container with its settings in files.

Write the Dockerfile

The Dockerfile is the Docker course’s, from Runtime safety, with the names of this project and one new line: COPY bundles ./bundles.

FROM python:3.14.8-slim-trixie

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_ROOT_USER_ACTION=ignore

RUN useradd --create-home --uid 10001 app

WORKDIR /app

COPY requirements-run.txt .
RUN pip install -r requirements-run.txt

COPY bundles ./bundles
COPY pyproject.toml .
COPY escalation ./escalation

USER app

EXPOSE 8000
CMD ["fastapi", "run", "--host", "0.0.0.0", "--port", "8000"]

COPY bundles ./bundles puts the model bundle into the image. People call this baking the bundle into the image. The other way is to keep the bundle in storage, for example a cloud storage account, and download it when the container starts.

Tomás: “Why bake it? A new model then needs a new image.” Yes, and that is the reason. The image escalation-service:1.0.0 then holds the code and the model that the team tested together. Every version of the image is complete. So one way to roll back is to run the old image, with the old code and the old model. In Module 5 the image holds more than one bundle, and Rollback and retirement also rolls back the model alone, with a setting in .env. The image starts without any network call to storage, so the cold start has one step less. The cost: every new model needs a build, and the image grows with the model. For a 45 kB bundle that costs nothing. For a 5 GB language model, teams load the file from storage at start-up. They still check its digest, as in Artifact trust and integrity.

The line comes after pip install and before the code, because of the layer cache from Build efficiently: The layer cache in the Docker course. Packages change least, the code changes most, and the bundle is in between. MODEL_SHA256 is not in the image. It comes from outside when the container starts, so that the image cannot confirm its own bundle.

Add this line to The decision in docs/infrastructure.md:

- The bundle is baked into the image: each image holds the code and the model that were tested together.

Make requirements-run.txt and .dockerignore

requirements-run.txt lists every package of the image with its exact version. Make it in a Linux container, as in Dockerfile essentials in the Docker course, because the image runs Linux:

docker run --rm python:3.14.8-slim-trixie sh -c 'pip install -q --root-user-action=ignore --disable-pip-version-check "fastapi[standard-no-fastapi-cloud-cli]==0.142.2" scikit-learn==1.9.1 pandas==3.0.6 numpy==2.5.3 joblib==1.6.0 && pip freeze' > requirements-run.txt

It took 18 s. wc -l requirements-run.txt gives 60 requirements-run.txt. The five packages pin the versions that the bundle checks; the other 55 come with them, for example SciPy with scikit-learn.

Then .dockerignore, as an allowlist (as in the Docker course). The bundles are the new line:

# Send only what the image needs. Data, tests, .env and .venv stay out.
*
!requirements-run.txt
!pyproject.toml
!escalation/
!bundles/
**/__pycache__/

Build the image

docker build -t escalation-service:1.0.0 .

The output (shortened; a build without the cache):

#4 [internal] load build context
#4 transferring context: 46.61kB done
...
#9 [5/8] RUN pip install -r requirements-run.txt
#9 DONE 19.1s
#10 [6/8] COPY bundles ./bundles
#10 DONE 0.1s
...
#13 naming to docker.io/library/escalation-service:1.0.0 done

The build took 31 s on the course’s computer. Now look at the image and its layers:

docker image ls escalation-service
docker history escalation-service:1.0.0 --format "{{.Size}}\t{{.CreatedBy}}"

The output (shortened):

IMAGE                      ID             DISK USAGE   CONTENT SIZE   EXTRA
escalation-service:1.0.0   1ca084f354f7        787MB          166MB
...
69.6kB  COPY escalation ./escalation # buildkit
12.3kB  COPY pyproject.toml . # buildkit
45.1kB  COPY bundles ./bundles # buildkit
461MB  RUN /bin/sh -c pip install -r requirements-r…
...
110MB  # debian.sh --arch 'arm64' out/ 'trixie' '@1…

The layers of escalation-service:1.0.0 (docker history). The model bundle is too small to see next to the libraries that run it.

Text description of the chart

A column chart of the layers of escalation-service:1.0.0, in MB. The base image (Debian and Python): 160 MB. RUN pip install: 461 MB. COPY bundles, the model bundle: 45.1 kB, too small to show as a column. COPY escalation, the code: 69.6 kB, also too small to show.

The model is 5 kB, and the libraries that run it are 461 MB: SciPy alone is about 149 MB. The image that a platform downloads is the content size, 166 MB, compressed (the Docker course’s ticket-api was 73 MB). A larger image takes longer to download at each cold start in the cloud.

Compose: the size in one file

The size of the container goes into compose.yaml, so that every start uses the same settings:

# The escalation service on this computer, with the resource limits of a
# small cloud container. MODEL_SHA256 comes from .env (copy .env.example).
services:
  escalation:
    image: escalation-service:1.0.0
    build: .
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      MODEL_BUNDLE: bundles/escalation-${MODEL_VERSION:-1.0.0}
      MODEL_SHA256: ${MODEL_SHA256:?Set MODEL_SHA256 in .env}
      SHOW_DOCS: "true"
    command: ["fastapi", "run", "--host", "0.0.0.0", "--port", "8000", "--workers", "${WORKERS:-1}"]
    cpus: ${CPUS:-1.0}
    mem_limit: ${MEMORY:-512m}
    memswap_limit: ${MEMORY:-512m}  # the same value: no swap, as in most cloud containers
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/ready')"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 10s

cpus is the CPU limit: 1.0 is one CPU, 0.5 is half of one.
mem_limit is the memory limit. memswap_limit with the same value means no swap. Swap is disk space that the system uses as extra, slow memory. Docker allows it by default, and most cloud containers have none, so the laptop must fail like the cloud.
--workers sets the number of workers, the server processes that answer requests. Each worker loads its own copy of the bundle, as in Load and initialize.
The health check calls /ready, so Compose knows when the bundle is ready.

${WORKERS:-1} reads a variable from .env, with 1 as the default. ${MODEL_SHA256:?…} has no default: without it, Compose stops with the message after ?. The values go in .env, which Git ignores. .env.example is the template that Git keeps:

# Copy this file to .env and fill in MODEL_SHA256.
# MODEL_VERSION picks the bundle: bundles/escalation-<MODEL_VERSION>.
MODEL_VERSION=1.0.0
# The digest that make_bundle.py printed for that bundle (64 characters).
MODEL_SHA256=
# The container's limits (C17 Module 4).
WORKERS=1
CPUS=1.0
MEMORY=512m

Amira copies it to .env and starts Compose before she adds the digest:

cp .env.example .env
docker compose up -d --wait

The output:

error while interpolating services.escalation.environment.MODEL_SHA256: required variable MODEL_SHA256 is missing a value: Set MODEL_SHA256 in .env

Good: no digest, no start. She pastes her bundle’s digest after MODEL_SHA256= (shasum -a 256 bundles/escalation-1.0.0/SHA256SUMS, as in Module 1; yours is different from the course’s) and starts again:

docker compose up -d --wait
docker stats --no-stream

The output (shortened):

 Container escalation-service-escalation-1 Started
 Container escalation-service-escalation-1 Waiting
 Container escalation-service-escalation-1 Healthy
CONTAINER ID   NAME                              CPU %     MEM USAGE / LIMIT   MEM %   ...
eeeea9b9ac05   escalation-service-escalation-1   0.41%     148.7MiB / 512MiB   29.05%  ...

One worker uses about 149 MiB in the container, of a limit of 512 MiB.

Check parity in the container

In Check parity, you checked the service on your computer. Now the same tickets go to the container, with Linux, the image’s packages and 1 CPU. Each new place can bring its own differences, so run the check again:

python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0

The output (shortened):

case     ticket      expected     served difference
normal   T-116940    0.187598   0.187598    2.8e-17
normal   T-116941    0.064735   0.064735    0.0e+00
normal   T-116942    0.129584   0.129584    1.1e-16
...
missing  T-900001    0.037929   0.037929    9.7e-17
...
14 of 14 cases match within 1e-09.

The differences are up to 1.1e-16, a little larger than on macOS, and far below the tolerance. The exit code is 0. Run this check after every new image and every deployment.

CPU limits and throttling

How does Docker give a container half a CPU? It does not make the CPU slower. It counts time in short periods of 100 ms. With cpus: 0.5, the container may use 50 ms of CPU time in each period. When it has used its 50 ms, the system pauses it until the next period starts. The name of this pause is throttling.

Two timelines of 300 ms, split into three periods of 100 ms. With cpus 1.0, a busy container runs all the time. With cpus 0.5, it runs for 50 ms, is paused for 50 ms, and repeats in every period. Below, one request arrives at 60 ms in the 0.5 CPU container, after the 50 ms of that period are used: it waits 40 ms until the next period starts, then needs 3 ms of work, so it takes about 43 ms instead of 3 ms. — Throttling with cpus: 0.5. In each 100 ms period, a busy container runs for 50 ms and is paused for the rest. A request that arrives during the pause waits for the next period.

This explains the 44.1 ms p95. The benchmark sends the next request as soon as the last answer arrives. So the service is busy most of the time, and it uses its 50 ms early in many periods. Most requests still finish before the pause: p50 stays 4.1 ms. Some arrive during a pause and wait for it: p95 jumps. In the cloud, the size you choose (0.25, 0.5 or 1 vCPU) is a CPU limit of this kind. Throttling needs a busy container, though: a service that answers one request now and then never uses up its 50 ms. The next lesson asks whether Larkfield’s service is ever that busy.

Cold start and CPU

The CPU limit also changes the start. Start-up is CPU work: importing the libraries and loading the bundle, in Load and initialize. Amira started the image eight times with each limit, with the image already on the computer, and timed docker run until /ready answered:

Cold start on the course's laptop, image already on the computer. Half the CPU, about twice the start-up time.

Text description of the chart

A column chart of the median time from docker run to the first answer of /ready, eight starts each. 1 CPU: 2.0 s (from 1.6 to 3.0 s). 0.5 CPU: 3.5 s (from 2.7 to 5.6 s). 0.25 CPU: 6.7 s (from 5.9 to 11.1 s).

Half the CPU, about twice the time: the start is almost all CPU work. The spread (1.6 to 3.0 s at 1 CPU) comes from other programs on the same laptop. In the cloud, the platform must also find a machine and download the 166 MB image. The course team measured it on Azure Container Apps (live). The first request after zero replicas took 26.4 s with 0.5 vCPU and 23.9 s with 1 vCPU. Twice the CPU saved only 2.5 s: in the cloud, most of a cold start is the platform, not the CPU.

Workers and concurrency

Concurrency is the number of requests that are in progress at the same moment. One worker runs Python, and Python in one process uses about one CPU for this work. So one worker answers about one request at a time, and the others wait in a queue.

More workers can answer more requests at once, but only if there are CPUs for them, and each one costs memory. With two workers, docker top shows the processes in the container:

PID     RSS      COMMAND
32317   58944    /usr/local/bin/python3.14 /usr/local/bin/fastapi run --host 0.0.0.0 --port 8000 --workers 2
32351   17412    /usr/local/bin/python3.14 -B -c from multiprocessing.resource_tracker import main;main(6)
32352   198664   /usr/local/bin/python3.14 -B -c from multiprocessing.spawn import spawn_main; ...
32353   197448   /usr/local/bin/python3.14 -B -c from multiprocessing.spawn import spawn_main; ...

RSS is the memory of each process, in KiB (1,024 bytes). A parent process of about 58 MiB watches the workers, and each of the two workers uses about 194 MiB. Each worker has its own Python, its own libraries and its own bundle. Two workers do not share one copy.

Why does docker stats show 149 MiB for the whole container with one worker, when one worker process here has 194 MiB? The two tools count differently. RSS counts every memory page that a process uses, also the pages of library files that the system can share or drop. docker stats counts the container’s memory once, without the file cache that the system can free. The memory limit counts like docker stats, so compare the limit with docker stats. To plan, use about 190 MiB per worker: it is a little more than each extra worker really adds, so it leaves room.

Amira measured more workers than CPUs: a limit of 2 CPUs, 16 requests at once, 2,000 requests.

Workers (2 CPUs)	Requests per second	p50	Memory
2	524	28.0 ms	350 MiB
4	317	49.3 ms	631 MiB
6	262	76.4 ms	914 MiB

Four workers on two CPUs are slower than two, and they use almost twice the memory. The extra workers do not add CPU time: they take turns on the same two CPUs, and the switching costs time. The rule for a CPU-bound service like this one: workers = CPUs, and memory ≥ workers × about 200 MiB, plus room.

Guided practice: size the container

In ~/projects/escalation-service, write the Dockerfile, .dockerignore, compose.yaml and .env.example from this lesson.
Make requirements-run.txt in a container. Check that it has 60 lines.
Add the bundle line to The decision in docs/infrastructure.md.
Build the image with docker build -t escalation-service:1.0.0 ..
Run docker history escalation-service:1.0.0. Find the COPY bundles layer and the pip install layer.
Copy .env.example to .env, and paste your bundle’s digest after MODEL_SHA256=.
Start with docker compose up -d --wait. Expected: Healthy.
Run docker stats --no-stream. Write down the memory.
Call /ready: curl -s http://127.0.0.1:8000/ready. Expected: {"status":"ready","model_version":"1.0.0"}.
Run python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0. Expected: 14 of 14 cases match within 1e-09.

Show the expected output of steps 5 and 8

Step 5: COPY bundles ./bundles is about 45 kB, and RUN pip install about 461 MB on Apple silicon. Sizes on an Intel or AMD computer (amd64) are a little different.

Step 8: about 150 MiB of 512 MiB, for example 148.7MiB / 512MiB. If the memory is about twice that, check WORKERS in .env: it must be 1.

Your turn: two workers on one CPU

In .env, set WORKERS=2 and keep CPUS=1.0. Before you start, write two predictions: the memory in docker stats, and whether two workers answer more requests per second than one. Then run docker compose up -d --wait and docker stats --no-stream.

Show the answer

The memory doubles, about: 337.5MiB / 512MiB, because each worker loads its own libraries and bundle. The speed does not improve: two workers share one CPU, so they take turns. The next lesson measures it: with 8 requests at once, one worker answered 216 requests per second, and two workers on one CPU 209. You pay twice the memory for nothing. Set WORKERS=1 again.

Common mistake

Healthy, then gone: four workers in 512 MiB

Symptom: Tomás wants more speed and sets WORKERS=4, with CPUS=2.0 and MEMORY=512m. docker compose up -d --wait says:

 Container escalation-service-escalation-1 Healthy

and /ready answers 200. Ten seconds later, the service does not answer at all.

Cause: four workers need more than 512 MiB. Each worker loads its libraries and its bundle, and the system’s out-of-memory killer stops processes when the container passes its limit. The log shows the workers dying, while the parent process starts new ones:

INFO:     Waiting for child process [89]
INFO:     Child process [89] died
INFO:     Waiting for child process [90]
INFO:     Child process [90] died
...
INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...) in 2501 ms

The health check passed, because one worker answered /ready while the others died. Then the killer also stopped the main process. docker inspect gives the reason:

docker inspect escalation-service-escalation-1 --format "{{.State.Status}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}"

exited OOMKilled=true exit=137

Exit code 137 means that something killed the process. OOMKilled=true says why: out of memory.

Memory of the container after start-up, with a limit of 512 MiB and no swap: 1 worker 148.7 MiB, 2 workers 340.3 MiB, 3 workers 481.5 MiB, 4 workers more than 512 MiB: OOMKilled, exit code 137. — Memory grows with each worker. Three workers fit in 512 MiB with little room; four do not.

Fix: measure the memory of one worker, multiply by the number of workers, and add room. With a limit of 2 GiB, four workers used 631 MiB (the table in Workers and concurrency), so give them at least 1 GiB. Or use fewer workers: on 2 CPUs, 2 workers are faster anyway. Check OOMKilled whenever a container stops for no clear reason.

For a real AI service: with swap allowed (Docker’s default), the same four workers stayed running at 511.9MiB / 512MiB. The laptop hid the problem, and only the cloud would show it. That is why compose.yaml sets memswap_limit. A large model makes this worse: a 2 GB model in each of four workers is 8 GB. Health checks that pass during start-up do not prove that the service survives its first minute.

Check your understanding

With cpus: 0.5, p50 stays 4.1 ms, but p95 goes from 6.1 to 44.1 ms. Explain why only the slow requests change.

Show the answer

Docker gives the container 50 ms of CPU time in each 100 ms period. A busy service uses its 50 ms early, and the system pauses it until the next period: throttling. Requests that run before the pause take the usual 4 ms, so the median does not change. Requests that arrive during a pause wait for the next period, up to about 50 ms. They are the slowest few percent, so p95 jumps.

Check your understanding

Predict: on a container with 1 CPU and 1 GiB, Tomás sets WORKERS=4. What happens to memory and to requests per second, compared with one worker?

Show the answer

Memory grows a lot. Amira measured 631 MiB for four workers and about 150 MiB for one, because each worker loads its own libraries and bundle. It still fits in 1 GiB, so nothing is killed. Requests per second do not grow. Four workers take turns on one CPU, and the switching costs a little time, so the service can get slower. Workers help only when there are CPUs for them.

Check your understanding

Why does the course bake the bundle into the image, and when would you load the model from storage at start-up instead?

Show the answer

Baked in, the image holds the code and the model that the team tested together. The image is complete, and running an old image brings back both. (Module 5 also rolls back only the model, with a setting.) The start needs no download from storage. Load from storage when the model is large (gigabytes), so that a new model does not need a new image and the image stays small. Then check the downloaded file’s digest at start-up, as load_bundle does.

Recap

The Dockerfile bakes the bundle into the image: each image holds the code and the model that were tested together. The model is 45.1 kB; the libraries are 461 MB.
compose.yaml sets the size: cpus, mem_limit, memswap_limit (no swap) and --workers, with the values in .env.
A CPU limit pauses a busy container in each 100 ms period (throttling). At 0.5 CPU, p95 was 44.1 ms, and the cold start was about twice as long.
Each worker is a process with its own copy of everything. Workers = CPUs; memory ≥ workers × about 200 MiB.
Too many workers for the memory: Healthy, then OOMKilled=true, exit code 137.

Next, you measure the service the same way every time: a fixed workload, p50 and p95, throughput and concurrency.

Next: Benchmark the service