Load and initialize

What you will be able to do

Make the slow work happen once, when the service starts, so that each request does only the fast work. If the model is wrong or missing, the service refuses to start.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The bundle and the six checks of load_bundle(), from Artifact trust and integrity: Six gates, in order.
• settings.py and score_tickets() with its three timed steps (frame, prepare, model), from Batch inference: scoring.py.
• jobs.py, from Asynchronous inference: The job runner.
• An application factory and a request ID, from Configuration and logging: An application factory in Build and Deploy Your First API. Liveness and readiness, from Publish safely: Liveness and readiness in the API course.

What you need on your computer

• The project ~/projects/escalation-service with its .venv, active in a terminal. On Windows: $HOME\projects\escalation-service.
• escalation/contract.py, escalation/bundle.py and bundles/escalation-1.0.0/, from Module 1. escalation/settings.py, escalation/scoring.py and escalation/jobs.py, from Module 2.
• python -m pytest gives 20 passed.
• Three files that main.py imports: escalation/errors.py, escalation/routes.py and escalation/security.py. The next lesson writes them line by line. For this lesson, copy only these three files from the snapshot end-of-m03 into your escalation folder. This lets you start the service now, and learn the start-up before the routes.
• If your project is missing or broken, copy the snapshot end-of-m02. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

A kitchen before it opens

A restaurant kitchen does the slow work before the doors open. The cooks chop the vegetables, make the stocks and heat the ovens. When an order comes, a dish takes minutes, not hours. No cook makes a new stock for each plate.

A prediction service has the same two kinds of work. Some work is slow and the same for every request: import the packages, check the bundle, load the model. Other work is fast and different for each request: check one ticket, score it, answer. This lesson puts each kind of work in its place, in escalation/main.py, the file that starts the service.

The problem: load the model in every request

Tomás has a simple idea. “The model is only 5 kB. Why not call load_bundle() inside the endpoint? Then we need no start-up code.”

Amira measures both ways in one Python process. First, she scores one ticket with a bundle that is already in memory. Then she loads the bundle again before each score, as Tomás suggests. Predict: the bundle is 5 kB. How much slower is each request?

One ticket, median of 50 calls, measured on this lesson's computer. Loading the bundle again in each request takes 3.6 ms, twice the time of the scoring itself.

Text description of the chart

Two bars. “Load once”: the bundle is already in memory, and scoring one ticket takes 1.7 ms. “Load every time”: the request first calls load_bundle() (3.6 ms), then scores the ticket: 5.3 ms in total. Each value is the median of 50 calls. The test ran three times in one Python process on macOS, Apple silicon.

Loading costs more than the scoring. load_bundle() reads five files, computes their SHA-256 hashes, checks the package versions, unpickles the pipeline and scores the 14 parity cases. All of that work gives the same result every time. In a container with 1 CPU, each later load took 3.6 ms too.

For a 5 kB model, 3.6 ms looks small. But this model is the smallest that you will meet. A text model of 500 MB takes seconds to read and check. The help desk would wait seconds for each ticket, for work that gives the same result each time.

Start-up work and request work

The rule is the kitchen’s rule: do the slow, unchanging work once, at start-up; do only the per-ticket work in each request. “Once” means once for each worker, one server process that answers requests (more about workers below). The diagram shows both paths, with the times measured in a container with 1 CPU.

The slow work runs once per worker, before the port opens. Each request uses the bundle that is already in memory. Times measured in the container with 1 CPU, which Module 4 builds.

Text description of the diagram

The left column is the start-up path. The process starts (fastapi run). It imports scikit-learn, pandas and FastAPI, which takes about 1.0 s. It reads the settings MODEL_BUNDLE and MODEL_SHA256. load_bundle() checks the digest, the file hashes and the package versions, and only then calls joblib.load and checks the features. The self-check scores the 14 parity cases. The first load_bundle(), with the self-check, takes 74 ms. Then the service opens its port, and /ready answers 200: about 2 s after docker run. If any check fails, the log says Start-up error, the process exits with code 3, and nothing is served.

The loaded bundle stays in memory. The right column is the path of each request. POST /v1/score brings one ticket as JSON. Then come the API key and contract checks, frame (0.6 to 0.8 ms); prepare (1.2 to 1.5 ms); model (0.05 ms); the answer and one log line. About 3 ms in all. A note says that load_bundle() in each request adds 3.6 ms, for a model of 5 kB.

FastAPI has one place for the start-up work: the lifespan. It is a function that runs once when the server starts, before the first request, and once more when the server stops. The name says what it covers: the whole life of the application.

Write main.py

Make escalation/main.py. The first part is the lifespan:

escalation/main.py

"""The escalation service: Larkfield's C05 model behind an HTTP API.

Start-up does the slow work once: read the settings, check and load the
bundle, score the parity cases. Only then does the service accept requests.
Each request does only the fast work: check the ticket, score it, answer.
"""

import logging
import time
import uuid
from contextlib import asynccontextmanager

from fastapi import FastAPI, Request

from escalation.bundle import BundleError, load_bundle
from escalation.errors import add_error_handlers, error_response
from escalation.jobs import JobRunner
from escalation.routes import router
from escalation.settings import Settings, SettingsError, load_settings

MAX_BODY_BYTES = 2_000_000  # about 5,000 tickets for a job; one ticket is about 300 bytes

logger = logging.getLogger("escalation")


def create_app(settings: Settings | None = None) -> FastAPI:
    settings = settings or load_settings()
    logging.basicConfig(level=settings.log_level, format="%(levelname)s %(name)s: %(message)s")

    @asynccontextmanager
    async def lifespan(app: FastAPI):
        started = time.perf_counter()
        try:
            app.state.bundle = load_bundle(settings.model_bundle, settings.model_sha256)
        except BundleError as error:
            logger.error("Start-up error: %s", error)
            raise
        app.state.jobs = JobRunner(
            lambda: app.state.bundle, settings.max_queued_jobs, settings.extra_ms_per_ticket
        )
        logger.info(
            "model %s loaded (bundle %s...) in %.0f ms",
            app.state.bundle.version,
            app.state.bundle.digest[:12],
            (time.perf_counter() - started) * 1000,
        )
        yield

Everything before yield is start-up work. The lifespan loads the bundle and keeps it in app.state.bundle, where every request can find it. It also makes the JobRunner from Module 2, which scores jobs with the same bundle. Code after yield would run when the server stops; this service has nothing to clean up. If load_bundle() raises a BundleError, the lifespan writes one clear log line and raises the error again. The server then stops, so it never accepts a request with a wrong model.

The second part makes the app. It adds a request ID to every response, rejects bodies that are too large, and adds the two health endpoints:

escalation/main.py (continued)

    app = FastAPI(
        title="Escalation Service",
        version="1.0.0",
        description="Scores new support tickets for the risk of escalation within 72 hours.",
        lifespan=lifespan,
        docs_url="/docs" if settings.show_docs else None,
        redoc_url=None,
        openapi_url="/openapi.json" if settings.show_docs else None,
    )
    app.state.settings = settings

    @app.middleware("http")
    async def request_id_and_limits(request: Request, call_next):
        request.state.request_id = uuid.uuid4().hex[:12]
        started = time.perf_counter()
        length = request.headers.get("content-length")
        if length and int(length) > MAX_BODY_BYTES:
            response = error_response(
                request, 413, "body_too_large", f"The body is larger than {MAX_BODY_BYTES} bytes."
            )
        else:
            response = await call_next(request)
        response.headers["X-Request-ID"] = request.state.request_id
        logger.info(
            "%s %s %d %.1fms id=%s",
            request.method,
            request.url.path,
            response.status_code,
            (time.perf_counter() - started) * 1000,
            request.state.request_id,
        )
        return response

    add_error_handlers(app)
    app.include_router(router)

    @app.get("/health", tags=["health"])
    def health() -> dict:
        return {"status": "ok"}

    @app.get("/ready", tags=["health"])
    def ready(request: Request) -> dict:
        bundle = getattr(request.app.state, "bundle", None)
        if bundle is None:
            return error_response(request, 503, "not_ready", "The model is not loaded.")
        return {"status": "ready", "model_version": bundle.version}

    return app


try:
    app = create_app()
except (SettingsError, BundleError) as error:
    raise SystemExit(f"Start-up error: {error}") from None

The middleware and the request ID work as in the API course’s ticket-api. The last four lines handle an earlier error: a wrong setting, such as a MODEL_SHA256 that is not 64 characters long. Then the process stops at once, with exit code 1.

main.py imports escalation.errors and escalation.routes, and routes.py imports escalation/security.py. These are the three files that you copied from the snapshot end-of-m03 (see Before you start). Without them, fastapi run stops with ModuleNotFoundError: No module named 'escalation.errors'. The next lesson, Connect the API, writes them line by line.

Last, tell the fastapi command where the app is. Make pyproject.toml in the project folder:

pyproject.toml

[project]
name = "escalation-service"
version = "1.0.0"
description = "Scores new support tickets for the risk of escalation within 72 hours."
requires-python = ">=3.12"
dependencies = [
    "fastapi[standard]==0.142.2",
    "scikit-learn==1.9.1",
    "pandas==3.0.6",
    "numpy==2.5.3",
    "joblib==1.6.0",
]

[tool.fastapi]
entrypoint = "escalation.main:app"

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.fastapi] entrypoint names the module and the variable: app in escalation/main.py. Now fastapi run and fastapi dev need no file name.

Start the service

The service needs the bundle’s digest in MODEL_SHA256. Set it from the SHA256SUMS file, then start the service:

macOS and Linux

Terminal, in ~/projects/escalation-service

export MODEL_SHA256=$(shasum -a 256 bundles/escalation-1.0.0/SHA256SUMS | cut -c1-64)
fastapi run

Windows (PowerShell)

PowerShell, in $HOME\projects\escalation-service

$env:MODEL_SHA256 = (Get-FileHash bundles\escalation-1.0.0\SHA256SUMS -Algorithm SHA256).Hash.ToLower()
fastapi run

In a real release, the digest comes from a trusted place outside the bundle, as in Artifact trust and integrity: Provenance. Here you compute it from a bundle that you made yourself.

The output (shortened):

 ⚡️ Starting FastAPI in production mode
 ...
 🌐 Server started at http://0.0.0.0:8000
    Documentation at http://0.0.0.0:8000/docs
 ...
INFO:     Started server process [62654]
INFO:     Waiting for application startup.
INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...) in 881 ms
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Read the lines in order. The banner says “Server started” before anything is loaded: it only means that the command started. Then the lifespan runs, and the service logs the model version, the first 12 characters of the digest and the load time. Only after “Application startup complete” does Uvicorn open the port. Your digest is different, because your metadata.json holds your own training time. Your time is different too. Here it was 881 to 1,416 ms in four runs, because the first load_bundle() also imports scikit-learn.

In a second terminal, call the two health endpoints:

Terminal 2

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/ready

The output:

{"status":"ok"}
{"status":"ready","model_version":"1.0.0"}

Omar asks: “What does the help desk get if it sends a ticket while the model loads?” Amira tried it. She called /health every 50 ms from the start. The first 37 calls got no connection at all, and the 38th got 200. The port stays closed until the lifespan finishes, so no request can reach a half-loaded service.

Cold start and warm-up

The time from “start the process” to “ready for the first request” is the cold start. Here it is mostly the package imports (about 1.0 s), not the model (74 ms). In the container with 1 CPU, docker run to /ready took about 2 s (the median of 8 starts). With a quarter of a CPU it took 6.7 s. In the cloud, the platform must also download the image, and it took about 25 s. Container resource settings measures it.

The first request after a start is often slower than the others. Python and scikit-learn prepare some things only when the code first runs. Running real work before users arrive is a warm-up. In this service, the self-check is the warm-up: the 14 parity cases go through the same pipeline before the port opens. In the container with 1 CPU, the first /v1/score after ready took 4.5 to 5.7 ms. Later requests took a median of 3.1 to 5.5 ms, so the first user waits almost no extra time. The warm-up costs no extra code, because the check must run anyway.

One copy for each worker

A worker is one server process that answers requests. fastapi run --workers 2 starts two. Each worker is a separate process with its own memory, so each runs its own lifespan and loads its own copy of the bundle. Try it:

Terminal

fastapi run --workers 2

The output (shortened):

INFO:     Started parent process [63495]
INFO:     Started server process [63499]
INFO:     Started server process [63498]
INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...) in 799 ms
INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...) in 799 ms

Two “loaded” lines: two copies. This is why the rule says “once per worker”, not “once”. It also sets the memory cost. In the container of Module 4, the service used about 150 MiB with one worker and about 340 MiB with two. Each worker process holds about 190 MiB, mostly the Python packages, not the 5 kB model. Container resource settings shows what happens when the workers do not fit in the memory limit. Stop the service with Ctrl+C.

Ready, or not at all

/health and /ready answer different questions, as in the API course and the Docker course. /health asks “is the process alive?”. A platform restarts a process that stops answering it. /ready asks “can it do its job now?”, and names the model version that answers. A platform sends no traffic to a replica that is not ready.

In this service, the lifespan decides first. If the bundle fails a check, the process exits, so it answers neither endpoint. /ready still checks for the bundle, so it stays correct if a later change loads the model in the background.

Tomás asks: “In the API course’s house-price API, the service started without a model and answered 503. Why exit here?” Both are valid choices, with different costs:

• Exit at start-up (this service): a wrong model can never answer, and the log says why. A platform with revisions keeps the old revision serving until the new one is ready, as in Operational configuration: Three probes, two endpoints in the Docker course. The cost: on a single container with no old revision, nothing answers until you fix it.
• Start and answer 503 (the API course’s house-price API): the process stays up, so you can read /ready and the logs over HTTP. The cost: a process that looks alive but cannot do its job. A platform that checks only /health keeps it running.

For a model service, a wrong model is worse than no answer, because a wrong score looks like a good one. So escalation-service refuses to start.

Guided practice: start, check and stop the service

1. Write escalation/main.py and pyproject.toml as above.
2. Copy escalation/errors.py, escalation/routes.py and escalation/security.py from the snapshot end-of-m03.
3. Set MODEL_SHA256 as in Start the service.
4. Run fastapi run.
5. Check that the log has one line model 1.0.0 loaded (bundle …) in … ms.
6. In a second terminal, run curl http://127.0.0.1:8000/ready. It gives {"status":"ready","model_version":"1.0.0"}.
7. Stop the service with Ctrl+C.
8. Start it with fastapi run --workers 2. Count the “loaded” lines.
9. Stop the service with Ctrl+C.

Show the expected result of step 8

Two lines INFO escalation: model 1.0.0 loaded (bundle …) in … ms, one for each worker. Each worker ran its own lifespan and holds its own copy of the bundle.

Your turn: change one file of the bundle

Copy the bundle to a new folder outside the project, and add one space at the end of its contract.json. Then start the service with that folder. MODEL_SHA256 stays the same.

Terminal

cp -R bundles/escalation-1.0.0 ../edited-bundle
echo " " >> ../edited-bundle/contract.json
MODEL_BUNDLE=../edited-bundle fastapi run

On Windows, copy the folder in File Explorer, add the space in VS Code, and run $env:MODEL_BUNDLE = "..\edited-bundle" before fastapi run. Predict first: does the service start? If not, which of the six checks in load_bundle() stops it, and what is the exit code? Then delete ../edited-bundle.

Show the answer

The service does not start. The log says:

ERROR escalation: Start-up error: ../edited-bundle/contract.json: the file changed after the bundle was made
ERROR:    Application startup failed. Exiting.

The exit code is 3. Check 1 passes: nobody changed SHA256SUMS, so its digest still equals MODEL_SHA256. Check 2 fails: the hash of contract.json is no longer the one in SHA256SUMS. One added space is enough, and joblib.load never ran.

Common mistake

The service stops at start-up with exit code 3

Symptom: after a new release, the service does not start. The log ends with:

INFO:     Waiting for application startup.
ERROR escalation: Start-up error: bundles/escalation-1.0.0: the bundle digest is 8f233029aa4c..., but MODEL_SHA256 expects 382077148b6c...
ERROR:    Traceback (most recent call last):
...
escalation.bundle.BundleError: bundles/escalation-1.0.0: the bundle digest is 8f233029aa4c..., but MODEL_SHA256 expects 382077148b6c...

ERROR:    Application startup failed. Exiting.

echo $? (PowerShell: $LASTEXITCODE) gives 3. Amira ran the same mistake with the container image of Module 4: docker inspect --format '{{.State.ExitCode}}' gave 3 too.

Cause: MODEL_SHA256 is the digest of another bundle. Somebody pasted the digest of escalation-1.1.0 (382077148b6c…), but MODEL_BUNDLE still names version 1.0.0. The first check compares the two digests, and they differ. Uvicorn exits with code 3 when the lifespan fails. A wrong setting stops the process earlier, with code 1.

Fix: set MODEL_BUNDLE and MODEL_SHA256 together, from the same release record. Read the two short digests in the log line: the first is the bundle on disk, the second is the one that you asked for.

For a real AI service: this failure is the protection doing its job. The service refuses a model that nobody approved, before any code in the joblib file runs. The cost is downtime if nothing else serves, so plan releases where the old version answers until the new one is ready. Rollback and retirement practises this.

Check your understanding

Tomás wants to call load_bundle() inside the score endpoint, because the model is only 5 kB. What did the measurement show, and why does the cost grow for other models?

Show the answer

Scoring one ticket with the bundle in memory took 1.7 ms. Loading the bundle again first added 3.6 ms, so the request took 5.3 ms. That is three times the work, for a result that never changes. Loading reads and hashes the files, checks versions, unpickles the pipeline and scores the parity cases. That work grows with the size of the model: a 500 MB model takes seconds, in every request.

Check your understanding

You start the service with four workers. How many times is the bundle loaded, and what does that mean for memory?

Show the answer

Four times. Each worker is a separate process that runs its own lifespan, so the log has four “loaded” lines. Memory grows with the number of workers. In the container of Module 4, one worker used about 150 MiB, two about 340 MiB and four 631 MiB. Four workers did not fit in a limit of 512 MiB. Choose the number of workers with the memory limit in mind.

Check your understanding

Why does escalation-service exit when the bundle check fails, instead of starting and answering 503 as the API course's house-price API did?

Show the answer

A wrong model gives scores that look right, so it is worse than no answer. Exiting makes the failure loud (a log line and exit code 3), and a platform with revisions keeps the old revision serving. Answering 503 keeps the process up, which helps you read the problem over HTTP. But the process looks alive while it cannot do its job, and a platform that checks only /health keeps it.

Recap

• Start-up work is slow and the same every time: imports, settings, load_bundle(), the self-check. It runs once per worker, in the FastAPI lifespan, before the port opens.
• Request work is fast and different each time: check one ticket, score it with the bundle in memory, answer. About 3 ms here.
• Loading in every request cost 3.6 ms more per request for a 5 kB model; for a large model, seconds.
• Cold start is the time to the first ready request. The self-check is also the warm-up.
• Each worker holds its own copy of the bundle, so memory grows with the workers.
• A wrong bundle stops the service with a Start-up error and exit code 3: nothing is served.

Next, you connect the routes: the score endpoint, the model information, the jobs and the API key.

Next: Connect the API