Release the demo

What you will be able to do

Start the tested image as a private demo on your computer. Its secrets stay outside the image. It starts in the right order and says when it is ready. Its logs help without showing what people asked. A smoke test proves the core journey in a few seconds.

Before you startWhat to know and to have readyAbout 90 minutes

What you need to know

• Settings and secrets that come from the environment, not from the image, from Runtime configuration in Docker and Cloud Essentials for AI Developers.
• Published ports and who can connect to them, from Ports and service access: Choose who can connect in the Docker course.
• Liveness and readiness, from Operational configuration: Three probes, two endpoints in the Docker course, and Load and initialize: Ready, or not at all in Deploying AI Models for Real Users.
• Smoke checks, from Verify and remove in the Docker course.
• Request IDs in logs and error bodies, from Configuration and logging in Build and Deploy Your First API.
• Migrations before the new version starts, from Migration safety in CI/CD, Testing and Safe Releases.
• In this course: the practice identity provider and python -m idp token, from Permission enforcement; the build pipeline and the image, from Build pipeline.

What you need on your computer

• The clean copy ~/projects/doc-assistant-ci from Build pipeline: Guided practice, with its virtual environment, .env, .idp/ and the image doc-assistant:1.0.0. If you do not have it, do steps 1, 2 and 9 of that practice first.
• Docker Desktop running, with at least 2 GB of memory free for the containers.
• Ports 8000, 8400 and 5432 free on your computer, or other ports in .env (API_PORT, IDP_PORT, DB_PORT).

Tested on macOS (Apple silicon) with Docker Engine 29.6.1. The Windows and Linux commands are written, not tested by the course team. The cloud steps at the end are written, not run by the course team.

Opening a shop in the morning

Think of a bakery owner who opens her shop at seven. She unlocks the door with the key in her pocket. The key is never taped to the door. She switches on the oven and the till, and she waits until the oven is hot. She sells the first loaf to herself, to check that the till prints a receipt. Only then does she turn the sign on the door from “Closed” to “Open”.

The lights being on does not mean that the shop can sell. If the till is broken, the sign stays on “Closed”. Customers wait outside, instead of queueing inside for nothing.

The problem: started is not the same as working

Sam’s image passed every check. Now it must run as a demo, with its database and its identity provider. Three things can go wrong, and two of them are silent.

First, the secrets. The API needs the database password, the session secret and the client secret. If they are missing, Compose refuses to start anything, before a single container runs:

Terminal

docker compose --profile app config -q

error while interpolating services.api.environment.SESSION_SECRET: required variable SESSION_SECRET is missing a value: Run python -m idp init first

That failure is loud and early, which is good. The message also says what to do.

Second, the order. The API must not start before the database accepts connections, and not before the migrations have run. Otherwise it fails on its first query, or it serves an old schema.

Third, and silent: a process that runs but cannot work. During Sam’s run, the database container was stopped on purpose while the API kept running. The API process was still alive:

Terminal

curl http://127.0.0.1:8000/health

{"status":"ok","version":"1.0.0"}

The process says “ok”, and it is right: it runs. But it cannot answer a single question. A check that only asks “does the process run?” would send Grace to an assistant that fails on every click.

The idea: a release that waits until it is ready

First, two names. A deployment puts one version of the product where its users can reach it, and starts it there. A release is the deployment of a new tagged version. In this course, the deployment is a private demo on your own computer.

The owner’s habits translate into five rules for a release:

• Secrets come from outside. A secret is a value that gives access: a password, a signing key, a session secret. It lives in .env or in the platform’s secret store, never in the image or in Git.
• Start in order, migrations first. The database must be healthy before the migrations run, and the migrations must finish before the API starts.
• Two health questions. Liveness asks “does the process run?” (/health). Readiness asks “can it do its job now?” (/ready: the database answers, and every migration is applied). A platform restarts a process that is not live. It sends no users to a process that is not ready.
• A smoke test after every start. A smoke test is a short set of checks of the core journey, from outside, in a few seconds. The name comes from electronics: switch the new device on, and see whether smoke comes out. It is the owner’s first loaf.
• Private by default. A demo is for the people you show it to, on your screen.

A tiny example: the start order

Sam’s compose.yaml has four services. db is PostgreSQL with pgvector. idp is the practice identity provider. migrate runs the migrations and stops. api is the assistant. The image is the same for the last three. The depends_on conditions and the health checks put them in order:

compose.yaml (parts)

  migrate:
    profiles: ["app"]
    image: doc-assistant:${APP_VERSION:-1.0.0}
    command: ["python", "-m", "doc_assistant.migrate"]
    depends_on:
      db:
        condition: service_healthy

  api:
    profiles: ["app"]
    image: doc-assistant:${APP_VERSION:-1.0.0}
    environment:
      SESSION_SECRET: ${SESSION_SECRET:?Run python -m idp init first}
      # ... the other settings
    ports:
      - "127.0.0.1:${API_PORT:-8000}:8000"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/ready')"]
      interval: 5s
      start_period: 20s
    depends_on:
      migrate:
        condition: service_completed_successfully
      idp:
        condition: service_started

The profile app keeps the three services out of docker compose up -d db, which you used while you developed. ${SESSION_SECRET:?…} makes the missing secret an error with a message, as you saw above. The health check of api calls /ready, not /health, so “healthy” means “ready to serve”.

The start order that Compose enforces: the database first, then the migrations, then the API, and --wait returns only when /ready says yes.

Text description of the diagram

The command starts the database and the practice provider first. The migrations run only when the database is healthy, which means that pg_isready succeeds. The API starts only when the migrations have finished with exit code 0 and the provider has started. The API loads the embedding model and opens its database pool. Its health check calls /ready every 5 seconds. With --wait, the command returns only when that check passes, so a script that runs next can trust the stack.

Without --wait, docker compose up -d returns as soon as the containers exist, before the models are loaded. A smoke test that runs right after it can fail for no real reason. With --wait, it fails only if the stack never gets ready.

Secrets stay outside the image

python -m idp init makes the practice provider’s signing key in .idp/, and it writes two new secrets into .env: OIDC_CLIENT_SECRET and SESSION_SECRET. Compose reads .env and gives each container only the values that its environment names. The image contains none of them. Sam checks it from the image itself, without starting the stack:

Terminal

docker run --rm --entrypoint sh doc-assistant:1.0.0 -c 'whoami; ls -a /app'

app
.
..
data
doc_assistant
evaluation
idp
migrations
recordings
requirements-run.txt
requirements.txt
scripts
traces

The user is app, and /app has no .env and no .idp, because .dockerignore kept them out of the build. The practice provider gets its key folder as a read-only mount, so the key also stays outside the image.

Where the secrets live and who can connect: the image carries no secret, .env and .idp/ enter at start, and only this computer can reach the ports.

Text description of the diagram

The image holds the code, the two models and the data pack. It has no .env and no .idp/. On your computer, .env holds the three secrets. Compose passes them as environment variables to the API and the database when they start. The .idp/ folder holds the provider’s signing key, and it is mounted read-only into the provider’s container. The browser on your computer reaches the API at 127.0.0.1:8000. Another computer on the same network cannot connect, because every port listens on 127.0.0.1 only. The course team did not test this from a second computer. It follows from how Docker publishes a port, as in the Docker course.

A live model’s key, LLM_API_KEY, would follow the same path: into .env, and from there into the API’s environment. The settings class keeps it out of every log and error (repr=False).

Logs that help, without the content

The API writes one JSON line per request. Here are four lines from Sam’s run, from the smoke test:

{"request_id": "fe1d2399e78449ce", "method": "GET", "path": "/v1/tenants/larkfield/documents", "status": 401, "ms": 0.8, "tenant": null, "user": null}
{"request_id": "2cf6f95f2e004f76", "method": "GET", "path": "/v1/tenants/larkfield/documents", "status": 200, "ms": 73.7, "tenant": "larkfield", "user": "usr-sam"}
{"request_id": "0b296ceba2b04e6b", "method": "POST", "path": "/v1/tenants/larkfield/answers", "status": 200, "ms": 76.5, "tenant": "larkfield", "user": "usr-sam"}
{"request_id": "78cfd1d6542341c5", "method": "GET", "path": "/v1/tenants/bramble/documents", "status": 404, "ms": 0.9, "tenant": null, "user": null}

Each line has a request ID, the method, the path, the status, the time in milliseconds, the shop and the user. It never has the question, the answer or a token. Questions can contain a customer’s name or order, and a log is read by more people than the shop’s staff. A test, test_the_request_log_has_ids_but_no_content, sends the question “A secret question?” and fails if the word “secret” or a bearer token reaches the log.

Every error body carries the same request_id. So when Grace sends a screenshot of an error, Sam searches the logs for that ID. The line gives the request, its shop, its user and its time. The question itself is in the database, behind the shop’s access rules, not in the log.

How much memory the demo needs

While the demo ran, Sam measured the memory of each container with docker stats --no-stream:

The memory of each container after the data was loaded and the smoke test ran. The API holds the embedding model and torch, and uses most of it.

Text description of the chart

A column chart with three columns. The api container uses 858 MiB, the idp container 42 MiB and the db container 36 MiB. The api column is about twenty times taller than each of the others.

The API is the heavy part, because it holds torch and the embedding model in memory. In the reranking modes, the cross-encoder adds more. The whole demo needs about 1 GB. So a computer with 8 GB runs it next to a browser and a video call. A cloud plan needs at least 2 GiB for the API, which is the number in docs/deployment.md.

Guided practice: start the demo and smoke-test it

1. In ~/projects/doc-assistant-ci, with the virtual environment active, stop the database from the last lesson, so that the whole stack starts fresh: docker compose down.

2. Check the settings. docker compose --profile app config -q prints nothing when every required secret is set. If it prints required variable … is missing a value, run python -m idp init.

3. Start the whole demo, and wait until it is ready:

   macOS / Linux

   APP_VERSION=1.0.0 docker compose --profile app up -d --wait

   Windows (PowerShell)

   $env:APP_VERSION = "1.0.0"
   docker compose --profile app up -d --wait

   The last lines say that db and api are Healthy. If the image does not exist yet, add --build.

4. See the state of the services: docker compose --profile app ps. You see:

   SERVICE   STATUS                   PORTS
   api       Up 5 seconds (healthy)   127.0.0.1:8000->8000/tcp
   db        Up 8 seconds (healthy)   127.0.0.1:5432->5432/tcp
   idp       Up 8 seconds             127.0.0.1:8400->8400/tcp

   migrate is not in the list: it ran, finished with code 0, and stopped. The ports start with 127.0.0.1.

5. Load the data pack inside a one-off container of the same image:

   docker compose --profile app run --rm api python -m scripts.load_pack

   The end of the output:

   Indexed 31 documents in 8.5 s (failed 0).
     bramble: 15 documents, 60 passages, 0 failed
     larkfield: 36 documents, 185 passages, 0 failed

   “31 documents” is not an error. The worker thread of the running API indexed the other 20 at the same time. The last two lines count every document of each shop: 51 in all, and 0 failed.

6. Ask the two health questions:

   curl http://127.0.0.1:8000/health
   curl http://127.0.0.1:8000/ready

   {"status":"ok","version":"1.0.0"}
   {"status":"ready","version":"1.0.0","answer_provider":"recorded","retrieval_mode":"hybrid","auth_mode":"oidc"}

   On Windows, use curl.exe instead of curl.

7. Run the smoke test with a practice token for Sam:

   macOS / Linux

   SMOKE_TOKEN=$(python -m idp token usr-sam) python scripts/smoke.py http://127.0.0.1:8000

   Windows (PowerShell)

   $env:SMOKE_TOKEN = python -m idp token usr-sam
   python scripts/smoke.py http://127.0.0.1:8000

   PASS  health  (version 1.0.0)
   PASS  ready: database and migrations  (recorded)
   PASS  the page is served
   PASS  no sign-in -> 401
   PASS  documents are indexed  (36 indexed)
   PASS  a question gets a cited answer  (recorded: We refund the extra charge within 3 business days after confirming the double ch)
   PASS  another shop -> 404
   7 of 7 checks passed.

8. Open http://127.0.0.1:8000, choose Sign in, pick Ines, and ask “How long is a gift card valid?”. You get a cited answer from Bramble Books’ documents. This browser sign-in was not run by the course team; the same routes were checked with curl.

9. Read the last log lines: docker compose --profile app logs api --tail 5. Find the line of your question: POST, the path with bramble, status 200, user usr-ines.

Keep the demo running for the next lesson, or stop it with docker compose --profile app down. That command keeps the data.

The seven checks are the core journey, from outside: the process runs, it is ready, the page is served, a request without sign-in gets 401, the shop has indexed documents, a question gets an answer with a citation, and another shop’s path gets 404. The smoke test uses only the standard library, so it runs on a CI runner or on a new computer without any install. It does not replace the evaluation: it asks one question, not 41.

Your turn: the smoke test without a token

Change one condition. Run the smoke test again, but without SMOKE_TOKEN (on Windows, first Remove-Item Env:SMOKE_TOKEN). Predict: how many checks run, and which ones are left out?

Show the answer

PASS  health  (version 1.0.0)
PASS  ready: database and migrations  (recorded)
PASS  the page is served
PASS  no sign-in -> 401
4 of 4 checks passed.

Four checks run: the ones that need no sign-in. The three checks that need a member of Larkfield are left out: the indexed documents, the cited answer and the 404 for another shop. A run without a token is useful on a computer that has no .idp/, but it does not prove the core journey. That is why the CI workflow makes a token first.

Common mistake

Healthy for the platform, useless for the user

Symptom: during Sam’s run, the database container was stopped with docker compose stop db. The API container kept running. /health still said ok, as you saw at the start of this lesson. But /ready gave:

HTTP/1.1 503 Service Unavailable
...
{"code":"not_ready","message":"the database does not answer","request_id":"1e6a14469ba74296"}

Cause: the process was alive, and the database was not. /health does not call the database, on purpose. /ready does.

Diagnosis: compare the two endpoints. Live but not ready means that a dependency is missing. The message names it: the database does not answer. docker compose --profile app ps shows that db is not running.

Recovery: start the database again: docker compose up -d --wait db. Then /ready gives:

{"status":"ready","version":"1.0.0","answer_provider":"recorded","retrieval_mode":"hybrid","auth_mode":"oidc"}

The API reconnected by itself, without a restart.

Consequence in AI work: some teams point the liveness check at /ready. Then a database outage makes the platform restart the API again and again. Each restart loads the models for seconds, and a restart cannot repair a database. Keep liveness on /health and readiness on /ready, as the Docker course did.

Private by default

Sam’s demo is private in four ways. Each one is a choice, written in docs/deployment.md:

• Ports listen on 127.0.0.1. Only this computer can open the demo. To show it, share your screen or record a video.
• The practice identity provider asks for no password. Anyone who can reach it can sign in as Grace. Never publish it.
• SESSION_COOKIE_SECURE=false lets the cookie work over plain http on your own computer. Anywhere else, use https and keep it true.
• Synthetic documents only. The demo shows the course’s made-up shops. Never load a real company’s documents into a demo that others can see, even “just for the meeting”. A citation shows the passage to whoever asks.

Omar asks: “Can we put it online for the client?” Sam’s answer: a private demo on Sam’s screen is the release of this course. A public one needs a real identity provider, https, a managed database and a budget, and somebody who removes it afterwards.

Optional: the cloud (written, not run)

If you have an Azure account and want a public address, docs/deployment.md lists the steps for Azure Container Apps. The course team wrote them from the steps that ran in Registry and deployment and Verify and remove. The team did not run them for this project, and created no cloud resource. In short:

1. A PostgreSQL flexible server with the vector extension allowed, reachable only from the app.
2. The image in a registry, referenced by its digest, as in the CI/CD course.
3. One container app: 1 vCPU and 2 GiB, at least 1 replica (a cold start would load the models), liveness on /health, readiness on /ready.
4. The secrets as platform secrets, not as plain settings.
5. A real identity provider instead of the practice one, and SESSION_COOKIE_SECURE=true.
6. The same smoke test against the app’s address. Then delete the resource group the same day.

Prices change, so check the current price pages before you start. The system design course priced a similar design at about US$50 a month, in Cost model. A one-day demo costs a few dollars, if you delete it. The next lesson shows the removal.

Your project

1. Add the services to your compose.yaml: the migrations as their own service, the API with a health check on its readiness endpoint, and every port on 127.0.0.1.
2. Give your API the two endpoints: /health without dependencies, and /ready that checks the database and the migrations, and any other dependency that it cannot work without.
3. Make every required secret an error with a message (${NAME:?message}), and check that your image contains no secret.
4. Write your smoke test: the core journey of your product in at most ten checks, standard library only, exit code 0 or 1.
5. Start the demo from a clean copy, load your data, and run the smoke test. Save the output.
6. Write your docs/deployment.md: the steps, what is private and why, and what you did not do.

Evidence to keep: the output of up --wait, ps, the two health endpoints and the smoke test from a clean copy; two log lines with request IDs; docs/deployment.md.

Checklist

• docker compose … config -q fails with a clear message when a secret is missing.
• The image has no .env, no keys, and runs as a user that is not root.
• The API starts after the migrations, and --wait waits for readiness.
• /health and /ready answer different questions, and you can explain the difference.
• The smoke test passes from a clean copy, with a sign-in.
• Logs carry request IDs and no question text, answer or token.
• The demo is private, and it uses only synthetic or permitted documents.

Knowledge check

Check your understanding

In the bakery, the owner sells the first loaf to herself before she turns the sign to 'Open'. Which part of Sam's release is that, and which part is 'waiting until the oven is hot'?

Show the answer

The first loaf is the smoke test: a short check of the core journey from outside, after the start. It has seven checks, with one real question and its citation. Waiting until the oven is hot is readiness. docker compose … up --wait returns only when the API’s health check on /ready passes. That means that the database answers and every migration is in place.

Check your understanding

Predict: the database stops while the demo runs. What do /health and /ready return? What should a platform do with the API container, and why?

Show the answer

/health returns 200 with "status":"ok", because the process runs and the endpoint does not call the database. /ready returns 503 not_ready with “the database does not answer”. The platform should send no users to the API until it is ready again, but it should not restart it. A restart cannot repair the database, and it reloads the models every time. When the database is back, /ready returns 200 without a restart.

Check your understanding

Omar wants to load Larkfield's real supplier contracts for a demo to a client 'just this once'. What do you tell him, and what do you show instead?

Show the answer

No. A demo shows passages to whoever asks, and its database, logs and screenshots keep copies. Real contracts need the owner’s permission, real access control and a plan to delete them. Show the synthetic pack instead: it has the same difficulties (two versions of a policy, a contradiction, a table-only answer, staff-only documents, a hidden instruction). If the client needs their own documents, that is a pilot with their permission, not a demo.

Recap

• A secret gives access. It comes from .env or the platform’s secret store at start, and never sits in the image or in Git. Compose refuses to start when a required one is missing.
• Compose starts the services in order: the database healthy, the migrations completed, then the API. --wait returns only when the API is ready.
• Liveness (/health) asks “does the process run?”. Readiness (/ready) asks “can it work now?”. With the database stopped, the first says yes and the second says no.
• A smoke test checks the core journey from outside in a few seconds: 7 of 7 for Sam’s demo.
• Logs carry a request ID, the shop and the user, never the question, the answer or a token.
• The demo is private: ports on 127.0.0.1, the practice provider, synthetic documents. The cloud steps are optional and written, not run.

Next, you make a release fail on purpose, roll it back, limit what the demo can spend, and remove everything.

Next: Recovery and cost control