Recovery and cost control

What you will be able to do

When a release goes wrong, go back to the last good one in seconds. While the demo runs, cap what it can spend. When it ends, remove everything it made. Each of these is a procedure that you test before you need it.

Before you startWhat to know and to have readyAbout 80 minutes

What you need to know

• Rolling back the application, and why data does not roll back, from Rollback strategy in CI/CD, Testing and Safe Releases, and Rollback and retirement in Deploying AI Models for Real Users.
• Expand and contract, so that an old version still works on a new schema, from Migration safety in the CI/CD course.
• Why a budget alert stops nothing, and a cleanup from an inventory, from Cost awareness and Verify and remove in Docker and Cloud Essentials for AI Developers.
• The cost per question and the budget lines C1 and C2, from Threat and cost review and Document trade-offs in this course.
• The release flow of this module, from Build pipeline and Release the demo.

What you need on your computer

• The running demo from Release the demo: Guided practice, in ~/projects/doc-assistant-ci, with the data loaded and the smoke test at 7 of 7.
• The virtual environment active in that folder.

Tested on macOS (Apple silicon) with Docker Engine 29.6.1. The Windows and Linux commands are written, not tested by the course team. The cloud cleanup is written, not run: the course team created no cloud resource.

Two safety nets from everyday life

When you buy a new phone, you keep the old one in a drawer until the new one works: your messages, your bank app, your contacts. If the new phone fails on the first day, you put the card back into the old one. You can call again in a minute. You do not try to repair the new phone while nobody can reach you.

A prepaid phone card works in another way. You put 20 dollars on it. When the 20 dollars are spent, calls stop. A child with a prepaid card cannot run up a bill of 500 dollars, whatever happens. A contract phone with no limit can.

This lesson builds both nets for Sam’s demo: the old phone in the drawer is the rollback, and the prepaid card is the usage limit. Then it removes everything, as you return a rented car empty.

The problem: a release that does not start

The course team ran a release drill on the course-end project. First, the good release 1.0.0 started and passed the smoke test, 7 of 7. Then they released a “1.0.1” with one bad setting: ANSWER_PROVIDER=live, but no LLM_BASE_URL. This is a constructed mistake, but a common one. Somebody switches to a real model and forgets the address. Sam repeats the drill on the clean copy:

Terminal

APP_VERSION=1.0.1 ANSWER_PROVIDER=live docker compose --profile app up -d --wait --wait-timeout 120 api

 Container doc-assistant-idp-1 Healthy
 Container doc-assistant-db-1 Healthy
 Container doc-assistant-migrate-1 Exited
container doc-assistant-api-1 exited (3)

The API container stopped with exit code 3, and --wait failed after 3.2 s. The logs say why:

  File "/app/doc_assistant/providers.py", line 228, in make_provider
    raise ValueError("ANSWER_PROVIDER=live needs LLM_BASE_URL (and LLM_API_KEY).")
ValueError: ANSWER_PROVIDER=live needs LLM_BASE_URL (and LLM_API_KEY).
ERROR:    Application startup failed. Exiting.

The smoke test cannot connect. Every check fails with Connection refused, and the last line is 0 of 7 checks passed.

So the old container is gone, the new one does not start, and nobody can ask a question. Grace’s team is waiting. This is the moment to go back, not to debug.

The idea: roll back to the previous tag

A rollback starts the previous release again: the previous image tag and the settings that went with it. Sam does not build anything, because doc-assistant:1.0.0 is still on the computer. That is why the last lesson said “keep the previous tag”. Then Sam runs the smoke test, because a rollback is a release too:

Terminal

APP_VERSION=1.0.0 docker compose --profile app up -d --wait api
SMOKE_TOKEN=$(python -m idp token usr-sam) python scripts/smoke.py http://127.0.0.1:8000

 Container doc-assistant-db-1 Healthy
 Container doc-assistant-idp-1 Healthy
 Container doc-assistant-api-1 Healthy
PASS  health  (version 1.0.0)
PASS  ready: database and migrations  (recorded)
PASS  the page is served
PASS  no sign-in -> 401
PASS  documents are indexed  (36 indexed)
PASS  a question gets a cited answer  (recorded: We refund the extra charge within 3 business days after confirming the double ch)
PASS  another shop -> 404
7 of 7 checks passed.

The bad setting is gone too. ANSWER_PROVIDER=live was given only on the command line of the bad release, so the rollback runs with the settings in .env again. If you change a setting in .env for a release, you must change it back for the rollback. A release is a tag plus its settings.

Here are the times of the course team’s first drill, on the same kind of laptop with other work running. Sam’s run gave similar times: 3.2 s for the failed start, 7.3 s for the rollback.

• good release
• bad release
• rollback

The release drill: building 1.0.0 is the slow part. Starting, failing and rolling back each take seconds, because no step builds anything.

Text description of the chart

A column chart with five columns, in the order of the drill. Building 1.0.0 takes 107.1 s. Starting it takes 9.7 s, and loading the data inside the container 15.7 s. The bad release 1.0.1 fails after 4.5 s. The rollback to 1.0.0 takes 7.5 s. The build column is about fourteen times as tall as the rollback column.

A rollback that needed a new build would take minutes, and a broken build on that day would leave you with nothing.

What a rollback cannot do

The rollback kept the database: Larkfield’s 36 documents were still indexed. That is safe only under one condition: every migration of the new release is additive. Additive means new tables and new columns that the old code ignores. All five of Sam’s migrations are additive. Migration 005 adds the table usage_days, and version 1.0.0 does not read it.

Imagine a constructed release that renames the column text of the table chunks. Version 1.0.0 would start, and then fail on its first search, because it looks for a column that no longer exists. A rename needs two releases, expand and then contract, as in Migration safety. A rollback also cannot undo data: a document deleted under 1.0.1 stays deleted. That needs a backup and a restore drill, as in Recovery and retention.

The easy case is a release that fails at the start, like this one: readiness never turns green, so nothing new serves. The hard case is a release that starts and answers wrongly. Only the smoke test and the evaluation find it, which is why both run before you call a release done.

The problem: a bill without a ceiling

Sam’s demo replays recorded answers, so it costs nothing. But a live model costs money for every question. Module 5 measured US$0.000167 for one question with chat-small, on the held-out set. That number looks too small to matter.

Now imagine a constructed accident. Tomás writes a script to test the page. A bug makes it ask the same question once per second, and he leaves it running over the weekend:

	Questions	Cost with chat-small
One day	60 × 60 × 24 = 86,400	86,400 × US$0.000167 = US$14.43
A 30-day month	2,592,000	about US$433

The budget alert of the cloud account sends an email, but it stops nothing, as the Docker course showed in Cost awareness. The provider’s bill arrives at the end of the month.

The idea: limits that refuse before the model is called

A usage limit is a ceiling that the application itself enforces. It refuses work before it costs money. Sam’s assistant has two, per shop, from the budget lines that Kwame approved in Module 2:

• DAILY_QUESTION_LIMIT=200: at most 200 questions per shop per day.
• MONTHLY_BUDGET_USD=5: at most US$5 of model cost per shop per month.

The check runs first in the answer pipeline, before the search and before the model. Over a limit, the question gets 429 limit_reached with a Retry-After header, and nothing goes to the model. After each answer, the pipeline adds one question and its cost to the shop’s row for the day, in the table usage_days of migration 005. That table has row-level security like every other shop table. Deleting documents or answers does not reset it, so nobody can reset the limit by cleaning up.

A tiny example. Sam restarts the API with a limit of 3, to see the limit without asking 200 questions. Then Ines asks the same question four times at Bramble Books. Sam’s run asked through the API with a short script; each line below shows the status and the start of the answer:

Terminal

APP_VERSION=1.0.0 DAILY_QUESTION_LIMIT=3 docker compose --profile app up -d --wait api

question 1: 200 A gift card is valid for 3 years from the date of purchase.
question 2: 200 A gift card is valid for 3 years from the date of purchase.
question 3: 200 A gift card is valid for 3 years from the date of purchase.
question 4: 429 Retry-After: 3600
{"code":"limit_reached","message":"This shop has asked 3 questions today, the daily limit. Try again tomorrow.","request_id":"5f780483622447e1"}

Three answers, then a refusal. The message says what happened and what to do. Retry-After: 3600 tells a script to wait an hour. Each shop can see its own usage:

Terminal

curl -H "Authorization: Bearer $(python -m idp token usr-ines)" http://127.0.0.1:8000/v1/tenants/bramble/usage

{"tenant":"bramble","questions_today":3,"daily_question_limit":3,"cost_this_month_usd":0.000325,"monthly_budget_usd":5.0}

The cost is not 0, although no model ran. A recording keeps the token counts of the real call that made it, so the assistant counts its cost as if the model had answered. The budget then behaves the same with recorded and live answers. The limit is per shop: at the same moment, Larkfield’s usage was {"tenant":"larkfield","questions_today":1,"daily_question_limit":3,"cost_this_month_usd":0.000133,"monthly_budget_usd":5.0}, and Sam could still ask. A test, test_the_daily_limit_stops_questions, proves both things on every run of the pipeline.

Which limit stops first?

The two limits protect against different things. A 30-day month is an assumption here, to keep the numbers simple:

• The daily limit allows 200 questions a day. With chat-small, that is 200 × 30 × US$0.000167 = US$1.00 a month, the number in Sam’s threat and cost review.
• The budget of US$5 allows 5 ÷ 0.000167 = 29,940 questions a month, about 998 a day. So with chat-small, the daily limit stops first.
• Now assume a model that costs ten times as much per question (an assumption, not a measured model). The budget then allows about 100 questions a day, and the budget stops first, after about 15 days at 200 a day.

• daily question limit
• monthly budget, chat-small
• monthly budget, model at 10× the price (assumed)

How many questions a day each limit allows. With chat-small the daily limit is the lower ceiling; with a model ten times the price, the budget is.

Text description of the chart

A column chart with three columns, in questions per shop per day. The daily question limit allows 200. The monthly budget of US$5 allows about 998 a day with chat-small, at US$0.000167 per question. With a model at ten times the price, an assumption, the same budget allows about 100 a day.

So you need both. The daily limit stops a runaway script on the first day. The budget stops a slow drift, for example a change to a bigger model that nobody priced. In Tomás’s weekend accident, the daily limit stops his script after 200 questions, about 3 minutes and 20 seconds. The cost is about US$0.03 instead of US$14.43 a day.

When not to use a hard limit: a product whose users must never be refused, such as an emergency line, needs alerts and a person on call instead. For a demo and a small internal tool, a hard limit is right. A refused question costs a little patience. An open bill can cost a lot of money.

Clean up everything

A demo leaves things behind: containers, a network, a database volume, two image tags that share about 3.7 GB of disk, the virtual environment, the practice provider’s keys and the secrets in .env. docs/runbook.md lists them all. Sam’s run removed them in this order:

Terminal

docker compose --profile app down --volumes
docker image rm doc-assistant:1.0.0 doc-assistant:1.0.1
docker image ls doc-assistant

Untagged: doc-assistant:1.0.0
Deleted: sha256:adfa58ddb3595608af6707453198cf31c62b53ef71c4177a379917ce2dc33843
Untagged: doc-assistant:1.0.1
Deleted: sha256:8ee403c67fe84fe4237aac2154f019b915bcb2e3a02e3efd4c3f0aa636ba4cc8
IMAGE   ID             DISK USAGE   CONTENT SIZE   EXTRA

The empty list proves that the image tags are gone. down --volumes also deleted the database volume. Without --volumes, the data stays on the disk: right between two demo days, wrong at the end.

Then, in the project folder: rm -rf .venv .idp traces .env on macOS and Linux, or Remove-Item -Recurse -Force .venv, .idp, traces, .env in PowerShell. The PostgreSQL image (pgvector/pgvector:0.8.7-pg18-trixie) and the two models in ~/.cache/huggingface (about 1 GB) stay, because other projects may use them. Delete them too if nothing else does.

If you ran the optional cloud steps, delete the resource group: az group delete --name <group> --yes. Then az group exists --name <group> must print false. A cleanup is finished only when a command proves that nothing is left.

Guided practice: the drill on your demo

1. In ~/projects/doc-assistant-ci, check that the demo runs: the smoke test says 7 of 7 checks passed.

2. Make a second tag of the same image: APP_VERSION=1.0.1 docker compose --profile app build api (PowerShell: $env:APP_VERSION = "1.0.1", then the command without the prefix). This takes seconds, because every layer is in the cache.

3. Release 1.0.1 with the bad setting:

   macOS / Linux

   APP_VERSION=1.0.1 ANSWER_PROVIDER=live docker compose --profile app up -d --wait --wait-timeout 120 api

   Windows (PowerShell)

   $env:APP_VERSION = "1.0.1"; $env:ANSWER_PROVIDER = "live"
   docker compose --profile app up -d --wait --wait-timeout 120 api

   The command fails, and its last line says that the API container exited with code 3.

4. Run the smoke test. Every check fails with Connection refused, and the last line is 0 of 7 checks passed.

5. Read the cause: docker compose --profile app logs api --tail 6. You see ANSWER_PROVIDER=live needs LLM_BASE_URL (and LLM_API_KEY).

6. Roll back. In PowerShell, first remove the bad setting: Remove-Item Env:ANSWER_PROVIDER, and set $env:APP_VERSION = "1.0.0". Then run APP_VERSION=1.0.0 docker compose --profile app up -d --wait api (PowerShell: without the prefix). Time it with your phone.

7. Run the smoke test again. It says 7 of 7 checks passed., and /health says "version":"1.0.0".

8. Try the limit: restart the API with DAILY_QUESTION_LIMIT=3, ask four questions as Ines on the page, and read /v1/tenants/bramble/usage. Then restart without the limit: APP_VERSION=1.0.0 docker compose --profile app up -d --wait api.

9. Clean up everything, as in the section above. Finish with docker image ls doc-assistant: the list is empty.

Your turn: a bigger daily limit

Change one condition. Grace asks for a daily limit of 1,200 questions per shop, and keeps the budget at US$5 a month with chat-small. Predict: which limit stops a shop that asks 1,200 questions every day, and on which day of a 30-day month?

Show the answer

The budget stops it first. 1,200 questions cost 1,200 × US$0.000167 = US$0.20 a day, so US$5 lasts 5 ÷ 0.2004 = about 24.95 days. On day 25, after about 1,140 of that day’s questions, the budget is used up. From then on, every question gets 429 limit_reached with “This shop has used its model budget for this month (US$5.00).” and Retry-After: 86400. A full month would cost 1,200 × 30 × US$0.000167 = US$6.01. If Grace wants the daily limit to stay the ceiling that people meet, the budget must be a little above US$6.01, for example US$7. Either choice can be right, but it must be Grace’s decision, written in the budget, not a surprise.

Common mistake

A rollback that forgets the settings

Symptom: a constructed case. Sam releases 1.0.1 by changing ANSWER_PROVIDER=live in .env, not on the command line. The start fails. Sam rolls back with APP_VERSION=1.0.0 docker compose --profile app up -d --wait api, and the rollback fails in the same way, with the same ValueError in the log.

Cause: the rollback started the old image with the new settings. The bad value is still in .env, so 1.0.0 reads it too. A release is an image tag plus its settings; Sam went back on only one of the two.

Diagnosis: compare the settings of the failing container with the last good release: docker compose --profile app config shows the values that Compose would use. Do not paste its output anywhere, because it contains the secrets.

Fix: keep the settings of each release with its tag, for example in the release notes or in a file per environment, as in Configuration separation. A rollback restores both.

Consequence in AI work: in an AI product, many releases change only settings: a model name, a prompt version, a retrieval mode. If those settings are not versioned with the release, “roll back” restores the code and keeps the change that broke it.

Your project

1. Write your docs/runbook.md with three parts: roll back, usage limits, clean up. Every step is a command, and every check is a command with its expected output.
2. Run a release drill on your own demo: a good release, a constructed bad one (one wrong setting), the rollback, the smoke test. Record the times and the outputs.
3. List your migrations, and mark each one as additive or not. If one is not, write the two releases that it needs.
4. Add usage limits that fit your product: a limit per tenant per day, and a model budget per tenant per month. Compute both from your own cost per question, as in the chart above, and write which one stops first.
5. Clean up from your list, and finish with a command that proves that nothing is left.

Evidence to keep: docs/runbook.md, the drill’s outputs and times, a 429 response with its message, the usage endpoint’s output, and the cleanup’s final listing.

Checklist

• The rollback restores the previous tag and its settings, and the smoke test passes after it.
• Every migration since the last release is additive, or it is split into expand and contract.
• The limits refuse before the model is called, per tenant, with a clear message and Retry-After.
• You can say which limit stops first, with the numbers.
• The cleanup removes every container, volume, image tag, key and secret that the demo made, and a listing proves it.

Module practice: a reproducible demonstration

You did each step of this module on Sam’s project. Now do them all for your own product.

Your task

Deploy your product if you have the resources, or provide a fully reproducible local demonstration. Either way, a person who has only your repository must be able to repeat it:

1. On a clean copy of your repository, run the checks of your pipeline and build the image with a version tag.
2. Start the demo with --wait, load the data, and run your smoke test.
3. Release a constructed bad version, roll back, and run the smoke test again.
4. Show one refusal by a usage limit.
5. Clean up, and prove it with a listing.
6. Write down what you did not do (for example: “the workflow was not run on GitHub”, “no cloud deployment”), in your docs/deployment.md.

Show a model answer: Sam’s run

Sam ran every step of this module on two fresh copies of the snapshot, in one script. The first copy ran the CI steps. The second copy ran the demo, the limit, the drill and the cleanup. The outputs in the three lessons of this module come from that run.

Step	Result
Lint, held-out lock	both pass in under a second
Tests	108 passed in about 41 s
Load the pack, isolation	51 documents indexed (0 failed); 414 probes, 0 leaks
Evaluation gate (floor 0.75)	correct 0.878, exit 0
Start without secrets	refused: required variable SESSION_SECRET is missing a value
Build and start 1.0.0 (up --build --wait)	63 s, mostly the build (some layers were in the cache)
Load inside the container	51 documents, 0 failed, in about 12 s
Smoke test	7 of 7
Daily limit 3	200, 200, 200, then 429 limit_reached
Bad release 1.0.1	up --wait failed: the API exited with code 3; smoke test cannot connect
Rollback to 1.0.0	7.3 s; smoke test 7 of 7
Cleanup	no image doc-assistant left, no volume left

What makes it reproducible: the snapshot, the pinned versions, the recorded answers (no key), the synthetic data, and the commands in README.md, docs/deployment.md and docs/runbook.md. What it does not show: a run on GitHub, Windows or Linux, and a cloud deployment. Sam writes those three gaps in docs/deployment.md.

Module check

Check your understanding

Retrieval: name the six checks of Sam's test job in order, and say which one protects the held-out set.

Show the answer

Lint (ruff check .), the held-out lock (python -m scripts.holdout check), the tests (python -m pytest), loading the pack (python -m scripts.load_pack), the isolation checks (python -m scripts.isolation, 0 leaks), and the evaluation smoke test (python -m scripts.evaluate --set heldout --min-correct 0.75). The held-out lock protects the held-out set: it compares the SHA-256 of heldout.jsonl with heldout.lock, and fails if a question was moved, added or changed after it was reserved.

Check your understanding

Predict and explain: release 1.1.0 adds migration 006, which renames a column that 1.0.0 reads. 1.1.0 starts, and the smoke test fails. Sam rolls back to 1.0.0. What happens, and what should Sam have done?

Show the answer

The rollback starts 1.0.0 on a database that migration 006 has already changed. A rollback does not undo migrations, so 1.0.0 looks for a column that no longer exists. /ready may still say ready, because every migration that 1.0.0 knows is applied. But the first search or answer fails, and the smoke test fails again. Sam should have split the rename into two releases. First expand: add the new column, fill it, and keep the old one. Later contract: remove the old column once no running version reads it. Then every release is additive for the version before it, and the rollback is safe.

Check your understanding

Completion check: verify the same core journey from a clean environment. Which commands do you run, in which order, and which outputs prove each step?

Show the answer

On a fresh copy of the repository, with a new virtual environment and a new .env:

1. python -m idp init: the keys and two secrets are made.
2. The checks: ruff check . says All checks passed!; python -m scripts.holdout check says the SHA-256 matches the lock; python -m pytest passes (108 for Sam); python -m scripts.isolation says 0 leaks; the evaluation gate says correct 0.878 and exits with 0.
3. APP_VERSION=1.0.0 docker compose --profile app up -d --build --wait: db and api are healthy.
4. docker compose --profile app run --rm api python -m scripts.load_pack: 51 documents, 0 failed.
5. curl http://127.0.0.1:8000/ready: "status":"ready" with the version.
6. SMOKE_TOKEN=$(python -m idp token usr-sam) python scripts/smoke.py http://127.0.0.1:8000: 7 of 7 checks passed. That includes one question answered with a citation, and a 404 for the other shop.
7. Clean up: docker compose --profile app down --volumes, docker image rm doc-assistant:1.0.0, and docker image ls doc-assistant is empty.

The proof is the outputs from the clean copy, not from your working folder.

Recap

Module 6 in short

• A build pipeline runs six checks on a clean computer, fast to slow, and builds one image only if they all pass. The evaluation gate catches answers that get worse while every test passes.
• The image pins every version, carries the models, runs as app and has no secret. You ship the image that passed, with a version tag.
• A release starts in order (database, migrations, API), waits for readiness, and ends with a smoke test of the core journey. Liveness and readiness answer different questions.
• A rollback starts the previous tag with its settings, in seconds, and keeps the data. It is safe only when every migration is additive.
• Usage limits refuse a question before the model is called: 200 a day and US$5 a month per shop. With chat-small, the daily limit stops first.
• A cleanup is finished when a listing proves that nothing is left.

Reset to a known state

1. Stop and remove the demo: docker compose --profile app down --volumes.
2. Remove your image tags: docker image rm doc-assistant:1.0.0 doc-assistant:1.0.1.
3. Copy the snapshot end-of-m06 to a new folder, as in Use a snapshot. Make its .venv and .env, and run python -m idp init.
4. Run docker compose up -d --wait db, then python -m pytest. You should see 108 passed.

Next, Module 7 turns the project into a portfolio piece: the README, the demo narrative and the reflection.

Next: Repository handoff