Progressive release

Lesson 12
50 min
Module 4: Release AI and database changes

Lesson 12 of 15 · CI/CD, Testing and Safe ReleasesProgressive release

Course overview

1. Understand the delivery path

2. Create a repeatable CI pipeline

3. Manage environments and secrets

4. Release AI and database changes

5. Recover and maintain the release process

What you will be able to do

The gate measured keywords-1.1 on 33 tickets. Production has thousands, and some are unlike any ticket in the set. This lesson shows the new classifier to a small share of real requests first, and watches that share on its own. Then it gives the classifier to everyone, or takes it back.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

The flag CLASSIFIER_VERSION and the gate, from AI evaluation gate: Worked example: keywords-1.1 behind a flag and The gate in CI.
Version 1.2.0, its migrations, and two versions on one database, from Migration safety: Expand now, contract later and Guided practice: two versions on one database.
Revisions and single revision mode in Azure Container Apps, from Operational configuration: Worked example: apply the file (live) in Docker and Cloud Essentials for AI Developers.
The Deploy workflow and the approval of production, from Promote to staging: The Deploy workflow and Approve the production deployment.

What you need on your computer

Version 1.2.0 merged into main and deployed. Simulated route: in your local production. Live route: in your two container apps. See Migration safety: Guided practice. Both run with CLASSIFIER_VERSION=1.0 in production.
Live route only: the Azure CLI 2.88.0 with the containerapp extension, signed in to the subscription of your apps. A canary adds a second revision for a short time. With minimum 0 replicas and a few hundred requests, it stays inside the monthly free grant, as in the Docker course. You delete the resource group in Module 5.
Simulated route only: Python 3 (the project’s virtual environment is enough) and Docker Desktop.
If your project is missing or broken, copy the snapshot end-of-m04 and promote it as in Migration safety.

The live steps ran with Azure CLI 2.88.0 (containerapp 1.3.0b5) in canadacentral. They changed the app ticket-api-production of Nextia-Intelligence/ticket-api-ci. The simulated steps were run the same day with Docker Desktop 4.81.0 and Python 3.14.6 on macOS (Apple silicon). The Windows and Linux steps are not tested.

Version 1.2.0 went to staging and production through the pipeline, after the approval: Deploy run 37667574234. Amira tagged it v1.2.0, and the Release workflow tagged the same image (run 37667587273). In production, CLASSIFIER_VERSION is still 1.0, so every user still gets keywords-1.0. The new code is there, switched off.

That is the first tool of a progressive release: the feature flag from the first lesson. Turning a flag on is a change of settings, not a new build. The image stays the same digest that CI tested. The help desk lead asks: “Can we turn it on for everyone today?” Amira says: “First for one ticket in ten.”

A canary is a new version that gets a small share of the real traffic first. The current version, the stable version, gets the rest. The name comes from the birds that miners took into mines: a problem shows in the canary before it reaches everyone. The share is the traffic weight: a canary weight of 10 means that each request has a chance of 10% to reach the canary.

Azure Container Apps can do this with revisions. A change of an environment variable makes a new revision (Operational configuration in the Docker course). In multiple revision mode, several revisions are active, and the ingress splits the requests between them by weight.

Amira's loop sends 200 requests with the ticket Package never arrived to the ingress of the container app ticket-api-production, which is in multiple revision mode, with traffic weights 90 and 10. 90% go to revision ticket-api-production--0000004: image 1.2.0, digest sha256:bd2316b5, CLASSIFIER_VERSION=1.0, weight 90. It gave 185 answers from keywords-1.0, category other. 10% go to revision ticket-api-production--kw11: the same image, CLASSIFIER_VERSION=1.1, weight 10. It gave 15 answers from keywords-1.1, category shipping. Log Analytics, requests by revision and status: 0000004, 186 with status 200 and 1 with 401 (a smoke check without a key); kw11, 15 with 200. Each request is chosen on its own: a weight of 10% gave 15 of 200 requests, 7.5%, not exactly 20. — The live canary on 2026-10-07: one image, two revisions that differ only in the flag, and a 90/10 split. The answers and the logs both say which revision answered.

How random is one in ten?

The ingress does not count “nine to stable, one to canary”. It chooses for each request on its own. So 200 requests at 10% give about 20 canary requests. The live run gave 15.

Explore

Canary weight: expected and one random sample of 200 requests

Canary: expected: 20 of 200
Canary: this sample: 17 of 200 (8.5%)
Most samples of 200 (95%): 12 to 29

Weight 10%: expect 20 of 200 requests on the canary. Sample 1 sent 17, 3 fewer than expected. Each request is chosen on its own, so most samples of 200 give 12 to 29. The live canary on 2026-10-07 had the same weight and got 15 of 200 (7.5%), also inside that range.

Sample 1: 200 requests, canary weight 10%
Revision	Expected	This sample	Share
Stable `keywords-1.0`	180	183	91.5%
Canary `keywords-1.1`	20	17	8.5%

Stable revision, keywords-1.0
Canary revision, keywords-1.1 (with a dot)

Move the weight and compare the expected count with one seeded random sample of 200 requests. Each square is one request, in the order it was sent. Another sample draws the next 200.

Text description of the explorer

The explorer has one slider: the canary’s traffic weight, from 0% to 100%. It sends 200 requests and decides each one on its own: a request goes to the canary when its random number is below the weight. The readout shows the expected number of canary requests (200 × weight) and the number in this sample. It also shows the range of 95% of samples of 200. A table shows the expected and the sampled counts for both revisions. A grid shows the 200 requests as squares: blue for the stable revision, orange with a dot for the canary.

At the start (weight 10%), 20 requests are expected. Sample 1 sends 17 to the canary and 183 to the stable revision. Most samples give 12 to 29. The live canary got 15, inside that range. The next samples give 20, 18, 22 and 21. At 0%, no request reaches the canary in any sample; at 100%, every request does. A fixed seed makes the sample the same on every visit. A request that goes to the canary at a low weight stays on the canary at a higher weight.

This matters when you read a canary. With 200 requests at 10%, the canary usually gets 12 to 29 of them. If one of them fails, that is an error rate between 3% and 8%, from a single request. A small canary finds big problems fast. To see small differences, it needs more requests or more time.

Worked example: the live canary

Amira does the canary by hand with the Azure CLI, as the owner of the subscription.

First she looks at the active revisions (live, from the run log):

az containerapp revision list -g rg-ticket-api-ci -n ticket-api-production \
  --query "[?properties.active].{name:name, traffic:properties.trafficWeight, image:properties.template.containers[0].image}" -o table

Name                            Traffic    Image
------------------------------  ---------  -------------------------------------------------------
ticket-api-production--0000003  0          ghcr.io/usmanahmed99/ticket-api-ci@sha256:e5157802...
ticket-api-production--0000004  100        ghcr.io/usmanahmed99/ticket-api-ci@sha256:bd2316b5...

0000004 is 1.2.0 and has all the traffic. She switches the app to multiple revision mode, and makes the canary revision: the same image, with the flag on. --revision-suffix kw11 gives the revision a name that says what it is:

az containerapp revision set-mode -g rg-ticket-api-ci -n ticket-api-production --mode multiple
az containerapp update -g rg-ticket-api-ci -n ticket-api-production \
  --set-env-vars CLASSIFIER_VERSION=1.1 --revision-suffix kw11 \
  --query properties.latestRevisionName -o tsv

ticket-api-production--kw11

Then she checks the traffic, and sees a problem:

Name                            Traffic    Health
------------------------------  ---------  --------
ticket-api-production--0000004  0          Healthy
ticket-api-production--kw11     100        Healthy

The new revision got 100% of the traffic at once. The common mistake below explains why, and the order that avoids it. She sets the weights at once:

az containerapp ingress traffic set -g rg-ticket-api-ci -n ticket-api-production \
  --revision-weight ticket-api-production--0000004=90 ticket-api-production--kw11=10 -o table

RevisionName                    Weight
------------------------------  --------
ticket-api-production--0000004  90
ticket-api-production--kw11     10

Now 200 requests with the same ticket, “Package never arrived”. keywords-1.0 says other and keywords-1.1 says shipping, so the answer shows which revision answered. She counts the values of model_version:

 185 keywords-1.0 other
  15 keywords-1.1 shipping

The answer is one source. The platform’s logs are a second one, independent of the app. Log Analytics counts the requests of the canary period by revision and status (live; the run log kept only the end of the query, summarize requests=count() by RevisionName_s, status):

RevisionName_s                  Requests    Status
------------------------------  ----------  --------
ticket-api-production--kw11     15          200
ticket-api-production--0000004  186         200
ticket-api-production--0000004  1           401
ticket-api-production--0000003  1           401
ticket-api-production--0000003  1           200

The canary answered all of its 15 requests with 200. The 401 lines are smoke checks without a key, which must answer 401. The 0000004 line has one request more than the loop counted, from another check. The two lines of 0000003 belong to the older revision, which had a weight of 0: they are not part of the canary. No revision answered 500.

When to stop a canary

Decide the stop rules before the canary starts, and write them in the release plan. If you decide during the canary, every result looks “probably fine”. For the ticket API, Amira wrote:

Watch	Where	Stop the canary if
Errors	Logs by revision: status `500` and `503`	The canary has any `500`, or more errors than the stable revision
Latency	Logs by revision, or the platform’s metrics	The canary is clearly slower
The model in production	The answers by `model_version`: the share of each category and of priority 1	The canary’s share of priority 1 or of `other` is far from the gate’s results
People	The help desk: tickets moved to another team by hand	The help desk reports wrong categories from the canary

The gate’s metrics need labels, and production tickets have none at first. So in production you watch the signs that you can see at once: errors, and the share of each answer. The help desk’s corrections come later. This course measured only the requests by revision and status. The other rows are what a team adds for a real model.

To stop, give the canary 0% (--revision-weight ticket-api-production--0000004=100 ticket-api-production--kw11=0), then return to single mode. Users are back on keywords-1.0 within seconds, with no new build.

Promote, then back to single mode

The canary was clean, so Amira gave it all the traffic and returned the app to single revision mode (live):

az containerapp ingress traffic set -g rg-ticket-api-ci -n ticket-api-production \
  --revision-weight ticket-api-production--kw11=100 ticket-api-production--0000004=0 -o table
az containerapp revision set-mode -g rg-ticket-api-ci -n ticket-api-production --mode single
az containerapp show -g rg-ticket-api-ci -n ticket-api-production \
  --query "properties.template.containers[0].env[?name=='CLASSIFIER_VERSION']" -o table

Name                Value
------------------  -------
CLASSIFIER_VERSION  1.1

Two things follow from these lines:

Return to single mode after a canary. The Deploy workflow runs az containerapp update --image. In multiple mode, that new revision either gets no traffic or takes all of it at once, depending on the traffic rule. Neither is what the pipeline expects.
The flag stays on. CLASSIFIER_VERSION=1.1 is now part of the app’s template. The next deployment of a new image keeps it. To turn the model off again, change the flag: that is a model rollback, without an application rollback (Module 5).

The simulated route: a canary on your computer

Docker has no traffic weights. You build the same picture with three small parts. The stable version is your local production on port 8082 (flag 1.0). The canary is a container of the same image on port 8084 (flag 1.1). split.py on port 8090 sends each request to one of them, at random, by weight. Clients call only 8090, as clients call only the cloud app’s address.

Check that the local production runs 1.2.0: scripts/smoke.sh http://127.0.0.1:8082 "$(cat secrets/production_api_key.txt)" prints four ok lines.
Start the canary: the same digest, on the same network and database, with the same settings as production except the flag. The command is below the steps.
Smoke-check the canary on its own port: scripts/smoke.sh http://127.0.0.1:8084 "$(cat secrets/production_api_key.txt)".
Save split.py next to your project folder, not in it: for example ~/projects/split.py. The code is below.
In a second terminal, start the splitter with a canary weight of 10. Leave it running.
In the first terminal, send 200 requests to port 8090 and count the answers by model_version.
Read the second terminal: one line per request, with the version that answered.
Stop the splitter with Ctrl+C. It prints the requests by version and status.

The canary container (step 2). Production hides /docs, so the canary does too: a canary must differ in one setting only.

NEW=ghcr.io/usmanahmed99/ticket-api-ci@sha256:bd2316b56067c738199d6203a23845c97fb7b85b550379411eddd7cc6a2ddab8
docker run -d --name ticket-api-canary --platform linux/amd64 \
  --network ticket-api-production_default -p 127.0.0.1:8084:8000 \
  -e API_KEY_FILE=/run/secrets/api_key -e REQUIRE_API_KEY=true -e SHOW_DOCS=false \
  -e CLASSIFIER_VERSION=1.1 \
  -v "$PWD/secrets/production_api_key.txt:/run/secrets/api_key:ro" \
  -e DATABASE_URL="postgresql://tickets:$(grep '^POSTGRES_PASSWORD=' .env | cut -d= -f2)@db:5432/tickets" \
  "$NEW"

Use your own 1.2.0 digest for NEW. The canary uses the production key file, so clients need no new key, and the production database, so its history is in the same table.

Show split.py

"""A small traffic splitter for the simulated canary (C19-M04-L03).

    python3 split.py --stable http://127.0.0.1:8082 --canary http://127.0.0.1:8084 --weight 10

Clients call http://127.0.0.1:8090 only. Each request is sent to the canary
with the chance --weight (in percent), chosen on its own, as in Azure.
Only the Python standard library. For practice on your computer only: no
TLS and no protection. Stop it with Ctrl+C.
"""

import argparse
import random
import signal
import sys
import urllib.error
import urllib.request
from collections import Counter
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

HOP_BY_HOP = {"connection", "keep-alive", "transfer-encoding", "host", "content-length"}


def make_handler(stable: str, canary: str, weight: int, choose, counts: Counter):
    class Split(BaseHTTPRequestHandler):
        def forward(self) -> None:
            target = "canary" if choose() < weight / 100 else "stable"
            base = canary if target == "canary" else stable
            length = int(self.headers.get("Content-Length") or 0)
            body = self.rfile.read(length) if length else None
            headers = {k: v for k, v in self.headers.items() if k.lower() not in HOP_BY_HOP}
            request = urllib.request.Request(base + self.path, body, headers, method=self.command)
            try:
                with urllib.request.urlopen(request, timeout=30) as answer:
                    status, data, answer_headers = answer.status, answer.read(), answer.headers
            except urllib.error.HTTPError as error:
                status, data, answer_headers = error.code, error.read(), error.headers
            except OSError:
                status, data, answer_headers = 502, b'{"error":"upstream not reachable"}', {}
            counts[(target, status)] += 1
            self.send_response(status)
            for k, v in answer_headers.items():
                if k.lower() not in HOP_BY_HOP:
                    self.send_header(k, v)
            self.send_header("X-Served-By", target)
            self.send_header("Content-Length", str(len(data)))
            self.end_headers()
            self.wfile.write(data)
            print(f"{target:<6} {status} {self.command} {self.path}", flush=True)

        do_GET = do_POST = forward

        def log_message(self, *args) -> None:  # one line per request is enough
            pass

    return Split


def stop(*_) -> None:
    raise KeyboardInterrupt


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--stable", required=True, help="address of the current version")
    parser.add_argument("--canary", required=True, help="address of the new version")
    parser.add_argument("--weight", type=int, default=10, help="percent of requests for the canary")
    parser.add_argument("--port", type=int, default=8090)
    parser.add_argument("--seed", type=int, help="repeat the same random choices (for tests)")
    args = parser.parse_args()
    if not 0 <= args.weight <= 100:
        parser.error("--weight must be from 0 to 100")

    choose = random.Random(args.seed).random
    counts: Counter = Counter()
    handler = make_handler(args.stable.rstrip("/"), args.canary.rstrip("/"), args.weight, choose, counts)
    server = ThreadingHTTPServer(("127.0.0.1", args.port), handler)
    print(f"127.0.0.1:{args.port}: {100 - args.weight}% to {args.stable}, "
          f"{args.weight}% to {args.canary}", flush=True)
    # Ctrl+C, or `kill` from a script (a background job ignores Ctrl+C).
    signal.signal(signal.SIGTERM, stop)
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        pass
    print("\nrequests by version and status:", file=sys.stderr)
    for (target, status), n in sorted(counts.items()):
        print(f"  {target:<6} {status}  {n}", file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())

The program reads each request and chooses a target with one random number. It sends the same method, path, headers and body there, and returns the answer with one more header, X-Served-By. It is the job that the ingress did in the cloud, in about 60 lines.

Steps 5 and 6:

python3 ~/projects/split.py --stable http://127.0.0.1:8082 --canary http://127.0.0.1:8084 --weight 10

KEY=$(cat secrets/production_api_key.txt)
for i in $(seq 200); do
  curl -s -X POST http://127.0.0.1:8090/v1/classify -H "X-API-Key: $KEY" \
    -H "Content-Type: application/json" -d '{"subject": "Package never arrived"}'
  echo
done | grep -o '"model_version":"[^"]*"' | sort | uniq -c

The output (simulated, the author’s first run):

 177 "model_version":"keywords-1.0"
  23 "model_version":"keywords-1.1"

The second terminal shows 127.0.0.1:8090: 90% to http://127.0.0.1:8082, 10% to http://127.0.0.1:8084, then one line per request, such as stable 200 POST /v1/classify or canary 200 POST /v1/classify. After Ctrl+C (simulated, a later run):

requests by version and status:
  canary 200  17
  stable 200  183

Five runs at weight 10 gave 23, 27, 20, 24 and 17 canary requests: the same spread as the explorer. Your numbers differ; they should be in the range 12 to 29 most of the time. --seed 7 makes the choices repeat: two runs with it both gave 175 and 25.

To promote on the local route, set CLASSIFIER_VERSION=1.1 in deploy/local/production.env, run scripts/promote-local.sh production "$NEW" again with the same digest, and remove the canary with docker rm -f ticket-api-canary. The author’s promotion took about 10 seconds, and “Package never arrived” then answered shipping with keywords-1.1. In a team, the change of production.env goes through a pull request like any other change. The reference repository kept 1.0 in that file.

Show the expected result of the simulated canary

Steps 1 and 3 each print four ok lines. One request to each port shows the difference: 8082 answers {"category":"other","priority":3,"confidence":0.2,"score":0.2,"model_version":"keywords-1.0"}, and 8084 answers {"category":"shipping","priority":1,"confidence":0.9,"score":0.9,"model_version":"keywords-1.1"}. The count of 200 requests has two lines that add up to 200, with roughly 20 for keywords-1.1. The summary of split.py gives the same two numbers, all with status 200.

Your turn: a canary that breaks

Change one condition: during the canary, stop the canary container, as if the new version crashed. Keep split.py running with weight 10, and send 200 requests again. Count the status codes instead of the versions, with this loop:

for i in $(seq 200); do
  curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:8090/v1/classify -H "X-API-Key: $KEY" \
    -H "Content-Type: application/json" -d '{"subject": "Package never arrived"}'
done | sort | uniq -c

Predict first: how many requests fail, and how would you see which version is broken?

Show the answer

After docker stop ticket-api-canary, the author’s run gave (simulated):

 183 200
  17 502

and the summary of split.py:

requests by version and status:
  canary 502  17
  stable 200  183

About one request in ten failed, and every failure was on the canary. The status codes alone say “something is wrong for 8.5% of the requests”. The count by version says what: the canary. That is why each revision must be watched on its own. With all traffic on one version, the same crash would fail every request. Stop the canary: set its weight to 0 (here, stop split.py and call 8082 directly). Then start the canary container again with docker start ticket-api-canary, or remove it.

All traffic at once

Common mistake

The canary revision takes 100% of the traffic

Symptom (live): right after az containerapp update … --revision-suffix kw11, the revision list said ticket-api-production--kw11 100 and ticket-api-production--0000004 0. For a short time, every user got keywords-1.1: the step that the canary was meant to avoid.

Cause: the app’s traffic rule followed the latest revision. In single mode, that is what you want: a new revision takes the traffic when it is ready, as in the Docker course. Switching to multiple mode does not change the rule. So the new revision took all the traffic, as every new revision had done before.

Fix: pin the traffic to the current revision by name, before you make the canary. Then the latest revision starts with 0%:

az containerapp revision set-mode -g rg-ticket-api-ci -n ticket-api-production --mode multiple
az containerapp ingress traffic set -g rg-ticket-api-ci -n ticket-api-production \
  --revision-weight ticket-api-production--0000004=100
az containerapp update -g rg-ticket-api-ci -n ticket-api-production \
  --set-env-vars CLASSIFIER_VERSION=1.1 --revision-suffix kw11
az containerapp revision list -g rg-ticket-api-ci -n ticket-api-production \
  --query "[?properties.active].{name:name, traffic:properties.trafficWeight}" -o table
az containerapp ingress traffic set -g rg-ticket-api-ci -n ticket-api-production \
  --revision-weight ticket-api-production--0000004=90 ticket-api-production--kw11=10

Check the list after the update: the canary must show 0 before you give it 10. The test run did these steps in the other order, so this order comes from the cause, not from a second run.

For a real AI service: a model can be good on the evaluation set and still fail on inputs that the set does not have. The first minutes of a release are when you know least. A canary that silently becomes 100% gives every user the riskiest version, with nothing left to compare it with. Check the weights after every step, and keep the check in the release plan.

Check your understanding

Why could Amira run the canary with the same image digest for both revisions, and why is that better than a canary of a new image?

Show the answer

Version 1.2.0 already contained keywords-1.1, switched off by the flag CLASSIFIER_VERSION. A revision with the flag on is a change of settings, so both revisions run the exact image that CI tested and the gate checked. The only difference between them is the model. If the canary shows a problem, the cause can only be keywords-1.1. To stop, change the weights or the flag, with no build.

Check your understanding

After the promotion, Amira forgets set-mode single. The next normal release runs the Deploy workflow with a new digest. What can happen?

Show the answer

In multiple mode, az containerapp update --image makes a new revision next to kw11. Depending on the traffic rule, it gets no traffic, or it takes all the traffic at once. With no traffic, the release seems to succeed, but users keep the old version. With all of it, there is no canary. The pipeline’s wait and smoke check do not see which one happened, because they check the latest revision. Return to single mode after every canary, and check the traffic of the active revisions.

Module practice

Guided practice → independent task. You saw Amira release two changes at once: a new API field (score, with migrations) and a new model version (keywords-1.1, behind a flag). Now plan a release yourself, before you write the code: one that changes an API field and a model version.

Choose your own change, or use this one. The API gets a new field queue: the team that should take the ticket, for example billing-team. The classifier gets a new version, keywords-1.2, with a rule that tells “lost parcel” from “lost password”. The plan is one file, docs/releases/<version>.md, and it answers these questions:

Changes for clients: what is new, what is deprecated, and until which version?
The model: what changes, how it is switched on, and the gate’s result (accuracy, recall, must-pass) against the baseline. Which must-pass ticket is new?
The database: which migrations, in which order, and which version works at each step? Is a backup needed?
The rollout: the order of the steps, the canary weight, how long, and the stop rules, written before the start.
Rollback: for each part (application, model, data), what you roll back and how.
Owner and watch: who owns the release, what is watched after it, and for how long.

Show a model answer: the plan of 1.2.0

This is the plan that Amira followed, from the reference repository’s docs/releases/1.2.0.md, with the rollout and the rollback steps added:

## ticket-api 1.2.0

- **New for clients:** every classification has `score`, from 0 to 1. `confidence`
  has the same value and is **deprecated**: it stays until version 2.0, so clients
  have time to change. History items have `score` too.
- **New, off by default:** classifier `keywords-1.1` (`CLASSIFIER_VERSION=1.1`).
  On the 33 holdout tickets: accuracy 0.909 (keywords-1.0: 0.636), every must-pass
  ticket right. We turn it on in production step by step.
- **Database:** migrations replace the table that the app made at start-up. Run
  `python -m ticket_api.migrate` before the new version starts (Compose does it in
  the `migrate` service). Migration `002_add_score` adds a column; version 1.1.0
  keeps working with it.
- **Owner:** Amira. **Watch after release:** the smoke checks, the error rate in
  the logs, and `GET /v1/history`, for one day.
- **Rollback:** deploy the 1.1.0 digest. The database needs no change: 1.1.0
  ignores the new column.

### Rollout (in this order)
1. PR #3: keywords-1.1 behind the flag, and the gate in CI (required check renamed).
2. PR #5: migrations 001 and 002, `score` next to `confidence`, database tests.
3. Deploy 1.2.0 with the flag off: staging, then production after approval.
   Migrations run before the app. Check old and new on one database.
4. Canary in production: pin 0000004=100, make revision kw11 (flag 1.1),
   then 90/10. Stop if kw11 has any 500, or more errors than 0000004.
5. Promote: kw11=100, then back to single revision mode.

### Rollback, by part
- Model only: CLASSIFIER_VERSION=1.0 (no new build; during the canary: kw11=0).
- Application: deploy the 1.1.0 digest. Users lose `score` and keywords-1.1.
- Data: none needed for 002. A backfill (003) only after no 1.1.0 runs.
- Contract (remove `confidence`): not in 1.2.x. In 2.0, after clients moved.

A reviewer marks a plan demonstrated when four things are true. Each step names the version that must still work at that step. The database changes before the code that needs them, and nothing is removed in the same release. The model has a gate result against the baseline and a way to turn it off without a build. The canary has a weight and stop rules written before the start. A plan with only “deploy, then roll back if needed” is not yet demonstrated: it does not say what a rollback takes away.

Module check

Check your understanding

Retrieval: name the three parts of the evaluation gate, and the four steps of expand and contract for the score field.

Show the answer

The gate:

Floors: accuracy at least 0.6, every recall at least 0.5.
No regression against the baseline in production: accuracy drop at most 0.02, recall drop at most 0.1 per category.
Must-pass tickets right, with the right priority: E019, E026, E033.

Expand and contract:

Expand: migration 002 adds score, which may be empty.
Move: 1.2.x writes both, the API marks confidence as deprecated, and clients change.
Backfill: 003 fills score from confidence, after no 1.1.0 runs.
Contract: 2.0 removes confidence.

Check your understanding

Predict: Amira runs a second canary at a weight of 5% and sends 100 requests. The canary gets 3 requests, and one of them answers 500. The stable revision gets 97 with no error. What does she conclude, and what does she do?

Show the answer

One error in 3 requests is a large error rate, but from very few requests: it could be one bad request or a real fault. Her stop rule says “any 500 on the canary”, so she stops it first. With weight 0, users are back on the stable version within seconds, with no build. Then she reads the canary’s log for that request ID to find the cause. At 5% of 100 requests, 5 are expected, and 95% of samples give 1 to 10. The explorer shows the same kind of spread for 200. So a small canary needs stop rules that act on single errors, or more requests before a decision.

Check your understanding

Completion check: demonstrate compatibility while old and new application versions coexist. What evidence do you show?

Show the answer

Show the evidence from the guided practice of Migration safety:

the migrate output, before the new version started;
versions 1.2.0 and 1.1.0 running on the same database at the same time (docker ps with both containers);
one classification from each, both 200: the new one with score, the old one without it;
the table, with a row from each version (the 1.1.0 row has an empty score).

From this lesson, add the canary: one image with two flag values on one database, both answering. Use the model_version counts, or the summary of split.py, or the live 185/15 and the logs by revision. Say which parts you did not check: here, the history endpoint of each version. Every result is labelled live or simulated. A reviewer marks it demonstrated only with outputs from both versions at the same time, not one after the other.

Recap

A feature flag (CLASSIFIER_VERSION) puts new model code into production switched off. Turning it on is a settings change on the same image.
A canary gives the new version a small share of real traffic. In Azure Container Apps: multiple revision mode, a second revision with the flag on, and traffic weights. Locally: a canary container and split.py.
Each request is chosen on its own: 10% of 200 gave 15 live, and 17 to 27 locally. Expect a range, not an exact number.
Pin the traffic to the current revision by name before you make the canary. Otherwise the new revision can take 100% at once, as it did on 2026-10-07.
Watch each version on its own (answers by model_version, logs by revision and status), with stop rules written before the start. Promote, then return to single revision mode.
The module: the gate blocked a worse model (PR #4). Expand and contract kept 1.1.0 and 1.2.0 working on one database. The canary showed keywords-1.1 to one request in ten first.

Reset. Simulated route: stop split.py (Ctrl+C), remove the canary (docker rm -f ticket-api-canary) and any ticket-api-old container, and keep or stop the local environments (scripts/stop-local.sh production; add --volumes to delete their databases too). Set CLASSIFIER_VERSION in deploy/local/production.env back to the value in Git (git diff shows it). Live route: az containerapp revision list … -o table must show one active revision with traffic 100, and az containerapp show … --query properties.configuration.activeRevisionsMode must say Single. Keep the apps for Module 5, or delete the resource group if you stop here. To reset the files, copy the snapshot end-of-m04: it contains the 1.2.0 code as released, with the bug that Module 5 finds.

Next: Rollback strategy

Progressive release

A flag and a small share

How random is one in ten?

Worked example: the live canary

When to stop a canary

Promote, then back to single mode

The simulated route: a canary on your computer

Your turn: a canary that breaks

All traffic at once

Module practice

Module check