Release communication

What you will be able to do

A release is finished when the people after you know what changed, who watches it, for how long, and how to go back. A short text that answers these questions turns a 2 a.m. surprise into a decision that someone else can make.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The release notes docs/releases/1.1.0.md and 1.2.0.md, and how the Release workflow makes the GitHub release from them, from Promote to staging: A version without a new build and Progressive release: Module practice.
• What to watch during a rollout (error rate, the gate’s metrics in production), from Progressive release: When to stop a canary.
• The rollback options and the drill, from Rollback strategy: Three things can go back and Guided practice: a rollback drill on your computer.
• Hotfix 1.2.1, from Hotfix procedure: Worked example: hotfix 1.2.1.

What you need on your computer

• Your repository ticket-api-ci with docs/releases/, from Hotfix procedure.
• For the module practice: Docker, and a copy of main with deploy/local/, .env and the files in secrets/, from Configuration separation: Guided practice: the local environment files and Promote to staging: Guided practice: promote your own digest. It is free: everything runs on your computer.
• If your project is broken, copy the snapshot end-of-m05, as in Use a snapshot.

Run with GitHub Actions (ubuntu-24.04 runners) in the public repository Nextia-Intelligence/ticket-api-ci, Docker Desktop 4.81.0 on macOS (Apple silicon), and Azure CLI 2.88.0 in canadacentral. The Windows and Linux steps are not tested.

Who reads a release note?

Three people read the notes of a release, each with one question:

• A client developer: “Must I change my code?” For 1.2.0: no, but confidence is deprecated. Move to score before 2.0.
• The help desk lead: “What do my people see?” For 1.2.0: no change yet; keywords-1.1 comes later, step by step.
• The person on duty tonight, maybe not Amira: “If it breaks, what do I do?” For 1.2.0: deploy the 1.1.0 digest.

Release notes are the short text that answers these questions for one version. They are written for people who did not see the work. “Fixed some bugs” answers none of the three questions.

The parts of the 1.2.1 notes

Here are the real notes of the hotfix. The Release workflow put the file on the release page, and added the digest line under it:

docs/releases/1.2.1.md

## ticket-api 1.2.1 (hotfix)

- **Fixed:** `GET /v1/history` answered `500` when the history had rows from version 1.1.0. Those rows have no `score`; the API now uses their `confidence`. This happened while 1.1.0 and 1.2.0 ran at the same time on one database, during the rollout of 1.2.0.
- **Why the checks missed it:** the database tests made all their rows with 1.2.0, and the smoke check did not call `/v1/history`. Both are added in this release: a test with a row from 1.1.0, and a smoke check that fails on a `5xx` from `/v1/history`.
- **Made from:** the tag `v1.2.0`, on the branch `hotfix/1.2.1`. The same fix is merged into `main`.
- **Owner:** Amira. **Watch after release:** `GET /v1/history` and the error rate, for one day.
- **Rollback:** deploy the 1.2.0 digest (the bug comes back), or 1.1.0.

Each line has a job:

Line	Question it answers	Why it is there
Fixed	What do clients see now?	The symptom in the client’s words (GET /v1/history → 500) and when it happens. Not the code change: that is in the commit.
Why the checks missed it	Can it happen again?	Only for a fix. It names the gap, and the check that closes it.
Made from	Which code is this?	A hotfix is not main. This line says so, and that main has the fix too.
Owner	Whom do I call?	One person, by name. “The team” is nobody at 2 a.m.
Watch after release	What can go wrong, and until when?	The signal to look at, and the monitoring window.
Rollback	If it breaks, what do I do?	The target, and what it costs. “The bug comes back” is honest.
Image (added by the workflow)	What exactly runs?	The digest. The rollback of the next release starts from it.

The notes are a file in the repository, not text typed into a web form. So they are reviewed in the pull request like code, and the release workflow publishes exactly what was reviewed.

Owners and monitoring windows

An owner is the person who decides about the release after the deployment: watch it, roll it back, or call others. A monitoring window is the time after a release when the owner actively looks at named signals. Most problems of a release show up soon. Compare the three releases:

Release	Watch	For	Why this long
1.1.0	the production smoke check and the logs	one hour	The code was the same as before. Only the delivery path was new.
1.2.0	the smoke checks, the error rate in the logs, GET /v1/history	one day	A schema change and a new field. Data problems come when real traffic writes rows.
1.2.1	GET /v1/history and the error rate	one day	A fix for a data-dependent bug. Old rows must be read correctly for a full day of traffic.

The window grows with the risk: a change to data or to a model needs more traffic before you can trust it. “Watch” must name a signal that someone can check: a query, an endpoint, a log search. In the reference project, the error rate came from the log query of Progressive release: requests by revision and status.

Notice what the 1.2.0 notes said: watch GET /v1/history for one day. The bug was in exactly that endpoint. The note was right, and Amira found the bug 9 minutes after the release, on a local copy. The cloud had no database, so the signal could not show it there. A monitoring signal must exist in the environment that you watch.

The incident handoff

An incident is an event that makes the service worse for its users and needs action now. Often the person who starts the work is not the person who finishes it: a shift ends, or a specialist takes over. An incident handoff is the short written state that lets the next person continue without asking. It has six parts:

1. What happened, in one or two sentences, with the symptom.
2. Impact: who is affected, since when, and how much.
3. Timeline: times and facts, with links to the evidence. No opinions.
4. Current state: what runs now, and what is still wrong.
5. Next steps: what is still to do, each with a condition (“after no 1.1.0 is left”).
6. Owner: who has it now, and who the next person must call.

The timeline is the part that people write last and need first. Here is the real one of 2026-10-07:

The real release and recovery timeline of the reference repository on 2026-10-07, from version 1.1.0 to the forward repair. Production is the live route.

Text description of the diagram

The timeline has three lanes. On main: v1.1.0 at 18:18, v1.2.0 at 18:32, PR #6 at 18:44, PR #7 merged at 18:58, PR #8 with the forward repair at 19:12. On hotfix/1.2.1, which starts at the tag v1.2.0: the fix commit at 18:49 with every check green, the tag v1.2.1 at 18:51, and at 18:55 the update of the branch because PR #7 was behind main. In production: 1.1.0 at 18:18; 1.2.0 at 18:32 and then the canary; the bug found on a local copy at 18:41; 1.2.1 deployed by hand from 18:51 to 18:53; the build of main at 19:03; the rollback drill from 19:03; the build with the forward repair at 19:16; and the deletion of the Azure resources at 19:17.

Worked example: Amira hands over at 19:17

Amira’s day ends. Sam is on duty tonight. She writes the handoff from the timeline, with links instead of descriptions:

Incident handoff: GET /v1/history 500 (ticket-api 1.2.0)

**What happened:** In 1.2.0, GET /v1/history answered 500 when the history had a row
written by 1.1.0 (no score). Found at 18:41 UTC on a local copy with both versions on
one database. Classification was never affected.

**Impact:** Only GET /v1/history, and only where 1.1.0 and 1.2.0 wrote to one database
(the local environments; the cloud apps have no database, so /v1/history answers 503
there by design). No client reported it.

**Timeline (UTC, 2026-10-07):**
- 18:32 v1.2.0 released and deployed (release v1.2.0).
- 18:41 Bug found: 500, log "items.0.score Input should be a valid number".
- 18:49 Hotfix fc9005a on hotfix/1.2.1 (from v1.2.0): failing test, fix, all checks green (CI run 37669759163).
- 18:51 v1.2.1 released (Release run 37669994015), deployed by hand (Deploy run 37670050850), production at 18:53.
- 18:58 PR #7 merged the fix into main (after Update branch: it was BEHIND).
- 19:03-19:09 Rollback drill: 1.1.0 (run 37671631615, 120 s), forward to 1.2.1 (run 37671961512, 116 s), flag-only rollback 43 s.
- 19:12 PR #8 merged: migration 003 fills the empty score values.

**Current state:** main = 30e0c7e (1.2.1 code + 003), deployed to staging and production at 19:16.
CLASSIFIER_VERSION=1.1 in production. The history smoke check passes. Locally, 003 left 0 empty rows.

**Next steps:**
1. Watch GET /v1/history and the error rate until 18:53 UTC tomorrow (the 1.2.1 window).
2. If history fails again: deploy the 1.2.1 digest sha256:84937455... (Run workflow); do not go to 1.2.0.
3. Before 2.0 removes confidence: ask clients to move to score (see the 1.2.0 notes).

**Owner:** Sam from now (on duty). Amira is the owner of 1.2.1: call her for code questions.

Read it as Sam. Every line is a fact with a time or a link. The rollback in “Next steps” names the digest and says where not to go: 1.2.0 would bring the bug back. “Current state” says what runs now, not what was planned. And it is short: Sam reads it in two minutes.

At 19:17, the practice ended, and Amira deleted the Azure resources. az resource list -g rg-ticket-api-ci listed 6 resources: the workspace, the environment, the two apps and the two identities. There were also 2 role assignments. Then she ran az group delete --name rg-ticket-api-ci --yes --no-wait. The GitHub repository, its releases and its runs stay: they are the evidence that this course links.

Guided practice: notes and a handoff for 1.2.0

1. Open docs/releases/1.2.0.md in your repository. For each line, write which reader of Who reads a release note? needs it.
2. Find the line that would have helped most at 18:41. Write why.
3. Write the handoff for the 1.2.0 bug as if Sam took over at 18:45: before the hotfix. Use the six parts.
4. In “Next steps”, write the two options that Sam has at 18:45, with what each one takes away.
5. Compare with the 19:17 handoff above. What changed between 18:45 and 19:17 in “Current state”?

Show a model answer

1. “New for clients” and the deprecation of confidence: the client developer. “New, off by default” (keywords-1.1): the help desk lead. “Database” (run the migrations first): the person who deploys. “Owner”, “Watch” and “Rollback”: the person on duty.
2. “Watch after release: … GET /v1/history, for one day”. It named the signal of the bug before anyone knew about the bug.
3. and 4. At 18:45:

Incident handoff at 18:45 (shortened)

**What happened:** 1.2.0 GET /v1/history answers 500 when a row from 1.1.0 (no score) is
in the history. Found 18:41 UTC on a local copy. Classification works.
**Impact:** GET /v1/history only, where 1.1.0 and 1.2.0 share a database.
**Timeline:** 18:32 v1.2.0 released; 18:41 bug found (500, items.0.score is None).
**Current state:** production runs the 6ae49dc build (1.2.0 code), flag 1.1. Not fixed.
**Next steps (choose one):**
  a) Hotfix 1.2.1 from v1.2.0 (Amira started at 18:44). Keeps score and keywords-1.1.
  b) Roll back to 1.1.0 (digest sha256:4eb3db8d...): stops the 500, but removes score
     and keywords-1.1. Tell clients and the help desk first.
**Owner:** Amira (hotfix). Sam: call Amira before option b.

5. At 18:45 the bug is not fixed, and the next step is a choice. At 19:17 the fix is in production and on main, the data is repaired, and the next step is only to watch. A handoff is a snapshot of the state: write a new one when the state changes, do not edit the old one.

Your turn: notes for the forward repair

PR #8 (migration 003) was merged and deployed, but it got no tag and no release notes. Is that right? Write docs/releases/1.2.2.md as if 003 were released as version 1.2.2. What must the “Database” and “Rollback” lines say?

Show one answer

Without a tag, the change has no release page, and the next person must read the PR to know that a data change happened. For a change of data, a release is better. One version:

docs/releases/1.2.2.md

## ticket-api 1.2.2

- **Changes for clients:** none. History items that came from version 1.1.0 already showed
  `score` (since 1.2.1); now the database has the value too.
- **Database:** migration `003_backfill_score` sets `score = confidence` where `score` is
  empty. It changes no structure. Run it after no 1.1.0 instance is left. Take a backup first:
  it changes rows.
- **Owner:** Amira. **Watch after release:** the count of rows with an empty `score` (must
  stay 0) and `GET /v1/history`, for one day.
- **Rollback:** deploy the 1.2.1 digest. The filled values stay, and 1.2.1 reads them
  correctly. A rollback does not undo `003`; only the backup has the old values.

The “Rollback” line is the important one: it says that the data change stays after a rollback, and where the old values are.

Common mistake

The notes promised a check that the deployment did not run

Symptom (live): the 1.2.1 notes say: “a smoke check that fails on a 5xx from /v1/history” is added. But the smoke check of the hotfix deployment (run 37670050850) printed only four lines in each environment:

ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)

No history line. The new check did not run on the deployment that it was written for. The next deployment, after PR #7 (run 37671226621), printed five lines, with ok history (503). Its approval comment said so: “Staging passed, including the new history check.”

Cause: the Deploy workflow runs from main, with main’s files, also scripts/smoke.sh. At 18:51, main did not have the hotfix yet, so it had the old smoke check. Only the image came from the hotfix. That is the right design (the deployment process has one version, on main), but the notes did not say which check runs when.

Fix:

1. Write in the notes what was verified, not only what was added: “The new history smoke check runs from the next deployment of main, after the merge back.”
2. Merge the hotfix back before you rely on its new checks. Check the run’s log for the new line.
3. For the hotfix deployment itself, check the endpoint by hand, and write the result in the handoff: curl -s -o /dev/null -w "%{http_code}" <URL>/v1/history -H "X-API-Key: …".

For a real AI service: a note says “we added an evaluation for this case”, but the release pipeline still runs the old evaluation set. That gives everyone false confidence. Notes must say what ran, with a link to the run.

Check your understanding

Why do the 1.2.0 and 1.2.1 notes ask for one day of monitoring, and 1.1.0 for one hour?

Show the answer

The monitoring window grows with the risk. 1.1.0 had the same code as before. Only the delivery path was new, so an hour of smoke checks and logs was enough. 1.2.0 changed the schema and the API (a new field), and 1.2.1 fixed a bug that depends on the data in the table. Data problems appear only when real traffic writes and reads rows, often hours later. So the owner watches named signals (GET /v1/history, the error rate) for a full day.

Module practice

Guided practice → independent task. Simulate a failed staging release and a production hotfix with disposable environments. The local route costs nothing: two Compose projects, staging (port 8081) and production (port 8082), each with its own database, and a test database for the checks. You make them, break them, repair them and delete them.

You saw a small worked example in each lesson. Now do the whole module on your own, from a clean state:

1. The failed staging release. Deploy 1.2.0 to staging with the tools of main (they have the history smoke check). Start 1.1.0 next to it on the staging network, as during a rollout, and classify one ticket with it. Run the smoke check of staging again. It must fail. Stop the promotion: production stays as it is.
2. The decision. Write which rollback option you would use and why, in two sentences.
3. The hotfix. In a practice repository, make hotfix/1.2.1 from v1.2.0: failing test, fix, every check, tag. Deploy the 1.2.1 image to staging, then to production. Merge back into main after an update of the branch.
4. Restore and verify. Show known-good behaviour (all smoke checks pass, /v1/history → 200), repair the data with 003, and show that no fix is lost from main.
5. Communicate. Write the release notes of 1.2.1 and a handoff with the six parts.
6. Clean up. Delete both environments with their databases.

Label each result simulated (or live, if you did step 3 with your Azure apps). Use your own outputs.

Show a model answer (simulated, shortened)

1. The failed staging release. The commands, from a checkout of main:

Terminal (simulated)

scripts/promote-local.sh staging ghcr.io/usmanahmed99/ticket-api-ci@sha256:bd2316b56067c738199d6203a23845c97fb7b85b550379411eddd7cc6a2ddab8
docker run -d --name ticket-api-staging-old --platform linux/amd64 --network ticket-api-staging_default \
  -e API_KEY=old-key -e DATABASE_URL=postgresql://tickets:<password from .env>@db:5432/tickets \
  ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a
docker exec ticket-api-staging-old python -c "import urllib.request,json; r=urllib.request.Request('http://127.0.0.1:8000/v1/classify', data=json.dumps({'subject':'Parcel lost'}).encode(), headers={'Content-Type':'application/json','X-API-Key':'old-key'}); print(urllib.request.urlopen(r).read().decode())"
scripts/smoke.sh http://127.0.0.1:8081 "$(cat secrets/staging_api_key.txt)"; echo "smoke exit $?"

The old container has no published port, so the classify request runs inside it, with Python. The output (simulated):

ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)
ok    history (200)
staging now runs ghcr.io/usmanahmed99/ticket-api-ci@sha256:bd2316b56067c738199d6203a23845c97fb7b85b550379411eddd7cc6a2ddab8
{"category":"shipping","priority":1,"confidence":0.7,"model_version":"keywords-1.0"}
ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)
FAIL  history: expected 200|503, got 500
smoke exit 1

The first smoke check passed: the staging database was empty, so no row lacked score. Only after one row from 1.1.0 did the check fail. A staging check is only as good as the data and traffic in staging. Production was not touched: the release stopped in staging.

2. The decision. The code is wrong, not the model, so a flag rollback does not help. A rollback to 1.1.0 would remove score and keywords-1.1 for every client, to fix one endpoint. A hotfix is the smaller change.

3. The hotfix. The test fails first (assert None == 0.7 ± 7.0e-07), then 3 passed; all checks: 31 passed, “No known vulnerabilities found”, “Gate passed.” The graph after the merge back is the one of Hotfix procedure: Guided practice. Deploy to staging first:

$ scripts/promote-local.sh staging ghcr.io/usmanahmed99/ticket-api-ci@sha256:84937455beb63744755622d539748c9adb1772f379c9eb7e4f961c5b7f1bc924
ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)
ok    history (200)
staging now runs ghcr.io/usmanahmed99/ticket-api-ci@sha256:84937455beb63744755622d539748c9adb1772f379c9eb7e4f961c5b7f1bc924

Then the same digest to production, with the same five ok lines. Live route: the same digest with Run workflow, as in run 37670050850.

4. Restore and verify. /v1/history → 200, and the old row shows score from confidence. Remove ticket-api-staging-old, then deploy the build with 003 to each environment: applied 003_backfill_score, and the count of empty score rows goes to 0 (in the test run of Rollback strategy, from 3 to 0 in production). In the practice repository: git branch --contains v1.2.1 lists main; git log main --oneline | grep Hotfix finds the commit; grep -n test_history_reads_rows_from_version_1_1 tests/test_database.py finds the test on main.

5. Communicate. The notes have the seven lines of The parts of the 1.2.1 notes. The handoff has the six parts, with your times.

6. Clean up. See the reset in the recap.

A reviewer marks the practice demonstrated when four things are true. The failed staging check is shown with its FAIL line and exit code 1, and production was not changed. The new test is shown failing before the fix and passing after it, and no check was skipped. Known-good behaviour is shown with all five smoke checks after the hotfix, and 003 brings the empty rows to 0. main provably contains the fix.

Module check

Check your understanding

Retrieval: name the three rollback options of this module, how each one is done for the ticket API, and how long each took in the drill.

Show the answer

Application rollback: deploy the previous digest with the Deploy workflow (Run workflow), through staging and the approval: 120 s to production for 1.1.0, 116 s forward to 1.2.1. Model rollback: change only the flag, CLASSIFIER_VERSION=1.0, on the same image: 43 s. Data: no rollback; repair forward with a new migration (003_backfill_score), one migration that took the empty rows from 1 to 0. Restore a backup only when data is lost.

Check your understanding

Predict: a team skips the merge back of a hotfix, and also has no test for the bug. Which of the module's checks would still catch the bug when it returns with the next release, and which would not?

Show the answer

Almost nothing. The new database test is only on the hotfix branch, so CI on main does not run it. The history smoke check is also only on the hotfix branch, and the Deploy workflow uses main’s scripts/smoke.sh, so the deployment checks would pass. The evaluation gate and lint test other things. Only the monitoring window of the next release could show the 500, after users get it. And git branch --contains v1.2.1 would show at once that main does not contain the hotfix. The merge back and the test are what keep a fix fixed.

Check your understanding

Completion check: restore known-good behaviour and verify that no fix is lost from the development branch. Show your evidence.

Show the answer

Known-good behaviour: the environment that had the bug runs the hotfix digest, and the smoke check prints five ok lines, including ok history (200) (local) or (503) (cloud, no database); GET /v1/history with a row from 1.1.0 answers 200; after 003, the count of rows with an empty score is 0. No fix lost: git branch --contains v1.2.1 lists main; git log main --oneline | grep Hotfix shows the hotfix commit; the test test_history_reads_rows_from_version_1_1 is in tests/test_database.py on main, and CI on main passed with it. Each result is labelled simulated or live. If one of these is missing, the recovery is assumed, not verified.

Recap

• Release notes answer three readers: what changes for clients, what the help desk sees, and what the person on duty does if it breaks. Each release names an owner, the signals to watch and for how long, and the rollback with its cost.
• The monitoring window grows with the risk: one hour for 1.1.0, one day for the data changes of 1.2.0 and 1.2.1.
• An incident handoff has six parts: what happened, impact, timeline with links, current state, next steps with conditions, owner. Write a new one when the state changes.
• Notes say what was verified, with a link to the run. The hotfix deployment ran main’s smoke check, without the new history check.
• In this module, you chose between an application, a model and a data rollback. You shipped a hotfix from the tag with every check, merged it back, and repaired the data forward.

Reset. Delete the local environments with their databases, from the project folder: scripts/stop-local.sh staging --volumes and scripts/stop-local.sh production --volumes. Remove the extra containers: docker rm -f -v ticket-api-old ticket-api-staging-old ticket-api-testdb (“No such container” is fine). Check with docker ps -a --filter name=ticket-api and docker volume ls --filter name=ticket-api: no lines from this module. Delete the practice repository folder. To start the module again, copy the snapshot end-of-m04; for the final assignment, start from end-of-m05. Live route: if you made Azure resources, delete the resource group as in Verify and remove in the Docker course, and check az group exists until it says false.

The course. You began with one change and the question “how does it reach production safely?”. Now every change in your repository runs lint, tests, a security check and an AI evaluation gate on a clean runner. The pipeline builds one image and traces it to its commit. It promotes that digest to staging and, after an approval, to production. Each environment has its own identity and no stored password. You released a model change and a schema change without breaking the old version, rolled back in three different ways, and shipped a hotfix without losing it. The final assignment asks you to design and run the pipeline again, on your own, with the evidence for each of these claims.

Next: Final assignment