Skip to content
NextiaNextia Learning

Final assignment

  • After Module 5
  • about 3 hr

What you will be able to do

Use your pipeline the way a team does, for one real change. You turn a user’s report into requirements and checks. You deliver the change through a pull request, and promote one image to staging and then, with an approval, to production. Then you prove that you can go back and repair it. You keep evidence of every step, so that someone else can check your work.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

  • You completed Modules 1 to 5. Each part below names the lessons that it uses.

What you need on your computer

The solution was tested with Python 3.14.6 and the reference project’s checks. The Windows and Linux steps are not tested.

The situation

Section titled “The situation”

Sam reports a problem. The help desk got the ticket “I lost my password” with priority 1, the most urgent. Classifier keywords-1.1 treats every “lost” as urgent. In Module 4, a first attempt removed “lost” from the urgent words, and the evaluation gate stopped it: “Parcel lost in transit” lost its priority 1.

The help desk lead asks for:

  1. A lost password is priority 2. A lost parcel stays priority 1. A ticket that is urgent for another reason (“lost my password and now I am locked out”) stays priority 1.
  2. No other ticket gets worse than with the classifier in production.
  3. No risk to production until the change has passed staging and an approval, and a fast way back if it goes wrong.

Rules for every part

Section titled “Rules for every part”
  • Make evidence.md outside the repository (for example ~/projects/c19-evidence.md). After each part, add the commands, outputs and links that the part asks for. Link to your own pull requests, runs and releases.
  • Every change goes through a pull request. Do not skip a check, also not for the hotfix in Part 5.
  • No secret in any file, log or piece of evidence.

Part 1: requirements and checks

Section titled “Part 1: requirements and checks”

Uses Define checks: Eight checks, eight purposes, AI evaluation gate: The gate: three questions and AI evaluation gate: Your turn: make Sam’s ticket a must-pass ticket.

  1. Write the three requirements of the help desk lead as a table: requirement, the check that proves it, and where the check runs (pull request, push to main, deployment).
  2. Check the start: wc -l evaluation/tickets.jsonl says 51, as in the snapshot end-of-m05. (The E052 line of Module 4 was an experiment that you did not commit.) Add two must-pass tickets to the holdout split. E052: “I lost my password” (login, priority 2). E053: “Lost password, locked out” (login, priority 1).
  3. Run the gate for the classifier in production: python -m evaluation.run --classifier 1.1. It must fail now. Write down why that is correct.
  4. Production runs keywords-1.1 since the canary in Module 4. Make it the baseline: run the evaluation for 1.1 and write its results to evaluation/baseline.json, in the same shape as the file has now. (Simulated route: deploy/local/production.env in Git still says CLASSIFIER_VERSION=1.0. Set it to 1.1, as in the promotion in Progressive release: The simulated route: a canary on your computer. Then your local production runs the baseline too.)

Evidence for part 1: the table; the output of step 3; the new baseline.json, with one sentence about why the baseline had to change.

What you should see

keywords-1.1 fails the must-pass check on the new “I lost my password” ticket: it gives priority 1. That is the reported bug, now caught by a check. If the baseline stays keywords-1.0, a new classifier could fall from 0.914 to about 0.63 and still pass “no regression”. The baseline must be the classifier that users have today. With the two new tickets, keywords-1.1 has accuracy 0.914 on 35 holdout tickets.

Part 2: the change, through a pull request

Section titled “Part 2: the change, through a pull request”

Uses Build and test: The same checks on your computer, AI evaluation gate: Worked example: keywords-1.1 behind a flag, AI evaluation gate: The gate in CI and Progressive release: Module practice.

  1. Make a branch, for example feature/lost-password.
  2. Add classifier keywords-1.2 in ticket_api/classifier.py: the same as 1.1, but a lost password alone is priority 2. Select it with CLASSIFIER_VERSION=1.2. The default stays 1.0.
  3. Add tests for the three cases of requirement 1.
  4. Make the gate test the new candidate: change --classifier 1.1 to --classifier 1.2 in the CI workflow and in scripts/check.sh.
  5. Change the version to 1.3.0, export the contract again, and write docs/releases/1.3.0.md.
  6. Run scripts/check.sh. Then push, open the pull request (feature/lost-password into main), and wait for the checks.

Evidence for part 2: the link to the pull request with its passed checks; the gate output from the CI log; the diff of classifier.py.

What you should see

The gate output for keywords-1.2 on 35 holdout tickets: accuracy 0.914, no regression against keywords-1.1, must-pass failures: 0, Gate passed. One correct solution adds a subclass of the 1.1 classifier. It lowers the priority to 2 when the text has “lost my password” and no other urgent word. Other solutions are correct if the tests and the gate pass.

Part 3: one image, promoted

Section titled “Part 3: one image, promoted”

Uses Publish evidence: build-info.json: digest, commit and run in one file, Configuration separation: One image, many environments, Promote to staging: Approve the production deployment, Promote to staging: A version without a new build and Promote to staging: The local route.

  1. Merge the pull request. Find the digest of the new image in the CI run’s build-info.json.
  2. Watch the Deploy run deploy it to staging and run the smoke check.
  3. In staging, turn on the new classifier: set CLASSIFIER_VERSION=1.2 on the staging app. Check “I lost my password” there.
  4. Approve the production deployment, with a comment that says what you checked.
  5. Tag the merge commit v1.3.0, and push the tag. Check the release that the Release workflow makes.
  6. Show the digest in four places: build-info.json, the image tag 1.3.0, the staging app and the production app.

Evidence for part 3: the four digests, which must be the same; the approval (a link or the written record); the response for “I lost my password” in staging; the link to the release.

Part 4: protected credentials and environments

Section titled “Part 4: protected credentials and environments”

Uses Credentials and access: Scoped identities (live), Credentials and access: The GitHub environments and Credentials and access: The token’s own permissions.

  1. Show the protection rules of your environments: gh api repos/<you>/ticket-api-ci/environments --jq '.environments[] | {name, rules: [.protection_rules[].type]}', or a description of the Settings page.
  2. Show that the repository has no secret in its files: git grep -n -i -E "api[_-]?key|password|secret" -- . ':!*.md'. Explain each line that it finds.
  3. Live route: show the subject of each federated credential (az identity federated-credential list …), and the role and scope of each deployment identity.
  4. Show the permissions: of each workflow job, and say why each permission is needed.

Evidence for part 4: the outputs, and one sentence for each: what an attacker could do if this protection were missing.

Part 5: rollback and hotfix drill

Section titled “Part 5: rollback and hotfix drill”

Uses Rollback strategy: Worked example: the model-only rollback (live), Rollback strategy: Worked example: the application rollback (live), Hotfix procedure: Worked example: hotfix 1.2.1 and Release communication: The incident handoff.

  1. Model rollback. In production, turn keywords-1.2 off with the flag (CLASSIFIER_VERSION=1.1). Measure the time until the answers change. Turn it on again.
  2. Application rollback. Deploy the 1.2.1 digest with the Deploy workflow (Run workflow, with the digest and the reason). Check what changed for clients. Then deploy the 1.3.0 digest again.
  3. Hotfix. Simulate an urgent bug in 1.3.0 in the local environments. Make the branch hotfix/1.3.1 from the tag v1.3.0. Write a failing test, fix the bug, let CI run every check, tag v1.3.1, deploy it, and merge it back into main with a pull request.
  4. Prove that the fix is not lost: git branch -r --contains <hotfix commit> lists origin/main, and the new test is on main.
  5. Write the release notes of 1.3.1 and a short incident handoff: what happened, the impact, the timeline, the current state, the next steps and the owner.

Evidence for part 5: the times of step 1 and step 2; what the application rollback took away from clients; the links to the hotfix branch’s CI run, the release and the merge pull request; the output of step 4; the release notes and the handoff.

What you should see

The flag changes the model in under a minute, and the API keeps the same fields. The application rollback to 1.2.1 takes about two minutes through staging and production, and it also takes keywords-1.2 away: 1.2.1 does not know CLASSIFIER_VERSION=1.2, so it fails at start (unknown CLASSIFIER_VERSION '1.2') unless you set the flag back to 1.1 first. Check that before you roll back. In the reference run, the rollback from 1.2.1 to 1.1.0 removed score from the responses for the same reason: an older version does not know the newer settings and fields. Step 4 lists origin/main and the hotfix branch.

Check your work

Section titled “Check your work”

There is no reviewer for this course yet. Review your evidence with this table, or ask a colleague to use it. Mark each item as demonstrated, partially demonstrated or not yet demonstrated, with a reason. The course is complete when every item is demonstrated. You can improve an item and review it again as often as you need.

Required evidence Demonstrated when
Checks tied to requirements Each requirement has a check that runs automatically; the new must-pass tickets failed for keywords-1.1 and pass for keywords-1.2; the baseline is the classifier in production.
Immutable artifact promotion One digest in build-info.json, the version tag, staging and production; no image was built again for an environment or for the release.
Protected credentials and environments Production needs an approval; deployments sign in without a stored password (or, simulated, the keys are only in ignored files); no secret is in the repository; every workflow permission has a reason.
Verified rollback and hotfix procedure The model and the application were rolled back and forward with measured times; the hotfix ran every check, was released and deployed, and its commit is on main.

A working happy path is not enough. Maybe the gate still compares with keywords-1.0, the release built a new image, or the hotfix is not on main. Then the item is not yet demonstrated, even if everything else works.

Common mistake

The rollback does not start

Symptom: after you deploy the 1.2.1 digest, the new revision never becomes ready, and the deploy job stops after 5 minutes.

Cause: production still has CLASSIFIER_VERSION=1.2. Version 1.2.1 knows only 1.0 and 1.1, so it stops at start with ValueError: unknown CLASSIFIER_VERSION '1.2'. The old revision keeps the traffic, so users see no error, but you are not rolled back.

Fix: a rollback of the application must also roll back the settings that only the newer version knows. Set CLASSIFIER_VERSION=1.1 first, then deploy the older digest. Write this in the rollback section of your release notes before you need it.

Recap

  • A user’s report becomes a requirement, a check and a must-pass ticket, before the fix.
  • The baseline is what users have today. When production changes, the baseline changes.
  • One image goes everywhere. Environments differ only in settings, and production waits for an approval.
  • Practise the way back before you need it: the flag, the previous digest, the hotfix from the tag, and the merge back.

You have built and run a complete release process. The next step, MLOps and LLMOps: Operating AI in Production, keeps the service healthy after the release: monitoring, drift and retraining.