Final assignment

What you will be able to do

Hand over Larkfield’s ticket service to a person who has never been on call for it. You write the operations pack: what the service emits, how every answer is traced to its versions, what is watched and why, which alerts wake someone, and what that person does. Then you prove that the pack works, with an incident and a rollback that you run yourself.

Before you startWhat to know and to have readyAbout 180 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them: Monitor a model on public data that changes or Post-incident review of a public outage.
• The service: your ticket-api at the end of Module 6, or a copy of the snapshot end-of-m06 (see the snapshots). python -m pytest prints 218 passed.
• Docker with PostgreSQL and Valkey; the simulated provider, the API and a worker, as in Run the live service; and the course’s saved telemetry in data/telemetry/.
• Grafana is optional. Every part can use the ops tool on telemetry files instead.
• No account, no model call and no money are needed. Every AI answer is a replayed recording or a simulated one: say so wherever you show one.
• You can work in more than one session. Run the tests after each change.

The situation: the first on-call shift

Sam takes the on-call phone for the first time next week. Amira is on holiday, and Sam has never seen an incident of this service. Grace asks the course’s question one last time: “If the assistant starts failing our customers during Sam’s shift, how would Sam know, and what would Sam do?”

Your pack answers it. It goes in a new folder of your project, operations/pack/, with one file for each part. Write every file for Sam: short sentences, a command or a panel for each step, and an owner for each decision.

The course gave you a start: the inventory, the SLOs, the manifests, the alert rules, six runbooks and PIR-001. Use them, and say which ones you use. But the pack must show your service, from your runs. The reviewer looks for your own telemetry, your own release IDs and your own decisions. The course’s outputs, copied, are not evidence.

Part 1: the telemetry schema

Uses Logs, metrics and traces, Quality and costs and Privacy-aware observability.

1. Write operations/pack/telemetry.md. List the events (for example ticket.received, ticket.triaged, ticket.reviewed, answer.served and release.changed), the metrics with their labels, and the span attributes that the alerts and the ops tool read. For each one, give the question that it answers and who reads it.
2. Say what is never exported: ticket text, reply text, e-mail addresses and card numbers. Say what replaces them: a length, a language guess, a hash.
3. State the sampling rule, the retention and who may read the telemetry, with the reason for each value.
4. Run the service with TELEMETRY_FILE set. Send at least 20 practice tickets and 3 questions. Then run python -m ops summary and python -m ops privacy on your own files, and paste both outputs.

Part 2: end-to-end version traceability

Uses Artifact lineage, LLM-specific configuration, Experiment records and LLM/RAG regressions.

1. Build the manifests of prompt v2 and prompt v3 for your code with python -m ops release build … --write. Run the service with RELEASE_ID set to your v2 ID, and run python -m ops release verify on it.
2. Choose one triage answer from your own run. Walk it back with python -m ops trace TICKET on your two telemetry files: the release, the code fingerprint, the migrations, the prompt and its fingerprint, the model and its expected version, and the evaluation evidence. Explain each line in one sentence.
3. Choose one answer to a question. Show its source_versions, the index version from python -m scripts.reindex --check, and how Sam would see that a source is stale.
4. Run python -m ops release diff on your two IDs. Say which differences change the answers.
5. Say what plays the part of a model registry in this project, and what a real registry would add.

Part 3: the monitoring plan and the dashboards

Uses Service objectives, Quality and costs, Data and concept drift and Delayed labels.

Write operations/pack/monitoring.md, with one table row for each signal:

Column	What to write
Signal	for example, the draft SLI, the agreement with the reviews, the cost per 1,000 tickets
Kind	quality, latency, cost, safety, drift or a proxy
Known	at once, or later (a delayed label), and how late
Target or threshold	with its source: an SLO, the budget, or a rule of thumb (say that it is one)
Window	in tickets or reviews when the rate can change
How to read it	the ops command, the Grafana panel, or both
Owner	one person

The table covers at least these signals: the draft and safety journeys, the agreement with the agents’ reviews, the triage time (p95), the time to a draft, the cost per 1,000 tickets, answers without text, stale sources, the model version, and one input-drift check with its method. Then:

1. Say how many reviews a day the quality number needs, and why. Delayed labels found 139 reviews for a margin of about 5 points near 90%. Say what Sam does on a day with fewer.
2. Write the rule for a drift alert: what it starts, and what it never starts.
3. Dashboards. Either provision a Grafana dashboard from a file and add a screenshot of it on your own run, or write operations/pack/dashboards.md. In that file, give each panel its title, its query or ops command, and the question that Sam reads from it. Put the panels that tell “available” apart from “failing its users” at the top.
4. Run python -m ops slo, python -m ops quality --by release and python -m ops labels on your own telemetry and on one saved run of the course. Fill in a “now” value for each row from them.

Part 4: actionable alerts and runbooks

Uses LLM/RAG regressions, Runbooks and Provider and dependency outages.

1. List every alert in operations/pack/alerts.md with its seven parts: signal, where, rule, severity, owner, action (the runbook) and test.
2. Add two alerts that the course did not finish: the quality alert on the last 139 reviews, from the module practice of LLM/RAG regressions, and the answers-slow alert, from the last task of Runbooks. Write each one in the style of operations/alerts.toml, and add it to Grafana or to ops/cmd_alerts.py.
3. Test every alert on saved runs. Each one must fire on a run where it should, and stay quiet on normal-week.jsonl.gz. Use python -m ops alerts FILE, with --after and a practice action to measure the delay. An alert that fires on the normal week needs a decision. The course’s no-answer fires twice there, on prompt v2’s one cut-off answer: say what you do about it.
4. Write or update a runbook for each alert, in five parts: detection, triage, mitigation, escalation and recovery checks. Every step names a command or a panel, and every recovery check is a number.

Part 5: a tested incident and rollback

Uses Evaluation gates, Controlled rollouts and Post-incident learning.

1. Run the gate on both prompts. Keep the outputs and the exit codes.
2. On your own service, run prompt v3 as a canary against prompt v2 with python -m ops rollout. Write the rollback rules before you start. Send enough practice tickets for at least 20 canary tickets, at a rate that the simulated quota allows: for example 0.5 a second, as in Provider and dependency outages. You may run python -m scripts.scenario canary with your IDs instead (see the project’s README).
3. When a rule is met, follow your bad-release runbook: roll back with a reason, then send new tickets.
4. Prove the recovery with the three parts of the proof of recovery: the history, no canary tickets after the change, and signals back at the active release’s numbers. Paste python -m ops rollout history, and python -m ops canary and python -m ops incident on your telemetry.
5. Write operations/reviews/PIR-002-canary-v3.md in the format of PIR-001: a blameless review of your canary as a near miss, with a timeline from your telemetry and at least two actions with owners.
6. Run a short tabletop exercise of a provider outage with your runbook, and list each gap that it found.

The failure paths you must check

Show each one with an output from your service or from a saved run, and say which. Label every constructed case as constructed.

1. A manifest that does not match the files: change one word of the prompt file, and show that the service refuses to start. Then undo the change.
2. The gate stops prompt v3 with exit code 1, before any customer sees it.
3. python -m ops privacy finds no text, no e-mail address and no card number in your telemetry.
4. After a document change, python -m scripts.reindex --check exits with 1, and an answer reports stale_sources. After the rebuild, both are clear.
5. With prompt v3 active, a ticket without a valid answer goes to a person (sent_to_person), not to nobody.
6. During a practice outage of the simulated provider, GET /v1/status says “delayed”, no ticket becomes a dead letter, and ops incident ticks every recovery box after the provider is back.

Fresh-environment run

Before you submit, prove that the pack works from nothing:

1. Copy the project to a new folder, without .venv, .env, telemetry/ and practice/.
2. Stop the containers and delete their data with docker compose down --volumes. Start PostgreSQL and Valkey again.
3. Make a new virtual environment, install from requirements.txt, and make .env from .env.example.
4. Apply the migrations, load the data, and run every test.
5. Build the v2 manifest and set RELEASE_ID. Start the services, send 5 practice tickets, and run python -m ops trace on one of them.

Paste the commands and their outputs into your report. If a step needed something that is not in the README, fix the README, and run the steps again.

What to submit

Your project folder (without .env, .venv and large run outputs), with operations/pack/, your alert and runbook changes, PIR-002 and REPORT.md:

# Larkfield ticket service: operations pack
## Telemetry schema          (events, metrics, attributes; what is never exported; sampling,
                              retention, access; ops summary and ops privacy on your run)
## Version traceability      (your manifests; one triage answer and one document answer traced;
                              the release diff; what plays the part of a model registry)
## Monitoring plan           (the signals table; reviews a day; the drift rule; the dashboards)
## Alerts and runbooks       (every alert in seven parts; the two new ones; each tested on saved
                              runs, with its delay and the normal week; the runbooks)
## Incident and rollback     (the gate's outputs; the canary; the rollback; the three-part proof of
                              recovery; PIR-002; the tabletop's gaps)
## Failure paths             (each of the six, with the output that shows it)
## Fresh-environment run     (the commands and their outputs)
## Limits                    (replayed and simulated answers, practice time, your computer, what
                              you did not test)

How it is marked

A reviewer marks each required item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all four must be demonstrated. You can improve an item and submit it again.

Required item	Demonstrated	Partially demonstrated, for example	Not yet demonstrated, for example
End-to-end version traceability	Your service runs with a release ID that matches its files. One triage answer and one document answer are traced from your own telemetry to the release, the prompt fingerprint, the model version, the index version and the evaluation evidence. The diff of your two releases names what changes the answers.	The triage answer is traced, but the document answer’s sources or index version are missing. Or the trace uses only the course’s saved files.	Answers carry no release ID. Or the manifest does not match the files, and nobody noticed.
Quality, latency and cost monitoring plan	Every signal has a kind, a target with its source, a window, a way to read it and one owner. Delayed and immediate signals are kept apart, with the review count and its reason. A drift alert opens an investigation, never a change. The dashboards show “failing its users” next to “available”.	The plan has latency and cost, but quality is only a proxy. Or the windows are in minutes where the rate changes.	Uptime is the only service signal. Or a drift alert triggers a retrain or a release.
Actionable alerts	Each alert has its seven parts and a runbook with a command in each step. Each one is tested on saved runs: it fires where it should, with its delay, and the normal week is checked. The two new alerts work.	The alerts are written, but one is not tested. Or a runbook step says “check that it is fine”.	An alert has no owner or no runbook. Or an alert that cannot fire, or that fires on every healthy run, is kept without a decision.
Tested incident and rollback scenario	The gate’s decisions are shown with their exit codes. The canary ran on your service with rules written first. A rule was met, and the rollback has all three parts of the proof of recovery, from new tickets after the change. PIR-002 is blameless, and its actions have owners.	The rollback is done, but the proof uses no new tickets, or one part is missing. Or the review has actions without owners.	No run of your own: only the course’s recorded canary. Or “the alert stopped” is the evidence of recovery.

Whatever else is strong, the work is not complete while your telemetry contains ticket text, an e-mail address or a card number, or while a ticket without a valid answer can reach nobody. A full dashboard does not make up for either.

The course’s own results can help you judge yours, but they are not targets. In Controlled rollouts, the recorded canary met its first rule at the 9th canary ticket, and 25 customers got a prompt v3 answer. In Runbooks, the outage alerts fired 0.6 and 3.1 practice minutes after the outage began, by the rules on saved telemetry. Your runs are smaller and on your own computer, so your numbers differ.

AI assistants

You may use an AI assistant to practise, to explain an error, or to suggest an alert rule or a runbook step. Check each suggestion, and be ready to explain your final work. For each alert, say what it measures, why you chose its threshold and window, and which run proves that it fires. Mark in your report the decisions that are your own, and the suggestions that you rejected, with the reason. An assistant may not produce your evidence: every output, count and time in your report comes from a run on your computer or from the course’s saved files, named as such.

Common mistake

ops trace stops: no release manifest none.toml

Symptom: python -m ops trace on one of your tickets prints the input and the result, then stops with no release manifest operations/releases/none.toml.

Cause: the ticket was triaged while RELEASE_ID was empty, so it carries no release. There is nothing to trace it to.

Fix: set RELEASE_ID to a manifest that matches your files, restart the API and the worker, and send new tickets. Tickets made before that have no lineage, and you cannot add it afterwards.

Common mistake

The canary gets no tickets

Symptom: after python -m ops rollout canary, python -m ops canary on your telemetry shows 0 canary tickets.

Cause: the worker reads the release state every 5 seconds. Or the share is small: the canary chooses tickets by a hash of the ticket ID, so a few tickets may all miss it.

Fix: check python -m ops rollout status, send more tickets, and wait for the worker. For a short practice run, use a larger share, as Controlled rollouts did with 0.5.

Recap

In this course, you made an AI service operable, one module at a time:

• Own it: an inventory of what is operated, SLOs for the journeys that matter, error budgets, severity levels and an owner for each.
• Trace it: a release manifest for every change, with its ID on every answer, so that each answer leads back to its versions.
• See it: telemetry without private text, quality and cost by version, and a trace that shows where the time and the money go.
• Watch it change: drift with a stated method, delayed labels with enough reviews, and the regressions that only a language-model service has.
• Release it safely: a gate with written limits, a shadow and a canary with rollback rules, and a change chosen from the cause.
• Run it on a bad day: runbooks, an honest status for customers, recovery proved with numbers, and reviews without blame.

If you have not done one yet, choose a case study for the certificate: Monitor a model on public data that changes or Post-incident review of a public outage.