Controlled rollouts

What you will be able to do

Let a candidate release meet real tickets without betting every customer on it. Run it in the shadow first, then as a canary for a share of the tickets, with rollback rules written before it starts. When a rule is met, roll back, and prove with numbers that the service recovered.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The release gate and the new manifests of v2 and v3, from Evaluation gates.
• Answers without text and the no-answer alert, from LLM/RAG regressions.
• A canary as a release technique, from Progressive release (optional: this lesson explains it again).

What you need on your computer

• Your project after the last lesson: python -m pytest prints 213 passed, and operations/releases/ has your new v2 and v3 manifests.
• Docker running PostgreSQL and Valkey. Three terminals for the simulator, the API and the worker, as in Run the live service.
• The saved telemetry in data/telemetry/. The steps that read saved files need no Docker.

A new dish in a busy restaurant

A chef wants to change a popular dish. She does not print new menus for the whole city on day one.

• First, she cooks the new version next to the old one and tastes it. No guest eats it yet.
• Then she serves it to one table in ten. The other tables get the old dish.
• Before she starts, she writes down when she stops: “one plate sent back cold, and it goes off the menu”.
• If she stops, she checks the next plates of the old dish. She wants to know that the kitchen is back to normal, not just hope so.

Each step limits the damage of a mistake. Each step also gives her evidence that a recipe card cannot give.

The problem: everyone at once

“Last week”, prompt v3 went to every ticket in one step. In the practice replay, the switch came at minute 5.0. From then on, every ticket got v3. 186 of the 1,798 v3 tickets got no answer, and 10 tickets that needed a person reached nobody. The alerts did fire, 5 to 9 practice minutes after the switch. But nothing limited how many customers met v3 before a person acted, and nobody had decided in advance what “act” meant.

The gate of the last lesson would have stopped v3. But some candidates will pass the gate and still fail on live tickets. A failure in 1 of 1,000 tickets hides easily in 98 holdout tickets. So the release itself must be safe.

The idea: shadow, canary, rollback rules, proof

Four names, one for each step of the chef’s plan:

• Shadow run: the candidate release runs beside the active release on the same tickets. The active release answers the customer. The candidate’s result is saved for comparison only. No customer sees it.
• Canary: the candidate answers a share of the tickets, for example 10%. The rest keep the active release. The name comes from the canaries that miners took underground: a small, early warning.
• Rollback rules: written conditions that send the canary back, decided before it starts and stored in a file. Each rule names a signal, a threshold and the release it compares with.
• Rollback and proof of recovery: the canary is removed. Then the evidence must show that the tickets after the change look like the active release again.

In the project, the release state in PostgreSQL says which release does the work. It names an active release, maybe a canary with its share, maybe a shadow. python -m ops rollout changes it, and the worker reads it every 5 seconds. Every change is saved, with who made it and why.

The canary chooses tickets by a hash of the ticket ID, not by a coin toss. So the same ticket always gets the same release. If the queue delivers a job twice, the same release does it both times.

The release state and the ops rollout commands that change it. The worker reads the state every 5 seconds, and every change is saved in release_events.

Text description of the diagram

The service starts with v2 as the only active release. A shadow adds v3 beside it: v3 runs on every ticket, but only v2 answers. “Stop” removes the shadow. A canary gives v3 a share of the tickets, chosen by the ticket ID. From the canary, “rollback” returns to v2 only, and “promote” makes v3 the active release. After a promotion, a rollback returns to the last other active release.

A tiny example: how many canary tickets until the rule fires?

The project’s first rollback rule is “one answer without text”. How many canary tickets does v3 need before it breaks the rule? Each v3 ticket has the same chance of no text: 88 of its 679 recordings were cut off.

Run Python in your browser

How fast does one rule catch prompt v3?

The box prints the chance for one ticket and the average number of tickets until the first failure. Then it prints the chance within 5, 10, 20 and 50 canary tickets, and how long 10 canary tickets take at three shares. Then try p = 1/679, prompt v2’s rate.

Python code (you can change it)# Prompt v3's recordings: 88 of 679 answers were cut off (no text). p = 88 / 679 print(f"chance that one v3 ticket has no text: {p:.1%}") print(f"average canary tickets until the first one: {1 / p:.1f}") # The chance that the rule (one answer without text) is met # within the first n canary tickets: 1 - (chance that all n are fine). for n in (5, 10, 20, 50): print(f"within {n:>2} canary tickets: {1 - (1 - p) ** n:.1%}") # How long that takes depends on the share: 2 tickets a second in practice. for share in (0.01, 0.1, 0.5): per_minute = 2 * 60 * share print(f"share {share:.0%}: {per_minute:.1f} canary tickets a minute, " f"10 canary tickets in {10 / per_minute:.2f} minutes") # Try: p = 1 / 679 (prompt v2: one cut-off recording). How many # canary tickets would the rule need on average?

chance that one v3 ticket has no text: 13.0%
average canary tickets until the first one: 7.7
within  5 canary tickets: 50.0%
within 10 canary tickets: 75.0%
within 20 canary tickets: 93.8%
within 50 canary tickets: 99.9%
share 1%: 1.2 canary tickets a minute, 10 canary tickets in 8.33 minutes
share 10%: 12.0 canary tickets a minute, 10 canary tickets in 0.83 minutes
share 50%: 60.0 canary tickets a minute, 10 canary tickets in 0.17 minutes

Two lessons hide in these lines. First, the number of canary tickets that a rule needs depends on the failure, not on the share: about 8 for v3. Second, the share decides how fast those tickets arrive. A small share protects customers per minute, but you learn later.

With v2’s rate, 1 in 679, the average is 679 tickets. A rule of “one answer without text” would also fire on a healthy release, sooner or later. The recorded canary shows that too.

Try it on your service: shadow, canary, rollback

You run v3 first as a shadow, then as a canary for half of 20 practice tickets, then roll it back. Half, not 10%, so that a few practice tickets reach v3. Use your own release IDs from the last lesson.

1. Check the release state. A new database has none:

   python -m ops rollout status

   no release state: the worker uses RELEASE_ID (or ASSISTANT_PROMPT)

2. Make v2 the active release, then add v3 as a shadow. --by and --reason are saved with each change:

   python -m ops rollout start 1.5.0-5a4f87 --by Amira --reason "prompt v2 on the Module 5 code"
   python -m ops rollout shadow 1.5.0-422985 --by Amira --reason "prompt v3 beside v2, unseen"

   start: active 1.5.0-5a4f87. The worker takes it within 5 seconds.
   shadow: active 1.5.0-5a4f87, shadow 1.5.0-422985. The worker takes it within 5 seconds.

3. Start the three processes, one terminal for each, as before:

   python -m simulator
   fastapi dev
   python -m ticket_api.worker

4. In a fourth terminal, send 10 practice tickets:

   python -m scripts.practice --tickets 10 --rate 2

   Sent 10 practice tickets in 4.5 s: 202: 10

   The worker’s terminal logs the state when it reads it:

   INFO ticket_api.worker [-]: release state: State(active='1.5.0-5a4f87', canary=None, canary_share=0.0, shadow='1.5.0-422985')

5. Wait about 30 seconds, until the worker has done the 10 tickets. Then compare the shadow with the active release:

   python -m ops canary telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl --shadow

                                 shadow 1.5.0-422985  active 1.5.0-5a4f87
   ----------------------------  -------------------  -------------------
   tickets                       10                   10
   answers without text          2 (20.0%)            0 (0.0%)
   cost per 1,000 tickets (US$)  0.504                0.344
   triage call p95 (s)           9.01                 4.17
   agrees with reviews           -                    -
   Rollback rules met (first time):
     no answer    at +0.3 min: after 6 shadow tickets (5 tickets in all)

   Every ticket ran twice. v3 had two answers without text, and no customer saw them: v2 answered all 10.

6. Stop the shadow, and start a canary for half of the tickets:

   python -m ops rollout stop --by Amira --reason "shadow run finished"
   python -m ops rollout canary 1.5.0-422985 --share 0.5 --by Amira --reason "prompt v3 for half the practice tickets"

   stop: active 1.5.0-5a4f87. The worker takes it within 5 seconds.
   canary: active 1.5.0-5a4f87, canary 1.5.0-422985 for 50% of tickets. The worker takes it within 5 seconds.

7. Send 20 practice tickets, wait about 30 seconds, and compare:

   python -m scripts.practice --tickets 20 --rate 2
   python -m ops canary telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl

                                 canary 1.5.0-422985  active 1.5.0-5a4f87
   ----------------------------  -------------------  -------------------
   tickets                       10                   20
   answers without text          2 (20.0%)            0 (0.0%)
   cost per 1,000 tickets (US$)  0.452                0.357
   triage call p95 (s)           9.01                 4.61
   agrees with reviews           -                    -
   Rollback rules met (first time):
     no answer    at +1.0 min: after 4 canary tickets (20 tickets in all)

   10 of the 20 tickets went to the canary. The fourth canary ticket had no text, so the rule says: roll back. The “active” column also counts the 10 shadow-run tickets. The canary’s ticket count depends on your ticket IDs, so yours may differ.

8. Roll back, and say why:

   python -m ops rollout rollback --by Amira --reason "answers without text on canary tickets"

   rollback: active 1.5.0-5a4f87. The worker takes it within 5 seconds.

9. Prove the recovery. Wait 30 seconds, send 10 more tickets, wait again, and run the comparison:

   python -m scripts.practice --tickets 10 --rate 2
   python -m ops canary telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl

   The last line is new:

   After the last release change (+1.9 min: active 1.5.0-5a4f87, canary None): 10 tickets, 0 without text, US$0.343 per 1,000, triage p95 4.17 s, canary tickets 0

   After the change, no ticket went to v3, none was without text, and the cost and time are v2’s again.

10. Read the timeline of the rollout:

    python -m ops rollout history

    when      action    state after                                                  by     reason
    --------  --------  -----------------------------------------------------------  -----  ---------------------------------------
    +0.0 min  start     active 1.5.0-5a4f87                                          Amira  prompt v2 on the Module 5 code
    +0.0 min  shadow    active 1.5.0-5a4f87, shadow 1.5.0-422985                     Amira  prompt v3 beside v2, unseen
    +0.7 min  stop      active 1.5.0-5a4f87                                          Amira  shadow run finished
    +0.7 min  canary    active 1.5.0-5a4f87, canary 1.5.0-422985 for 50% of tickets  Amira  prompt v3 for half the practice tickets
    +1.4 min  rollback  active 1.5.0-5a4f87                                          Amira  answers without text on canary tickets

Stop the API and the worker with Ctrl+C when you are done. Every result has its release ID, so the database can also say who got what. In ai_runs, your run has 30 triage rows for v2, 10 triage rows for v3, and 10 triage_shadow rows for v3.

The rollback rules

The rules are in operations/rollout.toml, next to the canary’s share:

[canary]
share = 0.10
# Quality, cost and time are judged only after this many canary tickets.
minimum_tickets = 50

[rollback]
# Any one of these sends the canary back (python -m ops rollout rollback).
answers_without_text = 1      # canary tickets with no answer (a safety ticket may be one)
cost_increase = 0.20          # cost per ticket, against the active release (+20%)
triage_p95_increase = 0.50    # p95 of the triage call, against the active release (+50%)
agreement_drop = 0.05         # agreement with agents' reviews (needs 50 reviews of the canary)

Each rule compares the canary with the active release, by release ID, on the same hours of the same tickets. That is fairer than a fixed number: if all tickets get slower at noon, both releases slow down together.

Some rules need a minimum count. With 5 canary tickets, one slow answer can move the p95 a lot. So cost and time are judged only after 50 canary tickets, and agreement after 50 reviews. The answers-without-text rule needs no minimum, because a single missing answer can be a safety ticket. ops canary checks the first three rules. Agreement needs the agents’ reviews, which arrive later: ops canary prints it, and a person judges it.

Read the recorded canary

The course recorded the same change as a practice scenario on the full stack. It sent 2,400 tickets at 2 a second, with v3 as a canary for 10% from minute 5.0. The scenario rolled back 60 seconds after Grafana’s no-answer alert fired. The 60 seconds are an assumption: one minute for a person to read the alert and act. The answers are replayed recordings; the situation is constructed.

python -m ops canary data/telemetry/canary.jsonl.gz

                              canary 1.5.0-767b56  active 1.5.0-4fafd1
----------------------------  -------------------  -------------------
tickets                       25                   2,375
answers without text          3 (12.0%)            2 (0.1%)
cost per 1,000 tickets (US$)  0.479                0.340
triage call p95 (s)           7.86                 5.18
agrees with reviews           11/14                998/1053
Rollback rules met (first time):
  no answer    at +5.9 min: after 9 canary tickets (695 tickets in all)
After the last release change (+7.3 min: active 1.5.0-4fafd1, canary None): 1,533 tickets, 2 without text, US$0.341 per 1,000, triage p95 5.17 s, canary tickets 0

The release IDs are the course’s own manifests, so they differ from yours. Read the timeline:

Practice minute	What happened
5.01	ops rollout canary 1.5.0-767b56 --share 0.1
5.85	the 9th canary ticket has no text: the rule is met, 0.85 minutes after the start
6.19	Grafana’s no-answer alert fires, 1.19 minutes after the start
7.18	the rollback, 60 seconds later (assumed)

22 canary tickets started before the rollback, and 3 more were already in progress: 25 customers got a v3 answer. 3 of them got no text, and none of the 3 was a safety ticket. Compare “last week”: 1,798 v3 tickets and 186 without text.

The cost and time rules never judged: the canary had only 25 tickets, fewer than 50. Only the rule without a minimum could act in time, and it was enough.

The Grafana panels show the same story per release. The second release ID is the canary:

The recorded canary in Grafana: valid drafts and cost per release. The canary's lines stop at the rollback. Grafana assigns the colours, so your colours may differ.

The proof of recovery

“We rolled back” is an action. “The service recovered” is a claim, and it needs evidence. The last line of ops canary gives it: after the change, 1,533 tickets, 0 canary tickets, US$0.341 per 1,000 and a triage p95 of 5.17 s. Those are the active release’s numbers: 0.340 and 5.18 s.

The 2 answers without text after the rollback are not v3. They are v2’s one cut-off recording, practice ticket T-65450, which the scenario draws again and again. Because of it, the no-answer alert fired again on the healthy release, 10.5 minutes after the canary started. Runbooks in Module 6 deals with that noise.

Recovery evidence has three parts:

1. The state: ops rollout history shows the rollback, with who and why.
2. The traffic: no ticket after the change used the canary release.
3. The signals: the numbers after the change match the active release’s own numbers, not a hope.

Explore: the canary’s share

The share changes how fast the canary learns and how many customers it touches. The explorer computes other shares from the recorded run.

Explore

Canary share: when is prompt v3 caught?

Rule met (practice time)

the 9th canary ticket, after 0.75 min

Customers with a v3 answer before the rollback

24

Cost and time rules can judge after

4.2 min

Rule met at Larkfield’s speed (822 tickets a day)

2.6 h

At 10%, 12 canary tickets arrive per practice minute. The 9th one is the first without text, at 0.75 minutes. The rollback comes at 2.07 minutes, so about 24 customers get a v3 answer. At Larkfield’s real speed, the 9th canary ticket comes after about 2.6 h, and about 9 customers get a v3 answer: the person reacts long before the next canary ticket. The cost and time rules would need 50 canary tickets, at 4.2 minutes: the rollback comes first, so only the answers-without-text rule can act. The recorded run at 10%: the rule was met at 0.85 minutes, and 22 canary tickets started before the rollback.

• Computed canary tickets
• Recorded canary at 10% (open ring: answer without text)

Computed from the recorded canary: the replayed v3 outcomes in their recorded order (the 9th canary ticket is the first without text), canary tickets spread evenly, and 1.32 minutes from the rule to the rollback. The dots are the recorded run at 10%.

Text description of the explorer

The control chooses the canary’s share: 1%, 2%, 5%, 10%, 25% or 50%. The chart shows practice minutes after the canary started, from 0 to 10, and the number of canary tickets, from 0 to 90. A line rises at the share’s speed until the rollback, then stays flat. Two marks show when the rule is met and when the rollback comes. A dashed line at 50 tickets shows when the cost and time rules could judge. Dots show the recorded 10% canary, with rings on the 3 answers without text. At 10%, the rule is met at 0.75 minutes and about 24 customers get v3, close to the recorded 25. At 1%, the rule takes 7.5 minutes and about 10 customers get v3. At 50%, it takes 0.15 minutes and about 88 customers get v3. The readout also gives the time to the rule at Larkfield’s real speed of 822 tickets a day: 2.6 hours at 10%, 26.3 hours at 1%.

At practice speed, a bigger share catches v3 sooner but hurts more customers. The person needs 1.32 minutes to react, and many tickets arrive in that time. At Larkfield’s real speed, the canary needs about 9 customers whatever the share, because the person reacts long before the next canary ticket. There, a smaller share mainly means waiting longer for the answer.

The recorded shadow run

The course also recorded a shadow run: 600 tickets, v2 active, v3 in the shadow on every ticket.

python -m ops canary data/telemetry/shadow.jsonl.gz --shadow

                              shadow 1.5.0-767b56  active 1.5.0-4fafd1
----------------------------  -------------------  -------------------
tickets                       543                  588
answers without text          42 (7.7%)            0 (0.0%)
cost per 1,000 tickets (US$)  0.471                0.337
triage call p95 (s)           8.22                 4.94
agrees with reviews           -                    -
Rollback rules met (first time):
  no answer    at +0.3 min: after 10 shadow tickets (9 tickets in all)
  cost         at +0.6 min: after 50 shadow tickets (49 tickets in all)
  triage time  at +0.6 min: after 50 shadow tickets (49 tickets in all)

Three rules were met, and no customer saw v3. The shadow is the only way to judge cost and time on 50 tickets without customers. It has limits too:

• The shadow cannot be reviewed. Agents see v2’s decision, so “agrees with reviews” is empty.
• It costs double: every ticket made two triage calls, so the AI cost of these tickets doubled.
• A shadow call can fail without harm. 2 shadow calls timed out, the 12.1-second recording against the 10-second limit, and the tickets still got v2’s answer.
• The shadow had 543 results for 588 tickets. The shadow stopped with the scenario, and the tickets that the worker took after that got no shadow run.

When to use which, and what it costs

Step	Use it when	It cannot	It costs
Shadow	the candidate passed the gate, and you want live cost, time and failures with no customer at risk	measure what customers or agents think of the answers	a second model call for every ticket
Canary	the shadow looks fine, and you need real outcomes and reviews	protect the customers in the canary share	some customers meet the candidate; slower learning at a small share
Rollback rules	always, written before the start	judge before their minimum count	a few lines in rollout.toml, and someone on call to act

Omar asks, “Why not skip the shadow, to save money?” At the recorded shadow cost of US$0.471 per 1,000 tickets, one day of Larkfield’s 822 tickets costs about US$0.39 more. That is cheap for a prompt change. For an expensive model, a shadow on a share of the tickets may be enough.

Common mistakes

Common mistake

ops rollout: no active release

Symptom: python -m ops rollout canary 1.5.0-422985 --share 0.5 on a new database exits with code 2:

ops rollout: no active release: run python -m ops rollout start RELEASE first

Cause: a canary is always “against” an active release. Without one, there is nothing to compare with or to roll back to.

Fix: run python -m ops rollout start <your v2 ID> first.

Common mistake

ops rollout: no release manifest

Symptom: python -m ops rollout start 1.5.0-000000 exits with code 2:

ops rollout: no release manifest operations/releases/1.5.0-000000.toml

Cause: the tool refuses any release that has no manifest, or whose manifest does not match the files. That is how every rollout stays tied to versions.

Fix: use an ID from python -m ops release list, or build the manifest first.

Common mistake

No recovery line after the rollback

Symptom: right after ops rollout rollback, python -m ops canary telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl prints the canary table and the rules, but no line that starts with After the last release change.

Cause: the worker logs a release change when it takes its next job. With no new tickets, the telemetry has no rollback yet, and no ticket after it.

Fix: send tickets, wait for the worker, and run it again. “No data” is never proof of recovery.

Two more refusals protect you: --share 10 gives --share must be more than 0 and at most 1, and promote without a canary gives there is no canary to promote.

Knowledge check

Check your understanding

What is the difference between a shadow run and a canary? Give one thing that each one can measure and the other cannot.

Show the answer

In a shadow run, the candidate runs beside the active release on every ticket, and only the active release answers. It can measure cost, time and answers without text on many tickets with no customer at risk. A canary answers a share of the customers with the candidate. It can measure what customers and agents experience, such as the agreement of the agents’ reviews, which a shadow cannot have.

Check your understanding

Predict: the canary of a new release has 30 tickets so far. Its triage p95 is twice the active release's. Does ops canary report the time rule as met? Should the team act?

Show the answer

No. ops canary judges the time rule only after 50 canary tickets, so it does not report it yet. With 30 tickets, a few slow answers can double a p95. The team keeps the canary running and watches. If a safety ticket has no answer, that rule needs no minimum and acts at once. A person can still roll back early on judgement, with a reason in the history.

Check your understanding

Practical check: show that your rollback restored the service. Which three pieces of evidence do you give?

Show the answer

The state: python -m ops rollout history shows the rollback, by whom and why. The traffic: the last line of python -m ops canary … shows canary tickets 0 after the change. The signals: that line shows 0 without text and a cost and p95 that match the active release (in the recorded run: US$0.341 and 5.17 s against 0.340 and 5.18 s).

Recap

• A shadow run tests a candidate on every ticket with no customer at risk; a canary gives it a share of the customers.
• Rollback rules are written before the start, compare by release ID with the active release, and some need a minimum count.
• The recorded canary of v3 met the first rule at the 9th canary ticket, 0.85 minutes after the start. 25 customers got a v3 answer, against 1,798 “last week”.
• The share sets how fast the canary learns and, at practice speed, how many customers it touches.
• Proof of recovery has three parts: the state history, no canary traffic after the change, and signals back at the active release’s numbers.

Next: when the evidence says something is wrong, which change fixes it? A new prompt, a new index, a data fix or a new model?

Next: Retrain or revise