Milestones and roadmaps

What you will be able to do

Plan AI work as a series of stages. Each stage ends at a gate, where the evidence decides to continue, change or stop. Then a failure shows up early and small, not at launch in front of every customer.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• Experiments, time boxes and the three decisions (continue, change, stop), from Why AI plans are different.
• Success criteria written before a pilot starts, from Run a fair pilot.

What you need open

In the course’s practice materials:

• templates/stage-gate-roadmap.md, the template that you fill in.
• evidence/pilot-evidence-pack/4-operations.md, what happened when prompt v3 reached real tickets.
• evidence/fixed-date-plan.md, a plan with fixed dates. You turn it into gates in the module’s practice, at the end of the next lesson.

How a dish gets onto the menu

A chef wants to add a new dish to her restaurant’s menu. She does not print it on the menu on the first day. First she cooks it in the kitchen and tastes it: can she make it well at all? Then she serves it to her staff at their meal: do people who are not the chef like it? Then she offers it as a special at a few tables for a week. Do paying guests order it? Can the kitchen cook it fast on a busy night? Only then does it go on the printed menu.

Each step asks a different question, and each one puts more people in front of the dish. A bad answer at any step keeps the dish off the menu. Most failures happen early, in the kitchen, where they cost one plate.

The problem: a plan that goes straight to everyone

Look at the practice file fixed-date-plan.md. It is a constructed plan for a product chat on Larkfield’s website. Here are four of its weeks:

Week	Milestone	Done when
1–2	Choose the model and build the chat	The chat answers on the test website
5–6	Make the answers better	The team is happy with the answers
7	Design and accessibility review	The chat looks like the website
9	Launch to all customers	The chat is on every product page

Every milestone is a piece of work, with a date. No week asks whether the chat answers well enough, and no result can stop it. In week 9, every customer meets the chat at the same moment. If it is wrong about a product’s safety, everyone can read the wrong answer.

Larkfield already saw what this looks like. In the evidence pack’s “last week”, a constructed situation with recorded answers, someone switched prompt v3 on for every ticket at once. Of 1,798 tickets, 186 got no answer, and 10 tickets that needed a person reached nobody. The dashboard still said that the service accepted 100.00% of tickets. The first people to test the change were the customers.

The idea: stages and gates

A stage is a period of work with one question and a time box. A gate is the meeting at the end of a stage. There, one owner reads the evidence and decides: continue to the next stage, change and repeat it, or stop. A stop criterion, or stop result, is the result, written before the stage starts, that ends or changes the work. It is the same idea as a pilot’s stop conditions in Run a fair pilot. The word “gate” fits: like a gate on a path, it stays closed until the evidence opens it.

A roadmap is the stages in order. For AI work, four stages answer four questions, and each one puts the work in front of more people:

1. Feasibility. Can AI do this task on our own examples at all? Nobody outside the team sees it.
2. Pilot. Does it work on real work, with review, at an acceptable cost? A few users, or nobody, see it.
3. Limited launch. Does it work in operation, for a small share of the real work? Some customers see it.
4. Full launch. Does it keep working for everyone, and do we watch the right signals? Everyone sees it.

This is the chef’s path: kitchen, staff meal, specials, menu.

A tiny example: one stop criterion at work

The evidence pack’s operations file has a second, careful version of the v3 change. This time v3 answered only 1 ticket in 10: a limited launch, called a canary. The name comes from the canary birds that miners once carried underground to warn them of bad air. Before it started, the team wrote its stop criterion. An automatic alarm watches the tickets without an answer, and a rule switches back to v2 when the alarm fires. The numbers below are recorded answers in a constructed situation.

Step	Value
Share of tickets that v3 answered	10%
v3 tickets until the alarm rule was met	9
v3 tickets until the switch back to v2	22
v3 tickets without an answer	3 of 25

The same failure as “last week” appeared. But the stop criterion was written first, so the stage ended after 22 tickets, not after 1,798. The gate did not need a meeting to say “stop”: the rule said it.

The chart puts the three ways of trying v3 side by side. The runs had different lengths, so read the bars as “who met the failure”, not as rates.

The same failure, met by 0, 3 or 186 customers' tickets. Recorded answers in constructed situations, from the MLOps course.

Text description of the chart

Three bars. In the hidden test (a shadow test), v3 answered 543 real tickets in the background and left 42 without an answer. Only v2’s answers were used, so 0 customers’ tickets were affected. With 1 ticket in 10 (the canary), 3 of its 25 v3 tickets got no answer, and the rule switched back to v2. With everyone at once, 186 of 1,798 tickets got no answer, and 10 tickets that needed a person reached nobody. The data come from the course MLOps and LLMOps: Operating AI in Production.

A shadow test is a pilot where the new version answers real work in the background and nobody uses its answers. It found the problem with no customer touched. Its price is the model fee twice while it runs: about US$0.47 per 1,000 tickets on top of the real answers (recorded). That is why the cheap questions come first. A failure at feasibility costs a few weeks. A failure at full launch costs customers.

Larkfield’s roadmap for the next prompt change

Prompt v3 stopped. But Omar will propose another prompt change one day. Nadia, the head of customer operations, asks the team to fill the stage-gate template now, before anyone has a result to defend. The limits come from recorded evidence; the time boxes are constructed. Nadia decides at every gate.

Stage	Question	Time box	Evidence to pass the gate	Result that stops or changes the work
1. Feasibility	Can the new prompt do the task on our test tickets?	2 weeks	Every line of the release gate on the development tickets, then once on the holdout tickets; five runs	One ticket that needs a person gets no answer, in any run
2. Pilot (shadow)	Does it work on real tickets, in the background?	1 week	No more tickets without an answer than prompt v2 (4 of 5,754 in a normal week); Grace reads a sample of drafts and finds no invented rule	One safety ticket without an answer
3. Limited launch (canary)	Does it work for 1 ticket in 10?	1 week	Agents agree with its decisions as often as with v2’s (94.3% in a normal week)	The alarm rule on tickets without an answer fires: switch back at once
4. Full launch	Does it keep working for everyone?	Ongoing	Every safety ticket reaches a person (320 of 320 in a normal week); agreement and cost per 1,000 tickets watched	One ticket that needs a person reaches nobody: switch back, then review

The diagram below says “no blocker”. A blocker is a failure that stops the change on its own, however good the other results are. An example is a ticket that needs a person and gets no answer. The next lesson uses blockers in its thresholds.

Larkfield's stage-gate roadmap for the next prompt change. Each stage shows more customers the change, and each has its pass evidence and its stop result written first.

Text description of the diagram

The roadmap has four stages in a row, joined by three gates. Each stage has three boxes. They hold its question, the evidence to pass in green, and the result that stops the work in red. Stage 1, feasibility, asks whether the new prompt can do the task on the test tickets; no customer sees it. It passes on all six lines of the release gate over five runs with no blocker. It stops if a ticket that needs a person gets no answer. Stage 2, the pilot, runs on real tickets in the background; no customer sees it. It passes if there are no more tickets without an answer than today and Grace’s sample is fine. It stops on one safety ticket without an answer. Stage 3, the limited launch, gives the new prompt 1 ticket in 10, so a few customers see it. It passes if agents agree with it as often as today, and it stops when the alarm rule fires. Stage 4, full launch, reaches every customer. The team watches that every safety ticket reaches a person, and pauses the change if one ticket that needs a person reaches nobody.

Read the roadmap from left to right, and two patterns appear. The evidence gets closer to real customers at each stage: test tickets, then real tickets in the background, then real customers. And the stop results get faster: a meeting can stop feasibility, but the canary needs an automatic rule, because customers wait. The next lesson turns “every line of the release gate” into exact thresholds.

Discuss: can the team skip the pilot?

Months later, Omar proposes the next prompt change. It passes the feasibility gate on every line. He writes to Nadia (constructed): “The evaluation is clean. Let’s skip the shadow week and go straight to 1 ticket in 10.”

• Omar: “A shadow week costs a week, and it shows us nothing that the evaluation did not.”
• Kwame: “The shadow costs us about US$0.47 per 1,000 tickets on top of the real answers. That is nothing. The week is the real cost.”
• Grace: “A canary touches real customers. A shadow touches none.”
• Priya: “Our test tickets are made up. Real tickets bring topics that the test set does not have.”

Before you read on, answer these:

1. Is it ever right to skip a stage? What would make it safe?
2. What can a shadow test measure, and what can it not measure?
3. Who makes this decision, and when should it have been made?

The course’s answer. A stage can be short, but its gate stays. Keep the shadow week: it is the only stage that meets real tickets with no customer at risk, and its money cost is tiny. Priya’s point is the reason: a clean result on made-up tickets says nothing about new topics. But Omar is right about one limit. In a shadow, no agent reviews the drafts, so it cannot measure agents’ agreement. That is the canary’s job, so the canary keeps its alarm rule. Nadia decides, and the roadmap should have settled it before any result existed. Changing the plan after a good result is how a stage disappears.

What would change the answer: a change with no new model behaviour, such as a typo fixed in a reply rule, could skip the shadow. So could a product where the cost of an error is low and every answer is checked by a person anyway. Write the reason down, with the name of the person who decided.

When to use it, and what it costs

Use stage gates for any AI feature where a wrong answer can hurt a customer, cost money or break a promise. Use fewer, shorter stages for a small internal tool whose errors are cheap and easy to see. Merge stages there, but keep a written stop result.

Gates cost something. Each gate needs evidence that someone must prepare, and a meeting with the owner. The first launch comes later than in a fixed-date plan. A stop also “wastes” the stage before it. In return, most failures happen at feasibility or in the pilot, where they cost weeks and no customer sees them. The operations side of a staged launch is in the course MLOps and LLMOps: Operating AI in Production. The conditions that pause a launch are in Launch and after.

Common mistake

A gate that can only say yes

Symptom: every gate passes, and every monthly status is green. Then the work stalls just before launch.

Cause: the gate had no stop result and no fixed evidence. Each month the team chose the number to show, so every result looked like progress. The gate became a status meeting.

Fix: write the evidence to pass and the stop result in the roadmap before the stage starts. Keep the same measure from stage to stage. At each gate, ask “what result would have stopped us?” If nobody can answer, it is not a gate.

Consequence in AI work: in the case study below, six months of reports were green. Meanwhile the reported measure changed from “accuracy 81%” to “AUC 0.91”, a different measure on a different scale. Nobody can compare the numbers, and nobody ever decided to stop.

See it end to endRecover a stalled AI projectCase study

The case study’s 90-day recovery plan is a stage-gate roadmap, written for a project that ran six months without one.

Knowledge check

Check your understanding

Name the four stages of an AI roadmap, and for each one say who sees the work.

Show the answer

Feasibility: only the team; the work runs on test examples. Pilot: a few users, or nobody in a shadow test, on real work. Limited launch: a small share of real customers, for example 1 ticket in 10. Full launch: everyone. Each stage puts the work in front of more people, so the cheap questions come first.

Check your understanding

In the careful canary, prompt v3 failed in the same way as in 'last week'. Why did only a few customers' tickets meet the failure?

Show the answer

Two reasons. Only 1 ticket in 10 went to v3. And the stop criterion was written before the canary started: an alarm on tickets without an answer, and a rule to switch back. The alarm rule was met after 9 canary tickets, and the switch back came after 22. Three of 25 canary tickets got no answer, against 186 of 1,798 when v3 went to everyone at once.

Check your understanding

A team says: 'Our gate passed, as it does every month.' What one question shows whether it is a real gate?

Show the answer

“Which result, written before the stage, would have stopped us?” A real gate has a stop result and fixed evidence, chosen before the result. If nobody can name the stop result, the gate can only say yes. It is a status meeting.

Recap

• A stage has one question and a time box. A gate at its end lets one owner decide continue, change or stop, from evidence. A stop criterion is written before the stage starts.
• AI work has four stages: feasibility, pilot, limited launch, full launch. Each one shows the work to more people.
• The same v3 failure met 0 customers’ tickets in a shadow test and 3 in a canary with a stop rule. It met 186 when v3 went to everyone at once.
• Larkfield’s roadmap for the next prompt change has pass evidence and a stop result at every stage, and Nadia decides at every gate.

Next, you turn “every line of the release gate” into thresholds on an agreed evaluation set, with one owner who signs.

Next: Define done with evidence