Why AI plans are different

What you will be able to do

Plan the first steps of AI work as experiments with a time limit. Nobody knows how well an AI feature works until it is tried on real examples. So a fixed launch date promises something that nobody can know.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

• The four labels on every number (recorded, price, assumption, constructed), from Read the evidence pack.
• Why a demo is not evidence, and why a trial uses the same examples for every option, from Run a fair pilot.

What you need open

• The pilot evidence pack of Larkfield’s support reply assistant, in the course’s practice materials: evidence/pilot-evidence-pack/README.md and 1-evaluation-results.md.

You read and decide. There is nothing to install or run.

A new dish for Saturday

You invite six friends to dinner on Saturday. You want to cook a dish that you have never cooked. You can promise the date of the dinner. You cannot promise that the dish will be good, because you have never tasted it.

A careful cook does two things. She cooks the dish once on Wednesday, for herself, and tastes it. And she keeps the ingredients for her usual pasta in the cupboard. If the Wednesday dish fails, Saturday’s guests get pasta, and nobody is hurt. The test on Wednesday has a fixed time and one question: “Is this dish good enough for guests?”

Now think of a bookshelf from a flat-pack box. You know that it will stand when you finish. The only question is how long it takes. Most software work is like the bookshelf. The team knows what “finished” looks like and knows that it can build it. The risk is in the time.

The problem: AI work is the new dish

An AI feature is like the new dish. Its quality is unknown until someone tries it on real examples, and the result can surprise everyone, including the experts.

Larkfield saw this with prompt v3, the new, shorter instruction that Omar, the product manager, wrote for the support reply assistant. You met it in Read the evidence pack. Omar expected v3 to be cheaper, because its request is shorter, and its replies to be warmer.

The team then ran today’s prompt v2 and the new v3 on the same 98 holdout tickets: test tickets kept apart for the decision. The table compares the expectation with the result. The results are recorded in the course Evaluating and Testing AI Systems.

	What the plan expected	Prompt v2 (today), recorded	Prompt v3, recorded
Words in the request (tokens, median)	fewer	1,902	1,477
Words the model wrote, with its “thinking” (tokens, median)	not considered	259	643
Model cost per 1,000 tickets	lower	US$0.34	US$0.48
Time to an answer (median)	not considered	2.8 s	5.6 s
Tickets with an answer	all	98 of 98	90 of 98

The request was shorter, as planned. But the model wrote more than twice as much before it answered, and the shop pays for every word that the model writes. So the “cheaper” prompt was dearer and slower. Worse, on all 271 test tickets, v3 gave no answer at all 25 times. Two of those tickets were about safety: a parasol that fell toward a child, and a hot, cracked pan. With no answer, no person was told.

Nobody on the team predicted this, and nobody could have. The result came from how one model behaved with one prompt on Larkfield’s tickets. No design document shows that. Only a test does.

Models change fast. These recorded numbers are a snapshot of one model on Larkfield’s made-up test tickets. The lesson that they teach lasts.

Why nobody knows in advance

There are three reasons, and the evidence pack shows each one.

The result depends on your examples. A model that writes good replies for one shop can fail on another shop’s tickets, policies and languages. A vendor’s demo and an engineer’s experience give a hint, not a result.

The same test gives different results. A model does not give the same answer every time. You saw this in What AI does well and badly: the team ran the same 102 development tickets five times with each prompt. Here are those runs again, as a planner reads them.

The same 102 development tickets, five runs per prompt. Each dot is one run. Recorded in the evaluation course.

Text description of the chart

Each dot is one run of the same 102 development tickets. “Both decisions right” means that the assistant chose the right team and decided correctly whether a person must handle the ticket. Prompt v2’s five runs scored 90, 96, 94, 92, 95. Prompt v3’s five runs scored 84, 81, 76, 80, 81. The two groups do not overlap, but each prompt moves by several tickets from run to run.

Prompt v3 was worse on every run. But look at the worst failure: a ticket that needs a person and gets no answer. It appeared in only 2 of the 5 runs (runs 1 and 3), on a different ticket each time (T-64740 and T-81036). A team that ran the test once could easily have seen no safety problem at all.

The world after launch is not the test. Real tickets bring new topics and new customers. The test tickets are made up, and one person labelled them. A good test result is the start of the evidence, not the end.

The idea: an experiment with a time limit

So the first steps of AI work cannot be “build feature X by date Y”. They are questions, such as “Can prompt v3 cut the model cost without worse decisions?” The plan answers the question with evidence, within a fixed time, and then decides.

This has a name. An experiment is a small piece of work that answers one question with evidence. A time box is the fixed time that the experiment may use. At the end of the time box, the team stops and decides, finished or not. There are three possible decisions: continue to the next step, change the approach and try again, or stop and keep today’s way of working. It is the cook’s Wednesday: a fixed time, one question, and pasta in the cupboard.

A fixed-date plan learns the quality at launch. An experiment learns it inside a time box, and the evidence decides the next step.

Text description of the diagram

The diagram has two rows. The top row is a plan for a feature with a fixed date. It has four steps: build the feature, make the answers better, launch on the date, and then, in red, “quality is known only now”. The bottom row is a plan for an experiment with a time limit. The first step, in green, is one question with the result that stops the work, written first. Then the team tries it on its own examples for a fixed time, and reads the evidence, also in green. Last comes a decision with three exits. “Continue” goes to the next stage. “Change” goes back to the question. “Stop” keeps today’s way of working.

Notice what the experiment fixes and what it leaves open. It fixes the time and the cost. It does not fix the result. A fixed-date plan does the opposite. It fixes the result (“launch”) and learns the quality last, when a change of course costs the most.

A tiny example: an experiment card for prompt v3

Before the test, the team could have written v3’s experiment on one card. The question, the evidence and the result come from the evidence pack. The two-week time box is constructed for this example.

Part of the card	Prompt v3
Question	Does v3 lower the model cost without worse decisions, and does every safety ticket still get an answer?
Time box	Two weeks (constructed)
Evidence	v2 and v3 on the same 102 development and 98 holdout tickets: cost per 1,000 tickets, decisions, tickets without an answer
Result that continues	Lower cost, decisions not worse, every ticket that needs a person answered
Result that stops	Any ticket that needs a person gets no answer
What happened (recorded)	Cost US$0.34 → US$0.48; a safety ticket without an answer. Stop.

The card cost two weeks and the model fees of a test. The stop cost nothing more, because prompt v2 kept working. Compare this with the evidence pack’s “last week”, a constructed situation with recorded answers. There, someone changed a setting and v3 answered every ticket, with no test and no decision. Ten tickets that needed a person reached nobody, and the dashboard showed 100.00% of tickets accepted. The Wednesday test fails in private. The Saturday failure reaches customers.

Discuss: can Nadia promise a date?

Nadia is Larkfield’s new head of customer operations. She owns the support team’s results, so she decides what changes in its tools. Before the evaluation, Omar sends her a plan (constructed): “Prompt v3 live for every ticket by the end of the month. The support team and finance want a date.”

• Omar: “Without a date, nothing happens. Finance plans the budget on it, and the agents need to know when the change comes.”
• Kwame, the solution architect: “We cannot promise quality. We can promise to know in two weeks.”
• Grace, the support lead: “The date matters less to me than one thing: every safety ticket still reaches a person.”
• Priya, the data scientist: “And one test run is not enough. The model gives different answers on different runs.”

Before you read on, answer these:

1. What can Nadia promise to finance and to the agents, and what can she not promise?
2. What should the first two weeks produce?
3. Which result should stop prompt v3, whatever the other results say?

The course’s answer. Nadia promises a decision date, not a launch date. In two weeks she will say “continue”, “change” or “stop”, with the evidence. Finance gets a date and a cost limit for the experiment. The agents hear that nothing changes until the decision. The two weeks produce the comparison on the same tickets, run more than once, with the stop result written first. One ticket that needs a person and gets no answer stops v3: that is Grace’s point, turned into a rule. With the recorded result, the decision is “stop”, and the shop loses only two weeks.

What would change the answer: if the change were not AI work at all, a normal date would be fine. New wording on a fixed help-page button is an example: its result is known before the work starts. A date also becomes safe later, after the experiments. Then the team knows the quality, and only the remaining work is uncertain.

When to use it, and what it costs

Use an experiment with a time box whenever the quality is unknown: a new model, a new prompt, new data or a new task. Plan ordinary work in the usual way, with dates: a form, a screen or a report around a model that is already proven. Most AI plans mix both. Put dates on the known parts and time boxes on the unknown parts.

Experiments have costs. Some people think that a plan without a launch date is weak. An experiment can end in “stop”, and then the time box produced only knowledge. The team also needs test examples and a way to score them before it starts. The evaluation course teaches how to build them. These costs are small next to a launch that fails in front of customers.

Common mistake

A time box that never ends

Symptom: the experiment’s report says “nearly ready, two more weeks” at every meeting. The status stays green, and the end date moves.

Cause: nobody wrote the result that stops the work, so no result can stop it. The time box became a wish.

Fix: at the end of the time box, the decision meeting happens, even if the work is not finished. “Not finished” is evidence too. Write the stop result before the next time box starts.

Consequence in AI work: a project without a stop result spends its whole budget before anyone decides. The case study Recover a stalled AI project shows six months of green reports and no launch.

Knowledge check

Check your understanding

Omar expected prompt v3 to be cheaper because its request is shorter. Why was it dearer, and what does this teach about planning AI work?

Show the answer

The model wrote more than twice as many words before it answered: 643 against 259 tokens (median). The shop pays for every word that the model writes. So v3 cost US$0.48 per 1,000 tickets against US$0.34. Nobody could see this in the design. Only a test on Larkfield’s tickets showed it. So the first step of AI work is an experiment that measures, not a feature with a fixed launch date.

Check your understanding

The team ran prompt v3 on the same 102 tickets five times. In how many runs did a ticket that needs a person get no answer? What would you ask a team that says 'we tested it, no problems found'?

Show the answer

In 2 of the 5 runs, on a different ticket each time (T-64740 and T-81036). One run could easily have shown no safety problem. Ask how many times they ran the test, on which examples, and which result would have stopped the change.

Check your understanding

A sponsor asks you for the launch date of a new AI feature whose quality nobody has measured yet. What do you promise instead, and why is that not a weaker answer?

Show the answer

Promise a decision date and a cost limit. For example: “In two weeks, I will tell you continue, change or stop, with the evidence.” The sponsor can plan on that date and that cost. A launch date for unknown quality is a promise that nobody can keep. If the feature fails, it fails in front of customers, as in Larkfield’s “last week”, where ten tickets that needed a person reached nobody.

Recap

• Nobody knows the quality of an AI feature before it is tried on real examples. Prompt v3 was planned as cheaper, and it was dearer (US$0.48 against US$0.34 per 1,000 tickets) and slower.
• A model gives different results on different runs. Prompt v3’s worst failure appeared in only 2 of 5 runs.
• So the first steps are experiments with a time box: one question, the stop result written first, then continue, change or stop.
• Promise a decision date, not a launch date, until the evidence exists.

Next, you join experiments into a roadmap: stages with a gate at the end of each, where the evidence decides.

Next: Milestones and roadmaps