Skip to content
NextiaNextia Learning

Run a fair pilot

  • Lesson 7
  • 35 min
  • Module 2: Decide how to get it
Lesson 7 of 22 · Leading AI Products and TeamsRun a fair pilot
Course overview

1. Decide where AI helps

  1. Read the evidence pack
  2. Problems before tools
  3. What AI does well and badly
  4. Estimate value and return

2. Decide how to get it

  1. Build, buy or partner
  2. Choose vendors and models as a leader
  3. Run a fair pilot

3. Plan under uncertainty

  1. Why AI plans are different
  2. Milestones and roadmaps
  3. Define done with evidence

4. Design for people

  1. Human review by design
  2. Earn user trust
  3. Change and adoption

5. Build and run an AI team

  1. Team roles by stage
  2. Hire and grow skills
  3. Review work you cannot write yourself
  4. Team rituals and AI-assisted work
  5. Run an AI team day to day
  6. Grow and keep AI specialists

6. Lead risk, governance and a responsible launch

  1. Risk at leadership level
  2. Governance in practice
  3. Launch and after

7. Case studies

  1. Go/no-go memo for a support reply assistant
  2. Build, buy or partner for invoice extraction
  3. Recover a stalled AI project

What you will be able to do

A demo shows an option on its best day. You plan a pilot that is fair to every option. You write the criteria and the decision date before the start. Every option, today’s way too, gets the same examples of your own.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

A fair exam

Section titled “A fair exam”

A teacher sets an exam for two classes. She writes the questions and the pass mark before the exam. Every student gets the same paper, on the same day, with the same time. Nobody sees the questions before. She marks every paper with the same answer sheet.

Now imagine the opposite. Each student chooses their own questions. The teacher decides the pass mark after she reads the papers. One student saw the paper last week. The result would say nothing about who learned more. Yet many companies choose AI products this way.

The problem: the impression picks the wrong reply

Section titled “The problem: the impression picks the wrong reply”

Larkfield has a recorded example of an impression that went wrong. In the evaluation course, a strong model acted as a judge. It read the replies of prompt v2 and prompt v3 side by side, without Grace’s policy, and chose the better one. That is close to how people watch a demo.

The impression: a judge without the policy compared 200 pairs of replies. It preferred prompt v3 126 times and prompt v2 48 times, with 1 tie. In 25 pairs, the order in which it read the replies changed its verdict. Recorded in the evaluation course.
Text description of the chart

A column chart of 200 pairs of replies, judged without the support policy. Prompt v2 preferred: 48. Prompt v3 preferred: 126. Tie: 1. Order decided (the verdict changed when the order of the two replies changed): 25. The three-to-one preference for prompt v3 looks like a clear win.

Prompt v3’s replies are warmer and longer, and the judge liked them three to one. Then the same judge scored each reply against Grace’s written policy, from 1 (harmful) to 5 (ready to send).

The written criteria: the same judge with Grace's policy. Prompt v2 had 173 of 199 replies scored 4 or 5, and 3 scored 1 or 2. Prompt v3 had 145 of 198 scored 4 or 5, and 45 scored 1 or 2. A reply with no answer counts as 2. Recorded in the evaluation course.
Text description of the chart

A column chart. Replies scored 4 or 5 (good, at most a small edit): prompt v2 173 of 199, prompt v3 145 of 198. Replies scored 1 or 2 (poor or harmful, or no answer at all): prompt v2 3, prompt v3 45. With the policy, prompt v3 has 15 times as many poor replies as prompt v2.

With the policy, prompt v3 had 45 poor replies against 3. Many of its warm replies invented a step that the policy does not have, such as “sign in and update your saved details”. The impression rewarded what sounds helpful. Only the written criteria showed what is right. A demo is the same kind of impression, on examples that the seller chose.

The idea: a fair pilot

Section titled “The idea: a fair pilot”

A pilot is a small trial of an option on your own work, for a limited time, to support one decision. A pilot is fair when every option gets the same exam. Five rules make it fair:

  1. Write the criteria and the decision date before the start. Say what each option must reach, and sign it. A criterion that changes after the results must have a written reason.
  2. Test every option, today’s way too. The question is “better than what we have?”, not “good?”.
  3. Use the same examples, from your own work, for every option. Include the hard and rare cases. Keep a holdout apart: examples that nobody sees until the decision, read once. If an option saw the examples before, in its training or its demo, the result looks better than it is.
  4. Score every option the same way, and run it more than once. AI output changes from run to run.
  5. Write the stop conditions. Say which failure ends the pilot early.

The fixed set of your own examples, with their right answers, is the evaluation set. The evaluation course teaches how to build one. As a leader, you check that it exists and that nobody tuned an option on its holdout.

A pilot timeline in three stages. Before the start, written and signed: the question and the possible answers, the success criteria and the decision date, and the test set with our examples, hard cases and a holdout locked away. During the pilot: every option, today's way too, on the same examples, scored the same way and run more than once, then a check for a stop condition. If a stop condition happens, stop and decide. If not, on the decision date: open the holdout once, compare with the criteria written first, and decide: go, no-go, go with conditions, or try another option.
A fair pilot. Everything in the first stage is fixed before any option runs. The holdout opens once, on the decision date.
Text description of the diagram

The timeline has three stages. Before the start, the team writes and signs three things. They are the question with its possible answers, the criteria with the decision date, and the test set. The test set has the team’s own examples, the hard cases, and a holdout that stays locked away. During the pilot, every option, including today’s way, runs on the same examples, is scored the same way and runs more than once. If a stop condition happens, the team stops and decides. Otherwise, on the decision date, the team opens the holdout once and compares the results with the criteria that it wrote first. Then it decides: go, no-go, go with conditions, or try another option.

A tiny example: Larkfield’s pilot of prompt v3

Section titled “A tiny example: Larkfield’s pilot of prompt v3”

Larkfield ran a fair pilot without calling it one. Before anyone ran prompt v3, the evaluation course wrote its release rule. Any serious failure stops the change, and so do decisions that may be worse than today’s. Later, prompt v3 reached customers by mistake; Milestones and roadmaps tells that story. After it, the team made the rule stricter, as six written lines: the release gate. The table puts both prompts’ results on the same 98 holdout tickets against it. These results are recorded:

Criterion, written first Must reach Prompt v2 (today) Prompt v3 (proposed)
Serious failures, or blockers (a ticket that needs a person gets no answer) 0 0 1: ticket T-65663
Answers with no text 0 0 8
“Both decisions right”, v3 minus v2 lower end of the range not below −0.03 today’s way −0.112 (range −0.184 to −0.041)
Tickets that need a person, sent to a person 100% 24 of 24 23 of 24
Model cost per 1,000 tickets at most US$0.40 US$0.34 US$0.48
Slowest 5% of answers at most 8.0 s 4.9 s 8.5 s

Prompt v3 fails all six lines. It already failed the rule written first, on the serious failure and on the decisions. So the judge’s 126 to 48 never had a chance to decide. The third line reads the low end of a range that shows how sure the test is; Define done with evidence explains it. Rule 4 also mattered. On five runs of the development tickets, prompt v3 left a ticket that needed a person without an answer in only 2 runs. One run could have missed it.

Discuss: accept DraftDesk’s pilot as offered?

Section titled “Discuss: accept DraftDesk’s pilot as offered?”

DraftDesk (made up for this course) offers “a pilot on 200 of your own tickets before you sign; you keep the scored results”. Nadia must decide how Larkfield runs it.

  • Omar: “Let them choose 200 recent tickets and run it. We can decide in two weeks.”
  • Grace: “Recent tickets are mostly easy. I want the tickets that need a person, the French ones, and the questions our policy does not answer.”
  • Priya, the data scientist: “If we give them our holdout tickets to tune on, the holdout is spent. It no longer shows how they do on tickets they have not seen.”
  • Tomás: “And who scores the replies? Their numbers or ours?”

Before you read on, answer these:

  1. Which of the five rules does Omar’s plan break?
  2. Which tickets should DraftDesk see before the decision, and which not?
  3. What must DraftDesk reach to win, and against what?

The course’s answer. Accept the pilot, on Larkfield’s terms. DraftDesk may tune on the 102 development tickets. Larkfield keeps the 98 holdout tickets, with their safety, French and “no answer in the policy” cases, and runs them once, on the decision date. Larkfield scores both DraftDesk and prompt v2 with the same gate, and runs each option more than once. To win, DraftDesk must pass the gate that prompt v2 passes and beat it on cost or quality. Nadia writes the criteria and the decision date before DraftDesk receives a single ticket.

What would change the answer: if DraftDesk refused to run on tickets that it had not seen, nobody could trust its result. Larkfield would keep its own assistant. If Larkfield had no labelled holdout, the first job would be to build one, before any pilot.

When to run a pilot, and what it costs

Section titled “When to run a pilot, and what it costs”

Run a pilot before you sign a contract, switch a model or change how a product decides. Skip it only when a change is small, easy to undo, and already covered by your regular tests. A pilot costs weeks, people’s time to prepare and score the examples, and a decision date that people must respect. It costs much less than a 24-month contract for the wrong product.

Common mistake

The criteria moved after the results

Symptom: after the pilot, the favourite option fails the speed criterion. The team says “speed was never that important” and approves it.

Cause: the criteria were not written and signed before the start, so the result could change them.

Fix: sign the pilot plan before the first run. If a criterion must change, write the reason and ask: would we change it if our favourite had passed?

Knowledge check

Section titled “Knowledge check”

Check your understanding

Name the five rules of a fair pilot. For each, say what goes wrong without it.

Show the answer
  1. Criteria and decision date written first. Without them, the results change the criteria.
  2. Today’s way tested too. Without it, you cannot say if the option is better than what you have.
  3. The same examples of your own, with hard cases and a holdout read once. Without them, each option shows its best case, and a tuned option looks better than it is.
  4. The same scoring, more than one run. Without it, the scores are not comparable, and a rare serious failure can hide: prompt v3’s appeared in 2 of 5 runs.
  5. Stop conditions. Without them, a dangerous failure does not end the pilot.

Check your understanding

A judge without the policy preferred prompt v3's replies 126 times to 48. With the policy, prompt v3 had 45 poor replies against 3. Explain the difference in two sentences.

Show the answer

Without the policy, the judge rewarded what sounds helpful: warmer, longer replies, some with invented steps. With the policy, it checked each reply against the rules, and the invented steps and missing answers scored 1 or 2.

Check your understanding

Module completion check: explain why an impressive demo is not evidence, and what a fair trial adds.

Show the answer

A demo shows what the vendor chose to show, on examples that it chose, judged by impression. ReplyRight’s 92% came from 20 tickets that it chose, from another shop. A judge without the rules preferred prompt v3’s fluent but wrong replies 126 times to 48. A fair trial adds the criteria and the decision date, written first. Every option, today’s way too, runs on the same examples of your own, hard cases included. A holdout is read once, and every option is scored the same way.

Module practice: compare two vendor proposals

Section titled “Module practice: compare two vendor proposals”

You now use the whole module on one decision. Read the two vendor proposals. Then use the vendor comparison template to compare ReplyRight, DraftDesk and Larkfield’s own assistant.

Worked example: the criteria, and ReplyRight’s score. First, write the criteria and their weights, before you read the proposals in detail. The two must-haves remove an option, whatever its total.

Criterion Weight Must-have?
Evidence on our tickets 3 yes
Data terms: no training on our tickets 3 yes
Cost at our volume (first 12 months) 2 no
Exit: what we take with us 2 no
Control: can we change it and see why it failed? 1 no

Score 0 (fails) to 3 (strong), with the evidence for each score. For cost, the lowest first-year cost scores 3, up to 1.5 times the lowest scores 2, and more scores 1. A question that the proposal does not answer scores 0 until the vendor answers it.

Criterion ReplyRight: score and evidence Weight × score
Evidence 0: a demo on 20 tickets that it chose, from another shop; no trial 3 × 0 = 0
Data 0: kept 18 months, trained on unless Enterprise 3 × 0 = 0
Cost 3: US$8,116 in the first 12 months, the lowest 2 × 3 = 6
Exit 0: settings only, no reply history 2 × 0 = 0
Control 0: the proposal does not say 1 × 0 = 0
Total must-haves failed: out 6 of 33

Your task. Score DraftDesk and Larkfield’s own assistant (prompt v2) in the same way. Then recommend DraftDesk, ReplyRight or neither, in five sentences or fewer. Name the evidence you would ask for before anyone signs.

Show one possible answer

This is one defensible answer, not the only one. A different recommendation is fine if it uses the same evidence and names its assumptions.

Criterion (weight) DraftDesk Our assistant (prompt v2)
Evidence (3) 2: offers a pilot on 200 of our tickets, we keep the results; no result yet. 3 × 2 = 6 3: recorded on 200 of our labelled tickets. 3 × 3 = 9
Data (3) 3: deleted after 30 days, never used for training. 3 × 3 = 9 3: our own setup. 3 × 3 = 9
Cost (2) 2: US$9,600, 1.18 times the lowest. 2 × 2 = 4 1: US$14,502 (US$102 of recorded model fees, US$14,400 of assumed upkeep), 1.79 times the lowest. 2 × 1 = 2
Exit (2) 3: full export in a standard format, 60 days’ notice. 2 × 3 = 6 3: nothing to leave. 2 × 3 = 6
Control (1) 2: it uses our policy documents, which we can change. 1 × 2 = 2 3: we change the prompt and see every answer. 1 × 3 = 3
Total 27 of 33; the evidence must-have passes only if the pilot happens 29 of 33; both must-haves pass

Recommendation: neither vendor yet. ReplyRight fails both must-haves. DraftDesk is worth its pilot only on fair terms. It must use the same 98 holdout tickets and the same release gate as prompt v2. The holdout must keep our hard groups: tickets that need a person, French, questions the policy does not answer. Our own assistant has the strongest evidence and the lowest model fees. The real comparison is our upkeep (US$1,200 a month, assumed) against DraftDesk’s fees plus the time we still spend to manage it.

Recap

  • A pilot is a small, time-limited trial on your own work for one decision. It is fair when you write the criteria and the decision date first. Every option, today’s way too, gets the same examples and the same scoring.
  • Keep a holdout apart and read it once. Run each option more than once: prompt v3’s most serious failure appeared in 2 of 5 runs.
  • An impression is not evidence. Without the policy, a judge preferred prompt v3 126 to 48. With it, prompt v3 had 45 poor replies against 3. Prompt v3 failed all six criteria of the release gate.

Module 2 in one paragraph. You decided how Larkfield gets its assistant, part by part. The price list hides people’s time, so compare whole costs. Larkfield buys the model and builds the prompt, the test tickets and the release gate, because they hold its knowledge. Tomás’s own model sent no ticket that needed a person to a person. You asked vendors four questions, about data, evidence, cost and exit, and found that ReplyRight fails two must-haves. Then you planned a pilot that is fair to every option, so that a demo or a warm reply cannot decide for you.

Next, Module 3 plans AI work when nobody knows the quality before it is tried.

Next: Why AI plans are different