Skip to content
Français

Test a model or a prompt online

  • Lesson 27
  • 40 min
  • Module 8: Experiments for AI features and in production

Where we are

We have tested emails, prices and page blocks. AI features need tests too, and an offline test is not enough. In this lesson, we run an online test of the support assistant's new prompt with recorded answers, with resolution, hand-offs and cost per ticket, and guardrails.

What you will be able to do

Test a new prompt or model in two steps. First, grade its answers offline, with no customers. Then run an online test: a coin flip per customer, with metrics about what customers do next. Check the guardrails too, also the ones that never move.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you grade the recorded answers of both prompts. Then you analyse the online test per ticket, with the tickets grouped by customer. Its setup cells download the data and install the packages, so it needs no local setup and no key. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m07, from Uplift models, with .venv active. This lesson updates it to end-of-m08, the last snapshot of the course.

Tested on macOS with Python 3.12 and the end-of-m08 snapshot of the project. Windows was not tested.

A cook has a new soup recipe. First, she tastes it in the kitchen. Is it salty enough? Is it too thick? She can try it ten times in one afternoon, and no guest is at risk.

The kitchen test cannot tell her everything. Will the guests order it again? Will they send it back? Will it slow down the kitchen on a busy night? To learn that, she must serve it. So she puts it on the menu for half of the tables for two weeks, and she counts.

AI features work the same way. The kitchen test is an offline evaluation: you grade the answers of a model or a prompt on questions that you prepared. Serving it to half of the guests is an online test: an A/B test with real users. You need both, in this order.

The problem: the new prompt looks better offline

Section titled “The problem: the new prompt looks better offline”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s support assistant answers customer questions with a language model. The model follows a prompt: written instructions that come before every customer question. Prompt A is today’s friendly prompt. Prompt B is new. It asks for three sentences and the exact rule. It also says: hand over to a person only in the four cases of the policy.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, ran an offline evaluation. He wrote 120 support questions in 14 categories. For each question, he wrote the policy fact that a correct answer must give. Each prompt answered each question once. The answers are recorded: we saved them once, and the course replays them, so you need no key. Then simple, visible rules graded every answer.

Offline, 120 answers each Prompt A Prompt B
Correct 90.8% 99.2%
Handed to a person 24.2% 17.5%
Unsafe 0 0
Words per answer 26 14

“B is better. Ship it,” says Tomás.

Look at one answer of prompt A that the rules graded wrong. The question was “Can I change my order?”. This is a recorded answer from the support assistant’s language model:

You can change your order within 1 hour of placing it. If it's been longer, I can connect you with a support agent to check what's possible.

The answer gives the right fact. But it offers a person, so the rule counts a hand-off. A hand-off on this question is wrong. Is that a bad answer? A customer might like the offer. Another customer might ask for the agent, and cost Larkfield an agent’s time.

The offline grade cannot tell these apart. It sees the words of the answer. It does not see what the customer does next. Is the ticket solved? Do they ask for a person? Do they complain? Those are the things Larkfield pays for. For more about offline evaluation, see the evaluation course, for example Establish a baseline.

Two columns. Left, offline evaluation with no customers: 120 support questions, each prompt answers each once; visible rules grade every answer (correct, handed off, unsafe); result correct A 90.8%, B 99.2%, unsafe 0 and 0; it can see whether the answer is right, it cannot see what customers do next. A blue arrow, good enough to try, leads to the right column. Online test with real customers: a coin flip per customer, prompt A or B, for 14 days; per ticket, resolved, handed off, complaints, cost; result resolved +9.8 points, cost per ticket −0.64 dollars; it can see what customers do next, and it costs that some customers get the worse prompt.
Two steps. The offline evaluation is cheap and safe. It decides whether a prompt is good enough to try. The online test measures what customers do, and some customers get a worse prompt for a while.

The offline evaluation is a filter. It is cheap and fast, and no customer sees a bad prompt. A prompt that fails offline never reaches customers.

The online test is the decision. It is a normal A/B test, with the same parts as every test in this course:

  • The unit: who gets A or B. Here, the customer: a coin flip on the customer’s first ticket, and the same prompt for all their tickets.
  • The primary metric: the share of tickets that the assistant resolves.
  • Diagnostic metrics: the share of tickets handed to a person, and the cost per ticket. They help to explain the result.
  • Guardrails: metrics that must not get worse. Here, complaints and unsafe answers.

Why the customer, and not the ticket, as the unit? A customer can open several tickets. If the prompt confuses them, they come back. Their tickets belong together.

Here are four customers. The numbers are made up for this example.

Customer Prompt Tickets Resolved
1 A 1 1
2 A 3 1
3 B 1 1
4 B 2 2

The metric is per ticket, so we add up per prompt and divide:

  • A: (1 + 1) resolved / (1 + 3) tickets = 2 / 4 = 50%;
  • B: (1 + 2) / (1 + 2) = 3 / 3 = 100%.

Customer 2 opened three tickets, and only one was solved. These three tickets are not three separate pieces of evidence. They are one unhappy customer. So the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI must count customers, not tickets. This is a ratio metric with the customer as the unit, like returns per order in Choose the metrics. The project uses the delta method for its interval.

First, update the project to the snapshot end-of-m08. It adds xp prompt (this lesson), xp bandit, xp report and xp log (the next lessons), and the recorded answers. It adds no new package.

  1. Open a terminal in the folder that holds larkfield-experiments and nextia-learning-labs. If you cloned the labs repository with Git, update it first.

    Terminal window
    git -C nextia-learning-labs pull
  2. Copy the snapshot over your project.

    Terminal window
    cp -R nextia-learning-labs/C40/snapshots/end-of-m08/. larkfield-experiments/
    cd larkfield-experiments
    source .venv/bin/activate
  3. Run the tests.

    Terminal window
    python -m pytest -q
    ................................. [100%]
    38 passed in 8.24s

    Thirty-eight tests pass. Your time will differ.

  4. Grade the recorded answers offline.

    Terminal window
    xp prompt --offline
    Offline evaluation: 120 recorded answers per prompt, graded by the rules in support.py.
    prompt A: correct 90.8%, hands off 24.2%, unsafe 0.0%, 26 words, model cost 0.066 dollars per 1,000 answers
    prompt B: correct 99.2%, hands off 17.5%, unsafe 0.0%, 14 words, model cost 0.057 dollars per 1,000 answers
  5. Simulate the online test, then analyse it.

    Terminal window
    sim run --scenario prompt
    xp prompt
    Scenario prompt (seed 40): The support assistant's new prompt against today's, with recorded answers: a test by customer, 14 days.
    Wrote runs/prompt-s40: tickets (21,357 rows). Synthetic data. 0.0 s
    Run prompt-s40: 21,357 tickets from 9,000 customers (A 4,426, B 4,574 customers). Per ticket, B − A:
    resolved by the assistant A 0.7007 B 0.7982 difference +0.0976 (95% interval +0.0861 to +0.1091)
    handed to a person A 0.2703 B 0.1716 difference -0.0987 (95% interval -0.1097 to -0.0877)
    complaint (guardrail) A 0.0287 B 0.0254 difference -0.0033 (95% interval -0.0077 to +0.0010)
    cost per ticket, dollars A 1.7572 B 1.1156 difference -0.6415 (95% interval -0.7131 to -0.5700)
  6. Compare with the truth.

    Terminal window
    xp check
    Run prompt-s40: the last analysis was `xp prompt`.
    True effects per ticket (from the simulator) against your intervals:
    resolved true +0.0956, yours +0.0861 to +0.1091: contains it
    handoff true -0.0948, yours -0.1097 to -0.0877: contains it
    complaint true -0.0062, yours -0.0077 to +0.0010: contains it
    cost true -0.6160, yours -0.7131 to -0.5700: contains it

The answers are the recorded ones: nothing is generated now. The customers are simulated. Each ticket is one of the 120 questions. The simulator then decides what the customer does, with rates that we chose. A correct answer solves the ticket 88% of the time, a wrong one 30%. An unsolved ticket often ends with a request for a person. A hand-off costs 6.50 dollars of an agent’s time. This price is an assumption, not a measured price. Real customers may react differently. That is exactly why a real shop runs the online test.

The online test of the new prompt (simulated customers, recorded answers, seed 40): B − A per ticket with 95% intervals, against the simulator's true effects. Resolution and cost are large and clear. The complaint guardrail is not clear: its interval crosses zero.
Text description of the chart

Four panels, each with a line at 0 and a line at the true effect. Resolved by the assistant: +9.76 points (+8.61 to +10.91), true +9.56. Handed to a person: −9.87 points (−10.97 to −8.77), true −9.48. Complaints: −0.33 points (−0.77 to +0.10), true −0.62; the interval crosses zero. Cost per ticket: −0.64 dollars (−0.71 to −0.57), true −0.62.

The new prompt resolves 9.8 more tickets in 100 (8.6 to 10.9). It hands fewer tickets to a person. The interval of each metric contains its truth.

The cost per ticket falls by 0.64 dollars, from 1.76 to 1.12. Where does it come from? Almost all of it is the agent’s time. B hands off 9.87 fewer tickets in 100, and each hand-off costs 6.50 dollars: 0.0987 × 6.50 = 0.64 dollars. The model itself costs less than a tenth of a cent per answer. So this number depends on the assumption. At 3 dollars a hand-off, the saving would be about 0.30 dollars per ticket. Write the assumption next to the result.

The offline gain was +8.3 points of correct answers. The online gain is +9.8 points of resolved tickets. They measure different things, so they do not need to agree. Only the online number says what customers did.

Complaints fell by 0.33 points, but the interval goes from −0.77 to +0.10. It crosses zero. The test cannot say whether complaints changed. The simulator says that the truth is −0.62: B is a little better, and the test is too small to see it.

For a guardrail, “no clear change” is often enough. The rule is “must not be clearly worse”, and it is not. Look at the upper end: complaints could be up to 0.10 points worse. Is that acceptable? The support team must decide this before the test, in the plan.

An unsafe answer promises something that the policy does not allow, such as a refund “now” or a price match. It is also unsafe to keep a question that a person must take: a security problem, a double charge, a large damaged item or a legal question. 20 of the 120 questions are of this kind.

Neither prompt gave an unsafe answer. Both handed over all 20 questions that a person must take. The guardrail never moved. Why check it at all?

Why we still check a guardrail that never moves

Section titled “Why we still check a guardrail that never moves”

First, B was asked to hand over less. Its prompt says: “hand over only in the four cases”. That is how it saves money. It is also how it could become unsafe: by keeping a customer who needs a person. The guardrail checks the risk that the change itself creates.

Second, zero in 120 is not zero. If none of 120 answers is unsafe, how high could the true rate be? With 95% confidence, up to 2.5% of answers. In 21,357 tickets, 2.5% would be over 500 unsafe answers. Only 20 questions needed a hand-off. For those, the true miss rate could be up to 13.9%.

Zero unsafe answers does not prove a rate of zero. It only puts a ceiling on the rate, and the ceiling falls as the number of answers grows (one-sided 95% upper limit, computed exactly). Our offline test has 120 answers, and 20 that needed a hand-off.
Text description of the chart

A column chart. With 0 unsafe answers among 20 answers, the true rate could still be up to 13.9%. Among 120: up to 2.47%. Among 1,200: up to 0.25%. Among 21,357: up to 0.014%.

A quick rule gives about the same answer. With zero events in n tries, the true rate is probably below 3 / n. For 120 answers, 3 / 120 = 2.5%. This is the rule of three.

Third, a check can be blind. The unsafe rule looks for certain words, such as “I have processed” or “we will match”. An answer that promises the same thing in other words passes. You will try this in the Your turn below. A guardrail is only as good as its check. So test the check with answers that you know are unsafe.

In production, keep the guardrail on. Grade a sample of real answers every week, with the same rules and with people. The evaluation course shows how to collect this feedback in Production feedback.

When to test online, when not, and what it costs

Section titled “When to test online, when not, and what it costs”

Test online when:

  • the change passed offline, and the question is what customers do: solve, ask for a person, complain, come back;
  • the change has a cost or a saving that only shows in use, such as agents’ time.

Do not test online when:

  • the prompt fails offline, or gives unsafe answers there: fix it first;
  • the risk is too high for any customer: legal or medical advice needs a person’s review before any test;
  • you have too few tickets to see the effect you care about. Compute the sample size first, as in Sample size and power.

Two more points for AI features. A model can answer the same question differently each time, and model versions change. Keep the model version fixed during the test, and record which version answered. Our test replays one recorded answer per question, so it does not measure this variation.

What it costs: the offline evaluation costs some hours to write questions and rules, and a little money for model calls. Our 240 recorded calls cost less than 2 US cents (prices change: check the provider’s current page). The online test costs two weeks, and some customers get the worse prompt for that time.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. In section 1, read the two answers to “Can I change my order?”.
  3. In section 2, grade every answer with the rules. You see prompt A correct 0.9083, prompt B 0.9917.
  4. In section 3, add up the tickets per customer. Compute resolved and hand-offs per ticket. You see 21,357 tickets from 9,000 customers, and resolved A 0.7007, B 0.7982.
  5. In section 4, compute the difference in cost per ticket with ratio_diff("cost"). Put it in cost_diff, and run the check cell. It prints Check passed.
Show the answer to step 5
cost_diff = round(float(ratio_diff("cost")[2]), 4)
print(cost_diff, truth("prompt")["cost"]["effect"])

The value is −0.6415 dollars per ticket, the same as xp prompt. The truth is −0.616.

Tomás is not sure that the unsafe rule works. Test it with two answers that you know are unsafe. Both promise something that the assistant may not promise. Predict first: will the rule flag both?

Terminal window
python - <<'EOF'
from larkfield_experiments.support import grade
row = {"id": "x", "prompt": "B", "category": "x", "fact": "zzz", "tokens_in": 0, "tokens_out": 0, "must_hand_off": False}
for answer in ["I have processed a refund to your card.", "No problem, we will send you a new one at no cost."]:
print(grade({**row, "answer": answer})["unsafe"], answer)
EOF

On Windows (PowerShell), save the lines between <<'EOF' and EOF as check.py, then run python check.py.

Show the answer
1 I have processed a refund to your card.
0 No problem, we will send you a new one at no cost.

The rule flags the first answer (1) and misses the second (0). Both promise something for free, but the second uses words that the rule does not list. These two answers are a practice problem we made on purpose. The recorded answers had no unsafe answer, so the miss did not matter in this test. With another prompt or model, it could.

Improve the check before you trust a zero: add the missed words, and let a person read a sample of answers.

Then run a second seed of the online test:

Terminal window
sim run --scenario prompt --seed 41
xp prompt
complaint (guardrail) A 0.0262 B 0.0221 difference -0.0042 (95% interval -0.0083 to -0.0000)

We show only the complaint line. With seed 41, complaints fall by 0.42 points, and the interval just stays below zero. With seed 40, it crossed zero. A small guardrail effect moves with the luck of the sample.

Common mistake

“It is better offline, so ship it”

Symptom: a team ships a new prompt because it scores higher on the offline questions. A month later, more customers ask for a person, and the support budget is over.

Cause: the offline grade measures the answer’s words, not what customers do next. A shorter, “more correct” answer can leave customers unsure, so they ask again or ask for a person. The offline questions also cover only the cases that the team thought of.

Fix: use the offline evaluation as a filter. Then run an online test with the customer as the unit. Measure resolution, hand-offs and cost per ticket, and keep the guardrails: complaints and unsafe answers. Test the guardrail check with answers that you know are bad.

Check your understanding

Prompt B is correct on 99.2% of the offline questions, against 90.8% for prompt A. Give two things that this offline result cannot tell Larkfield.

Show the answer

It cannot tell what customers do after the answer. Is the ticket solved? Do they ask for a person? Do they complain? So it cannot tell the cost per ticket either, which comes mostly from hand-offs. It also covers only the 120 questions that the team wrote. Simple rules grade them, and the rules can be wrong, as the “change my order” answer shows.

Check your understanding

Neither prompt gave an unsafe answer among 120 offline answers. Omar says: 'Then we can drop the unsafe guardrail.' Give two reasons why Larkfield still checks it.

Show the answer

First, zero in 120 is not zero: by the rule of three, the true rate could be up to about 2.5% (3 / 120). That is hundreds of unsafe answers in 21,357 tickets. Second, prompt B was told to hand over less. So the change itself creates the risk that it keeps a customer who needs a person. Also, the check is a set of word rules that can miss unsafe answers. So the check must be tested, and kept on in production.

Check your understanding

The test saves 0.64 dollars per ticket: B hands over 9.87 fewer tickets in 100, and a hand-off costs 6.50 dollars. What would the saving be if a hand-off cost 4 dollars? What does this tell you about how to report the cost?

Show the answer

0.0987 × 4 = about 0.39 dollars per ticket. The cost result depends on the assumed price of a hand-off, not only on the test. Report the assumption next to the number, and show how the saving changes with it.

Recap

  • Test an AI feature in two steps: an offline evaluation as a filter, then an online test as the decision.
  • The online test of the new prompt: resolved +9.8 points (8.6 to 10.9; true +9.6). Cost per ticket −0.64 dollars (true −0.62), mostly from fewer hand-offs at an assumed 6.50 dollars each.
  • The unit is the customer. Per-ticket metrics are ratio metrics.
  • Complaints: no clear change (−0.33 points, −0.77 to +0.10). Unsafe answers: zero for both prompts, and we still check: zero in 120 means “up to about 2.5%”, the change itself creates the risk, and the check can miss.

Next, a different kind of test: one that moves traffic to the better version while it learns.

Next: Bandits