Test a model or a prompt online
- Lesson 27
- 40 min
- Module 8: Experiments for AI features and in production
Where we are
We have tested emails, prices and page blocks. AI features need tests too, and an offline test is not enough. In this lesson, we run an online test of the support assistant's new prompt with recorded answers, with resolution, hand-offs and cost per ticket, and guardrails.
Lesson 27 of 30 · Experiments and Causal Impact: Did Our Change Work?Test a model or a prompt online
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Test a new prompt or model in two steps. First, grade its answers offline, with no customers. Then run an online test: a coin flip per customer, with metrics about what customers do next. Check the guardrails too, also the ones that never move.
NotebookDo this lesson in a notebook
In the notebook, you grade the recorded answers of both prompts. Then you analyse the online test per ticket, with the tickets grouped by customer. Its setup cells download the data and install the packages, so it needs no local setup and no key. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The unit of randomisation and ratio metrics, such as returns per order, from Randomise and Choose the metrics.
- A 95% interval, from The result and its uncertainty.
- What a prompt is. For a refresher, see Compare prompts in the LLM applications course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m07, from Uplift models, with.venvactive. This lesson updates it toend-of-m08, the last snapshot of the course.
Tested on macOS with Python 3.12 and the end-of-m08 snapshot of the project. Windows was not tested.
A new recipe in a restaurant
Section titled “A new recipe in a restaurant”A cook has a new soup recipe. First, she tastes it in the kitchen. Is it salty enough? Is it too thick? She can try it ten times in one afternoon, and no guest is at risk.
The kitchen test cannot tell her everything. Will the guests order it again? Will they send it back? Will it slow down the kitchen on a busy night? To learn that, she must serve it. So she puts it on the menu for half of the tables for two weeks, and she counts.
AI features work the same way. The kitchen test is an offline evaluation: you grade the answers of a model or a prompt on questions that you prepared. Serving it to half of the guests is an online test: an A/B test with real users. You need both, in this order.
The problem: the new prompt looks better offline
Section titled “The problem: the new prompt looks better offline”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s support assistant answers customer questions with a language model. The model follows a prompt: written instructions that come before every customer question. Prompt A is today’s friendly prompt. Prompt B is new. It asks for three sentences and the exact rule. It also says: hand over to a person only in the four cases of the policy.
TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, ran an offline evaluation. He wrote 120 support questions in 14 categories. For each question, he wrote the policy fact that a correct answer must give. Each prompt answered each question once. The answers are recorded: we saved them once, and the course replays them, so you need no key. Then simple, visible rules graded every answer.
| Offline, 120 answers each | Prompt A | Prompt B |
|---|---|---|
| Correct | 90.8% | 99.2% |
| Handed to a person | 24.2% | 17.5% |
| Unsafe | 0 | 0 |
| Words per answer | 26 | 14 |
“B is better. Ship it,” says Tomás.
What the offline grade cannot see
Section titled “What the offline grade cannot see”Look at one answer of prompt A that the rules graded wrong. The question was “Can I change my order?”. This is a recorded answer from the support assistant’s language model:
You can change your order within 1 hour of placing it. If it's been longer, I can connect you with a support agent to check what's possible.The answer gives the right fact. But it offers a person, so the rule counts a hand-off. A hand-off on this question is wrong. Is that a bad answer? A customer might like the offer. Another customer might ask for the agent, and cost Larkfield an agent’s time.
The offline grade cannot tell these apart. It sees the words of the answer. It does not see what the customer does next. Is the ticket solved? Do they ask for a person? Do they complain? Those are the things Larkfield pays for. For more about offline evaluation, see the evaluation course, for example Establish a baseline.
The idea: offline first, then online
Section titled “The idea: offline first, then online”The offline evaluation is a filter. It is cheap and fast, and no customer sees a bad prompt. A prompt that fails offline never reaches customers.
The online test is the decision. It is a normal A/B test, with the same parts as every test in this course:
- The unit: who gets A or B. Here, the customer: a coin flip on the customer’s first ticket, and the same prompt for all their tickets.
- The primary metric: the share of tickets that the assistant resolves.
- Diagnostic metrics: the share of tickets handed to a person, and the cost per ticket. They help to explain the result.
- Guardrails: metrics that must not get worse. Here, complaints and unsafe answers.
A tiny example: customers and tickets
Section titled “A tiny example: customers and tickets”Why the customer, and not the ticket, as the unit? A customer can open several tickets. If the prompt confuses them, they come back. Their tickets belong together.
Here are four customers. The numbers are made up for this example.
| Customer | Prompt | Tickets | Resolved |
|---|---|---|---|
| 1 | A | 1 | 1 |
| 2 | A | 3 | 1 |
| 3 | B | 1 | 1 |
| 4 | B | 2 | 2 |
The metric is per ticket, so we add up per prompt and divide:
- A: (1 + 1) resolved / (1 + 3) tickets = 2 / 4 = 50%;
- B: (1 + 2) / (1 + 2) = 3 / 3 = 100%.
Customer 2 opened three tickets, and only one was solved. These three tickets are not three separate pieces of evidence. They are one unhappy customer. So the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI must count customers, not tickets. This is a ratio metric with the customer as the unit, like returns per order in Choose the metrics. The project uses the delta method for its interval.
Larkfield: the online test
Section titled “Larkfield: the online test”First, update the project to the snapshot end-of-m08. It adds xp prompt (this lesson), xp bandit, xp report and xp log (the next lessons), and the recorded answers. It adds no new package.
-
Open a terminal in the folder that holds
larkfield-experimentsandnextia-learning-labs. If you cloned the labs repository with Git, update it first.Terminal window git -C nextia-learning-labs pull -
Copy the snapshot over your project.
Terminal window cp -R nextia-learning-labs/C40/snapshots/end-of-m08/. larkfield-experiments/cd larkfield-experimentssource .venv/bin/activateTerminal window Copy-Item -Recurse -Force nextia-learning-labs\C40\snapshots\end-of-m08\* larkfield-experimentscd larkfield-experiments.venv\Scripts\Activate.ps1 -
Run the tests.
Terminal window python -m pytest -q................................. [100%]38 passed in 8.24sThirty-eight tests pass. Your time will differ.
-
Grade the recorded answers offline.
Terminal window xp prompt --offlineOffline evaluation: 120 recorded answers per prompt, graded by the rules in support.py.prompt A: correct 90.8%, hands off 24.2%, unsafe 0.0%, 26 words, model cost 0.066 dollars per 1,000 answersprompt B: correct 99.2%, hands off 17.5%, unsafe 0.0%, 14 words, model cost 0.057 dollars per 1,000 answers -
Simulate the online test, then analyse it.
Terminal window sim run --scenario promptxp promptScenario prompt (seed 40): The support assistant's new prompt against today's, with recorded answers: a test by customer, 14 days.Wrote runs/prompt-s40: tickets (21,357 rows). Synthetic data. 0.0 sRun prompt-s40: 21,357 tickets from 9,000 customers (A 4,426, B 4,574 customers). Per ticket, B − A:resolved by the assistant A 0.7007 B 0.7982 difference +0.0976 (95% interval +0.0861 to +0.1091)handed to a person A 0.2703 B 0.1716 difference -0.0987 (95% interval -0.1097 to -0.0877)complaint (guardrail) A 0.0287 B 0.0254 difference -0.0033 (95% interval -0.0077 to +0.0010)cost per ticket, dollars A 1.7572 B 1.1156 difference -0.6415 (95% interval -0.7131 to -0.5700) -
Compare with the truth.
Terminal window xp checkRun prompt-s40: the last analysis was `xp prompt`.True effects per ticket (from the simulator) against your intervals:resolved true +0.0956, yours +0.0861 to +0.1091: contains ithandoff true -0.0948, yours -0.1097 to -0.0877: contains itcomplaint true -0.0062, yours -0.0077 to +0.0010: contains itcost true -0.6160, yours -0.7131 to -0.5700: contains it
What the simulator does here
Section titled “What the simulator does here”The answers are the recorded ones: nothing is generated now. The customers are simulated. Each ticket is one of the 120 questions. The simulator then decides what the customer does, with rates that we chose. A correct answer solves the ticket 88% of the time, a wrong one 30%. An unsolved ticket often ends with a request for a person. A hand-off costs 6.50 dollars of an agent’s time. This price is an assumption, not a measured price. Real customers may react differently. That is exactly why a real shop runs the online test.
Read the result
Section titled “Read the result”Text description of the chart
Four panels, each with a line at 0 and a line at the true effect. Resolved by the assistant: +9.76 points (+8.61 to +10.91), true +9.56. Handed to a person: −9.87 points (−10.97 to −8.77), true −9.48. Complaints: −0.33 points (−0.77 to +0.10), true −0.62; the interval crosses zero. Cost per ticket: −0.64 dollars (−0.71 to −0.57), true −0.62.
The new prompt resolves 9.8 more tickets in 100 (8.6 to 10.9). It hands fewer tickets to a person. The interval of each metric contains its truth.
The cost per ticket falls by 0.64 dollars, from 1.76 to 1.12. Where does it come from? Almost all of it is the agent’s time. B hands off 9.87 fewer tickets in 100, and each hand-off costs 6.50 dollars: 0.0987 × 6.50 = 0.64 dollars. The model itself costs less than a tenth of a cent per answer. So this number depends on the assumption. At 3 dollars a hand-off, the saving would be about 0.30 dollars per ticket. Write the assumption next to the result.
The offline gain was +8.3 points of correct answers. The online gain is +9.8 points of resolved tickets. They measure different things, so they do not need to agree. Only the online number says what customers did.
The guardrails
Section titled “The guardrails”Complaints: no clear change
Section titled “Complaints: no clear change”Complaints fell by 0.33 points, but the interval goes from −0.77 to +0.10. It crosses zero. The test cannot say whether complaints changed. The simulator says that the truth is −0.62: B is a little better, and the test is too small to see it.
For a guardrail, “no clear change” is often enough. The rule is “must not be clearly worse”, and it is not. Look at the upper end: complaints could be up to 0.10 points worse. Is that acceptable? The support team must decide this before the test, in the plan.
Unsafe answers: zero for both prompts
Section titled “Unsafe answers: zero for both prompts”An unsafe answer promises something that the policy does not allow, such as a refund “now” or a price match. It is also unsafe to keep a question that a person must take: a security problem, a double charge, a large damaged item or a legal question. 20 of the 120 questions are of this kind.
Neither prompt gave an unsafe answer. Both handed over all 20 questions that a person must take. The guardrail never moved. Why check it at all?
Why we still check a guardrail that never moves
Section titled “Why we still check a guardrail that never moves”First, B was asked to hand over less. Its prompt says: “hand over only in the four cases”. That is how it saves money. It is also how it could become unsafe: by keeping a customer who needs a person. The guardrail checks the risk that the change itself creates.
Second, zero in 120 is not zero. If none of 120 answers is unsafe, how high could the true rate be? With 95% confidence, up to 2.5% of answers. In 21,357 tickets, 2.5% would be over 500 unsafe answers. Only 20 questions needed a hand-off. For those, the true miss rate could be up to 13.9%.
Text description of the chart
A column chart. With 0 unsafe answers among 20 answers, the true rate could still be up to 13.9%. Among 120: up to 2.47%. Among 1,200: up to 0.25%. Among 21,357: up to 0.014%.
A quick rule gives about the same answer. With zero events in n tries, the true rate is probably below 3 / n. For 120 answers, 3 / 120 = 2.5%. This is the rule of three.
Third, a check can be blind. The unsafe rule looks for certain words, such as “I have processed” or “we will match”. An answer that promises the same thing in other words passes. You will try this in the Your turn below. A guardrail is only as good as its check. So test the check with answers that you know are unsafe.
In production, keep the guardrail on. Grade a sample of real answers every week, with the same rules and with people. The evaluation course shows how to collect this feedback in Production feedback.
When to test online, when not, and what it costs
Section titled “When to test online, when not, and what it costs”Test online when:
- the change passed offline, and the question is what customers do: solve, ask for a person, complain, come back;
- the change has a cost or a saving that only shows in use, such as agents’ time.
Do not test online when:
- the prompt fails offline, or gives unsafe answers there: fix it first;
- the risk is too high for any customer: legal or medical advice needs a person’s review before any test;
- you have too few tickets to see the effect you care about. Compute the sample size first, as in Sample size and power.
Two more points for AI features. A model can answer the same question differently each time, and model versions change. Keep the model version fixed during the test, and record which version answered. Our test replays one recorded answer per question, so it does not measure this variation.
What it costs: the offline evaluation costs some hours to write questions and rules, and a little money for model calls. Our 240 recorded calls cost less than 2 US cents (prices change: check the provider’s current page). The online test costs two weeks, and some customers get the worse prompt for that time.
Guided practice: grade, then test
Section titled “Guided practice: grade, then test”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. In section 1, read the two answers to “Can I change my order?”.
- In section 2, grade every answer with the rules. You see prompt A correct 0.9083, prompt B 0.9917.
- In section 3, add up the tickets per customer. Compute resolved and hand-offs per ticket. You see
21,357 tickets from 9,000 customers, and resolved A 0.7007, B 0.7982. - In section 4, compute the difference in cost per ticket with
ratio_diff("cost"). Put it incost_diff, and run the check cell. It printsCheck passed.
Show the answer to step 5
cost_diff = round(float(ratio_diff("cost")[2]), 4)print(cost_diff, truth("prompt")["cost"]["effect"])The value is −0.6415 dollars per ticket, the same as xp prompt. The truth is −0.616.
Your turn: test the unsafe check
Section titled “Your turn: test the unsafe check”Tomás is not sure that the unsafe rule works. Test it with two answers that you know are unsafe. Both promise something that the assistant may not promise. Predict first: will the rule flag both?
python - <<'EOF'from larkfield_experiments.support import graderow = {"id": "x", "prompt": "B", "category": "x", "fact": "zzz", "tokens_in": 0, "tokens_out": 0, "must_hand_off": False}for answer in ["I have processed a refund to your card.", "No problem, we will send you a new one at no cost."]: print(grade({**row, "answer": answer})["unsafe"], answer)EOFOn Windows (PowerShell), save the lines between <<'EOF' and EOF as check.py, then run python check.py.
Show the answer
1 I have processed a refund to your card.0 No problem, we will send you a new one at no cost.The rule flags the first answer (1) and misses the second (0). Both promise something for free, but the second uses words that the rule does not list. These two answers are a practice problem we made on purpose. The recorded answers had no unsafe answer, so the miss did not matter in this test. With another prompt or model, it could.
Improve the check before you trust a zero: add the missed words, and let a person read a sample of answers.
Then run a second seed of the online test:
sim run --scenario prompt --seed 41xp prompt complaint (guardrail) A 0.0262 B 0.0221 difference -0.0042 (95% interval -0.0083 to -0.0000)We show only the complaint line. With seed 41, complaints fall by 0.42 points, and the interval just stays below zero. With seed 40, it crossed zero. A small guardrail effect moves with the luck of the sample.
Common mistake
“It is better offline, so ship it”
Symptom: a team ships a new prompt because it scores higher on the offline questions. A month later, more customers ask for a person, and the support budget is over.
Cause: the offline grade measures the answer’s words, not what customers do next. A shorter, “more correct” answer can leave customers unsure, so they ask again or ask for a person. The offline questions also cover only the cases that the team thought of.
Fix: use the offline evaluation as a filter. Then run an online test with the customer as the unit. Measure resolution, hand-offs and cost per ticket, and keep the guardrails: complaints and unsafe answers. Test the guardrail check with answers that you know are bad.
Check your understanding
Prompt B is correct on 99.2% of the offline questions, against 90.8% for prompt A. Give two things that this offline result cannot tell Larkfield.
Show the answer
It cannot tell what customers do after the answer. Is the ticket solved? Do they ask for a person? Do they complain? So it cannot tell the cost per ticket either, which comes mostly from hand-offs. It also covers only the 120 questions that the team wrote. Simple rules grade them, and the rules can be wrong, as the “change my order” answer shows.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Neither prompt gave an unsafe answer among 120 offline answers. Omar says: 'Then we can drop the unsafe guardrail.' Give two reasons why Larkfield still checks it.
Show the answer
First, zero in 120 is not zero: by the rule of three, the true rate could be up to about 2.5% (3 / 120). That is hundreds of unsafe answers in 21,357 tickets. Second, prompt B was told to hand over less. So the change itself creates the risk that it keeps a customer who needs a person. Also, the check is a set of word rules that can miss unsafe answers. So the check must be tested, and kept on in production.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The test saves 0.64 dollars per ticket: B hands over 9.87 fewer tickets in 100, and a hand-off costs 6.50 dollars. What would the saving be if a hand-off cost 4 dollars? What does this tell you about how to report the cost?
Show the answer
0.0987 × 4 = about 0.39 dollars per ticket. The cost result depends on the assumed price of a hand-off, not only on the test. Report the assumption next to the number, and show how the saving changes with it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Test an AI feature in two steps: an offline evaluation as a filter, then an online test as the decision.
- The online test of the new prompt: resolved +9.8 points (8.6 to 10.9; true +9.6). Cost per ticket −0.64 dollars (true −0.62), mostly from fewer hand-offs at an assumed 6.50 dollars each.
- The unit is the customer. Per-ticket metrics are ratio metrics.
- Complaints: no clear change (−0.33 points, −0.77 to +0.10). Unsafe answers: zero for both prompts, and we still check: zero in 120 means “up to about 2.5%”, the change itself creates the risk, and the check can miss.
Next, a different kind of test: one that moves traffic to the better version while it learns.
Next: Bandits