Testing a calculator, testing a new colleague
- Lesson 1
- 25 min
- Module 1: How AI features fail
Where we are
You already know how to test ordinary software: you give an input and compare the result with one expected result. That works because the code gives the same answer every time. An AI assistant does not: ask it the same question twice and you can get two different replies, and both can be right. In this lesson, we see what to check instead of one expected text.
Lesson 1 of 24 · Testing AI Applications: Manual and AI-Assisted QATesting a calculator, testing a new colleague
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Learn why a test of an AI feature cannot compare the reply with one expected text, and what to check instead. A feature can pass once and still be wrong one time in three.
Before you startWhat to know and to have ready
What you need to know
- What a test case is: an input, the steps, and an expected result. If you have never written one, this lesson shows one.
- No knowledge of AI or machine learning. This lesson explains the one idea it needs.
What you need on your computer
Nothing for this lesson. You read the replies here, and you run the same questions yourself in Meet the bug-hunt app.
Every reply in this lesson comes from the course’s practice app, version 2.3.0. Its model is simulated. It is ordinary code that behaves like a model, and it has planted defects. Each reply is labelled simulated (practice app). Replies of a real model are labelled recorded (real model).
Two new things to test
Section titled “Two new things to test”Esme is the new QA lead at Larkfield, the online home-and-garden shop. She tested web shops for years: the basket, the checkout, the discount codes. Now Larkfield’s support assistant is getting a new release, and Esme must test it. It is the first AI feature she has tested.
Think of two things that you could test. The first is a calculator. You type 7 × 8, and it shows 56. You type it again, and it shows 56 again. If it ever shows 57, it is broken. One input has one right answer, and you can write that answer down before the test.
The second is a new colleague on the help desk. A customer asks how long they have to return an item. Today, the colleague says “You have 45 days from delivery”. Tomorrow, the same colleague says “Our policy gives you 45 days, counted from the delivery date”. The words differ, and both answers are right. You cannot write down the one right sentence. But you can still judge each answer, because you know what a good answer must contain.
Most software that Esme tested was a calculator. The support assistant is a colleague.
The problem: one expected text
Section titled “The problem: one expected text”Esme starts as she always does. She writes a test case with one expected result:
| Field | Value |
|---|---|
| Input | How many days do I have to return an unused item? |
| Expected result | You can return an unused item within 45 days of delivery for a full refund of the item price. |
| Pass rule | The reply is the same as the expected result. |
The returns policy changed on 1 November: the return window grew from 30 to 45 days. So the expected result is right. She sends the question three times. Each run is a new conversation. These are the first lines of the three replies, simulated (practice app):
seed 1: Good question. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]seed 4: Thanks for asking. You can return an unused item within 45 days of delivery for a full refund of the item price. The 45 days start on the delivery date shown in your account. [1] Is there anything else I can help with?seed 2: Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.A seed is a number that starts the random choices of the model. The same seed gives the same reply; a different seed can give a different reply. Meet the bug-hunt app shows how to choose one.
All three runs fail her test case, because no reply is exactly the expected text. But two of the failures are false alarms. Seed 1 adds “Good question”, and seed 4 adds a useful second sentence. A customer would be happy with both. Only seed 2 is really wrong: it says 30 days, the old rule.
So Esme tries a looser pass rule: “the reply contains the word days”. Now all three runs pass, including the wrong one. A check that is too strict fails good answers. A check that is too loose passes bad ones. Neither tells her whether the feature works.
The idea: judge the answer by rules
Section titled “The idea: judge the answer by rules”Testers have a name for the way you decide whether a result is right: the test oracle. For a calculator, the oracle is easy: compare with 56. Software that always gives the same output for the same input is deterministic. The support assistant is not deterministic: the same input can give a different output on each run.
For a colleague, you judge with rules instead of one text. Each rule is something every good answer must do, or must never do. For the return question, Esme writes two:
- Must say 45 days.
- Must cite the returns policy, version 4 (the version in force).
These rules accept many different texts. They reject only the answers that are wrong. A range of acceptable answers replaces the one expected answer.
Text description of the diagram
On the left, a calculator. The input 7 × 8 gives 56 in all three runs. Each run is compared with one expected answer, 56: one right answer, so you compare the text.
On the right, the assistant. The same question gives three different replies. Seed 1 says 45 days and cites the returns policy, version 4: acceptable. Seed 4 says 45 days in other words, adds that the days start on the delivery date, and cites version 4: also acceptable. Seed 2 says 30 days and cites version 3: it sounds just as sure, and it is wrong. A box under the replies gives the rules that decide: the reply says 45 days and cites the returns policy, version 4.
A tiny example: three replies, two rules
Section titled “A tiny example: three replies, two rules”Here are the same three replies, judged by the two rules:
| Run | Says 45 days? | Cites version 4? | Result |
|---|---|---|---|
| seed 1 | yes | yes | pass |
| seed 4 | yes | yes | pass |
| seed 2 | no: 30 days | no: version 3 | fail |
Seed 1 and seed 4 now pass, although their texts differ. Seed 2 fails both rules, so the defect is still caught. Two testers who apply these rules get the same results, and that is the point of an oracle.
Look again at seed 2. It does not say “I think” or “maybe”. It quotes a rule, it cites a source, and it is polite. This is a confident wrong answer: a wrong answer that looks exactly like a right one. A customer has no way to see the difference. Only a tester who knows the rule, or checks the source, can catch it.
Why the same question gets different answers
Section titled “Why the same question gets different answers”A language model writes its reply one small piece of text at a time, called a token. Before each token, it gives every possible next token a probability. Then it picks one, usually at random, with those probabilities as the chances. The chart shows a real example from Predicting continuations in LLMs: How They Work and How to Choose One. A model continued “The parcel arrived late, so the customer asked for a”.
Text description of the chart
Six bars. “refund” has a chance of 91.3%. “full” has 3.4%, “discount” 1.9%, “partial” 1.8% and “return” 0.5%. All other tokens together have 1.1%.
“refund” is the most likely next token, but about 9 draws in 100 pick another word. After a different word, the rest of the reply goes another way too. A setting called the temperature controls how bold these random choices are. So the same question can get a different reply on every run. This is normal behaviour, not a defect. The defect is when one of those replies is wrong.
The practice app copies this behaviour. Its simulated model, sim-1, uses a random number generator, started by the seed. So one seed always gives the same reply, and different seeds give different replies, as a real model does.
The real model behind Larkfield’s assistant varies too. In the course’s recorded runs, the real assistant answered the return question 10 times, in 8 different texts. All 10 said 45 days and cited version 4. Recorded (real model). The words changed, but on this question the facts did not. You cannot switch the variation off on this model, either. It accepts only its default temperature: a request with temperature: 0 gets an error.
Larkfield: twenty runs of one question
Section titled “Larkfield: twenty runs of one question”Esme wants to know how often the wrong answer comes. She asks the same question 20 times, with seeds 1 to 20. Each run starts from fresh test data.
Text description of the chart
Two rows on an axis of seeds from 1 to 20. The pass row has 16 dots: seeds 1, 3, 4, 5, 6, 7, 9, 10, 12, 13, 15, 16, 17, 18, 19, 20. The fail row has 4 dots: seeds 2, 8, 11, 14. The failures are spread among the passes, with no pattern.
16 of the 20 runs pass, and 4 fail. The 16 passing replies use 14 different texts, so an exact-text test would fail almost all of them. In seeds 1 to 100, 30 runs give the wrong answer: about 1 run in 3.
Now imagine that Esme had run the question once, with seed 1. It passes both rules. She would mark the test case “passed” and move on. The feature fails in about a third of all conversations, and her single run could not show it. One passing run of an AI feature tells you that it can work. It does not tell you that it works. Repeat runs and reproduction rate shows how many runs you need.
Three things you will see
Section titled “Three things you will see”When you send the same question several times, you meet three kinds of answer:
- Different answers, all right. The words change from run to run, but every reply passes the rules. This is normal. Do not report it.
- Several acceptable answers. Some questions have more than one good answer. “Can I return a used pressure washer?” could get “no, it must be unused” or “only if it is faulty, under the warranty”. Your rules must accept every good answer, not only the one you thought of first.
- A confident wrong answer. A reply that breaks a rule and sounds as sure as the right ones. This is a defect. Report it with how often it happened, for example “4 of 20 runs”.
When the calculator way still works
Section titled “When the calculator way still works”Not every part of the assistant is a colleague. Some parts are ordinary code, and they are calculators. An order’s status comes from the order database. The trace ID, the version numbers and the refund amount in a tool’s result come from code. For these, an exact expected result is right, and one run is enough. Test them the way you always have.
Use rules for the parts that the model writes: the wording, the choice of facts, the sources it cites, the language and the tone. Rules take more thought than one expected text. You must say what every good answer has in common. You also need more runs, because a defect may appear only in some of them. That costs time, and with a real model it costs money for each run. This course shows you how to spend that time where the risk is highest.
Guided practice: judge five replies
Section titled “Guided practice: judge five replies”Here are five more replies to the return question, simulated (practice app). Judge each with the two rules: says 45 days, cites version 4. Write pass or fail before you open the answer.
seed 3: Here is what our policy says. You can return an unused item within 45 days of delivery for a full refund of the item price. [1] Sources: [1] Returns policy, version 4: Return windowseed 5: You can return an unused item within 45 days of delivery for a full refund of the item price. The 45 days start on the delivery date shown in your account. [1] I hope this helps. Sources: [1] Returns policy, version 4: Return windowseed 8: Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. The 30 days start on the delivery date shown in your account. [1] I hope this helps. Sources: [1] Returns policy, version 3: Return windowseed 9: You can return an unused item within 45 days of delivery for a full refund of the item price. [1] Is there anything else I can help with? Sources: [1] Returns policy, version 4: Return windowseed 14: You can return an unused item within 30 days of delivery for a full refund of the item price. The 30 days start on the delivery date shown in your account. [1] Is there anything else I can help with? Sources: [1] Returns policy, version 3: Return windowAnswer
| Run | Says 45 days? | Cites version 4? | Result |
|---|---|---|---|
| seed 3 | yes | yes | pass |
| seed 5 | yes | yes | pass |
| seed 8 | no: 30 days | no: version 3 | fail |
| seed 9 | yes | yes | pass |
| seed 14 | no: 30 days | no: version 3 | fail |
Seeds 8 and 14 are wrong in the same way as seed 2. Seed 14 has no “Thanks for asking”, so a search for that phrase would not help. The rules look only at what matters: the number of days and the source.
Your turn: rules for another question
Section titled “Your turn: rules for another question”A customer asks: “How long does a refund take to reach my card?” This part of the refunds and payments policy answers it:
When a refund is approved, we send it to the original payment method within 2 business days.The money then usually appears within 5 to 10 business days, depending on your bank.PayPal refunds usually appear within 5 business days. Refunds to a gift card appear at once.- Write two or three Must rules for a good answer. Keep each rule short enough to check in a few seconds.
- Judge these three replies, simulated (practice app), with your rules:
seed 1: Good question. Refunds to a gift card appear at once. [1] See the refunds and payments policy for how long it takes to arrive. [2]seed 4: Thanks for asking. When a refund is approved, we send it to the original payment method within 2 business days. Refunds to a gift card appear at once. [1] Is there anything else I can help with?seed 5: When a refund is approved, we send it to the original payment method within 2 business days. Refunds to a gift card appear at once. [1] I hope this helps.A model answer
Rules: the reply must say that the money usually appears within 5 to 10 business days. It must mention the 2 business days to send the refund. It must cite the refunds and payments policy.
All three replies fail the first rule. Seed 1 talks only about gift cards and sends the customer to read the policy. Seeds 4 and 5 give the 2 days, but not the 5 to 10 days that the customer really waits. In seeds 1 to 6, 0 of 6 replies mention 5 to 10 days.
This is a different lesson from the return question. Here, the replies vary, but they are wrong in the same way every time. Varied wording does not mean that some runs are right. And a reply that is wrong on every run is the easiest failure to reproduce.
Common mistake
A check that passes the wrong answer
Symptom: your test case passes on every run, but a customer reports a wrong return window.
Cause: the pass rule is too loose. “The reply contains days” or “the reply cites a source” is true for the 30-day answer too.
Diagnose: take a reply that you know is wrong, such as seed 2 above, and apply your rules to it. If it passes, the rules do not catch this defect.
Recover: make each rule about the fact that matters (“says 45 days”, “cites version 4”). Then test your rules on one right reply and one wrong reply before you use them.
Knowledge check
Section titled “Knowledge check”Check your understanding
Esme sends the return question once, with seed 1, and the reply is correct. Her manager asks: 'So the feature works?' What does she answer, and why?
Show the answer
One passing run shows only that the feature can give the right answer. The model makes random choices, so another run can give another answer. With the same question, seed 2 says 30 days and cites the outdated version 3. In seeds 1 to 100, 30 runs were wrong. Esme must run the question many times and report how many runs failed, for example “4 of 20”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a test case expects the exact text 'You can return an unused item within 45 days of delivery for a full refund of the item price.' What happens when you run it 20 times? Why is that a problem?
Show the answer
Almost every run fails, also the correct ones. The 16 correct replies use 14 different texts, with openers such as “Good question.” and different endings. The exact-text oracle is too strict for a reply that the model writes. It produces false alarms, and soon nobody trusts the test. Rules such as “says 45 days” and “cites version 4” pass the good replies and still fail the wrong ones.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Which of these parts of the assistant can you test with one exact expected result, and which need rules? (a) the trace ID format, (b) the wording of a policy answer, (c) the status of order LK-200103 in the order database, (d) whether the reply is in French when the customer writes in French.
Show the answer
(a) and (c) are calculators: ordinary code and data give one right answer, so an exact check and one run are enough. (b) and (d) are written by the model: use rules (“cites the policy”, “the reply is in French”) and several runs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A calculator has one right answer. The assistant is more like a colleague: its answers vary, and many different answers can be right.
- An exact expected text fails good answers. A loose check passes bad ones. Judge each reply with short Must and Must not rules: this is your test oracle.
- The same question can get different replies, because the model picks each token at random with its probabilities. The practice app copies this with a seed.
- A confident wrong answer looks exactly like a right one. In 20 runs of the return question, 4 said 30 days and cited the outdated policy.
- One passing run does not show that a feature works. Count the failures over many runs.
- Parts that are ordinary code, such as an order’s status, are still calculators: test them with exact results.
Your workspace now: nothing yet. You can now say that a reply is wrong. But “wrong” alone does not tell a developer what to fix. Next, you give each failure a name that points to its cause.