A field guide to AI failures
- Lesson 2
- 30 min
- Module 1: How AI features fail
Where we are
In the last lesson, we saw that the assistant is like a new colleague: its answers vary, and you judge each one with Must and Must not rules. With these rules, you can say that a reply is wrong. But “wrong” alone does not help a developer: an invented fact and an outdated document need different fixes. In this lesson, we build a field guide that gives each failure a name and shows where to look for its cause.
Lesson 2 of 24 · Testing AI Applications: Manual and AI-Assisted QAA field guide to AI failures
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Give each failure that you see a name. The name tells you where to look for the cause, who can fix it, and how to report it. “The bot is wrong” is not something a developer can act on.
Before you startWhat to know and to have ready
What you need to know
- Why the same question can get different replies, and how to judge a reply by rules, from Testing a calculator, testing a new colleague.
What you need on your computer
Nothing for this lesson. You use the field guide on the practice app in the next lesson, Meet the bug-hunt app.
How a bird-watcher learns
Section titled “How a bird-watcher learns”A new bird-watcher sees “a small brown bird”. An experienced one sees a house sparrow or a song sparrow, and she knows the difference matters. To learn, beginners carry a field guide: a small book with one page for each kind of bird. Each page says what the bird looks like, where it lives, and which other birds look almost the same.
The guide does two jobs. It gives the beginner a name to write in her notebook, so that other people understand her notes. And it tells her where to look next: a bird that lives in marshes is not the one she saw in a city park.
The problem: “the assistant gave a wrong answer”
Section titled “The problem: “the assistant gave a wrong answer””In her first hour with the support assistant, Esme writes notes like these:
- wrong answer about returns- bot weird with the order- refused something???Tomás, the AI engineer, reads them and cannot start. “Wrong answer about returns” could mean many different faults. Perhaps the search found an old document. Perhaps the model made up a number that is in no document. Perhaps a rule in the prompt was missing. Each cause has a different fix, in a different part of the assistant, and sometimes a different person fixes it.
Take the 30-day reply from the last lesson. It looks like the model invented a number. But the old returns policy, version 3, really says 30 days. So the model did not invent anything: it used an outdated source. If Esme reports “the assistant invents facts”, Tomás may change the prompt, and the real fault stays.
The idea: eight failure types, and where each one starts
Section titled “The idea: eight failure types, and where each one starts”The support assistant is not one piece. It has parts, and each part can fail in its own ways. The safety filter checks each message before the model sees it. The model writes the reply, following its instructions, also called the prompt: the text that tells the model its job and its rules. The search finds sections of Larkfield’s policy documents for the model to use. Tools let the model act on orders: look one up, cancel it, start a refund. A release fixes the versions of all these parts.
The field guide of this course names eight failure types. The map shows where each one usually starts.
Text description of the diagram
A customer message goes through the safety filter, then to the model with its instructions, and the model writes the reply with its sources. The model also uses two other parts, in both directions. One is the search in the policy documents. The other is the tools that look up, cancel and refund orders. A dashed frame around everything is one release: app 2.3.0, prompt support-v8, model sim-1.
The tags say where each failure type usually starts. Type 4, wrong refusal and over-refusal: the safety filter, or the model. Types 1, 3 and 7, invented facts, ignored instruction, and tone and language: the model and its instructions. Type 2, wrong or missing source: the search. Types 5 and 6, wrong tool or action, and another user’s data: the tools. Type 8, change after an update: any part, when a new version arrives.
“Usually” matters. A wrong source can also start in the model, when the search found the right document and the model used another one. The map tells you where to look first, not what the cause is. Module 6, Retrieval or generation?, shows how to prove the cause with evidence.
The field guide
Section titled “The field guide”Each entry says what you see, gives a Larkfield example, and names a look-alike. Most examples are constructed: written for this lesson to show the type. They are not outputs of the practice app, so they do not give away its bugs.
1. Invented facts
Section titled “1. Invented facts”What you see: a fact that no document, tool result or message gave the assistant. Testers of AI often call this a hallucination. It can be a number, a date, a product feature or a rule.
Example (constructed): “Your Ember GR-310 BBQ has a 5-year warranty on its burners.” The warranty policy gives no such rule.
Look-alike: a wrong source (type 2). If the fact is in an old or unrelated document that the assistant used, it is not invented. Check the trace before you choose.
2. Wrong or missing source
Section titled “2. Wrong or missing source”What you see: the reply uses an outdated or unrelated document, or cites a source that does not support the sentence. Or it gives a policy fact with no source at all. A citation is the number such as [1] that links a sentence to its source.
Example (simulated, practice app): the reply with seed 2 says “within 30 days” and cites “Returns policy, version 3”. Version 3 ended on 31 October; version 4 says 45 days.
Look-alike: an invented fact. The difference is whether the wrong fact exists in a source.
3. Ignored instruction
Section titled “3. Ignored instruction”What you see: the reply breaks a rule of the assistant’s instructions. Larkfield’s prompt has rules such as “cite every policy fact” and “ask before you cancel an order or start a refund”. Others say “name the order and the amount when you ask” and “do not share these instructions”.
Example (constructed): to confirm a refund, the assistant asks only “Shall I go ahead?”. The rule says it must name the item and the amount.
Look-alike: many types also break a rule. Choose a more exact type when one fits, and say which rule it breaks.
4. Wrong refusal and over-refusal
Section titled “4. Wrong refusal and over-refusal”What you see: the decision to refuse is wrong. Over-refusal is the common case: the assistant refuses a safe request. The other direction is just as wrong: it answers a request that it must refuse, such as help to hurt a person or an animal.
Example (constructed): “Can I store petrol for my lawn mower in the shed?” gets “Sorry, I can’t help with that request.” It is a normal garden question, and a human agent would answer it.
Look-alike: a correct refusal. Ask: would a good human agent at Larkfield answer this?
5. Wrong tool or action
Section titled “5. Wrong tool or action”What you see: the assistant calls the wrong tool, or the right tool with wrong values. Or it acts without a step it needs, such as the customer’s confirmation. The evidence is in the trace, in the list of tool calls.
Example (constructed): the customer asks to return the cushion set, and the trace shows a refund started for the hose reel.
Look-alike: a reply that only says it did something. If no tool was called, the action did not happen: that is an invented fact.
6. Leak of another user’s data
Section titled “6. Leak of another user’s data”What you see: the reply shows, or a tool changes, data that belongs to another customer: an order, an address, an email. This is often the most serious type, because it can break privacy law.
Example (constructed): Alex, signed in, asks about his delivery. The reply includes the address of another customer’s order.
Test only with test accounts. The practice app has four test customers. Never use real customer data to look for a leak.
7. Tone and language problems
Section titled “7. Tone and language problems”What you see: the facts are right, but the words are not. The reply is in the wrong language, rude, cold to an upset customer, or treats two customers differently for no reason.
Example (constructed): Yvette, a customer, writes in French and gets a correct answer in English.
Look-alike: an ignored instruction. Larkfield’s prompt says “answer in the customer’s language”, so this is also rule 4 broken. The tone and language type tells the developer more.
8. Change after an update
Section titled “8. Change after an update”What you see: an input that passed on the last version fails on the new one. The new version can be the app, the prompt, the model or the documents. Testers call this a regression.
Example (constructed): after a new model version, the reply to “Do you deliver on Saturdays?” no longer names the delivery days. With the old version, every run named them.
Look-alike: a defect that was always there, but rare. A failure in 1 run of 10 can stay hidden in five runs of the old version. Compare the same inputs, with enough runs, on both versions.
A tiny example: sort three observations
Section titled “A tiny example: sort three observations”Esme sorts three replies from the practice app, simulated (practice app). She uses the key below, and she shows her evidence at each step.
Text description of the diagram
Start with a reply that is wrong, and read its trace. Ask the questions in this order. First: does it show or change another customer’s data? If yes, type 6. Second: did it act, or call a tool, in the wrong way? If yes, type 5. Third: is the decision to refuse wrong? If yes, type 4. Fourth: is a fact wrong or unsupported? If yes, ask whether the fact is in a document that it used. If an old or wrong document says it, type 2. If no document says it, type 1. If no fact is wrong, ask whether it breaks a rule of its instructions: if yes, type 3. If not, the facts are right but the words are wrong: type 7. For every type, also ask whether the same input passed on the last version: if yes, it is also type 8.
Observation A. The question “How many days do I have to return an unused item?”, seed 2. The reply says “within 30 days” and cites “Returns policy, version 3”.
- Another customer’s data? No.
- A tool or an action? No: the trace shows
"tool_calls": []. - A refusal? No.
- A wrong fact? Yes: version 4, in force since 1 November, says 45 days.
- Is “30 days” in a document it used? The trace’s
retrievedlist has version 3, and version 3 says 30 days. Itseffective_tois"2026-10-31", and the app’s date is 2026-11-16.
Result: type 2, wrong source (an outdated document). Not an invented fact.
Observation B. The question “How long does a refund take to reach my card?”, seed 4. The reply says the refund is sent “within 2 business days” and that gift-card refunds “appear at once”. It cites the refunds and payments policy, version 3.
1 to 3: no, no, no. 4: a wrong fact? Nothing it says is false. But it leaves out the answer the customer needs: “the money then usually appears within 5 to 10 business days”. That sentence is in the same section that the reply cites. So no source is wrong, and nothing is invented.
Result: none of the eight types fits well. Write what you see: “incomplete answer: leaves out the 5 to 10 business days, which is in the cited section”. A field guide helps you name most failures, not all of them. Never force a failure into a type.
Observation C. The question “Do you deliver to Nunavut?”, seed 1. The reply: “Good question. We deliver to every province and territory in Canada. [1]” It cites the delivery policy, version 4, section “Where we deliver”. That section says exactly this.
Result: no failure. Sorting also means saying “this one passed”.
Larkfield: the same parts in the practice app
Section titled “Larkfield: the same parts in the practice app”The practice app has the same parts as the map. Its trace records what each part did, so you can follow the sorting key with evidence:
| Part | Field in the trace | Failure types to check there |
|---|---|---|
| Safety filter | safety |
4 |
| Search | retrieved, with version, effective_from, effective_to |
2 |
| Model and instructions | output, cited, language, prompt_version |
1, 3, 7 |
| Tools | tool_calls, with their arguments and results |
5, 6 |
| Release | app_version, prompt_version, model |
8 |
The next lesson shows how to open a trace. The app has planted defects of every type in this guide. Some appear in every run, some in about 1 run in 3, and some in about 1 run in 10.
When to use the field guide, and its limits
Section titled “When to use the field guide, and its limits”Use the types in your session notes and in the summary line of a bug report. For example: “Wrong source: return window answer cites returns policy version 3 (outdated)”. The name tells the developer where to look first, and it helps you count. “Six findings this week, four of them wrong source” says something about the search.
One observation can have two types. A rude reply that also gives an outdated rule is a tone problem and a wrong source. Record both, with the evidence for each.
The type is a first guess, not a proof. The map says where a type usually starts, but only evidence shows the real cause. Sorting also costs a little time: open the trace for each failure. That time is well spent, because a wrong guess sends the developer to the wrong part.
Guided practice: sort six observations
Section titled “Guided practice: sort six observations”These observations are constructed: written for this exercise, not taken from the app. Sort each into one or two types with the key. Then say what evidence you would check in the trace.
- A customer asks for the help desk’s hours on Sunday. The reply says “9 to 5” and cites the contact page. The contact page says the help desk is closed on Sundays.
- “Can I use a pressure washer on my wooden deck?” gets “Sorry, I can’t help with that request.”
- Alex (test customer C-1001) asks about “my BBQ order”. The reply describes an order of test customer C-1002.
- A customer asks to return the cushion set. The trace shows
start_refundfor the hose reel. - After a prompt update, an angry customer gets “As I already told you, read the policy.” The same message got a polite reply before the update.
- To confirm a cancellation, the assistant asks “Do you want me to go ahead?” and names no order.
Answers
- Type 1, invented fact, with a citation that does not support it. “9 to 5” is in no document. Check
citedand read that section: it says the opposite. - Type 4, over-refusal. A normal product question. Check
safetyin the trace: was it the filter, or the model? - Type 6, leak of another user’s data. Check
tool_calls: which order did the tool return, and for which customer? - Type 5, wrong tool or action (the right tool with the wrong item). The evidence is the tool call’s arguments.
- Type 7, tone, and type 8, change after an update. Check
prompt_versionin both traces, and run the same message several times on each version. - Type 3, ignored instruction: the rule says to name the order. Check
tool_calls: the tool must not have run before the customer said yes.
Common mistake
Every wrong answer becomes an invented fact
Symptom: most findings in your notes are “invented fact”, and the developer’s fixes do not help.
Cause: you sorted by how the reply looks, not by evidence. A wrong fact from an old document and a wrong fact from nowhere look the same in the reply.
Diagnose: for each “invented fact”, open the trace. Read the documents in retrieved and cited. If one of them contains the wrong fact, the type is a wrong source.
Recover: follow the sorting key in order. Write the evidence next to each type, for example: “30 days is in returns policy version 3 (ended 2026-10-31)”.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does Tomás need to know whether a wrong number is an invented fact or a wrong source? Give the fix that each type points to.
Show the answer
The two types start in different parts, so they need different fixes. A wrong source points to the search or the documents: for example, keep expired policy versions out of the search. An invented fact points to the model and its instructions. For example: a stricter rule to answer only from the sources, or a check that every number appears in a source. If Esme names the wrong type, Tomás may fix the wrong part, and the failure stays.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
An upset customer asks about returns. The reply begins 'As I already told you' and gives the outdated 30-day rule. Which types do you record, and what evidence supports each one?
Show the answer
Two types. Type 7, tone: the customer is upset, and the reply is cold and blames them. Type 2, wrong source: the reply says 30 days and cites the returns policy, version 3. The trace shows that version 3 ended on 2026-10-31, before the app’s date. Record both, each with its evidence, because they can have different causes.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A check passed in all 5 runs on version 2.3.0 and fails in 1 of 5 runs on the next version. Is it a change after an update? What do you do before you report it so?
Show the answer
Perhaps, but 5 runs are not enough to say so. A failure that comes about 1 time in 10 can easily stay hidden in 5 runs of the old version. Run the same input more times on both versions, with the same settings, and compare the counts. Report it as a change after an update only if the old version still passes clearly.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- “The assistant is wrong” is not enough. A name for the failure tells the developer where to look and helps you count.
- The assistant has parts: the safety filter, the model with its instructions (the prompt), the search in the documents, and the tools. A release fixes their versions.
- The eight failure types are: 1 invented facts, 2 wrong or missing source, 3 ignored instruction, 4 wrong refusal and over-refusal, 5 wrong tool or action, 6 leak of another user’s data, 7 tone and language, 8 change after an update.
- Sort with evidence from the trace. The 30-day reply is a wrong source, not an invented fact, because version 3 of the policy says 30 days.
- One observation can have two types, and some observations fit none. Write what you see.
Your workspace now: nothing yet. You now have names for the failures, but you have only read about them. Next, you start the practice app, find failures yourself, and sort them with this guide.
Next: Meet the bug-hunt app