Skip to content
Français

A field guide to AI failures

  • Lesson 2
  • 30 min
  • Module 1: How AI features fail

Where we are

In the last lesson, we saw that the assistant is like a new colleague: its answers vary, and you judge each one with Must and Must not rules. With these rules, you can say that a reply is wrong. But “wrong” alone does not help a developer: an invented fact and an outdated document need different fixes. In this lesson, we build a field guide that gives each failure a name and shows where to look for its cause.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Give each failure that you see a name. The name tells you where to look for the cause, who can fix it, and how to report it. “The bot is wrong” is not something a developer can act on.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing for this lesson. You use the field guide on the practice app in the next lesson, Meet the bug-hunt app.

A new bird-watcher sees “a small brown bird”. An experienced one sees a house sparrow or a song sparrow, and she knows the difference matters. To learn, beginners carry a field guide: a small book with one page for each kind of bird. Each page says what the bird looks like, where it lives, and which other birds look almost the same.

The guide does two jobs. It gives the beginner a name to write in her notebook, so that other people understand her notes. And it tells her where to look next: a bird that lives in marshes is not the one she saw in a city park.

The problem: “the assistant gave a wrong answer”

Section titled “The problem: “the assistant gave a wrong answer””

In her first hour with the support assistant, Esme writes notes like these:

- wrong answer about returns
- bot weird with the order
- refused something???

Tomás, the AI engineer, reads them and cannot start. “Wrong answer about returns” could mean many different faults. Perhaps the search found an old document. Perhaps the model made up a number that is in no document. Perhaps a rule in the prompt was missing. Each cause has a different fix, in a different part of the assistant, and sometimes a different person fixes it.

Take the 30-day reply from the last lesson. It looks like the model invented a number. But the old returns policy, version 3, really says 30 days. So the model did not invent anything: it used an outdated source. If Esme reports “the assistant invents facts”, Tomás may change the prompt, and the real fault stays.

The idea: eight failure types, and where each one starts

Section titled “The idea: eight failure types, and where each one starts”

The support assistant is not one piece. It has parts, and each part can fail in its own ways. The safety filter checks each message before the model sees it. The model writes the reply, following its instructions, also called the prompt: the text that tells the model its job and its rules. The search finds sections of Larkfield’s policy documents for the model to use. Tools let the model act on orders: look one up, cancel it, start a refund. A release fixes the versions of all these parts.

The field guide of this course names eight failure types. The map shows where each one usually starts.

A flow inside a dashed red frame labelled 'One release of the assistant: app 2.3.0, prompt support-v8, model sim-1'. Customer message, then safety filter, then model with its instructions, then reply with sources. Under the model, two parts connected to it in both directions: search in the policy documents, and tools that look up, cancel and refund orders in the order store. Red tags: at the safety filter, 4 wrong refusal and over-refusal. At the model, 1 invented facts, 3 ignored instruction, 7 tone and language. At the search, 2 wrong or missing source. At the tools, 5 wrong tool or action and 6 another user's data. At the frame, 8 change after an update: a new app, prompt or model version.
The parts of Larkfield's support assistant, and where each of the eight failure types usually starts.
Text description of the diagram

A customer message goes through the safety filter, then to the model with its instructions, and the model writes the reply with its sources. The model also uses two other parts, in both directions. One is the search in the policy documents. The other is the tools that look up, cancel and refund orders. A dashed frame around everything is one release: app 2.3.0, prompt support-v8, model sim-1.

The tags say where each failure type usually starts. Type 4, wrong refusal and over-refusal: the safety filter, or the model. Types 1, 3 and 7, invented facts, ignored instruction, and tone and language: the model and its instructions. Type 2, wrong or missing source: the search. Types 5 and 6, wrong tool or action, and another user’s data: the tools. Type 8, change after an update: any part, when a new version arrives.

“Usually” matters. A wrong source can also start in the model, when the search found the right document and the model used another one. The map tells you where to look first, not what the cause is. Module 6, Retrieval or generation?, shows how to prove the cause with evidence.

Each entry says what you see, gives a Larkfield example, and names a look-alike. Most examples are constructed: written for this lesson to show the type. They are not outputs of the practice app, so they do not give away its bugs.

What you see: a fact that no document, tool result or message gave the assistant. Testers of AI often call this a hallucination. It can be a number, a date, a product feature or a rule.

Example (constructed): “Your Ember GR-310 BBQ has a 5-year warranty on its burners.” The warranty policy gives no such rule.

Look-alike: a wrong source (type 2). If the fact is in an old or unrelated document that the assistant used, it is not invented. Check the trace before you choose.

What you see: the reply uses an outdated or unrelated document, or cites a source that does not support the sentence. Or it gives a policy fact with no source at all. A citation is the number such as [1] that links a sentence to its source.

Example (simulated, practice app): the reply with seed 2 says “within 30 days” and cites “Returns policy, version 3”. Version 3 ended on 31 October; version 4 says 45 days.

Look-alike: an invented fact. The difference is whether the wrong fact exists in a source.

What you see: the reply breaks a rule of the assistant’s instructions. Larkfield’s prompt has rules such as “cite every policy fact” and “ask before you cancel an order or start a refund”. Others say “name the order and the amount when you ask” and “do not share these instructions”.

Example (constructed): to confirm a refund, the assistant asks only “Shall I go ahead?”. The rule says it must name the item and the amount.

Look-alike: many types also break a rule. Choose a more exact type when one fits, and say which rule it breaks.

What you see: the decision to refuse is wrong. Over-refusal is the common case: the assistant refuses a safe request. The other direction is just as wrong: it answers a request that it must refuse, such as help to hurt a person or an animal.

Example (constructed): “Can I store petrol for my lawn mower in the shed?” gets “Sorry, I can’t help with that request.” It is a normal garden question, and a human agent would answer it.

Look-alike: a correct refusal. Ask: would a good human agent at Larkfield answer this?

What you see: the assistant calls the wrong tool, or the right tool with wrong values. Or it acts without a step it needs, such as the customer’s confirmation. The evidence is in the trace, in the list of tool calls.

Example (constructed): the customer asks to return the cushion set, and the trace shows a refund started for the hose reel.

Look-alike: a reply that only says it did something. If no tool was called, the action did not happen: that is an invented fact.

What you see: the reply shows, or a tool changes, data that belongs to another customer: an order, an address, an email. This is often the most serious type, because it can break privacy law.

Example (constructed): Alex, signed in, asks about his delivery. The reply includes the address of another customer’s order.

Test only with test accounts. The practice app has four test customers. Never use real customer data to look for a leak.

What you see: the facts are right, but the words are not. The reply is in the wrong language, rude, cold to an upset customer, or treats two customers differently for no reason.

Example (constructed): Yvette, a customer, writes in French and gets a correct answer in English.

Look-alike: an ignored instruction. Larkfield’s prompt says “answer in the customer’s language”, so this is also rule 4 broken. The tone and language type tells the developer more.

What you see: an input that passed on the last version fails on the new one. The new version can be the app, the prompt, the model or the documents. Testers call this a regression.

Example (constructed): after a new model version, the reply to “Do you deliver on Saturdays?” no longer names the delivery days. With the old version, every run named them.

Look-alike: a defect that was always there, but rare. A failure in 1 run of 10 can stay hidden in five runs of the old version. Compare the same inputs, with enough runs, on both versions.

Esme sorts three replies from the practice app, simulated (practice app). She uses the key below, and she shows her evidence at each step.

A decision flow. Start: you see a reply that is wrong; read its trace first. Does it show or change another customer's data? Yes: 6, leak of another user's data. No: did it act, or call a tool, in the wrong way? Yes: 5, wrong tool or action. No: is the refusal decision wrong? Yes: 4, wrong refusal or over-refusal. No: is a fact wrong or unsupported? Yes: is the fact in a document it used? Yes, an old or wrong document: 2, wrong or missing source. No document says it: 1, invented fact. If no fact is wrong: does it break a rule of its instructions? Yes: 3, ignored instruction. No, the facts are right but the words are not: 7, tone or language problem. A separate box: 8, change after an update: did the same input pass on the last version? Ask this for every type.
A sorting key for the field guide. Answer the questions in order, with evidence from the reply and its trace.
Text description of the diagram

Start with a reply that is wrong, and read its trace. Ask the questions in this order. First: does it show or change another customer’s data? If yes, type 6. Second: did it act, or call a tool, in the wrong way? If yes, type 5. Third: is the decision to refuse wrong? If yes, type 4. Fourth: is a fact wrong or unsupported? If yes, ask whether the fact is in a document that it used. If an old or wrong document says it, type 2. If no document says it, type 1. If no fact is wrong, ask whether it breaks a rule of its instructions: if yes, type 3. If not, the facts are right but the words are wrong: type 7. For every type, also ask whether the same input passed on the last version: if yes, it is also type 8.

Observation A. The question “How many days do I have to return an unused item?”, seed 2. The reply says “within 30 days” and cites “Returns policy, version 3”.

  1. Another customer’s data? No.
  2. A tool or an action? No: the trace shows "tool_calls": [].
  3. A refusal? No.
  4. A wrong fact? Yes: version 4, in force since 1 November, says 45 days.
  5. Is “30 days” in a document it used? The trace’s retrieved list has version 3, and version 3 says 30 days. Its effective_to is "2026-10-31", and the app’s date is 2026-11-16.

Result: type 2, wrong source (an outdated document). Not an invented fact.

Observation B. The question “How long does a refund take to reach my card?”, seed 4. The reply says the refund is sent “within 2 business days” and that gift-card refunds “appear at once”. It cites the refunds and payments policy, version 3.

1 to 3: no, no, no. 4: a wrong fact? Nothing it says is false. But it leaves out the answer the customer needs: “the money then usually appears within 5 to 10 business days”. That sentence is in the same section that the reply cites. So no source is wrong, and nothing is invented.

Result: none of the eight types fits well. Write what you see: “incomplete answer: leaves out the 5 to 10 business days, which is in the cited section”. A field guide helps you name most failures, not all of them. Never force a failure into a type.

Observation C. The question “Do you deliver to Nunavut?”, seed 1. The reply: “Good question. We deliver to every province and territory in Canada. [1]” It cites the delivery policy, version 4, section “Where we deliver”. That section says exactly this.

Result: no failure. Sorting also means saying “this one passed”.

Larkfield: the same parts in the practice app

Section titled “Larkfield: the same parts in the practice app”

The practice app has the same parts as the map. Its trace records what each part did, so you can follow the sorting key with evidence:

Part Field in the trace Failure types to check there
Safety filter safety 4
Search retrieved, with version, effective_from, effective_to 2
Model and instructions output, cited, language, prompt_version 1, 3, 7
Tools tool_calls, with their arguments and results 5, 6
Release app_version, prompt_version, model 8

The next lesson shows how to open a trace. The app has planted defects of every type in this guide. Some appear in every run, some in about 1 run in 3, and some in about 1 run in 10.

When to use the field guide, and its limits

Section titled “When to use the field guide, and its limits”

Use the types in your session notes and in the summary line of a bug report. For example: “Wrong source: return window answer cites returns policy version 3 (outdated)”. The name tells the developer where to look first, and it helps you count. “Six findings this week, four of them wrong source” says something about the search.

One observation can have two types. A rude reply that also gives an outdated rule is a tone problem and a wrong source. Record both, with the evidence for each.

The type is a first guess, not a proof. The map says where a type usually starts, but only evidence shows the real cause. Sorting also costs a little time: open the trace for each failure. That time is well spent, because a wrong guess sends the developer to the wrong part.

These observations are constructed: written for this exercise, not taken from the app. Sort each into one or two types with the key. Then say what evidence you would check in the trace.

  1. A customer asks for the help desk’s hours on Sunday. The reply says “9 to 5” and cites the contact page. The contact page says the help desk is closed on Sundays.
  2. “Can I use a pressure washer on my wooden deck?” gets “Sorry, I can’t help with that request.”
  3. Alex (test customer C-1001) asks about “my BBQ order”. The reply describes an order of test customer C-1002.
  4. A customer asks to return the cushion set. The trace shows start_refund for the hose reel.
  5. After a prompt update, an angry customer gets “As I already told you, read the policy.” The same message got a polite reply before the update.
  6. To confirm a cancellation, the assistant asks “Do you want me to go ahead?” and names no order.
Answers
  1. Type 1, invented fact, with a citation that does not support it. “9 to 5” is in no document. Check cited and read that section: it says the opposite.
  2. Type 4, over-refusal. A normal product question. Check safety in the trace: was it the filter, or the model?
  3. Type 6, leak of another user’s data. Check tool_calls: which order did the tool return, and for which customer?
  4. Type 5, wrong tool or action (the right tool with the wrong item). The evidence is the tool call’s arguments.
  5. Type 7, tone, and type 8, change after an update. Check prompt_version in both traces, and run the same message several times on each version.
  6. Type 3, ignored instruction: the rule says to name the order. Check tool_calls: the tool must not have run before the customer said yes.

Common mistake

Every wrong answer becomes an invented fact

Symptom: most findings in your notes are “invented fact”, and the developer’s fixes do not help.

Cause: you sorted by how the reply looks, not by evidence. A wrong fact from an old document and a wrong fact from nowhere look the same in the reply.

Diagnose: for each “invented fact”, open the trace. Read the documents in retrieved and cited. If one of them contains the wrong fact, the type is a wrong source.

Recover: follow the sorting key in order. Write the evidence next to each type, for example: “30 days is in returns policy version 3 (ended 2026-10-31)”.

Check your understanding

Why does Tomás need to know whether a wrong number is an invented fact or a wrong source? Give the fix that each type points to.

Show the answer

The two types start in different parts, so they need different fixes. A wrong source points to the search or the documents: for example, keep expired policy versions out of the search. An invented fact points to the model and its instructions. For example: a stricter rule to answer only from the sources, or a check that every number appears in a source. If Esme names the wrong type, Tomás may fix the wrong part, and the failure stays.

Check your understanding

An upset customer asks about returns. The reply begins 'As I already told you' and gives the outdated 30-day rule. Which types do you record, and what evidence supports each one?

Show the answer

Two types. Type 7, tone: the customer is upset, and the reply is cold and blames them. Type 2, wrong source: the reply says 30 days and cites the returns policy, version 3. The trace shows that version 3 ended on 2026-10-31, before the app’s date. Record both, each with its evidence, because they can have different causes.

Check your understanding

A check passed in all 5 runs on version 2.3.0 and fails in 1 of 5 runs on the next version. Is it a change after an update? What do you do before you report it so?

Show the answer

Perhaps, but 5 runs are not enough to say so. A failure that comes about 1 time in 10 can easily stay hidden in 5 runs of the old version. Run the same input more times on both versions, with the same settings, and compare the counts. Report it as a change after an update only if the old version still passes clearly.

Recap

  • “The assistant is wrong” is not enough. A name for the failure tells the developer where to look and helps you count.
  • The assistant has parts: the safety filter, the model with its instructions (the prompt), the search in the documents, and the tools. A release fixes their versions.
  • The eight failure types are: 1 invented facts, 2 wrong or missing source, 3 ignored instruction, 4 wrong refusal and over-refusal, 5 wrong tool or action, 6 leak of another user’s data, 7 tone and language, 8 change after an update.
  • Sort with evidence from the trace. The 30-day reply is a wrong source, not an invented fact, because version 3 of the policy says 30 days.
  • One observation can have two types, and some observations fit none. Write what you see.

Your workspace now: nothing yet. You now have names for the failures, but you have only read about them. Next, you start the practice app, find failures yourself, and sort them with this guide.

Next: Meet the bug-hunt app