Skip to content
Français

Retrieval or generation?

  • Lesson 17
  • 30 min
  • Module 6: Test retrieval, agents and safety

Where we are

In Module 5, we reported bugs and checked fixes. But a fix goes to the right place only when we know which part of the assistant made the mistake. In this module, we test the parts one by one. We start with the two main layers: the search that finds the documents, and the model that writes the answer from them.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

When an answer is wrong, find out which part made the mistake: the search that found the documents, or the model that wrote the answer. Prove it with evidence from the trace, so that the fix goes to the right place.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa, with practice-app/ and your workspace/ from Module 5. If you skipped a module, copy the files of workspace/snapshots/end-of-m05 into workspace. Each snapshot is the course’s version of the workspace at the end of a module.
  • A terminal in ai-qa/practice-app, with its virtual environment active, as in Meet the bug-hunt app.

Tested on macOS (Apple silicon) with Python 3.12. The commands are the same on Windows and Linux; they were not run there for this lesson.

A student writes an essay about a river. The teacher finds a wrong fact: “The river is 300 km long.” It is 340 km. Where did the mistake come from? There are three possible places.

  • The library gave the wrong book. The student asked for a book about this river and got a book about another one.
  • The student misread a good book. The book says 340 km. The student wrote 300.
  • The book was an old edition. The 1990 edition says 300 km, because the river was measured again later.

Each cause needs a different fix: train the librarian, teach careful reading, or take the old edition off the shelf. So the teacher first checks the book the student used.

The problem: “the assistant is wrong” does not say where

Section titled “The problem: “the assistant is wrong” does not say where”

Esme reported the 30-day answer as BUG-002, in A bug report template for AI behaviour. The reply says “30 days” and cites the returns policy, version 3. Her report guessed the layer: retrieval. Tomás reads it, but a guess is not proof. His first idea is to change the prompt: add a rule that says “Never use an old policy version.”

Then he opens the prompt, support-v8. Rule 3 already says: “Use the policy version that is in force on today’s date. Never quote an expired notice or an old version.” The rule is there, and the model still breaks it in some runs. A second rule with the same words is unlikely to help. Tomás could lose a day on a fix in the wrong place.

The bug report said what was wrong. It did not prove where the mistake starts. For an assistant that answers from documents, that question has two main answers, and they lead to different code and often different people.

Larkfield’s assistant answers policy questions in two steps.

  1. Retrieval. The search finds the policy sections that best match the question. In the trace, they are the list retrieved.
  2. Generation. The model reads the question and those sections, and writes the answer with its sources. In the trace, the answer is output, and its sources are cited.

This design is called retrieval-augmented generation, or RAG: the generation step is “augmented” with retrieved text. RAG: Building AI That Uses Your Data teaches how to build it. As a tester, you only need the two steps, because each step can fail on its own. They are the layers of the answer.

To isolate the layer means to find which step caused a wrong answer, with evidence. You ask the same three questions as the teacher, in a fixed order:

  1. Retrieved? Is the section with the right answer in retrieved? If not, the search failed: the model never saw the right text.
  2. Supported? Does the cited section say what the reply says? If not, the model failed: it added, dropped or changed a fact.
  3. Current? Is the cited version in force on the app’s date? If not, the answer used an outdated document.
A decision tree. Start: a wrong answer, open its trace. Question 1, Retrieved: is the section with the right answer in retrieved? If no: the search (retrieval) failed, it did not find the text. If yes, question 2, Supported: does the cited section say what the reply says? If no: the model (generation) failed, it added, dropped or changed facts. If yes, question 3, Current: is the cited version in force on the app's date? If no: outdated source, the search let an expired version through and the model used it. If yes: not retrieval or generation; check the citation numbers, the tools, the filter and the instructions.
The layer isolation tree. Ask the three questions in this order, and read each answer from the trace.
Text description of the diagram

The tree starts with a wrong answer and its trace. Question 1, “Retrieved?”: is the section with the right answer in retrieved? If no, the search failed. If yes, question 2, “Supported?”: does the cited section say what the reply says? If no, the model failed. If yes, question 3, “Current?”: is the cited version in force on the app’s date? If no, the source is outdated: the search let an expired version through and the model used it. If all three answers are yes, the cause is not in retrieval or generation. Check the citation numbers, the tools, the safety filter and the instructions.

Why this order? “Supported?” means nothing if the right text was never found, and “Current?” matters only for a reply that its source supports.

Three wrong answers to “How many days do I have to return an unused item?” are below. The right answer is 45 days, from the returns policy, version 4. These replies and traces are constructed for this example.

Reply retrieved cited Layer
“You can return it within 14 days.” the damaged-items policy only damaged-items policy: “report within 14 days” Retrieved? No: the search failed.
“You can return it within 60 days.” returns policy v4, Return window returns policy v4: “45 days” Retrieved? Yes. Supported? No: the model failed.
“You can return it within 30 days.” returns policy v4 and v3, Return window returns policy v3: “30 days” Retrieved? Yes. Supported? Yes. Current? No: outdated source.

The replies look alike: one wrong number each. The traces are different, and the traces tell Tomás where to work.

Worked example: the 30-day answer, layer by layer

Section titled “Worked example: the 30-day answer, layer by layer”

Esme now runs the three questions on the real 30-day reply of the practice app.

  1. Reset the data, and send the question with the seed 2:

    Terminal window
    python -m helpdesk reset
    python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2

    The output, simulated (practice app):

    The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.
    Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.
    Sources:
    [1] Returns policy, version 3: Return window (/policies/returns-policy/v3#return-window)
    [conversation c-0001 | trace tr-0001-1 | seed 2]
  2. Show the trace:

    Terminal window
    python -m helpdesk trace tr-0001-1

    These are the fields that the three questions need (the output, shortened):

    trace tr-0001-1 (part)
    "settings": { "seed": 2, "today": "2026-11-16", ... },
    "retrieved": [
    { "doc_id": "returns-policy", "version": 4, "section": "Return window",
    "effective_from": "2026-11-01", "effective_to": null, ... },
    { "doc_id": "returns-policy", "version": 3, "section": "Return window",
    "effective_from": "2025-03-01", "effective_to": "2026-10-31", ... },
    { "doc_id": "returns-policy", "version": 4, "section": "Condition of returned items", ... }
    ],
    "cited": [ "returns-policy@3#return-window" ]
  3. Open the cited section. In the web app, select the link in the reply. In your editor, open helpdesk/data/policies/returns-policy.v3.md and find the heading “Return window”.

Now the three questions, with the evidence for each:

  1. Retrieved? Yes. The right section, returns policy version 4, “Return window”, is first in retrieved. It says “within 45 days of delivery”.
  2. Supported? Yes. The cited section, version 3, says: “You can return an unused item within 30 days of delivery for a full refund of the item price.” The reply repeats it. The model did not invent the 30 days.
  3. Current? No. Version 3 has "effective_to": "2026-10-31". The app’s date, settings.today, is 2026-11-16. Version 3 expired 16 days before.

So the reply is an outdated source. Look again at retrieved: the search returned the expired version 3, next to version 4, with the same score. The model then had two sections with two answers, and it took the wrong one. Two layers took part, and the first mistake is in the search: it should never return a version that is no longer in force.

One trace is one run. Is the expired version in retrieved only in the runs that fail? The QA runner can check the search alone. The check not_retrieved fails when the search found a given policy version, whatever the reply says.

one case
id: ret-no-expired-version-retrieved
title: The search never returns a policy version that has expired on the app date
message: How many days do I have to return an unused item?
checks:
- not_retrieved: returns-policy@3

The policy ID is doc_id@version, as in the trace: returns-policy@3. You run it in the guided practice below. It fails in 5 of 5 runs, also in the runs where the reply says “45 days”. So the search returns the expired version every time, and the model picks it only sometimes. The chart shows the same over more runs, and next to it, the real model.

The return-window question: how often the search returned the expired version 3, and how often the reply cited it. The practice app and the recorded real model use the same search.
Text description of the chart

A dot chart in two panels. Practice app, simulated, 100 seeds: the search returned version 3 in 100% of runs, and the reply cited version 3 in 30%. Real model, recorded, 10 runs: the search returned version 3 in 100% of runs, and the reply cited it in 0%.

Read the two panels together. The search returns the expired version in every run of both: it is the same search. The real model ignored version 3 in all 10 recorded runs. The practice app’s model used it in 30 of 100 runs. So a real model can hide this search defect for a while. It does not remove it. Every answer still depends on the model choosing well between two versions.

This is what Esme adds to the bug: “Layer: retrieval. The search returns expired versions (case ret-no-expired-version-retrieved, 5 of 5 runs). The model then cites one in some runs.” Now Tomás knows where to work: in the search, which must leave out expired versions. The prompt rule can stay as a second guard.

A real model: the right source, the wrong answer

Section titled “A real model: the right source, the wrong answer”

The practice app’s model builds its sentences from the retrieved text, so its answers usually match their sources. A real model writes new sentences, and it can turn a right source into a wrong answer. The course recorded the real assistant, with the same prompt, search and tools, on the question “Do you deliver to Nunavut?” (Nunavut is a territory in the north of Canada.) The right answer: yes, Larkfield delivers there, but not large items.

In 2 of 10 runs (recorded, real model), the reply began with “No”. Here is run 4, with the sections it cites:

Left: the reply of the real model, run 4: No. Large items aren't delivered to Nunavut or any other territory [1][3]. For other items, we deliver throughout Canada, including Nunavut [1]. An arrow labelled cites goes to two sections on the right. [1] Delivery policy, version 4, Where we deliver: We deliver to every province and territory in Canada; large items are not delivered to the territories. [3] Large item delivery, version 2: We deliver large items in the provinces only, not in the territories. Below, three checks. 1 Retrieved: yes, both sections are in retrieved and cited. 2 Supported: no, the sources say yes, we deliver to Nunavut, except large items, and the reply starts with No. 3 Current: yes, both versions are in force. At the bottom: the layer is the model (generation); the search found the right text, and the model wrote the opposite.
A recorded reply of the real model next to the sections it cites. Retrieved and current, but not supported: the mistake is in generation.
Text description of the diagram

On the left is the reply of the real model in run 4: “No. Large items aren’t delivered to Nunavut or any other territory [1][3]. For other items, we deliver throughout Canada, including Nunavut [1].” It cites two sections, shown on the right. Source [1] is the delivery policy, version 4, “Where we deliver”: “We deliver to every province and territory in Canada.” It adds that large items are not delivered to the territories. Source [3] is the large item delivery policy, version 2: “We deliver large items in the provinces only, not in the territories.”

Below are the three checks. Retrieved: yes, both sections are in retrieved, and the reply cites them. Supported: no. The sources say “yes, we deliver to Nunavut, except large items”, and the reply starts with “No.” Current: yes, both versions are in force on the app’s date. The verdict: the layer is the model, in generation. The search found the right text, and the model wrote the opposite.

The rest of the reply is correct, but a customer who reads “No.” may stop reading and shop elsewhere. The fix is not in the search, which found exactly the right sections in all 10 runs. It is in generation: the prompt, the model, or a check of the answer against its sources. The evaluation course measures this as “faithfulness”: see Generated-output metrics.

The first branch of the tree also happens. In Metamorphic testing, the rephrased question “What is the time limit for sending back something I haven’t used?” never got the 45 days. Its trace with the seed 1 (simulated, practice app) shows why:

trace of the rephrased question, seed 1 (part)
"retrieved": [
{ "doc_id": "returns-policy", "version": 4, "section": "Condition of returned items", ... },
{ "doc_id": "returns-policy", "version": 3, "section": "Condition of returned items", ... },
{ "doc_id": "holiday-returns-2025", "version": 1, "section": "Extended return period",
"effective_to": "2026-01-31", ... }
]

No “Return window” section is in the list. Retrieved? No. The model never saw the 45 days, so no prompt change can make it say them. The real model, recorded, answered “I don’t have the policy information” for the same reason: it got the same three sections. The search matches words, and “time limit” and “sending back” are not the words of the policy. The RAG course shows how developers find and fix such search misses.

When to isolate the layer, and what it costs

Section titled “When to isolate the layer, and what it costs”

Isolate the layer for every wrong answer of a feature that uses documents, before you write the “suspected layer” of a bug report. It costs a few minutes per bug, and it saves the developer the guess.

It has limits:

  • You need the trace. The practice app shows retrieved and cited for every reply. A real product may not. Ask the developers early for a test mode or logs that show the retrieved sections. Without them, you can only guess the layer.
  • Two layers can share the blame, as in the 30-day answer. Name the first mistake, and say what the second layer did.
  • A layer can be right and the answer still wrong. If all three answers are yes, look elsewhere: at the numbers in the source list, the tools, the safety filter or an ignored instruction. The field guide’s other types start there.
  • Read the source yourself. “Supported?” needs a person, or a careful check, that compares the reply with the section. A citation that exists is not a citation that supports.

Common mistake

A check that can never fail

Symptom: Esme writes not_retrieved: returns-policy-v3. The runner shows PASS ... 5/5 passed. She is surprised, because the trace shows version 3 in retrieved.

Cause: the ID is wrong. The trace uses returns-policy with "version": 3, and the runner writes it as returns-policy@3. No retrieved section is called returns-policy-v3, so “not retrieved” is always true. python -m qa check accepts the file, because the format of the check is valid.

Diagnose: change the check once to the opposite, retrieved: returns-policy-v3, and run it. If it also fails in every run, the ID matches nothing.

Fix: copy the ID from the trace: doc_id, then @, then version. Run the case once on a version that you know is wrong, and see it fail.

Consequence in AI work: a check that never fails gives a green report and no protection.

Guided practice: check the layers with the runner

Section titled “Guided practice: check the layers with the runner”
  1. In ai-qa/workspace/cases, create the file retrieval.yaml with two cases:

    workspace/cases/retrieval.yaml
    # Retrieval or generation? (Module 6): check what was retrieved, what was cited, and whether it is current.
    id: ret-no-expired-version-retrieved
    title: The search never returns a policy version that has expired on the app date
    message: How many days do I have to return an unused item?
    checks:
    - not_retrieved: returns-policy@3
    ---
    id: ret-no-expired-notice-cited
    title: An expired notice (holiday returns 2025) is never cited
    message: What is the time limit for sending back something I haven't used?
    checks:
    - not_cites: holiday-returns-2025
    runs: 10

    The second case checks the answer, not the search. A reply must never cite the holiday notice, which ended on 31 January 2026.

  2. Predict: which case fails in more runs, and why?

  3. In practice-app, run the file:

    Terminal window
    python -m qa run ../workspace/cases/retrieval.yaml

    The output:

    QA run: 2 case(s) from ../workspace/cases/retrieval.yaml
    App: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8
    FAIL ret-no-expired-version-retrieved 0/5 passed (needs 5; seeds 1-5)
    first failing run: seed 1, trace tr-0001-1
    - not_retrieved: ['returns-policy@3'] was retrieved
    FAIL ret-no-expired-notice-cited 6/10 passed (needs 10; seeds 1-10)
    first failing run: seed 1, trace tr-0001-1
    - not_cites: cited ['holiday-returns-2025@1#extended-return-period']
    2 case(s): 0 passed, 2 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not run
    Report: reports/qa-<date>-<time>/report.md
  4. For each case, write its layer in one line under the file’s first comment line. Use the tree.

Show the layers

The first case fails in every run because it checks the search alone, and the search always returns version 3. Layer: retrieval.

The second case fails in 4 of 10 runs. The expired holiday notice is in retrieved in every run: the search does not filter it. The reply cites it only in some runs. Layer: retrieval first (an expired notice reached the model), then generation (the model used it). The same defect as the 30-day answer, in a second document. This shows that the search’s missing date filter affects every versioned document, not only the returns policy.

Your turn: find and report a retrieval defect

Section titled “Your turn: find and report a retrieval defect”

The app has more defects in its policy answers than the ones in this lesson. Find one more, and prove its layer.

  1. Choose questions that use the policies: damaged items, refunds, warranties, safety notices, or one question with two topics.
  2. Run each with the seeds 1 to 10. For each reply, read retrieved and cited in the trace, and open each cited section.
  3. Check the sources list too. Does each number in the reply point to a section that says that sentence?
  4. When you find a failure, walk the three questions. Write the answer to each, with the trace field that proves it.
  5. Add a case for it to workspace/cases/retrieval.yaml, with runs: 10. Run it, and check that it fails.
  6. Write workspace/bugs/BUG-006-<short-name>.md from the template in templates/bug-report.md. In the severity line, add a line “Layer (with evidence):”.

The course’s version of this task is in workspace/snapshots/end-of-m06. Open it only after you try.

Check your understanding

A reply says 45 days and cites the returns policy, version 4. Its trace shows version 3 in retrieved as well. Is there a defect? In which layer, and how do you show it?

Show the answer

Yes. The reply is right, but the search returned an expired version. That is a retrieval defect. It does not hurt this reply, but it can hurt any run in which the model picks version 3. Show it with a check on the search alone, not_retrieved: returns-policy@3, run several times. It fails in every run, also in the runs with a right reply.

Check your understanding

Practical: a reply says 'Gift cards expire after 2 years [1]'. The trace shows the gift-card policy in retrieved and cited, version in force, and its section says 'Gift cards never expire and have no fees.' Which layer failed? What do you write in the bug report?

Show the answer

Retrieved: yes, the right section is there. Supported: no, the section says the opposite of the reply. So the layer is generation: the model changed the fact. Current does not matter here. In the report: “Layer: generation. The cited section (gift-cards, version in force, retrieved and cited in trace tr-…) says ‘Gift cards never expire’; the reply says 2 years.” Add the reproduction rate over several runs. This example is constructed.

Recap

  • An assistant that answers from documents has two layers: retrieval (the search finds sections) and generation (the model writes the answer from them).
  • Isolate the layer with three questions, in order: Retrieved? Supported? Current? Read each answer from the trace: retrieved, cited and the section text, effective_to and settings.today.
  • The 30-day answer starts in retrieval: the search returns the expired version in every run, and the model picks it in some. A real model hid it in 10 recorded runs; the defect was still there.
  • A real model can also fail in generation with the right sources: “No” for Nunavut, in 2 of 10 recorded runs.
  • Test a check before you trust it: make it fail once on a known bad case.

Your workspace now: workspace/cases/retrieval.yaml with your retrieval cases, and workspace/bugs/BUG-006-…md with the layer and its evidence. Next, you test what the assistant does: its tools and actions.

Next: Tools, actions and permissions