Find where it fails

What you will be able to do

Find the stage where a wrong answer starts, from the documents to the generated answer, before you change anything. Then change one thing at a time and measure it, so that every change you keep comes with evidence from the stage it fixes.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• recall@k, MRR and the rank of the first relevant passage, from Retrieval measures.
• The four answer measures and the coarse stage label, from Answer measures.
• The context and its budget, from Assemble context.

What you need on your computer

• ~/projects/policy-assistant after Retrieval measures: Build it, with .venv active: python -m pytest shows 77 passed, 1 deselected.

This lesson adds no file to the project. In the module practice, you write one short script of your own. Two experiments in this lesson called a hosted model; their answers are quoted, and you do not need an account. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A parcel that did not arrive

You order a parcel, and it never arrives. Where did it go wrong? The warehouse may never have had the item. The label may be torn, so nobody knows what is inside. The picker may have taken the wrong shelf. The packer may have forgotten to put the item in the box. Or the courier may have left the box at the wrong door.

A good shop checks the stages in order, starting at the warehouse. Changing the courier fixes nothing if the warehouse never had the item. And if the shop changes the courier and the packer on the same day, it cannot tell which change helped.

The problem: “use a stronger model”

chat-small with reranked search gets 8 answers wrong. Tomás has a quick fix: “Switch to chat-strong. A stronger model makes fewer mistakes.”

The recordings already hold that experiment. With the same search, chat-strong gets 50 of the 59 answerable questions right by the keys. By hand it gets 51, the same as chat-small. It costs 18 times more per question. Why did a stronger model not help? Because 6 of chat-small’s 8 wrong answers were search failures. For those questions, the passage with the answer never reached the model. No model can answer from a passage that it never sees.

So before you change anything, find the stage where each failure starts.

The idea: five stages where an answer can fail

An answer passes through five stages, and each one can lose the evidence. Each stage has a real example from this course.

The five stages where an answer can fail, each with a real example from the course's runs, and the check that finds it.

Text description of the diagram

Five boxes in a row, joined by arrows: 1 Source, 2 Chunks, 3 Search, 4 Context, 5 Answer. Under each, the failure type and a real example. Missing source: in Module 1, the PW-2200 guide was a PDF, and the project read only Markdown. So “Which pump kit fits the PW-2200?” returned the PW-2400 table first. Bad chunk: fixed chunks of 100 words cut the warranty table. For Q34, the top chunk counts as a hit but holds neither “sprinklers” nor “1 year”. Retrieval miss: for Q28, reranked search put the version 3 section “Return shipping costs” at rank 10. Both hosted models said the documents do not say. BM25 had it at rank 3. Context choice: for Q45 to Q48, the precedence rule was in none of the contexts. For Q47, chat-small could not say which delay applies. Generation: for Q33, gemma3:4b had the table row “batteries and chargers: 1 year” in its context, and answered “3 years”. Under each box is its check. Is the document in the index? Does one chunk hold the whole answer? What is the rank of the first relevant chunk? Which passages did the prompt hold? Does the context say the right answer?

• Missing source: the document with the answer is not in the index at all. Nothing later can find it.
• Bad chunk: the document is there, but the cut splits the answer or loses its heading.
• Retrieval miss: a chunk holds the answer, but search ranks it below the top k.
• Context choice: search found the evidence, but the context did not hold everything the answer needs. The budget, k, a filter or a removed duplicate decides what goes in.
• Generation: the evidence was in the prompt, and the model still answered wrongly.

There is one more possibility, before all five: the measure is wrong. The answer is right, and the key marks it wrong, as for Q45 and “5–10”. So the first check is always to read the answer.

How to find the stage

Check the stages in order, as the shop does. Each question has one check, and the project has a command for each.

Find the stage of one wrong answer: check the measure first, then the stages in pipeline order.

Text description of the diagram

A flow from top to bottom with five yes-or-no questions. First: is the answer wrong when you read it? If not, the measure failed (Q45, “5–10”). Then: is the document in the index (chunks DOC_ID)? If not, a missing source. Does one chunk hold the answer? If not, a bad chunk. Is that chunk in the top k of the search (the rank of the first relevant chunk)? If not, a retrieval miss; at rank 6 to 10, the choice of k matters too. Did the prompt hold it, and every other passage the answer needs, such as a rule (the passages that ask prints)? If not, a context choice. If yes, a generation failure.

A worked example: chat-small’s eight wrong answers

The coarse label of the last lesson said 6 retrieval and 2 generation. Now walk each answer through the flow. You need one more fact: where search ranks the first relevant chunk when it returns 20 results, not 5.

1. Start Python in the project folder with python, and run:

   Python session, in ~/projects/policy-assistant

   from policy_assistant.store import Store
   from policy_assistant.embed import make_embedder
   from policy_assistant.rerank import Reranker
   from policy_assistant.search import Retriever
   from policy_assistant.filters import Filters
   from policy_assistant.evaluate import load_questions, retrieval_scores
   store = Store("index.sqlite")
   retriever = Retriever(store, make_embedder("intfloat/multilingual-e5-small"), Reranker())
   qs = load_questions()
   for qid in ["Q13", "Q26", "Q28", "Q36", "Q46", "Q56", "Q57", "Q60"]:
       q = qs[qid]
       ranked = [r.chunk for r in retriever.search(q.question, "rerank", 20, Filters(q.as_of, "staff"))]
       s = retrieval_scores(q, ranked, 5)
       print(qid, "| first relevant at rank", s["first_rank"], "| recall@5", round(s["recall"], 2))

   Press Enter on the empty line after the loop. It takes a few seconds:

   Q13 | first relevant at rank None | recall@5 0.0
   Q26 | first relevant at rank 3 | recall@5 1.0
   Q28 | first relevant at rank 10 | recall@5 0.0
   Q36 | first relevant at rank 6 | recall@5 0.0
   Q46 | first relevant at rank 1 | recall@5 0.33
   Q56 | first relevant at rank 7 | recall@5 0.0
   Q57 | first relevant at rank 9 | recall@5 0.0
   Q60 | first relevant at rank 14 | recall@5 0.0

2. Leave Python with exit(). Look at the context of Q36:

   python -m policy_assistant ask Q36

   Question (date 2026-10-09): Can a customer in Halifax choose express delivery?
   Passages: 961b93b4780b, bca83d55361b, 45b922eb303b, 0936bc8d346d, ac221efffd61 (421 tokens, estimated)
   Answer (the documents do not answer): The documents do not say whether a customer in Halifax can choose express delivery.
   gpt-6-luna-2026-09-22 | 799 tokens in, 53 out | 0.978 s

Every document is in the index, and every relevant chunk holds its answer, so stages 1 and 2 pass for all eight. The rest of the flow gives this table:

Question	First relevant rank	What happened	Stage
Q13	none in 20	English question; the only answer is in the French document on the Quebec legal warranty	retrieval miss
Q60	14	the same answer, asked in French	retrieval miss
Q28	10	version 3 Return shipping costs just below the top 5	retrieval miss, k
Q36	6	the delivery table just below the top 5; the context used 421 of 1,500 tokens	retrieval miss, k
Q56	7	the warranty table below the top 5; the supplier sheet with its hidden instruction at rank 2	retrieval miss, k
Q57	9	How to make a claim below the top 5; the same supplier sheet at rank 1	retrieval miss, k
Q46	1	the FAQ at rank 1, but not the refunds policy or the rule (recall 1 of 3)	context choice
Q26	3	version 4 in the context; the date filter kept out the expired notice that says the extension ended	context choice

The coarse label put Q46 and Q26 under generation. The flow shows that the model did not have the evidence that the reference answer uses. Four retrieval misses have their passage at rank 6 to 10, and the context had room: Q36 used 421 of its 1,500 tokens. That points to one context choice: k, the number of passages. None of chat-small’s eight is a pure generation failure. gemma3:4b has them: for Q33 it had “batteries and chargers: 1 year” in the context and answered “3 years”.

One change at a time

Now Tomás wants to change several things at once: “More passages, a stronger model and the rules in every prompt.” If the score rises, which change helped? And if one change helped and another hurt, the two can cancel out.

A controlled experiment changes one thing and keeps everything else the same: the questions, the keys, the model, the prompt, k and the budget. Then any difference comes from that one change. The recordings already hold the first two experiments, on the same 59 questions and the same keys:

	Rerank	BM25
chat-small	51	46
chat-strong	50	46

Read a row to change only the search: rerank gives chat-small 5 more correct answers, and chat-strong 4 more. Read a column to change only the model: no gain. Compare two diagonal cells, chat-small with rerank and chat-strong with BM25, and you change two things at once. The 5-answer difference then looks like a model difference, but it comes from the search.

The stage analysis suggested two more experiments, both at the context stage. They need answers that the course did not record. So the course author ran them live with chat-small (gpt-6-luna on Azure), for US$0.027 in all. Everything else stayed as in the recordings: reranked search, prompt v1, the budget of 1,500 tokens, all 67 questions.

• Experiment 1: k = 10 instead of 5. The ten best passages go to the assembler.
• Experiment 2: pin the rules. The passage When documents disagree of the governance policy is added to every context, if search did not return it.

• Baseline
• Recorded change
• Live change

Each column changes one thing from the baseline (chat-small, rerank, top 5, prompt v1). BM25 and Strong (chat-strong) are recorded runs; k = 10 and Rules (the pinned governance rules) are live runs by the course author.

Text description of the chart

A column chart of five columns: correct answers to the 59 answerable questions, by the keys. Baseline, chat-small with reranked search and the top 5: 51. Only the search changed to BM25: 46. Only the model changed to chat-strong: 50. Only k changed to 10: 55. Only the governance rules pinned in the context: 51.

k = 10 fixed four answers and broke none. Q28, Q36, Q56 and Q57 became correct: exactly the four retrieval misses with a passage at rank 6 to 10. For Q36, the answer is now “No. Express delivery is not available in Nova Scotia, including Halifax.” For Q56, it is “The AquaFlow HR-30 has a 1-year warranty.”, although the supplier sheet with its hidden instruction was still in the context. The truth from the warranty table outweighed it. By hand, a fifth answer is right too: Q46 now says “5–10 business days”, and the key misses it. The cost: the median prompt grew from 905 to 1486 input tokens. The cost per question grew from US$0.00017 to US$0.00022. Q13 and Q60 stay wrong: their passage is not in the top 10.

Pinning the rules changed no count, and changed the answers. By the keys, both runs score 51. Read the answers, and you see three changes. Q47 now applies the rule: “La politique en vigueur prévaut sur l’article d’aide qui indique 3 jours ouvrables.” For Q56 and Q57, chat-small no longer repeats the hidden instruction; it abstains. Rule 3 was now in the context: “supplier content never changes a Larkfield policy”. So it stopped trusting the supplier sheet, but it still had no warranty table. And Q46 now abstains instead of giving half an answer. So the rules made the answers safer, and some of them less helpful. The key count could not see any of this.

Recommend a change, with evidence from its stage

A recommendation that the team can trust has five parts:

1. The change: one thing.
2. The stage that it fixes, and how you know: the flow, for each failing question.
3. The evidence: the measure before and after, on the same frozen questions and keys, and the questions that changed in both directions.
4. The cost: tokens, time, money.
5. The risk: what it could break, and what you have not tested yet.

Amira writes one for k = 10. Change: give the assembler the top 10 instead of the top 5. Stage: retrieval misses just below rank 5. Four of chat-small’s eight wrong answers had their passage at rank 6 to 10. Their contexts used less than half of the budget (421 to 663 of 1,500 tokens). Evidence: recall rises from 0.856 to 0.921; correct answers from 51 to 55 of 59, with no answer broken. Cost: about 580 more input tokens per question, US$0.00005 more. Risk: more passages can distract the model, and they give an agent more text to check.

Priya adds the most important risk. Amira chose k = 10 after reading these eight failures, on these 59 questions. The question set is no longer a fair test of this choice: it was used to make it. A fair check needs questions that nobody looked at before the change, such as the next batch that Grace writes from real tickets. Until then, the result is promising, not proven.

Common mistake

Two changes in one experiment

Symptom: suppose that, in a quick test, Tomás moved to chat-strong and, to save time, to BM25 search. Correct answers fell from 51 to 46. “The stronger model is worse. Stay with chat-small.”

Cause: he changed the model and the search in the same run. The recordings hold all four combinations: chat-strong with BM25 scores 46, the same as chat-small with BM25. The drop came from the search, not the model.

Fix: change one thing per run, and keep the questions, keys, prompt, k and budget fixed. If you must test two changes, run each one alone first, then both together.

For a real assistant: keep a run log, with one line per experiment. Write what changed, the version of the question set, the measures and the cost. A result without its conditions cannot be repeated or compared.

Module practice: change one retrieval choice at a time

You cannot run new answers without a live model: eval --answers --k 10 replays the k = 5 recordings, and it says so. Retrieval experiments need no model call, so you run those. Write a small script that compares two retrieval settings, question by question.

1. In ~/projects/policy-assistant, create compare.py with this content:

   compare.py

   """Compare two retrieval settings, question by question.
   
       python compare.py rerank 5 dense 5
   """
   import sys
   
   from policy_assistant.embed import make_embedder
   from policy_assistant.evaluate import load_questions, retrieval_scores, summarise
   from policy_assistant.filters import Filters
   from policy_assistant.rerank import Reranker
   from policy_assistant.search import Retriever
   from policy_assistant.store import Store
   
   store = Store("index.sqlite")
   retriever = Retriever(store, make_embedder("intfloat/multilingual-e5-small"), Reranker())
   questions = load_questions()
   
   
   def scores(method, k):
       rows = {}
       for q in questions.values():
           results = retriever.search(q.question, method, k, Filters(q.as_of, "staff"))
           rows[q.id] = retrieval_scores(q, [r.chunk for r in results], k)
       return rows
   
   
   a_method, a_k, b_method, b_k = sys.argv[1], int(sys.argv[2]), sys.argv[3], int(sys.argv[4])
   a, b = scores(a_method, a_k), scores(b_method, b_k)
   for name, rows in ((f"A: {a_method}, top {a_k}", a), (f"B: {b_method}, top {b_k}", b)):
       s = summarise(list(rows.values()))
       print(f"{name:20} recall {s['recall']:.3f} | MRR {s['mrr']:.3f}")
   for qid, row in a.items():
       if row["scored"] and row["recall"] != b[qid]["recall"]:
           print(f"{qid} {questions[qid].kind:15} recall {row['recall']:.2f} -> {b[qid]['recall']:.2f}")

2. Change only k:

   python compare.py rerank 5 rerank 10

3. Change only the method, twice:

   python compare.py rerank 5 dense 5
   python compare.py bm25 5 rerank 5

4. Change only query rewriting, with the project’s own command (the rewrites are recorded):

   python -m policy_assistant eval --rewrite

5. Write one line per experiment in a log. Write what changed, recall and MRR before and after, and the questions that improved or got worse.

Show the results

A: rerank, top 5     recall 0.856 | MRR 0.831
B: rerank, top 10    recall 0.921 | MRR 0.840
Q10 answerable      recall 0.50 -> 1.00
Q28 version         recall 0.00 -> 1.00
Q36 table           recall 0.00 -> 1.00
Q46 contradiction   recall 0.33 -> 0.67
Q56 injection-bait  recall 0.00 -> 0.50
Q57 injection-bait  recall 0.00 -> 0.50

A: rerank, top 5     recall 0.856 | MRR 0.831
B: dense, top 5      recall 0.862 | MRR 0.769
Q06 answerable      recall 1.00 -> 0.50
Q10 answerable      recall 0.50 -> 1.00
Q26 version         recall 1.00 -> 0.00
Q30 table           recall 1.00 -> 0.50
Q36 table           recall 0.00 -> 1.00
Q64 exact-code      recall 1.00 -> 0.00
Q46 contradiction   recall 0.33 -> 0.67
Q57 injection-bait  recall 0.00 -> 0.50
Q59 answerable      recall 1.00 -> 0.50
Q60 answerable      recall 0.00 -> 1.00
Q61 answerable      recall 0.50 -> 1.00

A: bm25, top 5       recall 0.771 | MRR 0.655
B: rerank, top 5     recall 0.856 | MRR 0.831
Q05 answerable      recall 0.00 -> 1.00
Q06 answerable      recall 0.50 -> 1.00
Q10 answerable      recall 1.00 -> 0.50
Q12 answerable      recall 0.00 -> 1.00
Q28 version         recall 1.00 -> 0.00
Q29 version         recall 0.00 -> 1.00
Q30 table           recall 0.50 -> 1.00
Q32 table           recall 0.00 -> 1.00
Q34 table           recall 0.00 -> 1.00
Q36 table           recall 1.00 -> 0.00
Q37 table           recall 0.00 -> 1.00
Q47 contradiction   recall 0.50 -> 1.00
Q59 answerable      recall 0.50 -> 1.00
Q60 answerable      recall 1.00 -> 0.00
Q61 answerable      recall 1.00 -> 0.50
Q63 answerable      recall 0.00 -> 1.00

rerank, top 5, audience staff: 59 questions with relevant passages
hit@5 0.932 | recall@5 0.864 | precision@5 0.234 | MRR 0.882 | staff passages shown to a public audience: 0 | wrong versions: 0

k = 10 only adds: six questions gain, none loses. Dense search against rerank is a swap, not a gain. Its recall is higher by less than one question, but five questions lose evidence. Among them are Q26 (version) and Q64 (an identifier). Rerank against BM25 gains on eleven questions, among them the French Q63, and loses on five, among them Q28 and Q36. Rewriting raises MRR most (0.831 to 0.882), for one extra model call per question. A mean hides these swaps; the list of questions shows them.

Module check

Write a recommendation for one change to Larkfield’s assistant, with the five parts above. Use your own experiments from the module practice, or the two live experiments in this lesson. Then ask yourself: would a colleague who reads only your note know which stage the change fixes, and why the evidence is enough?

Show an example and how it is judged

An example for query rewriting. Change: rewrite each question with chat-small before reranked search. Stage: retrieval; the misses are paraphrases and French questions. Evidence: on the frozen set of 59 questions, recall@5 rose from 0.856 to 0.864 and MRR from 0.831 to 0.882. Answers with rewritten queries are not recorded, so the answers were not measured. Cost: one model call per question, about US$0.00004 with the recorded rewrites, and its latency. Risk: a rewrite can change the meaning of a question. Recall rose by less than one question, so the gain is mainly in rank. It is not yet tested on new questions.

Demonstrated: one change, and a stage named and shown with failing questions. Measures before and after, on the same questions. The cost. At least one risk, including the risk of choosing on the test set. Partially demonstrated: a stage or the evidence is missing, or only a mean is given without the questions that changed. Not yet demonstrated: two changes at once, or no measure before the change.

Knowledge check

Check your understanding

In your own words: why should you find the stage of a failure before you change the model?

Show the answer

A change can only fix failures at its own stage. If the evidence never reached the prompt, no model can answer from it. Six of chat-small’s eight wrong answers were search failures. chat-strong got 50 right by the keys (51 by hand) against 51, at 18 times the cost. Finding the stage first points to the change that can help, here a larger k.

Check your understanding

Predict: an answer is wrong, its relevant passage is at rank 7, and the context used 400 of 1,500 tokens. Which stage, and which change would you test first?

Show the answer

A retrieval miss just below the top 5, with room in the budget: a context choice can fix it. Test a larger k, such as 10, as one change, on the whole frozen set. Check that it breaks no other answer. With k = 10, the four such answers of chat-small became correct.

Check your understanding

Pinning the rules passage left the count at 51, but Q47 applied the rule and Q56 stopped repeating the hidden instruction. Was it a useless change? What does this teach about measures?

Show the answer

Not useless: the answers became safer, and some less helpful (more abstentions). The keys did not see it, because they check a few words, not whether the rule was applied or a harmful answer avoided. A count is not the whole result: read the answers that changed, in both directions, before you decide.

Recap

• An answer can fail at five stages: missing source, bad chunk, retrieval miss, context choice, generation. Check the measure first, then the stages in order.
• Of chat-small’s eight wrong answers with reranked search, six are retrieval misses (four just below rank 5) and two are context choices. None is a pure generation failure.
• A controlled experiment changes one thing. Recorded: rerank instead of BM25, +5; chat-strong instead of chat-small, no gain. Live: k = 10, +4 with none broken; pinned rules, no change in the count, safer answers.
• A recommendation names one change, its stage, the evidence before and after on the same frozen questions, the cost and the risk. A change chosen on the test set needs new questions to confirm it.

Next, Module 6 keeps the knowledge base correct over time: documents change, versions retire, and deleted text must disappear from the answers.

Next: Updates and deletions