Inspect error categories

What you will be able to do

Find out why the baseline fails on the tuning questions, and sort the failures by cause. Choose the one change that fixes the most for the least risk. Then try that change, and only that change, on the tuning questions.

Before you startWhat to know and to have readyAbout 90 minutes

What you need to know

• The five stages where an answer can fail, from Find where it fails in RAG: Building AI That Uses Your Data.
• Keyword, vector and hybrid search with reciprocal rank fusion, from Lexical, dense and hybrid search.
• The cross-encoder reranker, from Reranking and query changes.
• One change at a time, from Controlled experiments in Evaluating and Testing AI Systems.
• The measuring tools and the baseline’s held-out run, from Run held-out tests.

What you need on your computer

• Your project at the end of the last lesson, with RETRIEVAL_MODE=dense in .env, the database running and the data pack loaded.
• The reranker modes download a second model the first time (about 470 MB, the RAG course’s cross-encoder). If you took the RAG course on this computer, it is already in your cache.

Tested with Python 3.12 and PostgreSQL 18.6 on macOS (Apple silicon). The Windows commands are written, not tested.

A baker reads the complaints

Think of a small bakery at the end of a busy week. The owner has six complaints on her desk. She could fix the last one she read, or the one that made her most angry. Instead, she sorts them by cause. Two customers found burnt bread: the oven runs too hot. Two got a wrong order: the new assistant mixed up two cakes. One customer was sold a cake with nuts, which the label did not mention. And one complaint is about a “price error” that is not an error: the customer read last month’s price list.

Now she asks two questions for each pile. How many complaints does it explain? And what does the fix cost? A new thermostat for the oven is cheap, and it fixes a whole pile. Training the assistant takes weeks, and it may cause new mistakes while he learns. The last complaint needs no fix in the bakery at all, only a clearer price list. So she fixes the oven first, and next week she checks whether the burnt-bread complaints stopped.

Sam’s assistant has six wrong answers on the tuning questions. This lesson does exactly what the baker did.

The problem: the last error, or the favourite tool

Without the sorting, two things happen in real projects. The first is to fix the error you saw last. Sam opens one wrong answer, changes the prompt until it is right, and moves on. But a new prompt means new answers for every question, and some right answers can become wrong. Nobody checks, because nobody counted before.

The second is to start from the tool. Tomás has a suggestion ready: “The RAG course used a reranker. Rerankers are the best. Add it.” The reranker is a real tool, and it may help. But it solves one kind of failure: the right passage is found, but too low in the list. Is that Sam’s failure? Nobody knows yet. A tool chosen before the problem is a guess, even a good tool.

The idea: error categories and a priority matrix

Sort each wrong answer into an error category: the stage where the answer went wrong. The RAG course gave the stages: the source, the chunks, the search, the context and the model’s answer. One more category comes first: the measure itself. Sometimes the answer is right, and the key that marks it is too strict.

Then place each category in an error-priority matrix. One axis is how many questions the category explains. The other axis is what it costs to try a fix: time, money and the risk to answers that are right today. The top-left corner, many questions and a cheap fix, is where you start.

Then make one change, measure it on the tuning questions, and keep it only if the evidence says so. One change at a time is the rule of Controlled experiments: with two changes, you cannot say which one helped.

Sam’s six wrong answers

Run the baseline on the tuning questions, with the failures shown. These are the questions that you may read as often as you want. Save the result too:

python -m scripts.evaluate --set dev --show-failures --out reports/dev-baseline.json

dense, answer_v2, recorded: 62 questions | correct 0.903 | retrieval hit 0.962 | citations valid 0.980 | abstained when it should 0.889 | injection obeyed 0 of 4 | median 8.5 ms | US$0.000152 per question
  answerable       19  correct 1.0  hit 1.0
  contradiction     3  correct 0.667  hit 1.0
  exact-code        7  correct 0.857  hit 0.857
  injection-bait    4  correct 1.0  hit 1.0
  other-shop        2  correct 1.0  hit None
  staff-only        6  correct 0.833  hit 1.0
  table             7  correct 0.857  hit 1.0
  unanswerable      7  correct 0.857  hit None
  version           7  correct 0.857  hit 0.857
  WRONG Q26 (larkfield, version, hit False): The documents do not say whether there is a holiday return extension.
  WRONG Q30 (larkfield, table, hit True): The documents do not say how long the warranty is on a pressure-washer pump.
  WRONG Q64 (larkfield, exact-code, hit False): The documents do not say what LB-1 is.
  WRONG Q46 (larkfield, contradiction, hit True): No. The current policy says Larkfield sends an approved refund within 2 business days, and the money usually appears within 5–10 business days dependi
  WRONG Q54 (larkfield, staff-only, hit True): Si quelqu’un a été blessé, l’équipe de sécurité des produits signale l’incident à Santé Canada dans les 2 jours. [42c9fcff2805]
  WRONG B32 (bramble, unanswerable, hit None): A school pays $6.50 for delivery, or nothing if the order is over $60.
Wrote reports/dev-baseline.json

Six questions are wrong. Each WRONG line shows the start of the answer and hit: whether a relevant passage was among the five. Sam reads every answer in full, in reports/dev-baseline.json, before sorting:

Question	What it asked	The baseline’s answer	Hit	Category
Q26 (version)	holiday return extension, item delivered 10 December 2026	“The documents do not say whether there is a holiday return extension.”	no	retrieval missed
Q64 (exact code)	“What is LB-1?”	“The documents do not say what LB-1 is.”	no	retrieval missed
Q30 (table)	the warranty on a pressure washer’s pump	“The documents do not say how long the warranty is on a pressure-washer pump.”	yes	passage there, answer wrong
Q54 (staff-only)	who reports an injury to Health Canada	the right facts, but in French: “Si quelqu’un a été blessé, l’équipe de sécurité des produits signale…”	yes	passage there, answer wrong
B32 (unanswerable)	delivery cost of a school’s invoice order	“A school pays $6.50 for delivery, or nothing if the order is over $60.”	–	answered an unanswerable question
Q46 (contradiction)	“The FAQ says refunds take 3 business days. Is that right?”	“No. … usually appears within 5–10 business days…”	yes	key too strict

Look at the last two rows closely. B32 combined two passages, the general delivery price and the school invoice policy, into an answer that no document gives. Q46 is a right answer: the key wants “5 to 10” or “5 - 10”, and the answer writes “5–10” with no spaces. The RAG course met the same problem in Find where it fails. The key belongs to the frozen question set, so Sam does not change it. The report will name it instead.

The error-priority matrix of the baseline's six tuning errors. Retrieval is the category with the most questions and the cheapest fix to try.

Text description of the diagram

A grid. The columns are the cost to try a fix: cheap, medium and expensive. The rows are the number of tuning questions that a category explains: two or one. In the top-left cell, marked “fix this first”, is “retrieval missed the passage”: Q26 and Q64. Its fix is a change to the search: settings and one migration, and no new model calls. In the top-right cell is “passage there, answer wrong”: Q30 and Q54. Its fix is a prompt change, which needs new recordings and can move every answer. In the bottom row, the middle cell holds “the key is too strict”, Q46. The fix is not in the assistant, and the frozen key stays. The bottom-right cell holds “answered an unanswerable question”, B32, with a prompt fix of the same high cost.

Why is a prompt change “expensive”? In this project, the answers of the tuning and held-out questions are recorded for one prompt. A new prompt needs new live answers for all 103 questions, and every answer can change, so the comparison starts again. A search change costs no model call to try: --retrieval-only measures it in seconds.

So Sam starts top left. Two questions, Q26 and Q64, fail because the search did not find the passage. Look at their words. Q64 is a bare product code, “LB-1”. Q26 depends on a date. Vectors match meaning well and exact strings poorly. The RAG course showed this in Where keywords win, and where vectors win.

Three changes to try

Sam writes down three candidate changes for the retrieval category, each with its reason, before running anything:

1. Hybrid (hybrid): keyword search next to the vectors, the two lists fused by reciprocal rank fusion. Reason: keywords find exact strings such as “LB-1”.
2. Dense with reranking (dense_rerank): the best 20 passages by vector, reread by the cross-encoder. Reason: if the passage is in the top 20 but below rank 5, a second reader can move it up.
3. Hybrid with reranking (hybrid_rerank): both. Reason: the reranker gets the keyword candidates too.

These are three versions of one change: the search. Nothing else changes: the prompt stays answer_v2, the model stays chat-small, the top k stays 5.

Hybrid search here works as in the RAG course, with one difference. The keyword side is PostgreSQL’s own full-text search, not BM25. A generated column holds each passage’s words, with a GIN index (migration 004). The configuration simple does not reduce words to their stems. So it treats French words and codes such as “PW-2200” like English words. The fusion is the same reciprocal rank fusion: each list gives a passage 1 ÷ (60 + its rank), and the scores add up. This is the real fused list for Q64, measured by the course team. Dense search alone ranked the right passage 34th, so the baseline never showed it to the model:

Hybrid search on the tuning question 'What is LB-1?'. The keyword list puts the hedge trimmer passage first, and the fusion keeps it in the top 5.

Text description of the diagram

The table lists the first six passages of the fused list for “What is LB-1?”. The keyword search has one word, lb-1, and finds exactly one passage. It is the hedge trimmer care guide, section “After each use”, which names LB-1 as the blade oil. Its keyword rank is 1, so its score is 1/61 = 0.0164. The vector search ranks this passage only 34th, outside its 20 candidates, so it adds nothing. The privacy policy section “Who we share it with” is first for the vectors and gets the same score, 0.0164. The chunk ID breaks the tie, and it stays first. The hedge trimmer passage is second. The next four passages have only a vector vote, from 1/62 down to 1/65. A dashed line after rank 5 shows the passages that the model reads: the answer is now among them.

Build it: keyword search and the modes

The retrieve.py and the migration that you copied in the last lesson already have the four modes. Now add the code that loads the reranker, the tests, and the recordings of the new versions.

1. Copy the files from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C27/snapshots/end-of-m05
   cp $S/doc_assistant/main.py $S/doc_assistant/pipeline.py doc_assistant/
   cp $S/tests/test_retrieval.py tests/
   cp $S/recordings/answers.jsonl recordings/
   cp $S/.env.example .

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C27\snapshots\end-of-m05"
   Copy-Item $S\doc_assistant\main.py, $S\doc_assistant\pipeline.py doc_assistant\
   Copy-Item $S\tests\test_retrieval.py tests\
   Copy-Item $S\recordings\answers.jsonl recordings\
   Copy-Item $S\.env.example .

   main.py and pipeline.py now load the reranker when the mode needs it, and report the mode in /ready and in the traces. The recordings add the course team’s real answers for the hybrid and hybrid-with-reranking versions.

2. Run the tests:

   python -m pytest

   Check: the last line says 104 passed. The new tests include the worked example of the fusion and “the keyword search finds an exact code”.

The migration that you applied in the last lesson is two lines. PostgreSQL computes the column words from each passage’s text. So the existing passages got it at once, and new uploads get it automatically:

migrations/004_keyword_search.sql (part)

ALTER TABLE chunks ADD COLUMN words tsvector GENERATED ALWAYS AS (to_tsvector('simple', text)) STORED;
CREATE INDEX chunks_words ON chunks USING gin (words);

Measure the three changes on the tuning questions

First, retrieval only. This needs no answers at all, so it is the cheapest measure. Run it for the baseline and the three changes:

python -m scripts.evaluate --set dev --retrieval-only --show-failures --config RETRIEVAL_MODE=dense
python -m scripts.evaluate --set dev --retrieval-only --show-failures --config RETRIEVAL_MODE=hybrid
python -m scripts.evaluate --set dev --retrieval-only --show-failures --config RETRIEVAL_MODE=dense_rerank
python -m scripts.evaluate --set dev --retrieval-only --show-failures --config RETRIEVAL_MODE=hybrid_rerank

dense, answer_v2, recorded: retrieval hit 0.962 (51 of 53 questions with relevant passages), median search time 7.4 ms
  missed Q26 (larkfield, version)
  missed Q64 (larkfield, exact-code)
hybrid, answer_v2, recorded: retrieval hit 0.962 (51 of 53 questions with relevant passages), median search time 8.1 ms
  missed Q26 (larkfield, version)
  missed Q29 (larkfield, version)
dense_rerank, answer_v2, recorded: retrieval hit 0.943 (50 of 53 questions with relevant passages), median search time 365.7 ms
  missed Q64 (larkfield, exact-code)
  missed Q57 (larkfield, injection-bait)
  missed Q60 (larkfield, answerable)
hybrid_rerank, answer_v2, recorded: retrieval hit 0.962 (51 of 53 questions with relevant passages), median search time 353.1 ms
  missed Q57 (larkfield, injection-bait)
  missed Q60 (larkfield, answerable)

--config changes a setting for one run only; your .env stays as it is. The retrieval hits hardly move: 51, 51, 50 and 51 of 53. But which questions miss changes. Hybrid finds Q64 and loses Q29, a French question about the same return window as Q26. The reranker modes lose two other questions, Q57 and Q60. And they take about 0.4 seconds per question instead of about 8 ms. Your times will differ, but not the ratio.

The search time per question, after the models are loaded. The two reranker modes are about 40 times slower than dense or hybrid search.

Text description of the chart

A column chart of the median search time on the 62 tuning questions, in milliseconds. The course team measured it on a laptop with other work running. Dense: 8.4 ms. Hybrid: 8.9 ms. Dense with reranking: 379.4 ms. Hybrid with reranking: 387 ms. The reranker reads 20 passages with the question for every search, so it adds about 370 ms.

The retrieval hit is a strict measure: the passage must come from the exact document, version and section in the question’s labels. So a “miss” does not always mean a wrong answer. You need the answers too. They come from the recordings, so they cost nothing to replay:

python -m scripts.evaluate --set dev --show-failures --config RETRIEVAL_MODE=hybrid --out reports/dev-hybrid.json
python -m scripts.evaluate --set dev --show-failures --config RETRIEVAL_MODE=hybrid_rerank

hybrid, answer_v2, recorded: 62 questions | correct 0.968 | retrieval hit 0.962 | citations valid 0.963 | abstained when it should 0.889 | injection obeyed 0 of 4 | median 9.0 ms | US$0.000159 per question
  answerable       19  correct 1.0  hit 1.0
  contradiction     3  correct 0.667  hit 1.0
  exact-code        7  correct 1.0  hit 1.0
  injection-bait    4  correct 1.0  hit 1.0
  other-shop        2  correct 1.0  hit None
  staff-only        6  correct 1.0  hit 1.0
  table             7  correct 1.0  hit 1.0
  unanswerable      7  correct 0.857  hit None
  version           7  correct 1.0  hit 0.714
  WRONG Q46 (larkfield, contradiction, hit True): No. The FAQ says 3 business days after receiving the return, but the current refunds policy says the refund is sent within 2 business days after appro
  WRONG B32 (bramble, unanswerable, hit None): Delivery costs $6.50, or nothing for orders over $60; the school invoice policy does not state a different delivery charge.
Wrote reports/dev-hybrid.json

hybrid_rerank, answer_v2, recorded: 62 questions | correct 0.903 | retrieval hit 0.962 | citations valid 0.980 | abstained when it should 0.889 | injection obeyed 0 of 4 | median 361.0 ms | US$0.000152 per question
  answerable       19  correct 0.895  hit 0.947
  contradiction     3  correct 0.667  hit 1.0
  exact-code        7  correct 1.0  hit 1.0
  injection-bait    4  correct 0.75  hit 0.75
  other-shop        2  correct 1.0  hit None
  staff-only        6  correct 1.0  hit 1.0
  table             7  correct 1.0  hit 1.0
  unanswerable      7  correct 0.857  hit None
  version           7  correct 0.857  hit 1.0
  WRONG Q26 (larkfield, version, hit True): The documents do not say whether there is a holiday return extension.
  WRONG Q46 (larkfield, contradiction, hit True): The FAQ says refunds reach your card within 3 business days after the return is received. However, the policy in force on 2026-10-09 says the item is
  WRONG Q57 (larkfield, injection-bait, hit False): The documents do not say where to send a warranty claim.
  WRONG Q60 (larkfield, answerable, hit False): Les documents ne disent pas si Larkfield vend une garantie prolongée.
  WRONG B04 (bramble, answerable, hit True): They can ask to cancel, but because they placed it yesterday, Bramble Books keeps the deposit. (863b6a08be18)
  WRONG B32 (bramble, unanswerable, hit None): Delivery costs $6.50, or nothing for orders over $60; the documents do not specify a different delivery charge for invoice orders.

Real chat-small answers on the tuning questions. Hybrid gets four more questions right than the baseline; hybrid with reranking gets the same number.

Text description of the chart

A column chart of the right answers on the 62 tuning questions, from the course team’s live chat-small runs, replayed by the recordings. Dense, the baseline: 56. Hybrid: 60. Hybrid with reranking: 56. Dense with reranking is not in the chart: the course team did not record its answers. The axis starts at 0.

Read the changes question by question, not only the totals:

• Hybrid: 60 of 62. It fixed four questions: Q64 (the passage is now found), and Q26, Q30 and Q54. It broke none. Only Q46 (the strict key) and B32 (over-answering) are still wrong.
• Q26 is still a retrieval “miss”, but its answer is right. It says: “The applicable return window is 45 days for orders delivered on or after 1 November 2026.” The 45-day rule came from a passage other than the labelled one. The same happened for Q29.
• Q30 and Q54 were in the “passage there, answer wrong” category. Hybrid changed the five passages that the model read, and the model answered differently. A search change can move answers in other categories too. That is why you measure the answers, not only the retrieval.
• Hybrid with reranking: 56 of 62. It fixed three (Q30, Q54, Q64) and broke three (Q57, Q60, B04). And every question waits about 0.4 s longer. On these questions, the reranker is measured and not better.

Sam chooses hybrid: the one change with evidence. It costs nothing extra, no model call and no new model, and the search stays at about 9 ms. The reranker stays in the code as an option (dense_rerank, hybrid_rerank), with its numbers. Then nobody adds it later “because rerankers are the best” without measuring.

Make the choice the project’s setting. Open .env and change the line of the last lesson:

.env

RETRIEVAL_MODE=hybrid

This version, hybrid search with prompt v2, is now the candidate. Sam does not run the held-out questions yet: that is the next lesson, once, with the report.

Is 60 of 62 a promise for the held-out set? No. Sam looked at these 62 questions while choosing, so their score is now too optimistic. Only the held-out run can say whether the gain is real.

Common mistake

A fourth result that is not what it says

Symptom: you also replay the answers of the fourth mode, and save them:

python -m scripts.evaluate --set dev --config RETRIEVAL_MODE=dense_rerank --out reports/dev-dense-rerank.json

dense_rerank, answer_v2, recorded: 62 questions | correct 0.887 | retrieval hit 0.943 | citations valid 0.980 | abstained when it should 0.889 | injection obeyed 0 of 4 | median 358.0 ms | US$0.000152 per question

It looks like a fair fourth result: 55 of 62, a little below the baseline. It is not.

Cause: the course team recorded no answers for this mode. The recorded provider replays an answer when every passage that it cites is among your passages, even if the recording was made with other passages. So for this mode, it replays answers that other versions wrote from other contexts. You measured a mixture, not chat-small with dense_rerank.

Diagnose: open reports/dev-dense-rerank.json and read the note of each row. In this run, 20 of the 62 rows say “Recorded with passages …; yours: …”. In 17 of them, the recorded passages are not the same as yours. The same passages in another order are harmless: the hybrid run has 7 such notes. When a cited passage is missing, the note says that the mock answered.

Recover: do not put this number next to the others. Compare the modes on retrieval only, as above, or record live answers for this mode with your own key (ANSWER_PROVIDER=live). Every number in a report must say where its answers came from.

Your project

On the standard track, run the steps above. Your numbers should match Sam’s. Then do the sorting yourself before you read Sam’s table. Open reports/dev-baseline.json, read the six wrong answers in full, and write a category for each one. Compare your table with Sam’s.

On the own-idea track, do the same with your own tuning questions:

• Run --set dev --show-failures, read every wrong answer in full, and sort them into the categories. Add a category if yours do not fit: a data problem (the document is wrong or old) is common.
• Draw your own error-priority matrix: questions per category, and the cost to try a fix in your project. A fix that needs a model call for every question is expensive; a setting is cheap.
• Write two or three candidate changes for the top-left category, each with its reason, before you run them. Then run each one on the tuning questions only.
• If no change helps, keep the baseline. “We tried three changes, and none was better on the tuning questions” is a valid, honest result.

Evidence to keep: reports/dev-baseline.json, your category table, your matrix, the outputs of each change you tried (retrieval and answers), and the commit of the candidate.

Checklist

• Every wrong tuning answer is read in full and has a category.
• The matrix shows the number of questions and the cost to try a fix.
• The changes were written down, with reasons, before they ran.
• Each change was measured on the tuning questions only, one at a time.
• Answers replayed from another version, or made by the mock, were not counted as model answers.
• The chosen change is in .env (or the code), and the held-out set was not run.

See it end to endCapstone on a public document collectionCase study

The case study reads its three held-out failures: one key that was too strict, and two search misses. For the stand-up question, the vector search put the right section first and the keyword search missed it, so fusion pushed it to sixth, below the cut of 5.

Check your understanding

Q46's answer says '5–10 business days', and the key wants '5 to 10' or '5 - 10'. Sam could change the key, and the baseline would get 57 of 62. Why does Sam not do it, and what does Sam do instead?

Show the answer

The questions and their keys were frozen in Module 1, and the held-out questions use the same kind of keys. A key changed after reading the answers tunes the measure to the answer. The baseline would look better, with no change to the assistant. Sam keeps the key and counts Q46 as “key too strict”: a problem of the measure, not of the assistant. The evaluation report names it, so that a reader can correct the number in their head.

Check your understanding

Predict: the reranker got the same retrieval hits as hybrid (51 of 53). Why did Sam still not choose hybrid_rerank?

Show the answer

Because the answers and the cost decide, not the hit count alone. With real answers, hybrid_rerank got 56 of 62, the same as the baseline: it fixed three questions and broke three (Q57, Q60, B04). It also adds about 0.4 seconds to every search and a second model of about 470 MB. Hybrid got 60 of 62 at the baseline’s speed. Same hits, but different questions and a higher cost.

Check your understanding

Hybrid fixed Q30 and Q54, which were in the 'passage there, answer wrong' category, not in retrieval. How can a search change fix them?

Show the answer

The model answers from the five passages that the search gives it. Hybrid changed which five passages it read, and in what order. With other passages in its context, the model answered Q30 with the 3-year pump warranty and answered Q54 in English. A change at one stage can change the results at a later stage, for better or worse. That is why Sam measures the answers of every change, not only the stage it was aimed at.

Recap

• Sort the wrong tuning answers by the stage that failed. Sam’s six: retrieval missed (Q26, Q64), passage there but answer wrong (Q30, Q54), answered the unanswerable (B32), key too strict (Q46).
• The error-priority matrix weighs questions affected against the cost to try a fix. Retrieval was top left: two questions, a setting and one migration, no model calls.
• Three versions of one change, written before running: hybrid, dense_rerank, hybrid_rerank. Retrieval hits hardly moved; the answers decided: hybrid 60 of 62, hybrid_rerank 56, the baseline 56.
• The reranker is measured, slower, and not better here. It stays as an option.
• The candidate is RETRIEVAL_MODE=hybrid. Its tuning score is too optimistic: only the held-out run can confirm it.

Next: Document trade-offs