Retrieval measures

What you will be able to do

Measure how well search finds the evidence, with more than one number. Recall at k says how much of the evidence reaches the model. Precision at k says how much of the context is noise. The mean reciprocal rank says how near the top the evidence is.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• hit@5, the share of questions with a relevant passage in the top 5, from A keyword-search baseline.
• Precision@k and recall@k on one ranked list, from The precision–recall trade-off in Machine Learning: From Problem to Reliable Model.
• The retrieval methods (BM25, dense, hybrid, rerank) and query rewriting, from Lexical, dense and hybrid search and Reranking and query changes.

What you need on your computer

• ~/projects/policy-assistant at the end of Module 4, with .venv active: python -m pytest shows 74 passed, 1 deselected. If your project is broken, copy the snapshot end-of-m04 from the labs repository, as its README explains.
• The index index.sqlite from python -m policy_assistant ingest.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested. The numbers in this lesson are a snapshot of that date. Models and tools change fast; the way to measure them lasts.

A stack of library books

You write an essay about bees. You ask a librarian for books, and she brings a stack of five. How good is the stack? One question is not enough to say.

• Is at least one useful book in the stack?
• You needed three books: one on bee biology, one on honey, one on pesticides. How many of the three are in the stack?
• How many of the five books are useful, and how many only add weight?
• Is a useful book on top, or must you dig to the bottom?

A stack can pass the first question and fail the others. It holds the honey book, so it is “not empty”. But the two other books that you needed are missing, and you must open four books to find the useful one.

The problem: hit@5 says “found” when evidence is missing

Since Module 1, the project has one retrieval measure: hit@5. It asks only the first question. Look at Q46: “The help centre FAQ says refunds take 3 business days. Is that right?”

To answer it well, the model needs three passages. The FAQ says 3 business days. The refunds and payments policy says 5 to 10 business days. The governance rule says that a policy wins over a help article. Reranked search put the FAQ at rank 1, so hit@5 counts Q46 as a success. But the other two passages are not in the top 5. The recorded answer could only say that the FAQ is not confirmed.

hit@5 also ignores the order. Three of Larkfield’s retrieval methods have exactly the same hit@5, 0.898. Are they equally good? This lesson answers with measures that can tell them apart. Chunk deliberately showed another blind spot: a chunk can count as a hit and still miss the answer. These measures do not fix that one. The last lesson of this module shows how to find it.

The idea: relevance labels and three measures

Every measure needs to know which passages are the right ones. A relevance label says, for one question, which passages hold the evidence. Grace wrote them for all 67 questions, by reading the documents. For each question, she listed the document, its version and its section. A script then checked every label: the passage exists, is in force on the question’s date, and is public when the question is public. No model wrote or labelled a question. It is one person’s reading, and you can check every label in questions/questions.jsonl.

With the labels, the project measures three things for each question, and then takes the mean over the questions.

• Recall at k (recall@k): the share of the question’s relevant passages that the top k covers. Q46 needs three, and the top 5 covers one: recall@5 = 1 ÷ 3 = 0.33. It answers “how much of the evidence reaches the model?”
• Precision at k (precision@k): the share of the top k chunks that are relevant. It answers “how much of the context is evidence, and how much is noise?”
• Reciprocal rank: 1 ÷ the rank of the first relevant chunk. Rank 1 gives 1, rank 3 gives 1/3, and no relevant chunk in the top k gives 0. Its mean over the questions is the mean reciprocal rank (MRR). It answers “how near the top is the evidence?”

These are the precision@k and recall@k of the machine learning course, computed for each question and then averaged. hit@k stays as the simplest form of recall: “at least one”. Unanswerable questions have no relevant passage, so they are left out of all four measures.

Search engines also use nDCG (normalised discounted cumulative gain). It gives partial credit to passages that are only somewhat relevant, and less credit lower in the list. Grace’s labels say only “relevant” or “not relevant”, so this course does not need it.

A tiny worked example: five questions

Take five made-up questions, A to E, and the top 3 results for each (k = 3). These five are the test in tests/test_evaluate.py, so you can check every value.

The worked example of tests/test_evaluate.py: five questions, top 3 each. A blue box is a relevant result. E has no relevant passage, so it is left out of every mean.

Text description of the diagram

A table of five rows. A: result 1 relevant, results 2 and 3 not; hit 1, recall 1 of 1 = 1, precision 1 of 3, reciprocal rank 1. B: only result 3 relevant; hit 1, recall 1, precision 1 of 3, reciprocal rank 1/3. C needs two passages, and result 1 is one of them. C: hit 1, recall 1 of 2 = 0.5, precision 1 of 3, reciprocal rank 1. D has no relevant result, so every value is 0. E has no relevant passage, so it is left out. Under the rows are the means over A to D: hit@3 0.75, recall@3 0.625, precision@3 0.25, MRR 0.583.

Now take the means over the four questions that have relevant passages:

Measure	A	B	C	D	Mean over A to D
hit@3	1	1	1	0	3 ÷ 4 = 0.75
recall@3	1	1	0.5	0	2.5 ÷ 4 = 0.625
precision@3	1/3	1/3	1/3	0	1 ÷ 4 = 0.25
reciprocal rank	1	1/3	1	0	2.333 ÷ 4 = 0.583

Read the columns. A and B have the same hit, recall and precision. Only the reciprocal rank shows that B made you read two useless results first. C and A have the same hit, but recall shows that C is missing half its evidence. hit@3 gives one number for all of this; the other three measures say where the problems are.

Build it: the full measures in the project

The project’s new evaluate.py computes recall@k, precision@k and MRR, and the new __main__.py prints them in eval. evaluate.py also has the answer checks of the next lesson. Copy the three files from the snapshot end-of-m05.

1. Copy evaluate.py, __main__.py and the new test. On macOS and Linux:

   cp ../nextia-learning-labs/C09/snapshots/end-of-m05/policy_assistant/evaluate.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m05/policy_assistant/__main__.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m05/tests/test_evaluate.py tests/

   On Windows, in PowerShell:

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m05\policy_assistant\evaluate.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m05\policy_assistant\__main__.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m05\tests\test_evaluate.py tests\

2. Open policy_assistant/evaluate.py. Find retrieval_scores, and in it the four lines that compute hit, recall, precision and rr for one question.

3. Run the tests. You see 77 passed, 1 deselected:

   python -m pytest

   The new test test_five_questions_by_hand is the worked example above.

4. Measure the course default, reranked search with the top 5:

   python -m policy_assistant eval

   The command loads the two local models first, so it takes about 30 seconds. The output:

   rerank, top 5, audience staff: 59 questions with relevant passages
   hit@5 0.898 | recall@5 0.856 | precision@5 0.231 | MRR 0.831 | staff passages shown to a public audience: 0 | wrong versions: 0
   kind              n    hit  recall    MRR
   answerable       18  0.889   0.833  0.889
   contradiction     4  1.000   0.625  1.000
   exact-code       11  1.000   1.000  1.000
   injection-bait    3  0.333   0.333  0.333
   staff-only        7  1.000   1.000  1.000
   table             8  0.875   0.875  0.729
   version           8  0.875   0.875  0.525

Read the line under the title first. hit@5 is the 0.898 of Module 3. recall@5 is lower, 0.856, because some questions need two or three passages and get only part of them. The two last values are the checks of the Module 3 filters. No wrong version reaches the top 5. The leak count is always 0 for the staff audience; it counts only with --audience public, which Module 6 uses.

Then read the table by kind. The contradiction questions have hit 1.000 and recall 0.625. The top 5 always holds one side of the conflict, but often not the other side or the rule. The version questions have recall 0.875 and MRR 0.525: search finds the right version, but often not at rank 1. You could not see either problem with hit@5 alone.

Why is precision@5 so low, 0.231? Most questions have one relevant section, held in one chunk. Then at most 1 of the 5 chunks can be relevant, and precision@5 cannot pass 0.2. A low precision@5 is normal here. It shows the cost: most of the passages that the model reads are not evidence.

Recall at k for every retrieval method

Amira ran every method of Module 3 at k = 1, 3, 5 and 10, and Priya drew recall@k. A longer list can only add evidence, so every line rises. The question is how fast.

• Rerank
• Dense (e5)
• Hybrid (RRF)
• BM25

recall@k at k = 1, 3, 5 and 10 on Grace's 59 questions with relevant passages (structure chunks, date and access filters on, audience staff). Rerank is hybrid search's top 20 reread by the cross-encoder. Rerank leads at k = 1, 3 and 10; dense leads at k = 5 by less than one question; BM25 is lowest at every k.

Text description of the chart

A line chart with four lines. The x axis is k: 1, 3, 5 and 10. The y axis is recall@k, from 0.4 to 1. Rerank: 0.641, 0.825, 0.856, 0.921. Dense with e5: 0.531, 0.785, 0.862, 0.895. Hybrid with reciprocal rank fusion: 0.54, 0.743, 0.811, 0.904. BM25: 0.421, 0.689, 0.771, 0.828. Rerank is highest at k = 1, 3 and 10; dense is highest at k = 5. BM25 is lowest at every k.

Three things stand out. At k = 1, rerank finds 0.641 of the evidence and BM25 0.421: the cross-encoder’s work shows at the top of the list. At k = 5, dense search has the highest recall, 0.862, just above rerank’s 0.856. At k = 10, rerank reaches 0.921. So the best method depends on how many passages you give the model.

The table at k = 5, with the hosted embedding model and query rewriting as well:

Method	hit@5	recall@5	precision@5	MRR
BM25	0.831	0.771	0.207	0.655
Dense (e5, local)	0.915	0.862	0.234	0.769
Dense (embed-small, hosted)	0.881	0.836	0.231	0.827
Hybrid (e5, RRF)	0.864	0.811	0.220	0.745
Hybrid (embed-small, RRF)	0.898	0.845	0.227	0.787
Rerank (course default)	0.898	0.856	0.231	0.831
BM25 + rewrite	0.898	0.825	0.227	0.745
Rerank + rewrite	0.932	0.864	0.234	0.882

Now the question from the start of the lesson. Three methods have hit@5 0.898: BM25 with rewriting, hybrid with embed-small, and rerank. Their MRR is not the same.

Three methods with exactly the same hit@5 (0.898): BM25 with query rewriting, hybrid search with the hosted embed-small model, and rerank. They put the first relevant passage at different depths; rerank puts it highest.

Text description of the chart

A column chart of three columns, all for methods with hit@5 0.898. BM25 with query rewriting: MRR 0.745. Hybrid search with the hosted embed-small model: 0.787. Rerank: 0.831.

Each of the three finds evidence for 53 of the 59 questions. Rerank puts it higher. An MRR of 0.831 means that the first relevant passage is usually at rank 1. An MRR of 0.745 means that it is often at rank 2 or 3. Does the order matter if the model reads all five? It matters when the context budget cuts the list, when an agent reads the passages, and when a model trusts the first passage most.

Priya adds a warning. With 59 questions, one question is 1 ÷ 59 = 0.017 of hit@5. Dense search (0.915) and rerank (0.898) differ by one question. That is not proof that one is better. Look at which questions changed before you believe a difference.

When to use each measure, and what it costs

• recall@k is the main measure for RAG. The model can use only the evidence that reaches it, so recall@k limits how many answers can be good. Measure it at the k that you really send to the model.
• MRR matters when the order matters: a small budget, a person who reads the top results, or a reranker that you want to judge.
• precision@k measures the noise and the cost. Every passage that is not evidence costs tokens and can mislead the model. Do not chase it alone: k = 1 gives the best precision and loses evidence.
• hit@k stays useful as a quick first check.

All four need relevance labels, and the labels are the expensive part. Someone who knows the documents must write them by hand, and update them when a document changes. They are also one person’s reading. A second reader may find a relevant passage that Grace did not list, and a method that finds it gets no credit.

Common mistake

The question set changed between two comparisons

Symptom: suppose that Tomás compares the methods twice, a week apart. The first time, hosted embed-small has the best ranking: MRR 0.832 against 0.819 for rerank. The second time, rerank is best: 0.831 against 0.827. Nobody changed the search code. “Which one is true?”

Cause: the question set changed. The first runs used 55 questions with relevant passages. Those runs showed that product codes inside full sentences did not separate keyword and vector search. So Grace added four questions that are only an identifier, such as Q64 “What is LB-1?”. Here are the same recorded results, measured on both sets:

Method	MRR on the first 55	MRR on all 59	hit@5 on 55	hit@5 on 59
BM25	0.630	0.655	0.818	0.831
Dense (e5)	0.809	0.769	0.927	0.915
Dense (embed-small)	0.832	0.827	0.891	0.881
Rerank	0.819	0.831	0.891	0.898

Keyword search ranks the identifiers first, and dense e5 misses them. So the four new questions raise BM25 and lower dense search, and the order of the two best methods swaps. Both orders are within one or two questions, so neither comparison proves much.

Fix: freeze the question set, its labels and its keys before a final comparison, and give the set a version. Measure every method on the same version. Adding questions during development is good: Grace added these because the first runs showed a gap. But a number measured on 55 questions cannot be compared with a number measured on 59.

For a real assistant: write the version of the question set next to every number in a report. When the set changes, measure every method again.

Your turn: change k

Predict before you run: when k goes from 5 to 1, does recall go up or down? And precision? Then run:

python -m policy_assistant eval --k 1
python -m policy_assistant eval --k 10

Show the answer

The second line of each output:

hit@1 0.797 | recall@1 0.641 | precision@1 0.797 | MRR 0.797 | staff passages shown to a public audience: 0 | wrong versions: 0

hit@10 0.966 | recall@10 0.921 | precision@10 0.125 | MRR 0.840 | staff passages shown to a public audience: 0 | wrong versions: 0

From k = 1 to k = 10, recall rises from 0.641 to 0.921 and precision falls from 0.797 to 0.125. It is the trade-off of the machine learning course. At k = 1, precision, hit and MRR are the same number, because one result is either relevant or not. MRR rises only a little at k = 10, from 0.831 to 0.840. A relevant passage at rank 6 to 10 adds at most 1/6. In the table by kind, the version and table questions reach 1.000 at k = 10. Their evidence was there, just below rank 5. The last lesson of this module tests whether 10 passages give better answers.

See it end to endSearch technical documentationCase study

The technical documentation case study measures every search method with recall at k, MRR and hit at 1 on a labelled question set.

Knowledge check

Check your understanding

In your own words: what does recall@5 measure that hit@5 does not? Use Q46 as the example.

Show the answer

hit@5 asks whether at least one relevant passage is in the top 5. recall@5 asks what share of all the relevant passages is there. Q46 needs three passages: the FAQ, the refunds policy and the governance rule. The top 5 holds only the FAQ. So hit@5 is 1, and recall@5 is 1 ÷ 3 = 0.33. The model gets one side of the conflict and no rule.

Check your understanding

Predict: a question has one relevant passage, found at rank 4. What are its hit@5, recall@5, precision@5 and reciprocal rank?

Show the answer

hit@5 = 1, because it is in the top 5. recall@5 = 1 ÷ 1 = 1, because the only relevant passage is found. precision@5 = 1 ÷ 5 = 0.2. Reciprocal rank = 1 ÷ 4 = 0.25. The same passage at rank 1 would change only the reciprocal rank, to 1.

Check your understanding

Dense search has recall@5 0.862 and rerank 0.856, but rerank has the higher MRR (0.831 against 0.769). Omar asks which one to use. What do you tell him, and what would you check first?

Show the answer

The recall difference is less than one question in 59, so it proves nothing. The MRR difference is larger: rerank puts the evidence nearer the top. Keep rerank if the order matters (a small budget, agents who read the passages) and if 0.3 seconds more per question is acceptable. First, check which questions each method finds and misses. Two methods with the same mean can fail on different questions.

Recap

• A relevance label lists the passages that hold the evidence for one question. Grace wrote them by hand, and a script checked them. Every retrieval measure depends on them.
• recall@k: the share of the relevant passages in the top k. precision@k: the share of the top k that is relevant. MRR: the mean of 1 ÷ the rank of the first relevant passage. hit@k is the simplest recall.
• The course default is rerank with the top 5: hit@5 0.898, recall@5 0.856, precision@5 0.231, MRR 0.831. Recall shows missing evidence on contradiction questions; MRR shows deep ranks on version questions.
• Methods with the same hit@5 can rank very differently, and the best method depends on k. One or two questions in 59 are not proof.
• Freeze and version the question set before a final comparison: four added questions swapped the order of the two best methods.

Next, the same care for the answers: correctness, support, citations and usefulness.

Next: Answer measures