Retrieval metrics

What you will be able to do

Measure a search component on its own, from saved results and relevance labels. When an answer is wrong, these measures tell you three things. Did the search find the evidence at all? How much of it? How near the top?

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Precision@k and recall@k on one ranked list, from The precision–recall trade-off in Machine Learning: From Problem to Reliable Model. If you have not taken it, this lesson explains what you need.
• Component evaluations, the middle layer of the evaluation pyramid, from Choose evaluation layers.

You do not need RAG: Building AI That Uses Your Data. This lesson supplies the search results that it produced.

What you need on your computer

• The project ~/projects/eval-harness with .venv active: python -m pytest shows 13 passed. If it does not, copy the snapshot end-of-m02, as Use a snapshot explains.
• The labs repository in ~/projects/nextia-learning-labs, from Choose evaluation layers. You copy four files from it.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

Five folders from the filing cabinet

You ask a colleague a question: “How long is the warranty on a garden sprinkler?” She goes to the filing cabinet and brings back five folders, the best one on top. How good was her search?

• Is the right folder anywhere in the five?
• The answer is in two places. Did she bring both?
• How many of the five folders were useful, and how many did you open for nothing?
• Was a useful folder on top, or at the bottom of the pile?

If the right folder is not in the five, you cannot answer, however carefully you read. That failure belongs to the search, not to you.

Why this course measures a search

Many AI systems first retrieve: they search a collection of documents and pass the best passages to a language model. Larkfield’s policy assistant, built in the RAG course, works this way. When it answers wrongly, there are two suspects. Either the search did not find the evidence, or the model did not use it.

A test of the final answer cannot tell them apart. So you test the search on its own: a component evaluation. You need only two things, and both are saved in files:

• Saved search results. For each question, the passages that the search returned, best first. These come from four search methods of the RAG course.
• Relevance labels. For each question, the passages that hold the evidence. The RAG course’s author marked them, and a script checked them. It is one person’s reading.

No search runs in this lesson. You only score its saved output, exactly as you score the assistant’s saved replies.

The problem: the search never showed the answer

Here is a real question, Q34: “How long is the warranty on a garden sprinkler?” The evidence is in the warranty policy, in a table split over two passages. Here are the top 5 results of keyword search, and of the reranked search:

Rank	Keyword search (document / section)	Reranked search (document / section)
1	product-care-garden-furniture / Warranty	warranty-policy / Warranty periods by product category ✓
2	help-centre-faq / How long does a refund take?	product-care-garden-furniture / Warranty
3	refunds-and-payments / How long a refund takes	product-care-hedge-trimmer-ht550 / Warranty
4	privacy-policy / How long we keep it	damaged-or-wrong-items / After 14 days
5	returns-policy / Refunds	warranty-claims-procedure / Pressure washer pumps

With keyword search, no relevant passage reaches the model. The words “how long” pull in passages about refund times and about how long data is kept. Any model would fail here, or say that it does not know. With reranked search, the right passage is at rank 1. The problem is in the search, and only a search measure can show it.

The idea: four measures on a ranked list

For one question, call the passages that the search returned the ranked list, and the labelled passages the relevant ones. Choose k, the number of results that the model receives: here, 5.

• hit@k: 1 if at least one relevant passage is in the top k, else 0. “Is the evidence anywhere?”
• recall@k: the share of the relevant passages that are in the top k. “How much of the evidence arrives?”
• precision@k: the share of the top k that is relevant. “How much of what the model reads is useful?”
• reciprocal rank (RR): 1 ÷ the rank of the first relevant passage. Rank 1 gives 1, rank 2 gives 1/2, rank 3 gives 1/3; none in the top k gives 0. “How near the top is the evidence?”

Each measure is computed per question, then averaged over the questions. The mean of the reciprocal ranks is the mean reciprocal rank (MRR). Some questions have no relevant passage, because the documents do not answer them. They have nothing to find, so the measures leave them out.

These are the precision and recall of the last lesson, applied to a list instead of a yes-or-no decision.

A tiny worked example: five questions

Five made-up questions, top 5 each, with every value. They are the worked example in tests/test_retrieval.py.

The worked example: five questions, the top 5 results of each, best first. r1 and r2 are relevant passages; a dash is not relevant. The means of the five give MRR 0.567.

Text description of the diagram

Question A needs 1 passage, found at rank 1: hit 1, recall 1, precision 0.2, RR 1. Question B needs 1 passage, found at rank 3: hit 1, recall 1, precision 0.2, RR 1/3. Question C needs 2 passages, found at ranks 2 and 4: hit 1, recall 1, precision 0.4, RR 1/2. Question D needs 1 passage, not found: all values 0. Question E needs 2 passages, one found at rank 1: hit 1, recall 0.5, precision 0.2, RR 1. Means of the five: hit@5 0.8, recall@5 0.7, precision@5 0.2, MRR 0.567.

Follow two questions closely.

• B finds its passage, but only at rank 3. hit and recall say “perfect”. Only RR, 1/3, shows that the model must read past two useless passages first.
• E has its passage at rank 1, so RR is 1. But it needs two passages, and the second is missing. Only recall, 0.5, shows it.

So each measure sees a failure that another misses. The MRR is (1 + 1/3 + 1/2 + 0 + 1) ÷ 5 = 2.833 ÷ 5 = 0.567. Precision@5 is low everywhere, 0.2. Most questions have only one or two relevant passages, and the top 5 has five places. A low precision@5 is normal here; compare it between methods, not with 1.

Build it: the measures in the project

1. Get the newest labs files. In the terminal, go to ~/projects:

   cd ~/projects
   git -C nextia-learning-labs pull

   If you downloaded the labs as a ZIP file, download it again instead.

2. Go to the project, and copy the saved search results, the scorer and its test from the snapshot end-of-m03:

   macOS and Linux

   cd eval-harness
   mkdir -p data/retrieval
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/data/retrieval/rankings.json data/retrieval/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/data/retrieval/README.md data/retrieval/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/harness/scorers/retrieval.py harness/scorers/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m03/tests/test_retrieval.py tests/

   Windows

   cd eval-harness
   New-Item -ItemType Directory -Force data\retrieval
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\data\retrieval\rankings.json data\retrieval\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\data\retrieval\README.md data\retrieval\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\harness\scorers\retrieval.py harness\scorers\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m03\tests\test_retrieval.py tests\

3. Open harness/scorers/retrieval.py in VS Code. Find the four functions hit_at, recall_at, precision_at and reciprocal_rank. Each is one or two lines, the definition above written in Python. Then read evaluate(): it skips the questions with no relevant passage, and averages the four values over the rest.

4. Open harness/__main__.py. Add this function above def main(:

   harness/__main__.py

   def cmd_retrieval(a) -> None:
       from .scorers.retrieval import evaluate, load_rankings
   
       data = load_rankings()
       print(f"Saved search results: {len(data['questions'])} questions, top {a.k}")
       print(f"{'method':9}{'hit':>7}{'recall':>8}{'precision':>11}{'MRR':>7}")
       for method in data["methods"]:
           m = evaluate(data, method, a.k)
           print(f"{method:9}{m['hit']:7.3f}{m['recall']:8.3f}{m['precision']:11.3f}{m['mrr']:7.3f}")

5. In main(), add the new command below the lines of score, after s.set_defaults(fn=cmd_score):

   harness/__main__.py

       s = sub.add_parser("retrieval", help="ranking measures on the saved search results")
       s.add_argument("--k", type=int, default=5)
       s.set_defaults(fn=cmd_retrieval)

6. Run the tests:

   python -m pytest

   The last line is 15 passed. test_worked_example checks every value of the five questions above.

7. Score the four search methods:

   python -m harness retrieval

   Saved search results: 67 questions, top 5
   method       hit  recall  precision    MRR
   hybrid     0.864   0.774      0.220  0.745
   keyword    0.831   0.751      0.207  0.655
   rerank     0.898   0.808      0.231  0.831
   vector     0.915   0.816      0.234  0.769

If retrieval gives invalid choice, the parser lines of step 5 are missing or sit inside another block: check their indentation. If it gives FileNotFoundError, rankings.json is not in data/retrieval/.

Read the four methods

The 67 questions include 8 that the documents do not answer, so every mean is over 59 questions. There are four methods. Keyword search (BM25) counts matching words. Vector search compares meanings. Hybrid merges both lists. Rerank reorders hybrid’s top 20 with a second model.

The best method depends on the measure. Vector search has the best hit@5 (0.915) and recall@5 (0.816): it brings the most evidence into the top 5. Rerank has the best MRR (0.831): when it finds the evidence, it puts it near the top. Keyword search is last on all four. If the model reads only the first passage, rerank is the better choice. If it reads all five, vector search is slightly ahead: for hit@5, the difference is one question (54 against 53).

The value of k is a choice too. A larger k brings more evidence and more noise:

• vector
• rerank
• hybrid
• keyword

Recall@k of the four search methods at k = 1, 3, 5 and 10, on the 59 questions with relevant passages. Recall rises with k for every method; keyword search stays lowest.

Text description of the chart

A line chart with k (1, 3, 5, 10) on the horizontal axis and recall@k from 0 to 1 on the vertical axis. Vector: 0.52, 0.757, 0.816, 0.845. Rerank: 0.613, 0.777, 0.808, 0.87. Hybrid: 0.528, 0.723, 0.774, 0.856. Keyword: 0.41, 0.678, 0.751, 0.808. At k = 1, rerank is far ahead; at k = 5 and 10, vector and rerank are close.

At k = 1, rerank’s recall is 0.613 and vector’s only 0.52. At k = 10, they are almost equal. Precision falls the other way: at k = 10, about one passage in eight is relevant. Each extra passage costs tokens and gives the model more to ignore.

One detail if you take the RAG course too. This harness counts recall per relevant passage (chunk). The RAG course counts relevant sections, so its keyword recall@5 is 0.771, not 0.751. Hit, precision and MRR are the same in both. Q34 shows the difference. Its table is one section in two passages. So rerank’s recall for Q34 is 0.5 here, and 1.0 in the RAG course.

Your turn: one question, then k = 1

1. Start Python in the project folder, and score Q34 yourself:

   >>> from harness.scorers.retrieval import load_rankings, recall_at, precision_at, reciprocal_rank
   >>> data = load_rankings()
   >>> q = next(q for q in data["questions"] if q["id"] == "Q34")
   >>> q["question"]
   'How long is the warranty on a garden sprinkler?'
   >>> relevant = set(q["relevant"])
   >>> q["relevant"]
   ['d9a5bd8ea531', 'b47ef9ac1993']
   >>> for method in ("keyword", "rerank"):
   ...     ranked = data["rankings"][method]["Q34"]
   ...     print(method, recall_at(ranked, relevant, 5), precision_at(ranked, relevant, 5), reciprocal_rank(ranked, relevant, 5))
   ...
   keyword 0.0 0.0 0.0
   rerank 0.5 0.2 1.0

   Leave Python with exit().

2. Predict before you run: at k = 1, how do hit, precision and MRR compare for one method? Then run python -m harness retrieval --k 1.

Show the result and the reason

Saved search results: 67 questions, top 1
method       hit  recall  precision    MRR
hybrid     0.678   0.528      0.678  0.678
keyword    0.542   0.410      0.542  0.542
rerank     0.797   0.613      0.797  0.797
vector     0.678   0.520      0.678  0.678

With one result, all three ask the same question: is the first result relevant? If it is, hit is 1, precision is 1 ÷ 1 and RR is 1/1. If not, all three are 0. Recall is lower, because a question with two relevant passages can get at most half of them in one place. At k = 1, rerank beats vector search clearly (0.797 against 0.678); at k = 5, vector search has the higher hit@5.

When to use each measure, and what it costs

Measure	Use it when	It cannot see
hit@k	The model needs one passage to answer	A second passage that is missing; the rank
recall@k	An answer needs several passages (a rule and its exception)	Where in the list they are
precision@k	The context is expensive, or noise misleads the model	Whether the right passage is there at all
MRR	The model or the reader mostly uses the top result	Every relevant passage after the first

The measures are cheap once the labels exist: plain counts on saved files. The labels are the cost. Someone must read the documents and mark the evidence for every question, and then check the marks. And they have limits: a passage that holds half the answer counts as relevant, and an unlabelled passage that also answers counts as noise. Retrieval measures say whether the evidence arrived, not whether the final answer used it. That needs the checks of the next lesson.

Common mistake

Counting the questions that have no answer

Symptom: a colleague scores keyword search with a quick script. It gives hit@5 0.731, not 0.831. Rerank drops from 0.898 to 0.791. “Our search is worse than the report says.”

Cause: the script divides by all 67 questions. Eight of them (Q14 to Q21) have no relevant passage, because the documents do not answer them: “Do we offer a student discount?” Their hit can only be 0. Keyword search finds the evidence for 49 of the 59 answerable questions: 49 ÷ 59 = 0.831, but 49 ÷ 67 = 0.731.

Fix: leave out the questions with no relevant passage, as evaluate() does, and say how many questions the mean covers. Measure the unanswerable questions in another way: the system should say that the documents do not answer them. That is a check on the reply, not on the search.

Check your understanding

Two search methods have the same hit@5 on a set of questions. What can still differ, and which measure shows it?

Show the answer

The rank of the evidence and how much of it arrives. MRR shows the rank: one method can put the relevant passage first, the other at rank 4 or 5. Recall@5 shows how much of the evidence arrives, for questions that need more than one passage. In the worked example, B and E both have hit 1; B’s RR is 1/3, and E’s recall is 0.5.

Check your understanding

Larkfield's policy assistant gives the wrong warranty period for Q34 when it uses keyword search. Before you change the prompt, what do the retrieval measures say?

Show the answer

With keyword search, Q34 has hit@5 0, recall@5 0 and RR 0: neither warranty-table passage is in the top 5. The model never saw the evidence, so a better prompt cannot fix it. Change the search first: reranked search puts one table passage at rank 1 (recall@5 0.5, RR 1).

Check your understanding

Why is precision@5 low for every method (about 0.2), and is that a problem?

Show the answer

Most questions have only one or two relevant passages, and k = 5 always has five places. So even a perfect search scores at most 0.2 for a question with one relevant passage. It is not a problem in itself; compare precision@k between methods at the same k, and remember that every extra passage costs tokens.

Recap

• A component evaluation scores the search alone, from saved results and relevance labels. No search runs.
• hit@k: is any evidence there? recall@k: how much of it? precision@k: how much of the top k is useful? MRR: how near the top?
• Each sees a failure that the others miss. In the worked example, the MRR is 0.567.
• On Larkfield’s 59 answerable questions at k = 5, vector search has the best hit and recall (0.915, 0.816). Rerank has the best MRR (0.831). At k = 1, rerank leads clearly.
• Leave out questions with no relevant passage, and say how many questions the mean covers.

Your project now has harness/scorers/retrieval.py, data/retrieval/ and the retrieval command, and python -m pytest shows 15 passed. Next, the hardest component to measure: the written reply.

Next: Generated-output metrics