Reranking and query changes

What you will be able to do

Put the right passage at the top of the list, not only somewhere in the top 20. You add a second, slower reader that rereads the best candidates, and you let a language model rewrite the question. Then you measure what each one gains, what it costs in time, and how it can fail.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Keyword, vector and hybrid search, the filters and the top k, from Lexical, dense and hybrid search.
• A provider adapter, the mock provider and recorded answers, from Building Reliable Applications with LLM APIs (the course). This lesson uses the same pattern.

What you need on your computer

• The project from Lexical, dense and hybrid search, with .venv active. python -m pytest shows 44 passed, and rerank.py is in policy_assistant/.
• About 0.5 GB of free disk space for the reranker model.

No account and no key: the query rewrites come from recordings. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A second reader

A company gets 200 applications for one job. A recruiter skims all 200 in an hour, looking for a few words: the right degree, the right tool, the right city. She keeps 20. Then a manager reads those 20 carefully, from start to end, and puts them in order. The manager is much slower, so she could never read all 200. But she reads 20 well.

Search can work the same way: a fast first search keeps 20 candidates, and a slower, more careful model reads only those.

The problem: the right passage is in the list, but too low

Camille asks: “Combien de temps dure la garantie d’un barbecue?” (How long is the warranty on a barbecue?) The answer is in the English warranty table: “BBQs and grills | the Ember GR-310 | 2 years”. Here is hybrid search:

python -m policy_assistant search "Combien de temps dure la garantie d'un barbecue?" --method hybrid

1.   0.0315  [a540f2d18f86] garantie-legale-quebec v1 | Garantie prolongée | public
2.   0.0315  [09e9e322e55c] garantie-legale-quebec v1 | Comment faire une demande | public
3.   0.0313  [578b6d5a7284] garantie-legale-quebec v1 | Deux garanties différentes | public
4.   0.0310  [ad9138005226] garantie-legale-quebec v1 | Après la fin de la garantie L… | public
5.   0.0301  [4bf0b43abe06] faq-quebec v2 | Comment retourner un article? | public
hybrid, top 5, as of 2026-10-09, audience staff

(The text line under each result is left out here.) All five chunks are French, because they share “garantie” with the question. None of them gives a warranty period for a barbecue. The warranty table is in the list of 20, at rank 10. It was found, but it was not put at the top.

The same happens to “What is LB-1?”: hybrid search has the hedge trimmer passage at rank 6. In both cases, the first search did its job, which is to collect candidates. The ordering is the weak part.

The idea: read the question and the passage together

An embedding model reads the question alone and the passage alone, and it makes one vector for each. Only then are the two vectors compared. The model never sees both texts at the same time. It cannot notice that “barbecue” matches the table row “BBQs and grills”. It cannot see that “garantie” in a French heading is about another topic.

A cross-encoder reads the question and one passage together, as one input, and returns one score: how well this passage answers this question. The name says what it does: it encodes across the two texts. A model that embeds each text separately is, by contrast, a bi-encoder. Reranking means: take the top candidates of a fast search, score each one with a cross-encoder, and sort them again by that score.

An embedding model reads each text alone, so passage vectors can be made once, in advance. A cross-encoder reads the pair together: more careful, but it must run again for every question and every candidate.

Text description of the diagram

On the left, the embedding model e5. It reads “query: What is LB-1?” alone and returns 384 numbers. Separately, it reads “passage: After each use … blade oil LB-1 …” and returns 384 numbers. The two vectors are compared with cosine similarity, which gives one number. The passage vectors are made once, at ingest, so a search only embeds the question. On the right, the cross-encoder. It reads “What is LB-1?” and “After each use … blade oil LB-1 …” together, as one input, and returns one score: −0.87. It must run once for each candidate passage, for every new question.

So why not use a cross-encoder for everything? Because nothing can be prepared in advance. For every question, it must read every passage again. With 185 chunks that would take seconds per question; with a million, hours. That is why it only rereads the top 20 of a fast search, like the manager who reads 20 applications, not 200.

Larkfield’s reranker is cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, an open model under the Apache 2.0 licence, about 470 MB. It was trained on question and passage pairs in 14 languages. The course also tried an English-only reranker, cross-encoder/ms-marco-MiniLM-L6-v2. It took less than half the time: 167 ms against 391 ms for 20 candidates, in the model comparison run. But on the French questions, its hit@5 fell to 0.625, against 0.875.

Worked example: LB-1, reranked

Here is “What is LB-1?” with --method rerank. The project takes hybrid search’s top 20, after the filters, and lets the cross-encoder score all 20:

1.  -0.8719  [86a0a5266439] product-care-hedge-trimmer-ht550 v1 | After each use | public
   After each use Unplug it, clean the blade, and put a few drops of blade oil LB-1 along the blade. F…
2.  -5.2345  [d753fb02bc8c] privacy-policy v3 | Why we use it | public
   Why we use it We use your information to deliver orders, answer your questions, prevent fraud, and …
3.  -5.4942  [a31b42d3e40d] damaged-or-wrong-items v2 | Wrong item | public
   Wrong item We send a free return label. We ship the right item as soon as the courier scans the ret…
4.  -5.5988  [e50c70a93a32] warranty-claims-procedure v3 | Approval limits | staff
   Approval limits - An agent may approve a repair, replacement or refund worth up to 300 dollars. - F…
5.  -5.6818  [13f00fce8eaa] privacy-policy v3 | Our help-desk assistant | public
   Our help-desk assistant Our help-desk staff use an assistant that drafts answers from Larkfield's p…
rerank, top 5, as of 2026-10-09, audience staff

The hedge trimmer passage moves from rank 6 to rank 1. Look at the scores. The first is −0.87, and the second is −5.23: a gap of more than 4. The cross-encoder is much more decided than e5 was, whose scores sat close together.

The scores are not probabilities and not cosine similarities. They are raw numbers from the model, and only their order counts. A negative score does not mean “wrong”: −0.87 is the best of the 20 here. Do not set a fixed cut-off such as “keep scores above 0” without testing it on your questions.

The French barbecue question gets the same help. After reranking, the warranty table is first, with −0.55, and the French chunks about the legal warranty leave the top 5.

What reranking adds, and what it costs

One way to measure ranking is the mean reciprocal rank (MRR). For each question, take 1 divided by the rank of the first relevant passage. Then take the average over the questions. A passage at rank 1 gives 1, at rank 2 gives 0.5, at rank 5 gives 0.2, and a miss gives 0. So MRR rewards a relevant passage near the top, not only in the top 5. Retrieval measures teaches it in full.

How high the first relevant passage comes, on average. Reranking gives the best order; rewriting the question first improves it again.

Text description of the chart

A column chart of the mean reciprocal rank on Larkfield’s 59 questions with a relevant passage, top 5, filters on. BM25: 0.655. Vector search with e5: 0.769. Hybrid search: 0.745. Rerank, the hybrid top 20 reread by the cross-encoder: 0.831. Rerank with the rewritten question: 0.882. The axis goes from 0 to 1.

Reranking the hybrid candidates raises hit@5 from 0.864 to 0.898, and MRR from 0.745 to 0.831. In questions: it brings 4 relevant passages into the top 5, and it pushes 2 out. The 4 are LB-1, the French barbecue, a French returns question and a holiday-returns question. One of the 2 is Q60, “Larkfield vend-elle une garantie prolongée?” (Does Larkfield sell an extended warranty?): rank 3 before reranking, rank 14 after. Every change moves some questions up and some down. Look at both lists, not only the total.

Vector search alone still has the best hit@5, 0.915. But rerank has the best order: its first result is relevant for 80% of the questions, against 68% for vector search. The answer step in Module 4 reads the passages in rank order, so the course keeps rerank as the default method.

Now the price. These are medians on the test computer, a shared Mac with 2 CPU threads for the models:

Step, for one question	Time
BM25 search	0.26 ms
Embed the question with e5	5.2 ms
Compare it with 185 vectors	0.09 ms
Rerank 20 candidates	322 ms

Reranking is more than 50 times slower than the rest of the search together. For one agent with a question, 0.3 s is fine. For a service that answers 100 questions a second, it is a CPU problem. The cost grows with the number of candidates: 20 is a choice, not a law. Fewer candidates is faster, and more candidates can find more.

Query rewriting: ask for the right words

Back to the library. Before you search the catalogue, a helpful librarian asks: “What do you mean by ‘a gift for my mother who gardens’?” Then she types “gardening gloves” or “pruning shears”. She changes your question into the words that the catalogue uses.

Query rewriting does this with a language model. The model gets the agent’s question and returns a search query, and the search uses that query instead. It helps where the question’s words are far from the documents’ words: French questions over English documents, and everyday words over policy words.

The project’s prompt is in policy_assistant/prompts/rewrite.md:

policy_assistant/prompts/rewrite.md

[system]
You turn a question from a help-desk agent into one search query for Larkfield's policy documents (an online shop for home and garden products in Canada). The documents are mostly in English, some in French.
- Write the query in English, even when the question is in French.
- Keep every product code, part number, amount and date exactly as written.
- Use the words that a policy document would use (for example "return window", "warranty period", "refund").
- Do not answer the question. Return only the query, on one line.

[user]
{question}

The rewrites of all 67 questions were recorded with chat-small (gpt-6-luna on Azure). The mock provider replays them, so you need no key. Here are some of them, with the rank of the first relevant passage for BM25 before and after the rewrite:

Question	Recorded rewrite	BM25 rank: before → after
Puis-je retourner une carte-cadeau?	{rw.Q63.rewrite}	not in top 20 → 2
How long is the warranty on a garden sprinkler?	{rw.Q34.rewrite}	16 → 1
Which pump kit fits the PW-2200?	{rw.Q38.rewrite}	1 → 1
How long do we keep chat transcripts with customers?	{rw.Q08.rewrite}	1 → 17
Larkfield vend-elle une garantie prolongée?	{rw.Q60.rewrite}	2 → not in top 20

The first three show the help. The French gift card question becomes English words, and BM25 now finds the gift card passage. “Garden sprinkler warranty period” is shorter and without the question words, and the warranty table jumps from rank 16 to rank 1. The product code PW-2200 stays exactly as written, as the prompt asks.

The last two show the harm. “Chat transcripts” became “retention period customer service records”. The privacy section that answers it, “How long we keep it”, drops from rank 1 to 17. Q60 is the worst case: the failure case below looks at it.

Look also at what the model added to some rewrites. These are real recorded outputs:

• Q05: {rw.Q05.rewrite}
• Q45: {rw.Q45.rewrite}
• Q46: {rw.Q46.rewrite}
• Q43: {rw.Q43.rewrite}

Characters from other languages and scripts, and a stray &returns, at the end of a query. Nothing in the prompt asked for them. Keyword search ignores most of them, because they match no chunk. But they show that a model’s output is never clean by default. Q46 also lost part of its meaning: the agent asked “The help centre FAQ says refunds take 3 business days. Is that right?”. The rewrite keeps “3 business days” and drops the doubt about the FAQ. It now searches for the claim, not for the question.

What rewriting adds, and what it costs

Method	hit@5	MRR
BM25	0.831	0.655
BM25 + rewrite	0.898	0.745
Rerank	0.898	0.831
Rerank + rewrite	0.932	0.882

Rewriting helps BM25 most: 8 questions gain a relevant passage in the top 5, and 4 lose one. With reranking, the gain is smaller, but rerank with rewriting is the best combination in this module.

It is not free. It adds a model call before every search: a median of 1.33 s here, four times the reranking time. The 67 recorded rewrites cost US$0.0029 in all, so the money is small. The bigger costs are elsewhere. The agent’s question now goes to a model provider. The search fails when the provider fails. And the model can change the meaning. So in this course, rewriting is an option (--rewrite), measured, and not the default.

The whole funnel

Here is the full retrieval of Larkfield’s assistant, from the 185 chunks to the 5 passages that the model will read. A rewrite, when it is on, changes the query before the first stage.

The retrieval funnel, now complete: the filters remove what is not allowed, two fast searches collect candidates, and the cross-encoder chooses the order of the top 5.

Text description of the diagram

Seven stages, from left to right. The 185 chunks of the index. The filters, first: only the version in force on the question’s date, and only the access labels that the audience may see. They remove other versions and, for a public assistant, staff-only chunks. The candidates: the top 20 from BM25 and the top 20 from vector search. Hybrid search fuses the two lists with reciprocal rank fusion and keeps 20. The cross-encoder reads the question with each of the 20 chunks and orders them again. The top 5 are the selected evidence, which goes to the model in Module 4.

Build it: reranking and rewriting

rerank.py came with the last lesson. Now the project gets the adapter for model calls, the rewrite prompt and its recordings, and rewrite.py. Work in ~/projects/policy-assistant, with .venv active.

1. Read policy_assistant/rerank.py. The method rerank is the whole idea of this lesson in a few lines:

   policy_assistant/rerank.py (part)

       def rerank(self, query: str, candidates: list, k: int = 5) -> list[Hit]:
           """candidates: chunks (with .chunk_id and .text). Returns the k best by the cross-encoder's score."""
           if not candidates:
               return []
           scores = self.model.predict([(query, c.text) for c in candidates], batch_size=16)
           order = sorted(range(len(candidates)), key=lambda i: (-float(scores[i]), candidates[i].chunk_id))
           return [Hit(candidates[i].chunk_id, round(float(scores[i]), 4), rank) for rank, i in enumerate(order[:k], 1)]

   predict gets one pair per candidate, (question, chunk text), and returns one score per pair. Like embed.py, the file pins the model’s revision, 1427fd652930e4ba29e8149678df786c240d8825.

2. Run the worked example. The first time, the reranker downloads about 470 MB:

   python -m policy_assistant search "What is LB-1?" --method rerank

   Check: the first line is 1. -0.8719 [86a0a5266439] product-care-hedge-trimmer-ht550 v1 | After each use | public.

3. Copy the adapter, its settings, the prompt folder, the example settings file and the recorded rewrites from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C09/snapshots/end-of-m03
   cp $S/policy_assistant/providers.py $S/policy_assistant/config.py policy_assistant/
   cp -R $S/policy_assistant/prompts policy_assistant/
   cp $S/.env.example .
   cp $S/recordings/rewrites.jsonl recordings/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C09\snapshots\end-of-m03"
   Copy-Item $S\policy_assistant\providers.py, $S\policy_assistant\config.py policy_assistant\
   Copy-Item -Recurse $S\policy_assistant\prompts policy_assistant\
   Copy-Item $S\.env.example .
   Copy-Item $S\recordings\rewrites.jsonl recordings\

   providers.py is the adapter of the LLM applications course, without tools and streaming: MockProvider replays recordings, and OpenAICompatibleProvider calls a real model. config.py reads the settings from environment variables or a .env file. The default provider is the mock, so you need no .env now.

4. Make the file policy_assistant/rewrite.py:

   policy_assistant/rewrite.py

   """Query rewriting: ask a model to turn the agent's question into a better search query.
   
   It can help (a French question becomes English words that match English documents; "two years without
   buying" becomes "points expiry") and it can hurt (the model drops a date or a code, or answers the
   question instead of searching). Measure it like any other change. The course's rewrites were recorded
   with chat-small; the mock replays them.
   """
   
   from .config import load_prompt
   
   MAX_TOKENS = 1000
   
   
   def build_request(question: str, model: str) -> dict:
       system, user = load_prompt("rewrite")
       return {"model": model,
               "messages": [{"role": "system", "content": system}, {"role": "user", "content": user.format(question=question)}],
               "max_completion_tokens": MAX_TOKENS}
   
   
   def rewrite(question: str, provider, model: str = "chat-small") -> str:
       completion = provider.complete(build_request(question, model))
       text = (completion.text or "").strip().splitlines()
       return text[0].strip().strip('"') if text else question

   rewrite keeps only the first line of the answer and removes quotes around it. If the model returns nothing, the original question is used.

   Do not change rewrite.md or build_request. The mock finds a recording by a hash of the whole request. A changed prompt is a new request, and the mock has no recording for it.

5. Copy the last test file, and run all the tests:

   macOS / Linux

   cp ../nextia-learning-labs/C09/snapshots/end-of-m03/tests/test_rewrite.py tests/
   python -m pytest

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m03\tests\test_rewrite.py tests\
   python -m pytest

   The last line is 48 passed. One of the new tests checks the stray characters of the Q05 rewrite: the test keeps the real, imperfect output on record.

Guided practice: measure both changes

1. Rerank Camille’s barbecue question. Compare it with the hybrid output above:

   python -m policy_assistant search "Combien de temps dure la garantie d'un barbecue?" --method rerank

   Check: the first result is warranty-policy v5 | Warranty periods by product c…, with the score -0.5515.

2. Measure rerank on the whole set. It takes about half a minute, because the cross-encoder reads 20 candidates for each of the 67 questions:

   python -m policy_assistant eval --method rerank

   The first and last lines:

   rerank, top 5, audience staff: hit@5 0.898 (59 questions with relevant passages)
   ...
   missed: Q13, Q28, Q36, Q56, Q57, Q60

3. Search the French gift card question with BM25, without and with the rewrite:

   python -m policy_assistant search "Puis-je retourner une carte-cadeau?" --method bm25
   python -m policy_assistant search "Puis-je retourner une carte-cadeau?" --method bm25 --rewrite

   The second output starts with the rewritten query, then three chunks of gift-cards:

   Rewritten query: gift card return policy&returns
   1.   9.7154  [8c4bc4299042] gift-cards v1 | Buying a gift card | public
   2.   8.7215  [52ad3a57c1d1] gift-cards v1 | Lost cards and returns | public
   3.   8.0177  [f27bb50ce8b7] gift-cards v1 | Using a gift card | public

   (The text line under each result is left out here.) “Lost cards and returns” is the relevant passage.

4. Measure the rewrite with BM25 and with rerank:

   python -m policy_assistant eval --method bm25 --rewrite
   python -m policy_assistant eval --method rerank --rewrite

   The first lines:

   bm25, top 5, audience staff: hit@5 0.898 (59 questions with relevant passages)
   rerank, top 5, audience staff: hit@5 0.932 (59 questions with relevant passages)

   Compare the missed: line of BM25 with the one in Lexical, dense and hybrid search, without the rewrite: missed: Q05, Q12, Q13, Q29, Q32, Q34, Q37, Q56, Q57, Q63. The rewrite finds 8 of them, but Q08, Q28, Q60 and Q61 are now missed, and they were found without it.

5. Try a question that was never recorded:

   python -m policy_assistant search "Can a customer return a used pressure washer?" --method bm25 --rewrite

   It stops with a traceback, which ends with:

   policy_assistant.providers.RecordingNotFound: No recording for this request (key 8250f8ad99fd1353). The mock replays only the questions that were recorded for the course, with the same prompt and the same model.

   This is expected. The mock has rewrites only for the 67 questions of the set. Without --rewrite, the same search works. To rewrite your own questions, use a live model: .env.example explains how, with Ollama on your computer or your own key. That is optional, and it is not tested in this course.

Your turn: keep the original words

The rewrite lost Q08, Q60 and Q61 because it threw away the agent’s own words. One idea: search with both, the question followed by its rewrite, as one query. BM25 then matches the French words and the English words.

Before you run it, predict: does hit@5 for BM25 go up, go down, or stay at 0.898? Then run this in Python, in the project folder:

Python, in ~/projects/policy-assistant

from policy_assistant.config import Settings, make_provider
from policy_assistant.embed import LocalEmbedder
from policy_assistant.evaluate import load_questions, retrieval_scores, summarise
from policy_assistant.filters import Filters
from policy_assistant.rewrite import rewrite
from policy_assistant.search import Retriever
from policy_assistant.store import Store

store = Store("index.sqlite")
retriever = Retriever(store, LocalEmbedder())
provider = make_provider(Settings.from_env())
rows = []
for q in load_questions().values():
    query = q.question + " " + rewrite(q.question, provider)
    results = retriever.search(query, "bm25", 5, Filters(as_of=q.as_of))
    rows.append(retrieval_scores(q, [r.chunk for r in results], 5))

print(summarise(rows), [r["id"] for r in rows if r["scored"] and not r["hit"]])

Show the answer

{'questions': 59, 'hit': 0.898} ['Q13', 'Q28', 'Q29', 'Q37', 'Q56', 'Q57']

The total stays at 0.898, but the questions change. Q08, Q60 and Q61 are found again, because the original words are back. Q29, Q37 and Q56 are lost again. Q29 and Q37 are French questions: their French words now pull French chunks up, as before the rewrite. The same total hides six questions that moved. This is why you compare the lists of missed questions, and not only the score. It is also why one more idea needs one more measurement before it goes into the assistant.

Common mistake

A rewrite in the wrong language

Symptom: with --rewrite, Camille’s question “Larkfield vend-elle une garantie prolongée?” (Does Larkfield sell an extended warranty?) finds nothing relevant. Without the rewrite, BM25 has the answer at rank 2: garantie-legale-quebec | Garantie prolongée, “Nous ne vendons pas de garantie prolongée.” (We do not sell extended warranties.) With it:

Rewritten query: Does Larkfield offer an extended warranty?
1.   8.8860  [3c79dce4a330] safety-notice-kt180 v1 | What we offer | public
2.   6.2494  [144a40851685] warranty-policy v5 | What the warranty covers | public

The five results are about the kettle notice, the warranty policy, large-item delivery and the pressure washers. None of them says whether Larkfield sells extended warranties.

Cause: the prompt says “Write the query in English, even when the question is in French”. That rule helps when the answer is in an English document. But the only statement about extended warranties is in the French Quebec document. The rewrite translated the question away from the one document that answers it. The model followed the instruction; the instruction was wrong for this question.

Fix: treat the rewrite as one more retrieval choice, and measure it per question, not only in total. For this collection, there are three options. Keep rewriting off by default, as the course does. Or search with the original words and the rewrite together, and accept that this loses two other French questions. Or write a new prompt, record new rewrites and measure them all again. A test for Q60 with --rewrite would catch this case if it came back.

For a real AI service: a model in front of the search can quietly change what is searched. Log the rewritten query next to the original, so that an agent or a developer can see it. Measure the rewrite on questions in every language that your users write in.

See it end to endSearch technical documentationCase study

The technical documentation case study measures what reranking costs, and shows a question where the reranker pushes the right passage down.

Knowledge check

Check your understanding

In your own words: how does a cross-encoder differ from the embedding model, and why does the assistant use it only on the top 20?

Show the answer

The embedding model reads the question and each passage separately, and makes one vector for each. So it can embed the passages once, at ingest. A search then only needs to embed the question and compare vectors, which takes milliseconds. A cross-encoder reads the question and one passage together, as one input, and gives one score for the pair. It can see how the two texts fit, so it orders them better. But it must run again for every question and every passage, so it is slow: 322 ms for 20 candidates here. So a fast search collects 20 candidates, and the cross-encoder only puts those 20 in order.

Check your understanding

Rerank raised MRR from 0.745 (hybrid) to 0.831, and hit@5 from 0.864 to 0.898. Omar asks: 'So it is better for every question?' What do you answer?

Show the answer

No. The totals went up, but some questions went down. Reranking brought 4 relevant passages into the top 5, such as LB-1 and the French barbecue question. It pushed 2 out, such as Q60, which fell from rank 3 to rank 14. The total is the net result. Before a change goes in, look at the questions that it loses as well as those that it gains. Then decide if the losses are acceptable. It also costs about 0.3 s per question.

Check your understanding

Give one way that query rewriting helped Larkfield's search, and two ways that it failed, with an example of each.

Show the answer

It helped French questions over English documents. “Puis-je retourner une carte-cadeau?” became “gift card return policy”, and BM25 found the gift card passage for the first time. It failed by translating a question whose only answer is in French: Q60 lost its relevant passage. It failed by changing the meaning: Q46 “The help centre FAQ says refunds take 3 business days. Is that right?” became a search for “3 business days”, without the doubt. Also acceptable: it added stray characters from other scripts (Q45, Q46, Q05). Or: it replaced the agent’s words, as for the chat transcripts question (Q08, rank 1 to 17).

Module practice

This module gave Larkfield’s assistant four ways to search, two filters and two extra steps. The module practice puts them side by side on the three kinds of questions of the module.

Guided: one table for Grace

1. Start Python in the project folder. Make a retriever with all three models:

   Python, in ~/projects/policy-assistant

   from policy_assistant.embed import LocalEmbedder
   from policy_assistant.evaluate import covers, load_questions
   from policy_assistant.filters import Filters
   from policy_assistant.rerank import Reranker
   from policy_assistant.search import Retriever
   from policy_assistant.store import Store
   
   store = Store("index.sqlite")
   retriever = Retriever(store, LocalEmbedder(), Reranker())
   questions = load_questions()
   methods = ["bm25", "dense", "hybrid", "rerank"]

2. For each question, print the rank of the first relevant passage in the top 20 of each method:

   Python (continued)

   print("question", *methods, sep="\t")
   for qid in ["Q38", "Q40", "Q44", "Q34", "Q05", "Q12", "Q64", "Q65", "Q66", "Q67"]:
       q = questions[qid]
       ranks = []
       for method in methods:
           results = retriever.search(q.question, method, 20, Filters(as_of=q.as_of))
           first = next((r.rank for r in results if any(covers(r.chunk, p) for p in q.relevant)), "-")
           ranks.append(first)
       print(qid, *ranks, sep="\t")

   Press Enter on the empty line after the loop. It takes about 10 seconds.

3. Group the rows. Q38, Q40 and Q44 are exact terms in a sentence. Q34, Q05 and Q12 are paraphrases. Q64 to Q67 are rare identifiers alone.

4. Write a note of four or five sentences for Grace: which method to use, the evidence, and its cost.

Show a model answer

question  bm25  dense  hybrid  rerank
Q38  1  1  1  1
Q40  1  1  1  1
Q44  1  1  1  1
Q34  16  2  3  1
Q05  8  2  5  1
Q12  6  1  1  1
Q64  1  -  6  1
Q65  1  4  1  1
Q66  1  3  1  1
Q67  1  3  1  1

Note for Grace (59 questions, filters on, 2026-10-09): “Exact terms inside a sentence are easy: every method puts the right passage first. Paraphrases need vector search: BM25 ranks them 6th to 16th, and vector search 1st or 2nd. Rare identifiers alone need keywords: BM25 ranks all four first, and vector search misses LB-1. Rerank is the only method that puts the right passage first for all ten. On the whole set it has the best order (MRR 0.831). It costs about 0.3 s per question on a laptop, and no data leaves our computers. I recommend rerank as the default, and I would re-measure it when we add documents or questions.”

Your task: your own three questions

Write three questions of your own that the collection can answer. Make one of each kind: an exact term in a sentence, a paraphrase in everyday words, and a rare identifier alone. Look up the relevant section yourself with python -m policy_assistant chunks DOC_ID. Predict which method finds each one, then run search with all four methods, and explain each result in one or two sentences.

Show an example answer

Three questions run on the test computer (top 3 of each method):

• An identifier alone, “What is PK-2400-03?” All four methods have product-care-pw2400 | Parts and part numbers or warranty-claims-procedure | Pressure washer pumps first. Not every code defeats vector search. The chunks with “PK-2400-03” also say “pump kit” and “PW-2400”, so the vector has some meaning to hold on to. LB-1 has no such help.
• A paraphrase, “My kettle does not turn itself off when the water boils. What should I do?” Every method finds safety-notice-kt180 | The risk first. BM25’s second result is the BBQ gas section, because it shares “What should … do”; the vector methods return three sections of the kettle notice.
• An exact term in a sentence, “Is a gift card refundable?” BM25 ranks gift-cards | Buying a gift card first and “Lost cards and returns”, which answers it, third. Vector search and rerank put “Lost cards and returns” first.

The lesson of the example: predictions by kind are a good start, but each question can surprise you. That is why the course measures on a question set, and why you should add your own hard questions to it.

Module check

Check your understanding

Completion check: explain one case where lexical search outperforms vector search, why it happens, and what the assistant does about it.

Show the answer

“What is LB-1?” is a good case. BM25 ranks the hedge trimmer passage with “blade oil LB-1” first. LB-1 is a rare word that only that chunk contains, and BM25 gives rare shared words a high score. Vector search with e5 does not have it in its top 20. The model has no useful meaning for the code, so the question becomes a vague “What is …?”. It lands on general privacy sections. More generally, keyword search wins on exact identifiers that appear alone: product codes, part numbers, staff codes. The assistant keeps BM25 in the candidates (hybrid search) and lets the cross-encoder order them, so LB-1 comes first with rerank.

Check your understanding

Tomás changes the embedding model in the configuration and does not run ingest again. What happens, and how does the project protect against it?

Show the answer

The index still holds vectors made by the old model, and the questions would be embedded by the new one. Vectors of two models are not comparable, even with the same length. In the course’s test, an e5 index searched with MiniLM question vectors gave a hit@5 of 0.068, with no error. The project records the model name and revision in the index’s meta at ingest. search.py compares them with the model of the question and stops with IndexMismatch. The fix is to run ingest again with the new model.

Check your understanding

Hybrid search scored 0.864 and vector search 0.915 on Larkfield's questions, although many articles say that hybrid search is better. How do you explain this, and what do you recommend for another company's documents?

Show the answer

Reciprocal rank fusion adds votes from both lists. When one list is weak for a question, such as BM25 on French questions, its low ranks still vote. Then chunks that both lists rank low can push a good result out of the top 5. On Larkfield’s questions this cost more than it gained; with the hosted embed-small vectors, hybrid was slightly better. So no method is best everywhere. For another company, build a question set with relevant passages first, including codes, paraphrases and every language the users write in. Then measure each method on it, and look at the questions each one loses, not only the total.

End of Module 3

You can now find evidence in three ways and choose between them with numbers. Vector search finds paraphrases and French questions, keyword search finds rare codes, and reranking puts the best passage first. The filters keep the wrong versions and forbidden documents out before anything is ranked. Module 4 takes the top 5 and builds an answer from them, with a citation for every claim.

Reset to a known state

If your project does not give 48 passed, copy the snapshot end-of-m03 from the labs repository, in the same way as start in Set up the project. Then:

1. Open a terminal in ~/projects/policy-assistant. A snapshot has no .venv, so make a new one and activate it, as in step 5 of Set up the project.
2. Run python -m pip install -r requirements.txt. On Linux, install the CPU-only PyTorch first, as in Embeddings for search.
3. Run python -m pytest. The last line is 48 passed.
4. Run python -m policy_assistant ingest, then python -m policy_assistant eval. The first line of eval is rerank, top 5, audience staff: hit@5 0.898 (59 questions with relevant passages).

Recap

• A cross-encoder reads the question and a passage together and scores the pair. Reranking the 20 hybrid candidates raised MRR from 0.745 to 0.831, and hit@5 from 0.864 to 0.898. It costs about 322 ms per question. It is the course default.
• Query rewriting asks a model for better search words. It helped BM25 most (0.831 to 0.898) and gave the best result with rerank (0.932, MRR 0.882). It also lost questions, added stray characters and changed a meaning, so it stays optional and measured.
• Every change moves some questions up and some down. Compare the missed questions, not only the total.
• Module 3 ends with the snapshot end-of-m03: 48 tests, and eval gives hit@5 0.898 with rerank.

Next, choose which of these passages go into the prompt, within a budget of tokens, and keep the source of each one.

Next: Assemble context