Lexical, dense and hybrid search

What you will be able to do

Choose a search method for Larkfield’s assistant with evidence, not with a slogan. You compare keyword, vector and hybrid search on real questions. Then you make sure that the assistant sees only the policy version in force, and only the documents its users may read.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Embeddings, cosine similarity and the model record in the index, from Embeddings for search.
• BM25 and hit@5, from A keyword-search baseline.
• Versions, effective dates and access labels on every chunk, from Preserve metadata.

What you need on your computer

• The project from Embeddings for search, with .venv active: dense.py and embed.py exist, and python -m policy_assistant ingest printed embeddings intfloat/multilingual-e5-small.
• If your project is broken, copy the snapshot end-of-m02 and do the steps of that lesson’s Build it again.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A catalogue and a librarian

A good library has two ways to find a book. The catalogue finds an exact call number, such as QA76.73, in a second. The librarian finds a book from a vague description, but she does not remember every call number. You use the catalogue when you have the number, and the librarian when you only have an idea. A clever reader asks both and compares the two lists.

The problem: each method misses something

Tomás has seen the last lesson, and he is sure: “Vector search understands meaning. Keywords are old. Let’s use vectors for everything.” Grace tries one of her questions, a code that agents type as it is: “What is LB-1?” The answer is in the hedge trimmer guide: “put a few drops of blade oil LB-1 along the blade”. Here is vector search, with the project’s code from this lesson:

python -m policy_assistant search "What is LB-1?" --method dense

1.   0.7836  [8021b57fe6d3] privacy-policy v3 | Who we share it with | public
   Who we share it with We share your information with couriers, our payment processor and our help-de…
2.   0.7773  [13f00fce8eaa] privacy-policy v3 | Our help-desk assistant | public
   Our help-desk assistant Our help-desk staff use an assistant that drafts answers from Larkfield's p…
3.   0.7738  [6e0e1d1c7e4f] privacy-policy v3 | Contact | public
   Contact Write to our privacy office at privacy@larkfield.example.
4.   0.7616  [f78646d93435] policy-governance v2 | Access labels | public
   Access labels - public: The document may be shown or quoted to customers. - staff: Only for help-de…
5.   0.7609  [a31b42d3e40d] damaged-or-wrong-items v2 | Wrong item | public
   Wrong item We send a free return label. We ship the right item as soon as the courier scans the ret…
dense, top 5, as of 2026-10-09, audience staff

Three privacy chunks and nothing about blade oil. The hedge trimmer chunk is not even in the top 20: it comes 34th. “LB-1” means nothing to e5, so the question becomes “What is …?”, a vague question. Vague questions land on chunks that are close to everything, such as these short privacy sections. Keyword search, with the same question, puts the hedge trimmer chunk first, because “LB-1” is a rare word that only that chunk has.

So keywords lose on “sprinkler” and on French, and vectors lose on “LB-1”. Neither is “always better”. The only way to choose is to look at real questions of each kind.

Predict, then compare

The explorer shows the top 5 of three methods for four of Grace’s questions. Keyword search is BM25, as in Module 1. Vector search is e5, as in the last lesson. Hybrid search combines the two; the next section explains how. A passage with the “Relevant” badge is one that Grace marked as holding the answer.

Explore

Keyword, vector and hybrid search on four real questions

Keyword: first relevant

rank 1

Vector: first relevant

rank 1

Hybrid: first relevant

rank 1

Q38: An exact product code in a sentence. All three methods have a relevant passage in the top 5.

Question Which pump kit fits the PW-2200?

Relevant passages: product-care-pw2200 › Parts and part numbers; warranty-claims-procedure › Pressure washer pumps. Date filter: 2026-10-09.

Keyword search (BM25)

1. 1product-care-pw2200Parts and part numbersPart | Part number | Notes Pump kit | PK-2200-07 | fits the PW-2200 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…Relevant
2. 2product-care-pw2400Parts and part numbersPart | Part number | Notes Pump kit | PK-2400-03 | fits the PW-2400 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…
3. 3warranty-claims-procedure staffPressure washer pumps- If the pump of a PW-2200 fails within 3 years, send the pump kit PK-2200-07 instead of a new pressure washer. - For the PW-2400, send the pump kit…Relevant
4. 4product-care-pw2200Winter storageDrain all water from the pump and hoses before temperatures fall below 0 °C. Store it indoors, above 0 °C. Frost damage is not covered by the warrant…
5. 5product-care-pw2200Using it safelyNever run the pressure washer for more than 2 minutes with the trigger released: the pump overheats. Never point the spray at people, animals or elec…

Vector search (e5)

1. 1product-care-pw2200Parts and part numbersPart | Part number | Notes Pump kit | PK-2200-07 | fits the PW-2200 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…Relevant
2. 2warranty-claims-procedure staffPressure washer pumps- If the pump of a PW-2200 fails within 3 years, send the pump kit PK-2200-07 instead of a new pressure washer. - For the PW-2400, send the pump kit…Relevant
3. 3product-care-pw2400Parts and part numbersPart | Part number | Notes Pump kit | PK-2400-03 | fits the PW-2400 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…
4. 4product-care-pw2200Specifications- Maximum pressure: 2,200 PSI. - Water flow: 6.4 litres per minute. - Electric motor: 1,800 W. Plug into an outdoor outlet with a GFCI (ground-fault)…
5. 5product-care-pw2200Troubleshooting- Pressure pulses up and down: clean the inlet filter and check that the water tap is fully open. - Motor hums but does not start: release the trigge…

Hybrid search (RRF)

1. 1product-care-pw2200Parts and part numbersPart | Part number | Notes Pump kit | PK-2200-07 | fits the PW-2200 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…Relevant
2. 2product-care-pw2400Parts and part numbersPart | Part number | Notes Pump kit | PK-2400-03 | fits the PW-2400 only Nozzle set (0°, 15°, 25°, 40° and soap) | NZ-415 | fits the PW-2200 and PW-2…
3. 3warranty-claims-procedure staffPressure washer pumps- If the pump of a PW-2200 fails within 3 years, send the pump kit PK-2200-07 instead of a new pressure washer. - For the PW-2400, send the pump kit…Relevant
4. 4product-care-pw2200Winter storageDrain all water from the pump and hoses before temperatures fall below 0 °C. Store it indoors, above 0 °C. Frost damage is not covered by the warrant…
5. 5product-care-pw2200Using it safelyNever run the pressure washer for more than 2 minutes with the trigger released: the pump overheats. Never point the spray at people, animals or elec…

The top 5 of each method for four questions from Larkfield's question set, computed by the project's own search code (structure chunks, e5, the question's date, staff audience).

Text description of the explorer

The explorer has one control: the question. Under it, a readout gives the rank of the first relevant passage for each method. Three columns show each method’s top 5, with the relevant passages marked. The four questions and their results:

Product code (Q38, “Which pump kit fits the PW-2200?”): the relevant passage is first for keyword, vector and hybrid search. A code inside a full sentence is easy for all three.

Paraphrase (Q34, “How long is the warranty on a garden sprinkler?”): keyword search has the warranty table at rank 16, out of the top 5. Its first result is the garden furniture warranty. Vector search has it at rank 2 and hybrid search at rank 3.

French (Q63, “Puis-je retourner une carte-cadeau?”): keyword search finds no relevant passage in its top 20, only French chunks about returns. Vector search has the gift card passage “Lost cards and returns” at rank 2, and hybrid at rank 5.

Identifier only (Q64, “What is LB-1?”): keyword search has the hedge trimmer passage at rank 1. Vector search does not have it in its top 20; its top 5 are three privacy sections, the access labels and “Wrong item”. Hybrid has it at rank 6, just outside the top 5.

Look at the identifier-only question again. Keyword search has the answer first, and vector search does not have it at all. Hybrid search has it at rank 6, out of the top 5. Combining the two lists did not keep the best of both. The section after next shows why.

Where keywords win, and where vectors win

One question per kind is still a small sample. Here is hit@5 on Grace’s whole question set, by kind of question:

Questions	How many	Keyword (BM25)	Vector (e5)	Hybrid
Tables (the answer is in a table row)	8	0.625	1.000	0.875
French	8	0.625	1.000	0.750
Exact codes	11	1.000	0.909	0.909
Versions (dates in the question)	8	0.875	0.750	0.625
All	59	0.831	0.915	0.864

Lexical search, search by the exact words, wins where the words are rare and exact: product codes, part numbers, staff codes such as RMA-, and dates. Four questions in the set are only a code, for example “What is LB-1?” and “What is CV-310 for?”. BM25 ranks the right chunk first for all four. Vector search misses LB-1 and ranks the other three 3rd or 4th. Sometimes a code sits inside a full sentence, as in “Which pump kit fits the PW-2200?”. Then both methods find it, because the rest of the sentence carries the meaning.

Dense search, search by meaning, wins where the words differ. That is the case for paraphrases (“sprinkler” and “sprinklers”), for table rows with other words than the question, and for other languages. With BM25, the French questions reach 0.625; with e5, all of them find a relevant passage.

This answers Tomás. Vector search is better on average here, 0.915 against 0.831. But “on average” hides the questions where it fails completely, and those are the codes that agents copy from a box or a receipt. With 59 questions, a difference of one or two questions (0.017 to 0.034) proves nothing. A difference of 5 questions, as here, is a real signal on this set, and it may still not hold on another collection.

Hybrid search: let the two lists vote

If each method finds what the other misses, why not use both? Hybrid search runs keyword search and vector search, and combines the two ranked lists into one.

The scores cannot be added. BM25 scores go from 0 to about 30. The cosine similarities of e5 sit between 0.7 and 0.9. A sum would follow BM25 almost alone. So the usual way uses only the ranks. Each list gives each chunk a vote of 1 ÷ (60 + rank). Then the code adds the votes. This is reciprocal rank fusion (RRF): “reciprocal” because the vote is 1 divided by the rank. The constant 60 comes from the paper that proposed RRF, by Cormack, Clarke and Büttcher in 2009. It keeps the first place of one list from deciding everything.

A tiny example, with made-up lists:

Chunk	Keyword rank	Vector rank	RRF score
A	1	3	1/61 + 1/63 = 0.0323
D	not found	1	1/61 = 0.0164
B	not found	2	1/62 = 0.0161
C	2	not found	1/62 = 0.0161

A is in both lists, so it gets two votes and wins, although neither list had it alone at the top. That is the idea of hybrid search: a chunk that both methods like is probably relevant.

Now the real question “What is LB-1?”. BM25 ranks the hedge trimmer chunk first: 1/61 = 0.0164. Vector search does not have it in its top 20, so it gets no second vote. Four other chunks are in both lists, low in each. Their two small votes add up to more than one big vote:

Reciprocal rank fusion on the real question 'What is LB-1?'. Chunks with two weak votes beat the chunk with one strong vote.

Text description of the diagram

Four rows. Each row starts with one chunk and its rank in the two lists, and ends with its fused score and its rank in the hybrid list. The warranty table is 7th for BM25 and 11th for vector search: 1/67 + 1/71 = 0.0290, hybrid rank 1. The large-item delivery section is 4th for BM25 and 20th for vector search: 1/64 + 1/80 = 0.0281, hybrid rank 2. Two more chunks with two votes take ranks 3 and 4. The privacy section “Who we share it with” is 1st for vector search and not in the BM25 top 20: 1/61 = 0.0164, hybrid rank 5. The hedge trimmer passage with LB-1 is 1st for BM25 and not in the vector top 20: also 1/61 = 0.0164. The chunk ID breaks the tie, so it comes 6th and falls out of the top 5.

The honest result: hybrid is not better here

Over the whole set, hybrid search scores below vector search alone: hit@5 0.864 against 0.915. The reason is the one in the picture. When one list is weak for a question, its random low ranks still vote, and they can pull a good result down. On the French questions, BM25’s list is weak, and hybrid drops from 1.000 to 0.750.

Hybrid does help in one place. On the four identifier-only questions, it moves the right chunk up. TG-24, HR-50 and CV-310 go from rank 3 or 4 to rank 1. LB-1 goes from rank 34 to rank 6. If Larkfield’s agents typed many bare codes, that could matter. With the hosted embed-small vectors, hybrid is slightly better than vectors alone (0.898 against 0.881). The same method helps with one embedding model and hurts with another.

So the lesson is not “hybrid is bad”. It is: measure on your own questions. Blog posts and benchmarks often say that hybrid search wins, and on their data it may. On Larkfield’s 59 questions, with e5, it does not. There is no method that is best everywhere. The course keeps hybrid search for one job: it collects the 20 candidates that the next lesson reranks. That combination, called rerank, is the course default, because it has the best ranking.

Filters before the top k

Two questions must be settled before any ranking: which version of a document applies, and who may see it. The chunks carry the answers, from Preserve metadata: an effective date range, and an access label.

• The date filter keeps only the versions in force on a date: effective_from <= date <= effective_to. Which date? Larkfield’s policy-governance document says: for returns, the delivery date. Returns policy version 3 (30 days) is in force until 31 October 2026, and version 4 (45 days) from 1 November.
• The access filter follows the audience. The help-desk assistant is for staff, so it may see every chunk. A public assistant, for customers, may only see chunks labelled public. A chunk with no label counts as staff: when the label is missing, the safe choice is to hide it.

The filters run before the top k is chosen, inside the search. If they ran after it, 3 of the 5 places could go to staff chunks, and the filter would then remove them. A public user would get only 2 results. Worse, a later step might forget to filter at all.

The retrieval funnel of Larkfield's assistant. The filters come first, so nothing that is out of date or not allowed can take a place in the top 5.

Text description of the diagram

Seven stages, from left to right. 1: the 185 chunks in the index. 2: the filters, first: only the version in force on the question’s date, and only the access labels that the audience may see. A dashed arrow leads away from this stage: other versions, and staff-only chunks for a public assistant, are removed here. 3: the candidates: the top 20 from BM25 and the top 20 from vector search, both over the filtered chunks. 4: hybrid search fuses the two lists with RRF and keeps 20. 5: reranking, the topic of the next lesson: a cross-encoder reads the question with each of the 20 chunks. 6: the top k, here 5 chunks: the selected evidence. 7: the evidence goes to the model, in Module 4.

Choose the top k

The top k is how many chunks the search returns. Larkfield uses k = 5. A larger k finds more relevant passages, but it also sends more text to the model, which costs tokens and can bury the answer. Here is recall@k for vector search: the share of all relevant passages that are in the top k.

k	1	3	5	10
recall@k, vector search (e5)	0.531	0.785	0.862	0.895

From 3 to 5, recall gains 0.077. From 5 to 10, it gains only 0.033, for twice the text. That is why k = 5 is a fair choice here. Module 4 sets a budget in tokens for the context, and Retrieval measures explains recall@k in full.

No vector database yet

Tomás has one more idea: “Real RAG systems use a vector database.” A vector index is a data structure that finds close vectors without comparing every one. It pays off with millions of chunks. Larkfield has 185. Comparing the question with all 185 vectors in NumPy takes about 0.09 ms on the test computer. Embedding the question takes 5.2 ms, so the search itself is not the slow part.

A vector database would add a service to install, update and back up, for no gain. Measure first. When the collection grows to many thousands of chunks, try an index such as sqlite-vec and measure again.

Build it: three methods and the filters

The project gets three new files, hybrid.py, filters.py and search.py, and a new command-line file. Work in ~/projects/policy-assistant, with .venv active.

1. Make the file policy_assistant/hybrid.py:

   policy_assistant/hybrid.py

   """Hybrid search: combine keyword and vector rankings with reciprocal rank fusion (RRF).
   
   The scores of BM25 and cosine similarity are on different scales and cannot be added. RRF uses only
   the ranks: an item gets 1 / (K + rank) from every list it appears in, and the sums are sorted.
   K = 60 (the value of the original paper, Cormack, Clarke and Buettcher, 2009) keeps one list's top
   item from dominating. An item that both lists rank well comes first.
   
   Worked example (K = 60): chunk A is 1st by keywords and 3rd by vectors: 1/61 + 1/63 = 0.0323.
   Chunk B is 2nd by vectors only: 1/62 = 0.0161. A comes first.
   """
   
   from .lexical import Hit
   
   K = 60
   
   
   def rrf(rankings: list[list[Hit]], k: int = 5, constant: int = K) -> list[Hit]:
       scores: dict[str, float] = {}
       for ranking in rankings:
           for hit in ranking:
               scores[hit.id] = scores.get(hit.id, 0.0) + 1.0 / (constant + hit.rank)
       order = sorted(scores.items(), key=lambda item: (-item[1], item[0]))
       return [Hit(i, round(s, 6), rank) for rank, (i, s) in enumerate(order[:k], 1)]

2. Make the file policy_assistant/filters.py:

   policy_assistant/filters.py

   """Metadata filters: which chunks may be retrieved for this question at all.
   
   Applied BEFORE the top k is chosen (and so before anything reaches the model):
   - as_of: only versions in force on that date (effective_from <= as_of, and as_of <= effective_to
     when the document has an end date). Which date? The policy governance document says: for returns,
     the delivery date; otherwise the day of the question.
   - audience: "staff" (the help-desk assistant) may see every chunk; "public" (a customer-facing
     assistant) only chunks labelled public. A chunk without an access label counts as staff: when the
     label is missing, the safe choice is to hide it.
   """
   
   from dataclasses import dataclass
   
   AUDIENCES = ("staff", "public")
   
   
   @dataclass(frozen=True)
   class Filters:
       as_of: str | None = None        # YYYY-MM-DD, or None for no date filter
       audience: str = "staff"
   
       def __post_init__(self):
           if self.audience not in AUDIENCES:
               raise ValueError(f"audience must be one of {AUDIENCES}, not {self.audience!r}")
   
       def allows(self, chunk) -> bool:
           if self.audience == "public" and (getattr(chunk, "access", "") or "staff") != "public":
               return False
           if self.as_of:
               if chunk.effective_from and chunk.effective_from > self.as_of:
                   return False
               if chunk.effective_to and self.as_of > chunk.effective_to:
                   return False
           return True

   The dates are strings in the form YYYY-MM-DD, so comparing them as text gives the right order.

3. Make the file policy_assistant/search.py. It is the one entry point for every method:

   policy_assistant/search.py

   """One entry point for every search method, with the metadata filters applied first.
   
       bm25    keyword search (lexical.py)
       dense   vector search (dense.py) with the embedding model that built the index
       hybrid  bm25 + dense, combined with reciprocal rank fusion (hybrid.py)
       rerank  hybrid's top CANDIDATES, reread by the cross-encoder (rerank.py)
   """
   
   from dataclasses import dataclass
   
   from .dense import search as dense_search
   from .filters import Filters
   from .hybrid import rrf
   from .lexical import Hit
   from .metadata import Chunk
   from .store import IndexMismatch, Store
   
   METHODS = ("bm25", "dense", "hybrid", "rerank")
   CANDIDATES = 20   # how many results each first-stage method passes on to fusion and reranking
   
   
   @dataclass(frozen=True)
   class Result:
       chunk: Chunk
       score: float
       rank: int
   
   
   class Retriever:
       def __init__(self, store: Store, embedder=None, reranker=None):
           self.store, self.embedder, self.reranker = store, embedder, reranker
           self._chunks = {c.chunk_id: c for c in store.chunks()}
   
       def _allowed(self, filters: Filters):
           return lambda chunk_id: filters.allows(self._chunks[chunk_id])
   
       def _check_model(self) -> None:
           if self.embedder is None:
               raise ValueError("Dense search needs an embedding model.")
           meta = self.store.meta()
           if (meta.get("embedding_model"), meta.get("embedding_revision")) != (self.embedder.name, self.embedder.revision):
               raise IndexMismatch(
                   f"The index was built with {meta.get('embedding_model') or 'no embedding model'} "
                   f"({meta.get('embedding_revision', '')}), and the question would be embedded with "
                   f"{self.embedder.name} ({self.embedder.revision}). Vectors of two models are not comparable.")
   
       def bm25(self, query: str, k: int, filters: Filters) -> list[Hit]:
           return self.store.keywords.search(query, k, self._allowed(filters))
   
       def dense(self, query: str, k: int, filters: Filters) -> list[Hit]:
           self._check_model()
           ids, matrix = self.store.vectors()
           return dense_search(self.embedder.embed_queries([query])[0], matrix, ids, k, self._allowed(filters))
   
       def hybrid(self, query: str, k: int, filters: Filters) -> list[Hit]:
           return rrf([self.bm25(query, CANDIDATES, filters), self.dense(query, CANDIDATES, filters)], k)
   
       def rerank(self, query: str, k: int, filters: Filters) -> list[Hit]:
           if self.reranker is None:
               raise ValueError("Reranking needs a cross-encoder (rerank.Reranker).")
           candidates = [self._chunks[h.id] for h in self.hybrid(query, CANDIDATES, filters)]
           return self.reranker.rerank(query, candidates, k)
   
       def search(self, query: str, method: str = "hybrid", k: int = 5, filters: Filters = Filters()) -> list[Result]:
           if method not in METHODS:
               raise ValueError(f"Unknown method {method!r}: use one of {', '.join(METHODS)}.")
           hits = getattr(self, method)(query, k, filters)
           return [Result(self._chunks[h.id], h.score, h.rank) for h in hits]

   Part	Meaning
   _allowed(filters)	Turns the filters into the allowed function that dense.search and the keyword index take. Each method filters before it cuts the list to k.
   _check_model()	Compares the model in the index’s meta with the model of the question. A different name or revision stops the search.
   CANDIDATES = 20	Hybrid search takes the top 20 of each method, not the top 5. Then a chunk ranked 6th by one method can still get its vote.
   rerank	The method of the next lesson. Its file comes in the next step.

4. Copy the new command-line file and the reranker from the snapshot. The command-line file changes in many small places, and the next lesson explains the reranker:

   macOS / Linux

   cp ../nextia-learning-labs/C09/snapshots/end-of-m03/policy_assistant/__main__.py policy_assistant/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m03/policy_assistant/rerank.py policy_assistant/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m03\policy_assistant\__main__.py policy_assistant\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m03\policy_assistant\rerank.py policy_assistant\

   search and eval now take --method (bm25, dense, hybrid or rerank), --as-of and --audience. Without --as-of, the course’s date, 2026-10-09, is used, so that your results match this lesson. Use --as-of today for real use, and --as-of none for no date filter. eval uses each question’s own date.

5. Copy the two new test files:

   macOS / Linux

   cp ../nextia-learning-labs/C09/snapshots/end-of-m03/tests/test_retrieval_parts.py tests/
   cp ../nextia-learning-labs/C09/snapshots/end-of-m03/tests/test_search.py tests/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m03\tests\test_retrieval_parts.py tests\
   Copy-Item ..\nextia-learning-labs\C09\snapshots\end-of-m03\tests\test_search.py tests\

   test_retrieval_parts.py checks the tiny examples: the cosine of (3, 4) and (4, 3) is 0.96, and the RRF table above. test_search.py checks the real counterexamples on the collection: “What is LB-1?”, the French gift card question and the sprinkler question.

6. Run the tests:

   python -m pytest

   The last line is 44 passed. The first run takes a few seconds longer, because the tests load e5.

Guided practice: compare the three methods

1. Search for LB-1 with each method. Change only --method:

   python -m policy_assistant search "What is LB-1?" --method bm25
   python -m policy_assistant search "What is LB-1?" --method dense
   python -m policy_assistant search "What is LB-1?" --method hybrid

   Check: with bm25, the first result is product-care-hedge-trimmer-ht550 v1 | After each use. With dense and hybrid, it is not in the top 5. The hybrid list starts with warranty-policy v5 | Warranty periods by product c… at 0.0290.

2. Measure each method on the whole question set:

   python -m policy_assistant eval --method bm25
   python -m policy_assistant eval --method dense
   python -m policy_assistant eval --method hybrid

   The first lines of the outputs:

   bm25, top 5, audience staff: hit@5 0.831 (59 questions with relevant passages)
   dense, top 5, audience staff: hit@5 0.915 (59 questions with relevant passages)
   hybrid, top 5, audience staff: hit@5 0.864 (59 questions with relevant passages)

   BM25 is now 0.831, not the 0.797 of Module 2, because eval uses each question’s date. The wrong version of a document no longer takes a place in the top 5. The last line of each output lists the missed questions. For dense, it is missed: Q13, Q26, Q28, Q64, Q56.

3. See the date filter choose a version. The question has no date, so you give the delivery date:

   python -m policy_assistant search "Can a customer return an unused item?" --method bm25 --as-of 2026-11-05

   The first result is version 4, with 45 days:

   1.  11.6078  [963dae4229eb] returns-policy v4 | Return window | public
      Return window You can return an unused item within 45 days of delivery for a full refund of the ite…

   Run it again with --as-of none. Now version 3 and version 4 both come back, with the same score, 11.6078. Keyword search cannot tell them apart: only the date can.

4. See the access filter. Ask for the staff discount as a public assistant:

   python -m policy_assistant search "What is the staff discount?" --method bm25 --audience public

   No result comes from staff-discount-and-perks. The first result is promotions-and-discount-codes v2 | Using a discount code | public. Without --audience public, the first two results are the staff document’s Rules and Staff discount, labelled staff.

5. Try to search the e5 index with another model:

   python -m policy_assistant search "What is LB-1?" --method dense --embedder sentence-transformers/all-MiniLM-L6-v2

   The search stops before it compares any vector:

   The index was built with intfloat/multilingual-e5-small (614241f622f53c4eeff9890bdc4f31cfecc418b3), and the question would be embedded with sentence-transformers/all-MiniLM-L6-v2 (main). Vectors of two models are not comparable.

   This command downloads all-MiniLM-L6-v2 (about 90 MB) the first time, before the check. You can skip it: test_search.py checks the same refusal.

Your turn: three more questions

Predict first, then run search with --method bm25 and --method dense for each question. For each, write which method has a relevant passage in its top 5, and why.

1. “What is CV-310 for?” (relevant: product-care-bbq-gr310, section Parts)
2. “A customer has not bought anything from us for two years. What happened to their Rewards points?” (relevant: loyalty-program, section Expiry)
3. “What is the HR-50?” (relevant: supplier-aquaflow-hose-reels, section The AquaFlow range)

Show the answer

1. CV-310: BM25 ranks product-care-bbq-gr310 | Parts first, and dense search 3rd. Both have it in the top 5, but the code-only question suits keywords better. Dense search puts privacy-policy | Who we share it with first, as for LB-1: a short, general chunk that sits close to many vague questions.
2. Two years: BM25 ranks loyalty-program | Expiry 8th, so it misses. Dense search ranks it 2nd. The question says “has not bought anything for two years”. The policy says “Points expire after 18 months without any purchase”. It is the same idea in other words.
3. HR-50: BM25 ranks the AquaFlow range first, and dense search 3rd, again after the privacy chunk.

So the pattern of the explorer holds: codes favour keywords, and paraphrases favour vectors. On the code questions, dense search usually still finds the chunk in the top 5; on LB-1 it does not. That is why one failure in a set of 59 still matters. An agent who types “LB-1” gets nothing useful.

Common mistake

A public assistant without the access filter

Symptom: Omar wants to try the assistant on Larkfield’s website, for customers. Tomás runs the same search with no access filter: “It is the same documents, the answers will be fine.” The question “What is the staff discount?” returns two chunks of the staff procedure first. One gives the 20% discount, and one the rules (“Never share it with customers”). Over all 67 questions, BM25 without the access filter puts 59 staff-only chunks in the top 5 lists; hybrid puts 53, and rerank 70.

Cause: the access label is on every chunk, but nothing used it. Search finds the best text for the question, and for many questions the best text is a staff procedure.

Fix: pass the audience into the search, so that the filter runs before the top k: --audience public. With the filter, the count is 0 for every method. Test it in code too: test_search.py checks that a public search never returns a staff chunk. Permissions and isolation follows the labels through the whole assistant, including the answers and their citations.

For a real AI service: decide the audience of every search from who is asking, in code, not from the question. A filter after the search is not enough, and neither is a rule in the prompt. In both cases, the text has already reached the model.

Knowledge check

Check your understanding

Explain to Tomás, with one real example, why 'vector search is always better than keywords' is wrong.

Show the answer

On “What is LB-1?”, keyword search (BM25) ranks the hedge trimmer passage with “blade oil LB-1” first. Vector search with e5 does not have it in its top 20; its top 5 are privacy sections. “LB-1” is a rare, exact code that the embedding model gives no useful meaning, so the question becomes vague. BM25 finds it because a rare word that only one chunk has gets a high score. Vector search is better on average here, with hit@5 0.915 against 0.831, because it wins on paraphrases, tables and French. But “on average” is not “always”.

Check your understanding

Predict: with RRF and the constant 60, chunk X is 2nd in the keyword list and 2nd in the vector list. Chunk Y is 1st in the keyword list only. Which one comes first, and with what scores?

Show the answer

X gets 1/62 + 1/62 = 0.0323. Y gets 1/61 = 0.0164. X comes first, with about twice the score of Y. A chunk that both lists rank well beats a chunk that one list ranks best. That is the idea of hybrid search, and also its risk. If Y is the right answer and the vector list is weak, the fusion pulls Y down, as it did with LB-1.

Check your understanding

Why must the date and access filters run before the top 5 is chosen, and not after?

Show the answer

If the filters run after the top 5, the places held by wrong versions or forbidden chunks are removed. The user then gets fewer than 5 results, and a relevant chunk ranked 6th never comes back. If a later step forgets to filter, the forbidden text even reaches the model. Filtering first means that every one of the 5 places goes to a chunk that is in force and allowed.

See it end to endSearch technical documentationCase study

The case study compares keyword, vector and hybrid search with reranking on real documentation, where exact function names and paraphrases both matter.

Recap

• Lexical search (BM25) wins on rare, exact words: codes, part numbers, dates. Dense search wins on paraphrases, tables and other languages. On Larkfield’s set: BM25 0.831, vector search 0.915.
• Hybrid search with reciprocal rank fusion adds 1 ÷ (60 + rank) from each list. Here it scores 0.864, below vector search alone: a weak list can pull a good result down, as with “What is LB-1?”. Measure on your own questions.
• The date filter keeps the version in force, and the access filter keeps what the audience may see. Both run before the top k (here 5) is chosen.
• With 185 chunks, comparing every vector takes 0.09 ms. No vector database is needed yet.

Next, reread the candidates with a second, slower model, and let a language model rewrite the question. Both help, and both have a price.

Next: Reranking and query changes