Search a help centre or a scientific corpus
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In eight modules, we searched products. In this case study, we search documents from an open retrieval benchmark, compare lexical, dense and hybrid retrieval and re-ranking, and find when the cheap method is good enough.
Case study 3 of 3 · Search and Ranking: Find the Best MatchSearch a help centre or a scientific corpus
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Choose a search design for a corpus of long documents, with long queries, from real measurements. You compare BM25, dense search, two kinds of hybrid search and a cross-encoder on one open benchmark. Then you decide when the cheap method is good enough, and you write a decision that others can check.
The notebook runs every step of this page with the same code and output, plus four tasks with checks. Its setup cells download the benchmark and the course team’s files (about 13 MB). It needs no local setup, no account and no model. It runs in about one minute on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- NDCG@10, MRR and recall, from Ranking measures, and paired differences with intervals, from Build a test set that represents your users.
- BM25 and its settings, from BM25.
- Dense search and its failures, from Search by meaning and Where dense search fails.
- RRF and score blending, from Hybrid search.
- The cross-encoder and the re-ranking depth, from Cross-encoder re-ranking.
- p50, p95 and cost per 1,000 queries, from Latency and cost budgets.
What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)
- Python 3.12 with a new virtual environment, and the notebook downloaded from the link above, in VS Code or JupyterLab. Its first cell installs the packages: NumPy, pandas, matplotlib, bm25s and PyStemmer. The
larkfield-searchproject is not needed, and this case study does not change it. - About 15 MB of disk. The setup cell makes a new folder
arguana-search. To start again, delete that folder.
Tested on macOS (Apple M4, 16 GB) with Python 3.12, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.
The situation
Section titled “The situation”Think of a library desk. A student brings an essay and asks, “What is the strongest answer to it?” A good librarian does not look for books with the same words. She looks for a text that answers the essay, often in different words.
Leila is the data scientist of a small team that builds a debate-practice site. Students paste an argument, and the site shows the best counter-argument from a library of 8,674 written arguments. Kofi is the product lead. He asks, “Which search should we use? And do we need the expensive one?” Ines is the developer. She runs the site on one small CPU server, and she wants each search under 200 ms.
A help centre has the same problem. A customer pastes a long message, and the right article explains the fix in other words.
We looked for an open help-centre dataset with a licence that the course allows. We did not find one. The nearest set has no licence on its source page. So this case study uses ArguAna, an open benchmark with long queries and a CC BY 4.0 licence. For a scientific corpus, you can check SciFact for your own use. Its abstracts are under ODC-By 1.0 (see its licence file). That licence is not on this course’s list, so we did not use it.
What you will build
Section titled “What you will build”You will build five search designs and measure them on the same queries with one measure. The designs are BM25 (the cheap baseline), dense search, hybrid search by RRF and by blending, and a cross-encoder. You choose settings on validation queries and measure once on test queries. Then you look at slices by query length and topic, and add up time and cost. You write a decision and run it on a new argument.
The data
Section titled “The data”A debate page is about one question, for example vegetarianism and the environment. It has points: short arguments for one side. Each point has a counter-point: an answer written for that point. In ArguAna, each query is a point. The one judged document is its counter-point. The task is to find that counter-point among all 8,674 documents.
Text description of the diagram
A debate page on vegetarianism and the environment. The point “Being vegetarian helps the environment” is the query. Its counter-point, “You don’t have to be vegetarian to be green”, is the judged answer (grade exact). The search uses the whole point, a median of 174 words, over 8,674 documents. In the ranked list, the point itself is left out, because it has the same text. Another argument from the same page counts as wrong, because it is not judged.
The files are small:
| File | Rows | Columns we use | What one row is |
|---|---|---|---|
corpus.jsonl |
8,674 | _id, text |
One argument: a point or a counter-point, about 150 words |
queries.jsonl |
1,406 | _id, text |
One point, used as a query |
qrels/test.tsv |
1,406 | query-id, corpus-id |
The query’s one judged document: its counter-point |
split.csv (ours) |
1,406 | query_id, split, debate, words |
Validation or test, the debate page, the length in words |
We do not use the corpus’s title field. It is mostly empty, and otherwise holds topic tags.
The IDs have a pattern. test-environment-aeghhgwpe-pro02a is point 2 of the “pro” side of one page. Its counter-point has the same ID with b at the end. The part before the last dash names the debate page.
Split by debate page
Section titled “Split by debate page”The benchmark has only test judgments. We need validation queries too, to choose settings. So we split the 1,406 queries ourselves: 409 for validation and 997 for test. We split by debate page, not query by query. Points on one page share words and ideas. If one page had queries on both sides, a setting tuned on validation would have seen part of the test topics. This is a group split: all the queries of one page stay in one set. The machine learning course names it in Train, validation and test sets.
First checks
Section titled “First checks”Run the loading and checking cells of the notebook:
8,674 documents, 1,406 queries, 1,406 judgmentsjudged documents per query: {1: 1406}judged documents missing from the corpus: 5{'test': 997, 'validation': 409} | debates: 189words per query: median 174 | words per document: median 147queries that are also documents: 1298 | with the same text: 1298documents whose text appears twice or more: 218empty documents: 0queries whose text is also a document with another ID: 108What this tells us:
- One judged document per query. Other good counter-arguments count as wrong, so every score is a lower bound. At Larkfield, a query had a median of 199 judged products.
- Five judged documents are missing. Those queries score 0 for every method. We keep them, as the benchmark does.
- 1,298 queries are also documents, with the same text. The benchmark’s rule is to leave out a result whose ID is the query’s ID, and our helper does that (see the failure case below). For 108 queries, the same text is also in the corpus under another ID. The benchmark keeps that copy, and so do we, so that our numbers compare with other people’s.
- The queries are long: a median of 174 words. At Larkfield, a query had three or four words.
Frame the problem
Section titled “Frame the problem”The unit is one query: a point. The target is to put its counter-point as high as possible. The measure is NDCG@10, the course’s measure, on the test queries, with a 95% interval. We also report recall@100 (is the answer in the top 100, for a second stage?) and MRR.
With one judged document, NDCG@10 becomes simple. Its gain is the same wherever it is, so NDCG@10 depends only on the rank of that one document:
| Rank of the counter-point | 1 | 2 | 3 | 5 | 10 | 11 or lower |
|---|---|---|---|---|---|---|
| NDCG@10 | 1.000 | 0.631 | 0.500 | 0.387 | 0.289 | 0 |
Check the third column: log2(4) = 2, and 1 ÷ 2 = 0.5. So an average NDCG@10 of 0.5 means, roughly, “the answer is often third”. A gain of 0.1 is large.
We choose settings on the 409 validation queries, measure each design once on the test queries, and time it live on one laptop CPU.
The baseline: BM25
Section titled “The baseline: BM25”BM25 is cheap, needs no model, and is the usual first answer. We use the course’s settings: k1 1.2, b 0.75, English stemming, and common English words removed (bm25s 0.3.16 and PyStemmer). The notebook indexes the 8,674 documents and runs every query:
bm25 = bm25s.BM25(k1=1.2, b=0.75) # the course's settings (Lucene form)bm25.index(doc_tokens, show_progress=False)...out[q] = top_ids(s, doc_ids, n, drop=q if drop_self else None)indexed 8,674 documents in 0.5 sBM25: 0.7 ms a query (p50) on this computerTimes are from our laptop; yours will differ. On the 997 test queries:
| Run | NDCG@10 (95% interval) | recall@100 | MRR | Answer first |
|---|---|---|---|---|
| BM25 | 0.460 (0.437 to 0.483) | 0.935 | 0.386 | 23% |
BM25 has the counter-point in its top 100 for most queries, but puts it first for fewer than a quarter. A counter-point uses many of the point’s words (“vegetarian”, “environment”, “farming”), but so do other points on the same topic.
Common mistake
The query finds itself
Symptom: BM25 scores far lower than other people report for the same benchmark. Every first result looks perfect: it is exactly the query.
Cause: 1,298 of the 1,406 queries are also documents, with the same text. The best match for a text is the text itself, so the counter-point can be second at best.
Diagnosis: count the queries whose first result has the query’s ID. On 200 test queries, the first result was the query itself 100% of the time. NDCG@10 was 0.326 with the query’s own document, and 0.438 without it.
Fix: leave out a result whose ID is the query’s ID, as the benchmark does. Keep other documents with the same text and another ID.
Consequence in AI work: before you compare with a published number, check what the benchmark leaves out and how it counts. A small rule can move a score by 0.1.
The alternatives
Section titled “The alternatives”Each alternative is a method from the course. We recall the idea and give the result.
Dense search
Section titled “Dense search”The bi-encoder of Search by meaning (granite-embedding-97m-multilingual-r2, Apache-2.0) turns each text into 384 numbers. The course team encoded all documents and queries once, up to 512 tokens each. The 8,674 documents took 138 seconds on a laptop CPU.
Predict. At Larkfield, dense search lost to BM25: 0.519 against 0.620. Will it lose here too?
dense: NDCG@10 0.540; search 0.43 ms a query (p50), without encoding the queryDense search wins, by +0.081 (+0.059 to +0.104) against BM25: better on 402 test queries, worse on 250. A product query is short and full of exact words. An argument is long, and its counter-point answers it in other words. Meaning matters more here.
But a live search must also encode the query, and a long query costs much more to encode than “outdoor cushion” (see the times below).
Hybrid search
Section titled “Hybrid search”Hybrid search combines the two lists. RRF adds 1 ÷ (60 + rank) from each list. The blend scales each score to 0 to 1 and mixes them with a weight alpha for BM25. We chose alpha on the validation queries:
Text description of the chart
A line chart with alpha, the weight of BM25, on the horizontal axis from 0.1 to 0.9, and NDCG@10 on the validation queries on the vertical axis. alpha 0.1: 0.531; alpha 0.2: 0.534; alpha 0.3: 0.540; alpha 0.4: 0.540; alpha 0.5: 0.540; alpha 0.6: 0.540; alpha 0.7: 0.523; alpha 0.8: 0.506; alpha 0.9: 0.476. The line is flat between 0.3 and 0.6 and falls after 0.6.
The best alpha is 0.4, but 0.3 to 0.6 are all within 0.001 of it: that is noise. The direction matters. At Larkfield, alpha 0.8 was best, so BM25 led. Here dense search leads.
On the test queries, RRF scores 0.526 and the blend 0.562. RRF is not better than dense search alone. The difference is −0.014 (−0.031 to +0.001). It gives the weaker list, BM25, the same vote as the stronger one. The blend beats dense search by +0.021 (+0.012 to +0.031). Small, but clear.
Re-ranking with a cross-encoder
Section titled “Re-ranking with a cross-encoder”The cross-encoder of Cross-encoder re-ranking (granite-embedding-reranker-english-r2, Apache-2.0, English) reads the query and one document together. It is slow, so it re-orders only the top few documents of a first stage: here, the blend. The blend has the counter-point in its top 10 for 82% of the queries, and in its top 20 for 91%. So re-ranking 20 could, in principle, move most answers to first place.
The course team recorded the cross-encoder’s scores on the blend’s top 20 for every query, with each pair cut at 512 tokens. That took 43 minutes on the laptop’s CPU, at about 11 pairs a second. Then we chose the depth on validation:
Text description of the chart
A line chart with the number of documents that the cross-encoder re-orders on the horizontal axis (0, 5, 10 and 20) and NDCG@10 on the validation queries on the vertical axis. 0: 0.540; 5: 0.533; 10: 0.535; 20: 0.513.
So the rule “choose the depth on validation” gives a clear answer: depth 0, no cross-encoder. The test queries (measured only to learn) agree: re-ranking 10 documents scores 0.528 and 20 documents 0.496, against 0.562 for the blend.
Why does a strong model hurt? Our reading: a cross-encoder for search learns to score how well a text matches a query. Here, the right answer is the opposite claim on the same topic. The model also scores the other points of the same page as good matches. It cannot tell “agrees with” from “answers”, and with 20 documents it has more of these to put first. In Cross-encoder re-ranking, the same model hurt French queries because it reads only English. Here the task is not the one it learned.
One fair comparison
Section titled “One fair comparison”Here is every design on the same 997 test queries and the same measure:
Text description of the chart
NDCG@10 with its 95% interval for each design on the 997 test queries: BM25 0.460 (0.437 to 0.483); dense 0.540 (0.517 to 0.564); hybrid RRF 0.526 (0.504 to 0.549); hybrid blend 0.562 (0.539 to 0.585); re-rank 10 0.528 (0.504 to 0.550); re-rank 20 0.496 (0.472 to 0.519).
| Run | What it does | NDCG@10 (95% interval) | recall@100 | MRR | Difference with BM25 (95% interval) |
|---|---|---|---|---|---|
bm25 |
BM25 on the text, k1 1.2, b 0.75 | 0.460 (0.437 to 0.483) | 0.935 | 0.386 | baseline |
dense |
cosine of granite embeddings, 512 tokens | 0.540 (0.517 to 0.564) | 0.973 | 0.473 | +0.081 (+0.059 to +0.104) |
hybrid-rrf |
RRF (k 60) of the two top 100 lists | 0.526 (0.504 to 0.549) | 0.974 | 0.449 | +0.066 (+0.053 to +0.079) |
hybrid-blend |
0.4 × BM25 + 0.6 × dense, each scaled 0 to 1 | 0.562 (0.539 to 0.585) | 0.977 | 0.491 | +0.102 (+0.083 to +0.121) |
rerank-10 |
the blend, top 10 re-ordered by the cross-encoder | 0.528 (0.504 to 0.550) | 0.977 | 0.448 | +0.068 (+0.043 to +0.093) |
rerank-20 |
the blend, top 20 re-ordered | 0.496 (0.472 to 0.519) | 0.977 | 0.424 | +0.036 (+0.009 to +0.062) |
Read the last column first. It compares each design with BM25 query by query, so it is more precise than the overlapping intervals of the chart. Re-ranking takes the blend down: −0.034 (−0.052 to −0.017) with 10 documents, −0.066 (−0.088 to −0.045) with 20.
The intervals are about ±0.023, against ±0.045 at Larkfield. We have 997 test queries; Larkfield’s averages used 241. More queries give a more precise average (Build a test set that represents your users).
Long queries change what works
Section titled “Long queries change what works”The average hides the effect of query length. We cut the queries into three groups of equal size: short (up to 148 words), medium (149 to 208) and long (over 208).
- BM25
- dense
- hybrid blend
- dense, query cut at 128 tokens (dashed)
Text description of the chart
Four lines over three groups of query length: short (up to 148 words), medium (149 to 208) and long (over 208). BM25: 0.434, 0.482, 0.465; dense: 0.550, 0.540, 0.530; hybrid blend: 0.567, 0.568, 0.550; dense, query cut at 128 tokens: 0.551, 0.520, 0.488.
Two things happen as queries get longer:
- BM25 gets better: 0.434 on short queries, 0.465 on long ones. A long point repeats its key words, and more of them are in the counter-point. So the gap to dense search shrinks: +0.116 (+0.080 to +0.153) on short queries, +0.064 on long ones.
- Dense search needs to read the whole query. A bi-encoder reads a text only up to its token limit. A token is a piece of a word, as the model’s tokenizer cuts it. At Larkfield, 128 tokens were plenty. Here the median query has 221 tokens, and 1,287 of the 1,406 queries are longer than 128.
Change one thing: cut the query short
Section titled “Change one thing: cut the query short”What if we keep the Larkfield setting, and encode only the first 128 tokens of each query? Or give BM25 only the first 40 words?
dense-128 0.521bm25-first-40-words 0.397Both lose. Cut at 128 tokens, dense search loses −0.019 (−0.030 to −0.008) overall. It loses almost nothing on short queries, but −0.042 (−0.067 to −0.018) on long ones. The model never reads the second half of the argument. BM25 with only 40 words loses −0.063 (−0.084 to −0.042).
So read the whole query: set the token limit from the length of your real queries, not from a default. In a help centre, a model that reads only the first lines of a long message answers the greeting, not the problem.
Worst cases by topic
Section titled “Worst cases by topic”Here are the topics where BM25 is weakest:
| Topic | Test queries | BM25 | dense | blend | re-rank 10 |
|---|---|---|---|---|---|
| society | 46 | 0.328 | 0.472 | 0.448 | 0.432 |
| free | 41 | 0.360 | 0.405 | 0.443 | 0.461 |
| philosophy | 61 | 0.413 | 0.424 | 0.473 | 0.489 |
| economy | 95 | 0.413 | 0.532 | 0.561 | 0.523 |
| environment | 17 | 0.433 | 0.458 | 0.470 | 0.365 |
Small topics jump around. The cross-encoder is worst on environment (0.365, 17 queries). It is best on sport (0.737, 23 queries, not in the table). With 17 queries, one query can move the average by 0.06. Read small slices as hints, not as results.
The right page, the wrong answer
Section titled “The right page, the wrong answer”Every method finds the right debate page much more often than the right answer. The other points on that page share the topic and the words, but only one document counts as right.
A counter-point’s ID ends in b, and a point’s ID ends in a. So we can also count how often the first result is a counter-point at all:
| Run | Same page in the top 10 | First result is a counter-point | The answer first |
|---|---|---|---|
| BM25 | 52% | 39% | 23% |
| dense | 56% | 51% | 32% |
| hybrid blend | 58% | 50% | 34% |
| re-rank 10 | 58% | 51% | 29% |
The cross-encoder does not raise the share of counter-points first. Words or meaning find the page. Finding the answer on the page needs a model that understands “against”.
Time and cost
Section titled “Time and cost”Quality is half of the decision; time and money are the other half. The course team measured each stage live on its laptop (Apple M4, one thread, 40 test queries, other programs running):
| Stage | p50 ms, 1 thread | p95 ms, 1 thread | p50 ms, 4 threads |
|---|---|---|---|
| BM25 | 0.8 | 1.2 | 0.6 |
| Encode the query (512 tokens) | 34.4 | 65.6 | 23.6 |
| Dense search (8,674 vectors) | 0.6 | 2.5 | 0.6 |
| Cross-encoder, 10 documents | 1,843 | 1,908 | 1,112 |
| Cross-encoder, 20 documents | 3,675 | 3,753 | 2,183 |
Encoding a long query costs about 42 times the whole BM25 search. The cross-encoder is the expensive part: it reads 10 or 20 long pairs per search.
Text description of the diagram
The five designs as stages, with live times on one thread. A long query (a median of 174 words) goes to BM25 (0.8 ms) and to the encoder (34.4 ms for 512 tokens). The encoded query goes to the dense search (0.6 ms). BM25 alone scores NDCG@10 0.460 in 0.8 ms; dense alone 0.540 in 35.0 ms. The blend, 0.4 × BM25 + 0.6 × dense, scores 0.562 in 35.8 ms. The cross-encoder after the blend scores 0.528 with 10 documents, in 1,878 ms. With 20 documents it scores 0.496, in 3,711 ms.
Now add up the stages and turn milliseconds into money. We assume that a server with 4 vCPUs costs US$0.17 an hour on demand. Check your provider’s price. With one thread per search, it runs four searches at once.
Text description of the chart
Milliseconds per search, p50, one thread, stages added up: BM25 0.8; dense 35; hybrid blend 35.8; re-rank 10 1878.4; re-rank 20 3711.3. Ines’s budget is 200 ms.
| Design | NDCG@10 | p50 ms, 1 thread | Within 200 ms? | US$ per 1,000 searches |
|---|---|---|---|---|
| BM25 | 0.460 | 0.8 | yes | 0.00001 |
| dense | 0.540 | 35.0 | yes | 0.00041 |
| hybrid blend | 0.562 | 35.8 | yes | 0.00042 |
| re-rank 10 | 0.528 | 1,878 | no | 0.02218 |
| re-rank 20 | 0.496 | 3,711 | no | 0.04381 |
As at Larkfield, the money is small. Time is the real limit. The blend fits Ines’s budget of 200 ms more than five times over: 36 ms. The cross-encoder does not fit: 1.9 seconds with 10 documents, and still 1.1 seconds on four threads. At Larkfield, 10 product candidates took 194 ms with the whole hybrid stage. Here 10 arguments take nearly ten times as long, because each pair holds two long texts.
The recommendation
Section titled “The recommendation”Kofi asked: which search, and do we need the expensive one? Here is the decision, in the course’s format.
When is the cheap method good enough?
Section titled “When is the cheap method good enough?”BM25 is the cheapest design: one millisecond, no model on the server, nothing to re-encode when an argument changes. But it is 0.102 behind the blend, a large gap. The answer is first for 34% of the test queries with the blend, and 23% with BM25.
BM25 alone is good enough when you cannot run a model on the search server. It is also enough when it only feeds a later stage: it has the answer in its top 100 for 94% of the queries. It is not good enough when people read the first few results, as students do here. Then a gain of +0.102 is worth 36 ms.
And the cross-encoder? On this task it is expensive and worse: about 1,843 ms more per search, and NDCG@10 down by 0.034 with 10 documents. Measure a model on your own task before you pay for it.
An end-to-end run on new input
Section titled “An end-to-end run on new input”A design is not finished until it runs on a new query. The course team wrote an argument that is not in ArguAna:
Homework should be banned in primary schools. Young children learn best through play and rest, and long evenings of worksheets make them tired and unhappy. Homework also widens the gap between pupils whose parents can help them and pupils whose parents work late.
The notebook runs the chosen design: BM25, the recorded embedding of the argument, and the blend with alpha 0.4. These are the first five results:
| Rank | Document | Start of the text | Side, read by us |
|---|---|---|---|
| 1 | training-education-psthwbh-pro04a |
Homework puts students off learning Homework puts students off learning. Studies have shown that many children … | an argument for a ban (the same side) |
| 2 | training-education-psthwbh-pro04b |
If homework puts students off learning, then it has been badly planned by the teacher. As … | the answer to that argument: against the ban |
| 3 | training-education-psthwbh-con04a |
Homework is an essential part of education, allowing students to learn information beyond that which they … | an argument against a ban |
| 4 | training-education-psthwbh-con03b |
Homework is a class issue. In school everyone is equal, but at home some people have … | an answer that supports a ban (the same side) |
| 5 | training-education-psthwbh-pro03b |
Homework has not prevented students doing other activities; it takes very little time to complete. Recent … | an answer that says homework takes little time: against the ban |
We marked each result by hand. All five come from one debate page on homework, so the design found the right topic. But the first result is on the same side as our argument. The student gets three counter-arguments (ranks 2, 3 and 5) and two results on the same side (ranks 1 and 4). That is the pattern of the benchmark: the right page, often the wrong side first.
A new query has no judgment, so a person must read the results, as the decision says.
Insights and limits
Section titled “Insights and limits”- The course’s ranking of methods did not transfer. At Larkfield, BM25 beat dense search (0.620 against 0.519), and the blend needed alpha 0.8. The data decides, not the method’s reputation.
- Easier than Larkfield in some ways: edited English, with no misspellings, product codes or French. Harder in others: only one document per query counts as right.
- Larkfield’s log is synthetic; ArguAna’s queries are real arguments, written by debaters, not typed by searchers. Real searchers would probably write shorter, messier queries. Measure on your own queries.
- The models may have seen similar data. The Granite model cards (bi-encoder, cross-encoder) say that every training set has a permissive licence and that MS MARCO is not used. They do not name ArguAna, but the bi-encoder’s card reports a benchmark that includes it. These are vendor claims; we did not test them.
- Times are from one laptop. Your server will differ, but not the order: a search with the cross-encoder is about 50 times slower than with the blend.
Responsible use
Section titled “Responsible use”- An argument is not a fact. The documents are opinions. A counter-argument shows another view; it does not prove the first one wrong. Say so on the results page.
- Credit the source. CC BY 4.0 requires that you name the authors, link the licence and say what you changed. Our attribution is in the dataset card.
- Topics are not equal. Small topics (environment: 17, sport: 23) have uncertain numbers. Do not claim that the search “works for religion” from a handful of queries.
- Keep a person in the loop beyond practice. Do not use this search to decide what students may say, or to grade them.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This is a good public project. Put these in a repository:
- The notebook, with its outputs, and a link to the benchmark (do not copy the data).
- A
README.mdwith this outline:
# Counter-argument search: when is the cheap method good enough?
## ProblemFind the best counter-argument for a long argument, among 8,674 documents.
## Data and licenceArguAna (Wachsmuth, Syed and Stein, CC BY 4.0), BEIR version. Queries split bydebate page: 409 validation, 997 test. One judged document per query.
## ApproachBM25 baseline; dense (granite-embedding-97m-multilingual-r2); hybrid RRF andblend (alpha chosen on validation); cross-encoder re-ranking (depth chosen onvalidation). One measure: NDCG@10 with 95% bootstrap intervals.
## Results (test, 997 queries)| Design | NDCG@10 | p50 ms | US$ per 1,000 || ... | ... | ... | ... |
## Decision...
## LimitsOne judgment per query; model training data not checked; one laptop's times.- Your decision, in the format above, and one chart: NDCG@10 by query length.
In an interview: “On a public benchmark of long argument queries, a blend of BM25 and a small embedding model beat BM25 by +0.102 NDCG@10, in 36 ms per search. A cross-encoder made results worse and took almost two seconds, so I recommended the blend.”
Go further:
- Compare a second bi-encoder, with permissive weights and training data, on the same split.
- Encode the documents with 256 tokens instead of 512. How much quality do you lose, and how much encoding time do you save?
- Add a short prefix to each query, such as “An argument against this:”. Can a simple prompt turn similarity into opposition?
- Compare the two other case studies of this module: Product search for a furniture shop and Shopping queries with hard negatives. Which method won where, and why?
Knowledge check
Section titled “Knowledge check”Check your understanding
At Larkfield, BM25 beat dense search; on ArguAna, dense search beats BM25. Give two reasons from this case study, and say what this means for a new search project.
Show the answer
First, the queries: product queries are short and full of exact words (names, sizes, codes), and BM25 is good at exact words. ArguAna’s queries are long arguments. The counter-point answers them in other words, so meaning matters more. Second, the judgments: one answer per query rewards a method that understands the claim, not one that matches its topic. So the reputation of a method does not transfer from one corpus to another. In a new project, build a small test set first. Compare at least BM25, dense search and a blend on your own queries before you choose.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your team runs the benchmark and gets NDCG@10 0.33 for BM25, far below the 0.46 in this case study. What do you check first, and why?
Show the answer
Check that the evaluation leaves out each query’s own document. 1,298 of the 1,406 queries are also documents with the same text. Without that rule, the query itself comes first, and the counter-point can be second at best. On 200 test queries, forgetting this rule took BM25 from 0.438 down to 0.326. Before you compare your number with a published one, check what the benchmark leaves out and how it counts.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Kofi says: “The cross-encoder is the strongest model we have. Let us add it anyway; users will not notice one more second.” What do you answer, with numbers?
Show the answer
No. On this task the cross-encoder makes results worse, not better. Re-ranking 10 documents lowers NDCG@10 from 0.562 to 0.528 (−0.034, interval −0.052 to −0.017). With 20 documents, it falls to 0.496. It also makes a search take about 1.9 seconds on one thread, against 36 ms for the blend. That breaks the 200 ms budget by far. “Strongest” means strongest on the task of its training: matching a text to a query. Here the right answer is the opposite claim. Keep the blend. Test a re-ranker again only when one exists for counter-arguments.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On ArguAna, with long argument queries, dense search beats BM25 (+0.081), the reverse of Larkfield. The blend (alpha 0.4, chosen on validation) is best: NDCG@10 0.562, +0.102 over BM25.
- RRF is not better than dense search alone: it gives the weaker list a full vote.
- The cross-encoder makes results worse (−0.034 with 10 documents) and makes a search take about 1.9 seconds. A strong model can be wrong for a task.
- Long queries change what works. BM25 closes part of the gap on long queries. Cutting the dense query at 128 tokens costs −0.042 on long queries.
- Check what a benchmark leaves out (the query’s own document) before you trust a number.
- The decision: the blend, 36 ms per search, a person reading 20 new arguments before each release.
This was the last case study. In the final assignment, you apply the whole course to Larkfield’s new season, on unseen queries.
Next: Final assignment