Many languages
- Lesson 18
- 40 min
- Module 5: Dense and hybrid retrieval
Where we are
Hybrid search works well on English queries. Some Larkfield customers search in French. In this lesson, we compare a multilingual model with translating the query, and we measure the French slice apart.
Lesson 18 of 30 · Search and Ranking: Find the Best MatchMany languages
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Serve customers who search in French against an English catalogue, and know which method works, what it costs, and how far to trust the measurement.
NotebookMeasure the French slice in a notebook
The notebook puts each French test query next to the English query it came from. It measures BM25, dense search and the recorded translations on both. You compute the drop from English to French for each method, and find the French queries where dense search wins and where it loses. Its setup cells download the data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Dense search and hybrid search, from Search by meaning and Hybrid search.
- French titles in the index and the rewrite by a language model (the run
llm-rewrite), from Normalise text and Synonyms and query rewriting.
What you need on your computer
- The project
larkfield-searchwith the files of Search by meaning. This lesson adds no file.
A shop in Montréal
Section titled “A shop in Montréal”Larkfield ships to Québec. Its product labels are in English. A customer in Montréal asks for “une chaise ergonomique”. A shop assistant can help her in two ways.
A bilingual assistant understands the French directly. She walks to the office chairs. No step is added, but she must know both languages well.
Another assistant phones a translator first. The translator says “ergonomic chair”. Then the assistant searches the English labels as usual. This works for any language the translator knows. But every question waits for a phone call, and every call costs money.
Search has the same two choices: a model that understands both languages, or a translation of the query.
The problem: French queries against English text
Section titled “The problem: French queries against English text”The synthetic query log says it plainly. 12% of the searches are in French, and 94.3% of the French searches return nothing today. The most common empty search of all is “hibou” (owl), 216 times. “ampoule” (light bulb) is empty 119 times, and “canapé” (sofa) 79 times.
Remember what this data is. The French queries were machine-translated from English WANDS queries by a language model. The French titles of 7,597 products were machine-translated too. The log is synthetic. So these are realistic practice problems, not real customers’ words.
Module 4 already helped. The run bm25-norm+spell+syn searches the French titles with French stems. That found owls for “hibou” and bulbs for “ampoule”. But on the French test queries as a whole, it scores only 0.299, against 0.726 on English:
search eval --run bm25-norm+spell+syn --by languagebm25-norm+spell+syn on 243 test queries: NDCG@10 0.620 (95% interval 0.574 to 0.662)recall@100 0.546, MRR 0.626, zero results 0.0%, judged in the top 10 76.0%; 0.3 ms a query (p50), 0.8 ms (p95)
language queries ndcg@10 low high zero-result rate en 182 0.726 0.683 0.766 0.000 fr 61 0.299 0.215 0.384 0.000The 61 French test queries are 60 machine-translated queries and “canapé”, a planted query. Most products have no French title, so BM25 must match French words against English text. Sometimes the match is a false friend:
search query "chaise ergonomique" --run bm25-norm+spell+syn -k 3bm25-norm+spell+syn: "chaise ergonomique" -> 3 results (query F-422: judged results are marked) 1. LK-68838 cordella chaise lounge Indoor Chaise Lounges 327.99 [not judged] 2. LK-83161 marshalltown chaise lounge Indoor Chaise Lounges 216.99 [not judged] 3. LK-64061 meagan chaise lounge Indoor Chaise Lounges 172.99 [not judged]In French, “chaise” means “chair”. In English, “chaise” is part of “chaise lounge”, a long seat for lying down. BM25 matches the word, not the meaning. “table à cartes” (card table) has the same problem: the stem of “cartes” matches “cart”, and BM25 shows kitchen carts.
Idea 1: a multilingual model
Section titled “Idea 1: a multilingual model”A multilingual bi-encoder learned from texts in many languages, including pairs of texts that mean the same. So it places “table à cartes” near “card table”, in the same space of 384 numbers. A French query can then find English products directly. Our model, granite-embedding-97m-multilingual-r2, is such a model; its model card lists French among its languages.
A tiny example: French and English vectors
Section titled “A tiny example: French and English vectors”We encoded short French and English texts with the model and computed their cosines:
| French text | English text | Cosine |
|---|---|---|
| table à cartes | card table | 0.933 |
| table à cartes | kitchen cart | 0.861 |
| chaise ergonomique | ergonomic office chair | 0.872 |
| chaise ergonomique | chaise lounge | 0.814 |
| canapé | sofa | 0.768 |
| canapé | coffee table | 0.843 |
For “table à cartes” and “chaise ergonomique”, the right English text is closer than the false friend. The model understood the French. For “canapé”, the model is wrong: “coffee table” is closer than “sofa”. A single word gives the model little to work with, as in Where dense search fails. “canapé” also has a second meaning in French and English, a small snack, and the model may mix the two.
On the catalogue, the run dense shows the same two sides. For “chaise ergonomique”, its top 5 are five exact ergonomic chairs: NDCG@10 1.000, against 0.032 for BM25. For “hibou”, it fails:
search query "hibou" --run dense -k 3dense: "hibou" -> 3 results (query F-227: judged results are marked) 1. LK-84055 fiji shower hooks Shower Curtain Hooks and 163.99 [not judged] 2. LK-88452 hoaloha , god of the hawaiian isle tiki wall décor Wall Décor 63.99 [not judged] 3. LK-71998 kohler toobi rite-temp valve trim with lever handle Fixture Parts And Compon 71.99 [not judged]Shower hooks and a tiki wall décor. The model seems to read “hibou” by its letters, not its meaning. BM25 finds owls, because 21 products have “hibou” in their machine-translated French title.
Idea 2: translate the query
Section titled “Idea 2: translate the query”The other route translates the query into English first, then searches the English text with BM25. In Synonyms and query rewriting, a language model, chat-small, rewrote every validation and test query. For a French query, the rewrite is a translation. “chaise ergonomique” became “ergonomic chair”. The course recorded each answer, so you replay them without a key.
Text description of the diagram
The French query “chaise ergonomique” can take two routes. Route 1 searches by meaning. The multilingual bi-encoder reads the French query as it is. The first results are two ergonomic mesh task chairs and a shapleigh ergonomic office chair, all exact. On the French test queries, dense scores NDCG@10 0.212 and hybrid-blend 0.322. It takes about 8 ms a search, with no model call. Route 2 translates the query. A language model, chat-small, rewrites it as “ergonomic chair”. Then BM25 on the English text puts ergonomic chairs first. On the French test queries, it scores 0.690, with recorded rewrites. It takes 1.2 s a search (p50) and costs US$0.04 per 1,000 searches. A dashed box warns that the French queries were machine-translated from English by the same family of model. Translating them back is easier than translating what real French customers type.
Larkfield: measure the French slice apart
Section titled “Larkfield: measure the French slice apart”Now measure every method on the French queries. The fairest view puts each French test query next to the English query it was made from. Both have the same judgments. So the only difference is the language. For 59 such pairs:
- English source query
- French query (machine-translated)
Text description of the chart
The chart shows five methods, each with two rows: the 59 English source queries and their 59 French versions, with NDCG@10 and its 95% interval. BM25 (bm25-norm+spell+syn): English 0.694, French 0.301. dense: English 0.622, French 0.216. hybrid-rrf: English 0.706, French 0.315. hybrid-blend: English 0.712, French 0.324. llm-rewrite: English 0.701, French 0.690.
Read the gaps from English to French:
| Method | English | French | Drop |
|---|---|---|---|
BM25 (bm25-norm+spell+syn) |
0.694 | 0.301 | 0.393 |
dense (multilingual) |
0.622 | 0.216 | 0.406 |
hybrid-rrf |
0.706 | 0.315 | 0.391 |
hybrid-blend |
0.712 | 0.324 | 0.388 |
llm-rewrite (translate, then BM25) |
0.701 | 0.690 | 0.011 |
This is the surprise of the module. The multilingual model loses as much in French as BM25 does: about 0.4. It wins on some French queries, like “chaise ergonomique” and “table à cartes”. It loses on others, like “hibou”, “industriel” and “carreau rond penny” (penny round tile), where the French titles let BM25 match exactly. On average, the two effects cancel.
search compare gives the same picture with paired intervals, on all French test queries:
search compare bm25-norm+spell+syn hybrid-blend --by languagehybrid-blend minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.016 (95% interval +0.007 to +0.025); better on 90, worse on 41, the same on 110
language queries bm25-norm+spell+syn hybrid-blend difference low high worse en 181 0.726 0.740 0.014 0.003 0.026 37 fr 60 0.299 0.322 0.023 0.008 0.041 4The blend helps French a little (+0.023, interval +0.008 to +0.041), and hurts only 4 French queries. Dense search alone does not help French (−0.087, interval −0.189 to +0.011). The translation route is far ahead:
search compare bm25-norm+spell+syn llm-rewrite --by languagellm-rewrite minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.108 (95% interval +0.078 to +0.140); better on 69, worse on 23, the same on 149
language queries bm25-norm+spell+syn llm-rewrite difference low high worse en 181 0.726 0.740 0.014 -0.001 0.030 19 fr 60 0.299 0.690 0.390 0.308 0.475 4+0.390 on French. After translation, French queries score 0.690, almost as high as their English sources (0.701).
How far to trust this result
Section titled “How far to trust this result”Priya warns the team before anyone celebrates. The French test queries were written by a language model of the same family, from English queries. Translating them back is the easiest possible task: the model only has to undo its own work. Real French customers write differently. They use Québec words, they make spelling mistakes, and they mix French and English. On real French queries, the translation route will score lower. How much lower, this test set cannot say.
So the honest conclusion has three parts. The translation route is the strongest on these queries. Its advantage is overstated here. And the only fair test is a set of real French queries with judgments, which Larkfield does not have yet.
Which route to choose, and what it costs
Section titled “Which route to choose, and what it costs”| Route | French NDCG@10 (test) | Time per search | Cost | Risk |
|---|---|---|---|---|
| BM25 with French titles | 0.299 | about 1 ms | none at search time; French titles for the catalogue | needs translated titles; false friends |
| Dense, multilingual | 0.212 | about 8 ms | none at search time | single words and names fail |
| Hybrid blend | 0.322 | about 9 ms | none at search time | small gain only |
| Translate the query, then BM25 | 0.690 | 1.2 s (p50), 1.6 s (p95) | US$0.04 per 1,000 searches | measured on easy queries; a network call per search |
The French numbers in this table are the language slice of search compare. Module 4 grouped the queries by source (the machine-translated queries), so its numbers are a little higher: 0.301 for BM25, for example.
The translation route breaks Amira’s budget of 200 ms by six times. Omar likes its quality. There is a middle way: cache the translations of the most common texts. In the synthetic log, the 100 most common texts make 51.1% of the searches. A cache of translated head queries gives most searches the good answer in a millisecond. New and rare queries can use the hybrid search while their translation is computed in the background.
Use a multilingual model when you serve many languages, cannot call a model per search, and the queries are descriptions rather than single words. Measure each language on its own.
Use translation for the languages that matter most, with a cache, and measure it on real queries before you trust it.
Always report each language as its own slice. French is 61 of 243 test queries. In the average, its problems are diluted.
Common mistake
A language that the average hides
Symptom: a new search release raises NDCG@10 on the test set. A week later, the team in Québec reports that French searches are worse than before.
Cause: the change helped English queries and hurt French ones. English is about three quarters of the test queries, so the average went up.
Diagnosis: search compare old new --by language. Look at the French row and its interval, not only at the first line.
Fix: make each language a release gate: no release if any language falls with an interval below zero. Keep enough queries per language to measure it.
Consequence in AI work: a multilingual product needs a measurement per language. A model that is “multilingual” on its model card can still fail in one language on your data.
Try it yourself
Section titled “Try it yourself”Pick a French test query and predict which route wins. Then run all three:
search query "table à cartes" --run bm25-norm+spell+syn -k 3search query "table à cartes" --run dense -k 3search query "table à cartes" --run llm-rewrite -k 3What we got: BM25 put a cart coffee table, a bar cart and a kitchen cart first. Dense search and the translation both put exact card tables first. Then try “industriel”, where the result is the other way round.
Knowledge check
Section titled “Knowledge check”Check your understanding
BM25 finds owls for “hibou”, but dense search finds shower hooks. The multilingual model is supposed to understand French. What explains the result?
Show the answer
BM25 matches “hibou” in the French titles: 21 products have it in their machine-translated French title, so BM25 finds them exactly. The bi-encoder has to understand one rare French word with no context. It seems to place it by its letters, near “hooks” and “hoaloha”, not near “owl”. A single word gives a bi-encoder little meaning to work with, in any language.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The translation route scores 0.690 on the French test queries, against 0.322 for the hybrid blend. Hana asks: “So we translate every French search?” Give two reasons for caution and one practical plan.
Show the answer
First, time and cost: 1.2 s per search (p50) breaks the 200 ms budget, and each search costs a model call. Second, the measurement favours translation. A model of the same family machine-translated the French test queries from English. So translating them back is unrealistically easy. A practical plan: cache the translations of the most common French texts (the top 100 texts make 51.1% of all searches). Use the hybrid blend for the others. Build a set of real French queries with judgments before the final decision.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why is the comparison of 59 French queries with their English sources fairer than comparing the French slice with the English slice?
Show the answer
Each pair has the same intent and the same judgments. Only the language differs. So the drop from English to French measures the cost of the language alone. The English slice contains other queries, with other difficulties: misspellings, codes and different products. A difference between the two slices would mix the language with everything else.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- French queries against an English catalogue fail in two ways. No word matches at all (“hibou” was the most common empty search), or a false friend matches (“chaise” in “chaise lounge”).
- A multilingual bi-encoder places texts with the same meaning near each other, in any language. It wins on “chaise ergonomique” and “table à cartes”, and fails on single words like “hibou” and “canapé”.
- On 59 French test queries and their English sources, BM25 and dense search both lose about 0.4 in French. The hybrid blend helps French a little (+0.023).
- Translating the query with a language model scores 0.690 in French. But it takes 1.2 s and a model call per search, and we measured it on easy, machine-translated queries.
- Measure each language as its own slice, and test on real queries in that language before you trust a result.
You now have a hybrid search that scores 0.636 overall, and a French slice that is still weak. The hybrid list gives good candidates; their order can be better. Next, in Module 6, you split ranking into two stages: a fast search finds candidates, and a slower, more careful model orders them.
Next: Two-stage ranking