Skip to content
Français

Many languages

  • Lesson 18
  • 40 min
  • Module 5: Dense and hybrid retrieval

Where we are

Hybrid search works well on English queries. Some Larkfield customers search in French. In this lesson, we compare a multilingual model with translating the query, and we measure the French slice apart.

What you will be able to do

Serve customers who search in French against an English catalogue, and know which method works, what it costs, and how far to trust the measurement.

NotebookMeasure the French slice in a notebookColab · Kaggle · GitHub

The notebook puts each French test query next to the English query it came from. It measures BM25, dense search and the recorded translations on both. You compute the drop from English to French for each method, and find the French queries where dense search wins and where it loses. Its setup cells download the data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-search with the files of Search by meaning. This lesson adds no file.

Larkfield ships to Québec. Its product labels are in English. A customer in Montréal asks for “une chaise ergonomique”. A shop assistant can help her in two ways.

A bilingual assistant understands the French directly. She walks to the office chairs. No step is added, but she must know both languages well.

Another assistant phones a translator first. The translator says “ergonomic chair”. Then the assistant searches the English labels as usual. This works for any language the translator knows. But every question waits for a phone call, and every call costs money.

Search has the same two choices: a model that understands both languages, or a translation of the query.

The problem: French queries against English text

Section titled “The problem: French queries against English text”

The synthetic query log says it plainly. 12% of the searches are in French, and 94.3% of the French searches return nothing today. The most common empty search of all is “hibou” (owl), 216 times. “ampoule” (light bulb) is empty 119 times, and “canapé” (sofa) 79 times.

Remember what this data is. The French queries were machine-translated from English WANDS queries by a language model. The French titles of 7,597 products were machine-translated too. The log is synthetic. So these are realistic practice problems, not real customers’ words.

Module 4 already helped. The run bm25-norm+spell+syn searches the French titles with French stems. That found owls for “hibou” and bulbs for “ampoule”. But on the French test queries as a whole, it scores only 0.299, against 0.726 on English:

Terminal window
search eval --run bm25-norm+spell+syn --by language
bm25-norm+spell+syn on 243 test queries: NDCG@10 0.620 (95% interval 0.574 to 0.662)
recall@100 0.546, MRR 0.626, zero results 0.0%, judged in the top 10 76.0%; 0.3 ms a query (p50), 0.8 ms (p95)
language queries ndcg@10 low high zero-result rate
en 182 0.726 0.683 0.766 0.000
fr 61 0.299 0.215 0.384 0.000

The 61 French test queries are 60 machine-translated queries and “canapé”, a planted query. Most products have no French title, so BM25 must match French words against English text. Sometimes the match is a false friend:

Terminal window
search query "chaise ergonomique" --run bm25-norm+spell+syn -k 3
bm25-norm+spell+syn: "chaise ergonomique" -> 3 results (query F-422: judged results are marked)
1. LK-68838 cordella chaise lounge Indoor Chaise Lounges 327.99 [not judged]
2. LK-83161 marshalltown chaise lounge Indoor Chaise Lounges 216.99 [not judged]
3. LK-64061 meagan chaise lounge Indoor Chaise Lounges 172.99 [not judged]

In French, “chaise” means “chair”. In English, “chaise” is part of “chaise lounge”, a long seat for lying down. BM25 matches the word, not the meaning. “table à cartes” (card table) has the same problem: the stem of “cartes” matches “cart”, and BM25 shows kitchen carts.

A multilingual bi-encoder learned from texts in many languages, including pairs of texts that mean the same. So it places “table à cartes” near “card table”, in the same space of 384 numbers. A French query can then find English products directly. Our model, granite-embedding-97m-multilingual-r2, is such a model; its model card lists French among its languages.

A tiny example: French and English vectors

Section titled “A tiny example: French and English vectors”

We encoded short French and English texts with the model and computed their cosines:

French text English text Cosine
table à cartes card table 0.933
table à cartes kitchen cart 0.861
chaise ergonomique ergonomic office chair 0.872
chaise ergonomique chaise lounge 0.814
canapé sofa 0.768
canapé coffee table 0.843

For “table à cartes” and “chaise ergonomique”, the right English text is closer than the false friend. The model understood the French. For “canapé”, the model is wrong: “coffee table” is closer than “sofa”. A single word gives the model little to work with, as in Where dense search fails. “canapé” also has a second meaning in French and English, a small snack, and the model may mix the two.

On the catalogue, the run dense shows the same two sides. For “chaise ergonomique”, its top 5 are five exact ergonomic chairs: NDCG@10 1.000, against 0.032 for BM25. For “hibou”, it fails:

Terminal window
search query "hibou" --run dense -k 3
dense: "hibou" -> 3 results (query F-227: judged results are marked)
1. LK-84055 fiji shower hooks Shower Curtain Hooks and 163.99 [not judged]
2. LK-88452 hoaloha , god of the hawaiian isle tiki wall décor Wall Décor 63.99 [not judged]
3. LK-71998 kohler toobi rite-temp valve trim with lever handle Fixture Parts And Compon 71.99 [not judged]

Shower hooks and a tiki wall décor. The model seems to read “hibou” by its letters, not its meaning. BM25 finds owls, because 21 products have “hibou” in their machine-translated French title.

The other route translates the query into English first, then searches the English text with BM25. In Synonyms and query rewriting, a language model, chat-small, rewrote every validation and test query. For a French query, the rewrite is a translation. “chaise ergonomique” became “ergonomic chair”. The course recorded each answer, so you replay them without a key.

The French query chaise ergonomique goes two ways. Route 1, search by meaning: the multilingual bi-encoder reads the French query as it is; first results: ergonomic mesh task chair (exact), twice, and shapleigh ergonomic office chair (exact); on the French test queries NDCG@10 dense 0.212 and hybrid-blend 0.322; about 8 ms a search, no model call. Route 2, translate the query: a language model, chat-small, rewrites it as ergonomic chair; then BM25 on the English text puts ergonomic chairs first; on the French test queries 0.690 with recorded rewrites; 1.2 s a search (p50) and US$0.04 per 1,000. A dashed box warns: the French queries were machine-translated from English by the same family of model, so translating them back is easier than translating what real French customers type.
Two routes for a French query. The translation route scores much higher on these queries, but it is slow, it costs money, and the measurement favours it.
Text description of the diagram

The French query “chaise ergonomique” can take two routes. Route 1 searches by meaning. The multilingual bi-encoder reads the French query as it is. The first results are two ergonomic mesh task chairs and a shapleigh ergonomic office chair, all exact. On the French test queries, dense scores NDCG@10 0.212 and hybrid-blend 0.322. It takes about 8 ms a search, with no model call. Route 2 translates the query. A language model, chat-small, rewrites it as “ergonomic chair”. Then BM25 on the English text puts ergonomic chairs first. On the French test queries, it scores 0.690, with recorded rewrites. It takes 1.2 s a search (p50) and costs US$0.04 per 1,000 searches. A dashed box warns that the French queries were machine-translated from English by the same family of model. Translating them back is easier than translating what real French customers type.

Now measure every method on the French queries. The fairest view puts each French test query next to the English query it was made from. Both have the same judgments. So the only difference is the language. For 59 such pairs:

  • English source query
  • French query (machine-translated)
NDCG@10 of each method on 59 French test queries and on the English queries they were translated from, with 95% intervals. Every method loses about 0.4 in French, except the translation of the query.
Text description of the chart

The chart shows five methods, each with two rows: the 59 English source queries and their 59 French versions, with NDCG@10 and its 95% interval. BM25 (bm25-norm+spell+syn): English 0.694, French 0.301. dense: English 0.622, French 0.216. hybrid-rrf: English 0.706, French 0.315. hybrid-blend: English 0.712, French 0.324. llm-rewrite: English 0.701, French 0.690.

Read the gaps from English to French:

Method English French Drop
BM25 (bm25-norm+spell+syn) 0.694 0.301 0.393
dense (multilingual) 0.622 0.216 0.406
hybrid-rrf 0.706 0.315 0.391
hybrid-blend 0.712 0.324 0.388
llm-rewrite (translate, then BM25) 0.701 0.690 0.011

This is the surprise of the module. The multilingual model loses as much in French as BM25 does: about 0.4. It wins on some French queries, like “chaise ergonomique” and “table à cartes”. It loses on others, like “hibou”, “industriel” and “carreau rond penny” (penny round tile), where the French titles let BM25 match exactly. On average, the two effects cancel.

search compare gives the same picture with paired intervals, on all French test queries:

Terminal window
search compare bm25-norm+spell+syn hybrid-blend --by language
hybrid-blend minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.016 (95% interval +0.007 to +0.025); better on 90, worse on 41, the same on 110
language queries bm25-norm+spell+syn hybrid-blend difference low high worse
en 181 0.726 0.740 0.014 0.003 0.026 37
fr 60 0.299 0.322 0.023 0.008 0.041 4

The blend helps French a little (+0.023, interval +0.008 to +0.041), and hurts only 4 French queries. Dense search alone does not help French (−0.087, interval −0.189 to +0.011). The translation route is far ahead:

Terminal window
search compare bm25-norm+spell+syn llm-rewrite --by language
llm-rewrite minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.108 (95% interval +0.078 to +0.140); better on 69, worse on 23, the same on 149
language queries bm25-norm+spell+syn llm-rewrite difference low high worse
en 181 0.726 0.740 0.014 -0.001 0.030 19
fr 60 0.299 0.690 0.390 0.308 0.475 4

+0.390 on French. After translation, French queries score 0.690, almost as high as their English sources (0.701).

Priya warns the team before anyone celebrates. The French test queries were written by a language model of the same family, from English queries. Translating them back is the easiest possible task: the model only has to undo its own work. Real French customers write differently. They use Québec words, they make spelling mistakes, and they mix French and English. On real French queries, the translation route will score lower. How much lower, this test set cannot say.

So the honest conclusion has three parts. The translation route is the strongest on these queries. Its advantage is overstated here. And the only fair test is a set of real French queries with judgments, which Larkfield does not have yet.

Route French NDCG@10 (test) Time per search Cost Risk
BM25 with French titles 0.299 about 1 ms none at search time; French titles for the catalogue needs translated titles; false friends
Dense, multilingual 0.212 about 8 ms none at search time single words and names fail
Hybrid blend 0.322 about 9 ms none at search time small gain only
Translate the query, then BM25 0.690 1.2 s (p50), 1.6 s (p95) US$0.04 per 1,000 searches measured on easy queries; a network call per search

The French numbers in this table are the language slice of search compare. Module 4 grouped the queries by source (the machine-translated queries), so its numbers are a little higher: 0.301 for BM25, for example.

The translation route breaks Amira’s budget of 200 ms by six times. Omar likes its quality. There is a middle way: cache the translations of the most common texts. In the synthetic log, the 100 most common texts make 51.1% of the searches. A cache of translated head queries gives most searches the good answer in a millisecond. New and rare queries can use the hybrid search while their translation is computed in the background.

Use a multilingual model when you serve many languages, cannot call a model per search, and the queries are descriptions rather than single words. Measure each language on its own.

Use translation for the languages that matter most, with a cache, and measure it on real queries before you trust it.

Always report each language as its own slice. French is 61 of 243 test queries. In the average, its problems are diluted.

Common mistake

A language that the average hides

Symptom: a new search release raises NDCG@10 on the test set. A week later, the team in Québec reports that French searches are worse than before.

Cause: the change helped English queries and hurt French ones. English is about three quarters of the test queries, so the average went up.

Diagnosis: search compare old new --by language. Look at the French row and its interval, not only at the first line.

Fix: make each language a release gate: no release if any language falls with an interval below zero. Keep enough queries per language to measure it.

Consequence in AI work: a multilingual product needs a measurement per language. A model that is “multilingual” on its model card can still fail in one language on your data.

Pick a French test query and predict which route wins. Then run all three:

Terminal window
search query "table à cartes" --run bm25-norm+spell+syn -k 3
search query "table à cartes" --run dense -k 3
search query "table à cartes" --run llm-rewrite -k 3

What we got: BM25 put a cart coffee table, a bar cart and a kitchen cart first. Dense search and the translation both put exact card tables first. Then try “industriel”, where the result is the other way round.

Check your understanding

BM25 finds owls for “hibou”, but dense search finds shower hooks. The multilingual model is supposed to understand French. What explains the result?

Show the answer

BM25 matches “hibou” in the French titles: 21 products have it in their machine-translated French title, so BM25 finds them exactly. The bi-encoder has to understand one rare French word with no context. It seems to place it by its letters, near “hooks” and “hoaloha”, not near “owl”. A single word gives a bi-encoder little meaning to work with, in any language.

Check your understanding

The translation route scores 0.690 on the French test queries, against 0.322 for the hybrid blend. Hana asks: “So we translate every French search?” Give two reasons for caution and one practical plan.

Show the answer

First, time and cost: 1.2 s per search (p50) breaks the 200 ms budget, and each search costs a model call. Second, the measurement favours translation. A model of the same family machine-translated the French test queries from English. So translating them back is unrealistically easy. A practical plan: cache the translations of the most common French texts (the top 100 texts make 51.1% of all searches). Use the hybrid blend for the others. Build a set of real French queries with judgments before the final decision.

Check your understanding

Why is the comparison of 59 French queries with their English sources fairer than comparing the French slice with the English slice?

Show the answer

Each pair has the same intent and the same judgments. Only the language differs. So the drop from English to French measures the cost of the language alone. The English slice contains other queries, with other difficulties: misspellings, codes and different products. A difference between the two slices would mix the language with everything else.

Recap

  • French queries against an English catalogue fail in two ways. No word matches at all (“hibou” was the most common empty search), or a false friend matches (“chaise” in “chaise lounge”).
  • A multilingual bi-encoder places texts with the same meaning near each other, in any language. It wins on “chaise ergonomique” and “table à cartes”, and fails on single words like “hibou” and “canapé”.
  • On 59 French test queries and their English sources, BM25 and dense search both lose about 0.4 in French. The hybrid blend helps French a little (+0.023).
  • Translating the query with a language model scores 0.690 in French. But it takes 1.2 s and a model call per search, and we measured it on easy, machine-translated queries.
  • Measure each language as its own slice, and test on real queries in that language before you trust a result.

You now have a hybrid search that scores 0.636 overall, and a French slice that is still weak. The hybrid list gives good candidates; their order can be better. Next, in Module 6, you split ranking into two stages: a fast search finds candidates, and a slower, more careful model orders them.

Next: Two-stage ranking