Skip to content
Français

Shopping queries with hard negatives

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, our labels were exact, partial and irrelevant. In this case study, we rank real shopping queries with exact, substitute, complement and irrelevant labels, and we measure the cost of showing substitutes.

What you will be able to do

Rank real shopping queries where most wrong answers look right. You compare five ranking methods on Amazon’s public Shopping Queries dataset, in English and Spanish, with four labels: exact, substitute, complement and irrelevant. Then you put a price on showing a substitute, and decide what a shop should show when the exact product does not exist.

Case studyPortfolio piece

Licence
Apache-2.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Graded labels and gains
  • BM25 per field and language
  • Dense and hybrid ranking
  • Cross-encoder
  • LambdaMART with hard negatives
  • Slices by language
  • Cost of a substitute

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, plus four tasks with checks. Its setup cells install the packages and download the data (about 40 MB): no setup, no account. It runs in under a minute; the last section also downloads the bi-encoder (about 200 MB).

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)

  • Python 3.12 in a new virtual environment, and Jupyter or VS Code. The notebook’s first cell installs the pinned packages (NumPy, pandas, bm25s, PyStemmer, LightGBM, sentence-transformers and PyTorch).
  • On macOS, Homebrew’s OpenMP library for LightGBM: brew install libomp, as in Cross-encoder re-ranking.
  • The setup cell downloads the data into a new folder esci-search. To start again, delete that folder. It does not touch your larkfield-search project.

Tested on macOS (Apple M4, 16 GB) with Python 3.12, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.

Lucía Ortega leads search at an online marketplace that sells in the United States and in Spain. Her team does not want to replace the search engine. It wants a better order for the 16 to 40 products that the engine already finds for each query.

The problem is that most of those products look right. For “4k 3d tv”, the list has 3D TVs, but also 4K TVs without 3D, a 3D Blu-ray player and a 3D projector. All of them share words with the query, but some are near misses.

Lucía’s questions to Dev, her machine learning engineer, are these:

  • “Which ranking method puts the exact product first most often, in English and in Spanish?”
  • “How bad is it to show a similar product instead? A shopper might still buy it.”
  • “And when we do not sell the exact product, what should the page show?”

Her budget is the course’s: 200 ms for one search on a normal CPU server.

Lucía and Dev are invented; the data is real. Nothing here is about Amazon’s own search.

You will compare five method families against a BM25 baseline on the same 800 test queries. They are BM25 per field with stemming, dense search, hybrid search, a cross-encoder and LambdaMART. You will measure quality, the label mix of the top 10, time and cost, by language and by kind of query. Then you put a price on a substitute, run the chosen design on new queries, and write it up as a portfolio piece.

Larkfield searched all 40,066 products. Here, the shop’s own search has already chosen the candidates, so the task is re-ranking: putting a short list in the best order. The other case study, Product search for a furniture shop, searches a whole catalogue.

The data is a sample of the Shopping Queries Dataset, also called ESCI after its four labels. Amazon published it in 2022 for a research competition (the KDD Cup). It has real queries typed on Amazon’s sites, and for each query up to 40 products that Amazon’s search showed. Trained annotators labelled each product.

The full products file is 1.1 GB, too large for a notebook. So the course team took a fixed random sample from ESCI’s own train and test split: 400 test and 300 train queries per language. The licence allows a copy, so the sample lives next to the notebook. Dev loads it:

queries = pd.read_csv("data/esci_queries.csv", keep_default_na=False)
labels = pd.read_csv("data/esci_judgments.csv.gz")
products = pd.read_csv("data/esci_products.csv.gz", keep_default_na=False)

The output:

1,400 queries, 33,945 judgments, 32,759 products
{('test', 'es'): 400, ('test', 'us'): 400, ('train', 'es'): 300, ('train', 'us'): 300}
['product_id', 'locale', 'title', 'brand', 'colour', 'bullets', 'description']

What this tells us. One judgment is one label for one (query, product) pair. The test queries are for the comparison; the train queries are only for choosing settings and training LambdaMART.

The annotators gave each product one of four labels. Here they are on a real test query:

The query 4k 3d tv with four candidates. Exact, gain 1: a Sony 65-inch 4K Ultra HD 3D Smart TV. Substitute, gain 0.1: an LG 86-inch 4K LED UHD Smart TV without 3D. Complement, gain 0.01: a Sony 4K 3D Blu-ray disc player. Irrelevant, gain 0: a smart projector with 3D DLP-Link.
The four ESCI labels on the test query "4k 3d tv", with the gain that NDCG gives each label in this case study.
Text description of the diagram

A query box, “4k 3d tv”, with four arrows. Exact, gain 1: Sony 65-inch 4K Ultra HD 3D Smart TV. Substitute, gain 0.1: LG 86-inch 4K LED UHD Smart TV, no 3D. Complement, gain 0.01: Sony 4K 3D Blu-ray disc player. Irrelevant, gain 0: smart projector with 3D DLP-Link.

  • Exact (E): the product the person asked for, with every detail.
  • Substitute (S): it misses a detail, but it could replace the product. The LG TV has no 3D.
  • Complement (C): it does not match, but it goes with the product. The Blu-ray player plays 3D discs on the TV.
  • Irrelevant (I): it does not help. A projector is not a TV.

Larkfield had one grade, “partial”, for both near misses. ESCI splits it into two business meanings.

The columns that the case study uses:

File and column What it means Use
queries.query, locale The query text and its site (us English, es Spanish) Input, slice
queries.split ESCI’s split: train or test Train vs comparison
queries.source ESCI’s tag for how the query was collected: negations (“without”, “sin”), parse_pattern, behavioral, other Slice
judgments.esci_label E, S, C or I Target
products.title, brand, colour Short product text BM25 fields, bi-encoder, cross-encoder
products.bullets, description Long product text, cut to 1,000 characters BM25 fields

Before any ranking, Dev checks the labels:

esci_label E S C I
locale
es 0.403 0.353 0.076 0.168
us 0.427 0.363 0.043 0.166
repeated (query, product) pairs: 0
queries with no exact product: 0
queries with only exact products: 0
judged products per query: median 16 | min 8 | max 95
products without a title: 0
products without bullet points: 3875
product IDs in both languages: 23

What this tells us. About 4 in 10 candidates are exact, and about 1 in 3 are substitutes, in both languages. Spanish has almost twice as many complements. Every query has at least one exact product, so we must construct the case “no exact product” later. 23 product IDs are in both languages with different text, so a product is known by its language and its ID.

  • The unit is one query and its candidate list. The output is an order of that list.
  • The target is the ESCI label of each candidate.
  • The time of the decision is the moment of the search, when the query, the product text and their scores are known.
  • Leaks. The file order must not decide anything, so ties are broken by product ID, never by label. The train queries are the only place to choose settings.
  • The measure is NDCG@10 with gains for the four labels.

NDCG needs a gain for each label: how much one result in the list is worth. We use ESCI’s own gains, from the dataset’s ranking task: exact 1, substitute 0.1, complement 0.01, irrelevant 0. Dev gives the new gains to the course’s compare.py.

A tiny example: one query has four candidates, a, b, c and d, labelled E, S, C and I:

exact first: 1.0 | substitute first: 0.689 | irrelevant first: 0.642

What this tells us. With these gains, a substitute in first place costs 0.311 of NDCG, and an irrelevant product costs only 0.047 more. To this measure, a substitute is almost a miss.

Think of a shop assistant in training. If you test them by asking “Is this garden hose a TV?”, they always pass, and they learn nothing. A good test asks “Is this 4K TV the 3D TV that the customer wants?”. The useful wrong answers are the ones that look right.

A negative is a product that is not the answer. A hard negative looks like the answer: it shares words, the category or the brand. An easy negative is any random product.

The query 100w solar panel without controller. Positive, exact: an ACOPOWER 100W portable solar panel kit. Hard negative: a 20A solar charge controller for a solar panel, which shares 3 query words; the shop's search showed it, and a model must read 'without controller'. Easy negative: a Gillette razor blade refill, which shares no query word; the rule 'shares words = good' is enough to reject it.
A hard negative from the data and an easy negative. Every candidate in ESCI was shown by the shop's search, so its wrong answers are hard negatives.
Text description of the diagram

A query box, “100w solar panel without controller”, with three arrows. Positive, exact: ACOPOWER 100W portable solar panel kit. Hard negative: 20A solar charge controller for a solar panel, shares 3 query words; “the shop’s search showed it: a model must read ‘without controller’”. Easy negative: Gillette razor blade refills, shares no query word; “a random product: ‘shares words = good’ is enough to reject it”.

First, how good is a list with no ranking? Dev puts each list in a random order. Then he runs the baseline: BM25 on the title, bullet points and description as one field, with plain words.

random order NDCG@10 0.543 (0.528 to 0.559) exact first 41.5% S in top 10 35.8%
bm25 NDCG@10 0.682 (0.666 to 0.699) exact first 63.0% S in top 10 30.0%

Two new columns: exact first is the share of queries with an exact product in first place, and S in top 10 is the share of the top 10 that are substitutes.

What this tells us. A random order already scores 0.543, because 4 in 10 candidates are exact. Every other method must beat BM25’s 0.682 on the same 800 queries.

Each method below gives every candidate a score; the list is sorted by it.

Words in Spanish change their endings as in English (“fundas”, “funda”). So Dev uses the English or Spanish stemmer by the query’s site, and removes accents. He scores four fields apart (title, brand and colour, bullet points, description) and chooses their weights on the train queries only:

title brand_colour bullets description train ndcg@10
1.0 0.50 0.25 0.0 0.697
1.0 0.25 0.25 0.0 0.695
1.0 1.00 0.25 0.0 0.694
bm25-norm NDCG@10 0.687 (0.671 to 0.703) exact first 63.4% S in top 10 29.8%

What this tells us. The best weights ignore the description. The gain on test is small: +0.005, with an interval from −0.005 to +0.015, so it is not clear. Stemming helped Larkfield, which had to find the products. Here every candidate already shares words with the query.

The course team encoded every product (title, brand and colour) and every query once. The model is the course’s multilingual bi-encoder, IBM’s granite-embedding-97m-multilingual-r2. Dense search sorts the candidates by cosine similarity. Hybrid search fuses the bm25-norm order and the dense order with reciprocal rank fusion (RRF, k 60), as in Hybrid search.

dense NDCG@10 0.683 (0.667 to 0.700) exact first 63.5% S in top 10 31.8%
hybrid-rrf NDCG@10 0.702 (0.687 to 0.718) exact first 65.6% S in top 10 30.0%

What this tells us. Dense search alone is level with BM25. Fused, the two beat either one: +0.020 over BM25 (interval +0.009 to +0.031). They make different mistakes, so the fusion corrects some of each. Dense search puts the most substitutes in the top 10 (31.8%): a similar meaning is close to what a substitute is.

The cross-encoder reads the query and one product’s text together and gives one score, as in Cross-encoder re-ranking. It is the course’s model, IBM’s granite-embedding-reranker-english-r2. Its model card says it is trained for English, without MS MARCO (a vendor claim). The course team recorded its scores for all 19,379 test pairs; the notebook replays them.

19,379 recorded pairs for 800 test queries; model ibm-granite/granite-embedding-reranker-english-r2
rerank-ce NDCG@10 0.718 (0.701 to 0.734) exact first 69.0% S in top 10 29.4%

What this tells us. The cross-encoder is the best ranker in this case study: +0.036 over BM25 (+0.022 to +0.049), and an exact product first for 69% of queries. Reading the whole query helps with details such as “3d” or a model number. The cost is time.

LambdaMART learns how to order a list from labelled examples, as in Learning to rank. Dev trains it on the 600 train queries, with the label gains 0, 0.01, 0.1 and 1. Each candidate is a row of 14 simple features. Seven are scores: BM25 per field, plain BM25, the dense score and the RRF score. Two describe the match: the share of query words in the title, and whether the brand is in the query. Two are lengths: the number of words in the title and in the query. The last three are facts about the query: a number, a negation word, Spanish. The candidates of each train query are the products that the shop’s search showed: hard negatives.

ltr NDCG@10 0.707 (0.692 to 0.723) exact first 67.8% S in top 10 29.8%
{'rrf': 28.2, 'dense': 11.8, 'bm25_plain': 10.4, 'title_coverage': 10.3, 'bm25_title': 9.9}

What this tells us. LambdaMART beats BM25 by +0.025 (+0.014 to +0.036). Against hybrid search, its gain is +0.005 (−0.002 to +0.012): not clear. The second line is each feature’s share of the gain: the RRF score leads, so most of the model’s knowledge comes from hybrid search. Against the cross-encoder, it is 0.011 lower (−0.022 to −0.000).

Common mistake

Training with easy negatives

Labelled negatives are expensive, so a common shortcut is random products as negatives. Dev tries it: each train query keeps its exact products, with random products of other train queries in the same language as negatives.

ltr easy negatives NDCG@10 0.692 (0.677 to 0.708) exact first 64.6% S in top 10 30.3%

It loses 0.015 against training on the shop’s own candidates (−0.023 to −0.006). Its gain over BM25 is no longer clear (+0.010, −0.000 to +0.022). A random product is easy to reject: “shares words with the query” is enough. But every product in the real list shares words with the query. Fix: train on the candidates that your own first stage shows, labelled. If you must add negatives without labels, take them from your search’s results, not at random.

Every method ran on the same 800 test queries, labels, gains and measures. The time of one search is the sum of its stages. The course team measured it on 80 test queries, on a laptop with one CPU thread (Apple M4). The cost uses the course’s assumed price of US$0.17 an hour for a 4-vCPU server.

Method What it is NDCG@10 (95% interval) Exact first S in top 10 p50 / p95 ms US$ per 1,000
random order the floor 0.543 (0.528–0.559) 42% 36% 0 0
bm25 baseline: one field, plain words 0.682 (0.666–0.699) 63% 30% 0.1 / 0.4 0.00001
bm25-norm per field, stemmed per language 0.687 (0.671–0.703) 63% 30% 0.1 / 0.4 0.00001
dense bi-encoder cosine 0.683 (0.667–0.7) 64% 32% 6 / 9 0.0003
hybrid-rrf RRF of bm25-norm and dense 0.702 (0.687–0.718) 66% 30% 6 / 9 0.0003
rerank-ce cross-encoder on every candidate 0.718 (0.701–0.734) 69% 29% 340 / 862 0.0161
ltr LambdaMART, 14 features 0.707 (0.692–0.723) 68% 30% 8 / 11 0.0004
NDCG@10 of each method with its 95% bootstrap interval, on the same 800 test queries. The cross-encoder is highest; LambdaMART and hybrid search follow; training with easy negatives loses most of LambdaMART's gain.
Text description of the chart

A dot with an interval bar for each method. Random order 0.543 (0.528 to 0.559). BM25 0.682 (0.666 to 0.699). bm25-norm 0.687. dense 0.683. hybrid-rrf 0.702 (0.687 to 0.718). rerank-ce 0.718 (0.701 to 0.734), highlighted. ltr 0.707 (0.692 to 0.723). ltr with easy negatives 0.692.

What this tells us. Quality is narrow: 0.682 to 0.718 for real methods. Time is not. With one thread, the cross-encoder needs about 340 ms for the median search and 862 ms for the slowest 5%. That is far over the 200 ms budget. LambdaMART needs about 8 ms, most of it to encode the query. Every method costs a fraction of a cent per 1,000 searches: time is the limit, not money, as in Latency and cost budgets.

The share of substitutes in the top 10 hardly moves: 29% to 32% for every real method. All methods are tuned for exact products. None decides on purpose where substitutes go.

Not as measured: about 48 pairs a second on one thread, and a median list has 16 candidates. On Larkfield, 50 candidates still took 732 ms with four threads (not measured here). Re-ranking only LambdaMART’s top few might fit: a next experiment, not a result.

Lucía sells in two languages. A method that wins overall might lose in one of them.

Paired difference against BM25 with its 95% interval, by language. LambdaMART's gain is clear in Spanish and not clear in English; the English cross-encoder helps in both.
Text description of the chart

Two panels. English: dense −0.007 (−0.026 to +0.012); hybrid-rrf +0.015 (+0.000 to +0.030); rerank-ce +0.033 (+0.016 to +0.051); ltr +0.014 (−0.001 to +0.029); ltr with easy negatives +0.004. Spanish: dense +0.009 (−0.013 to +0.030); hybrid-rrf +0.025 (+0.008 to +0.042); rerank-ce +0.038 (+0.019 to +0.058); ltr +0.035 (+0.019 to +0.052); ltr with easy negatives +0.016.

What this tells us. Two results surprised the team.

  • The English cross-encoder helps Spanish (+0.038). On Larkfield, the same model hurt the French queries (−0.048). We do not know why. Two reasons are likely. Spanish titles are full of brand names, model numbers and English words. And people typed these queries; a model did not translate them. Do not assume this holds for your shop: measure it.
  • LambdaMART helps Spanish more than English (+0.035 against +0.014, not clear). Spanish BM25 starts lower (0.669 against 0.696), so there is more room.

ESCI tags 102 test queries as negations: queries that exclude something, such as “without hitch” or “sin relleno”. Here is NDCG@10 by kind of query:

Kind (test queries) random bm25 dense hybrid-rrf rerank-ce ltr
negations (102) 0.498 0.589 0.566 0.589 0.616 0.595
parse_pattern (111) 0.475 0.671 0.670 0.695 0.707 0.713
behavioral (37) 0.371 0.623 0.706 0.702 0.717 0.716
other (550) 0.577 0.706 0.706 0.724 0.739 0.726

What this tells us. Negations are the worst slice for every method, about 0.1 below the rest. Dense search loses to BM25 there (−0.023): a vector for “panel without controller” is close to one for “panel with controller”. Hybrid search gains nothing. LambdaMART’s negation feature has only 0.8% of the gain: 90 negation queries in the train set are too few to learn “without”.

Here is one negation query from the test set, in three methods:

"100w solar panel without controller" (bm25)
1. [S] ECO-WORTHY 100W Solar Water Pump Kit - 100W Solar Panel + 12V Water Pu
2. [E] ACOPOWER 100W Portable Solar Panel Kit (Expansion 100W)
3. [E] Renogy 100W Eclipse Lightweight Suitcase Without Controller, Panel Onl
"100w solar panel without controller" (rerank-ce)
1. [E] SOLPERK 100W Solar Panel 12V, Monocrystalline Solar Panel Kit with Hig
2. [E] PowerECO 100W 12V Solar Poly Panel RV Kit with 20A PWM Charge Controll
3. [S] ECO-WORTHY 100W Solar Water Pump Kit - 100W Solar Panel + 12V Water Pu
"100w solar panel without controller" (ltr)
1. [E] Renogy 100W Eclipse Lightweight Suitcase Without Controller, Panel Onl
2. [E] SOLPERK 100W Solar Panel 12V, Monocrystalline Solar Panel Kit with Hig
3. [C] HUINE 10A PWM Solar Charge Controller Waterproof IP68 12V 24V Solar Pa

What this tells us. BM25 likes the pump kit, which repeats “100W” and “solar panel”. LambdaMART finds the panel that says “Without Controller”, then shows a controller third. The cross-encoder’s second result is a kit with a controller, labelled exact: annotators make mistakes too.

NDCG says that a substitute is worth 0.1 of an exact product. Lucía wants that in shop terms. Dev uses the course’s click model as an assumption. A person looks at position k with a chance of 1/k. After looking, they click and then add to cart with these chances:

Label Click, after looking Add to cart, after a click Add-to-carts per look Source
Exact 0.65 0.30 0.195 the course’s click model (exact)
Substitute 0.25 0.08 0.020 the course’s click model (partial)
Complement 0.10 0.05 0.005 our assumption
Irrelevant 0.05 0.01 0.0005 the course’s click model
worth at position 1: {'E': 0.195, 'S': 0.02, 'C': 0.005, 'I': 0.0005}
a substitute earns 10% of an exact product, 40 times an irrelevant one
expected add-to-carts per 1,000 searches: {'random order': 260.8, 'bm25': 334.4, 'bm25-norm': 338.2, 'dense': 334.6, 'hybrid-rrf': 344.8, 'rerank-ce': 353.3, 'ltr': 349.0}

What this tells us. A substitute’s 10% is close to ESCI’s gain of 0.1, so the measure and the business view agree. A substitute instead of an exact product at the top costs about 0.175 add-to-carts. At the course’s assumed basket of 85 dollars, that is about 15 dollars of sales at risk for each such search. But a substitute earns 40 times as much as an irrelevant product. So show the exact product first when it exists; when it does not, a substitute is far better than nothing.

LambdaMART adds about 15 add-to-carts per 1,000 searches over BM25 (349.0 against 334.4). These are assumptions: only an online test, as in Test changes online, would measure the real number.

A real shop often does not sell the exact item. So Dev hides the exact candidates of every test query and ranks the rest with each method, without retraining. 740 test queries keep at least one substitute or complement.

ndcg@10 S first I first carts per 1,000
random order 0.794 0.672 0.211 37.933
bm25 0.806 0.673 0.162 38.491
dense 0.816 0.693 0.165 39.159
hybrid-rrf 0.816 0.685 0.158 39.061
rerank-ce 0.815 0.693 0.164 39.017
ltr 0.810 0.685 0.165 38.892
With the exact products hidden (740 test queries), how often each method puts an irrelevant product first (CE: the cross-encoder). Every method beats a random order by about 5 points, and they are all close to each other.
Text description of the chart

A column chart with six columns. Random order 21%. BM25 16%. Dense 16%. Hybrid search (hybrid-rrf) 16%. Cross-encoder (rerank-ce) 16%. LambdaMART (ltr) 16%.

What this tells us. Without an exact product, every method shows a substitute first for about 2 queries in 3. It shows an irrelevant product first for about 1 in 6. A random order does that for 1 in 5. The page earns about 39 add-to-carts per 1,000 searches, against about 350 with the exact products. The methods learned what makes an exact match, not a good substitute. So the shop cannot leave this case to the ranker alone.

Does the comparison depend on ESCI’s gains? Dev measures the same runs with the course’s gains, where substitutes and complements count as partial (exact 3, substitute 1, complement 1, irrelevant 0):

ESCI gains course gains
random order 0.543 0.648
bm25 0.682 0.758
hybrid-rrf 0.702 0.773
rerank-ce 0.718 0.783
ltr 0.707 0.777

What this tells us. All numbers rise, but the order of the methods does not change.

Dev’s recommendation for Lucía:

The recommended design. The query, in English or Spanish, goes to the shop's first stage, which finds about 16 to 40 candidates. LambdaMART with 14 simple features orders them in about 8 ms. If an exact product is sold, the results page shows the best match first, and a person checks negation queries. If not, the page shows a row 'Similar items' with substitutes, labelled, and a separate row 'Goes with' with complements.
The recommended design: LambdaMART re-ranks the shop's candidates; when the exact product is not sold, the page says so and shows substitutes as similar items.
Text description of the diagram

Query, English or Spanish → shop’s first stage, about 16 to 40 candidates → LambdaMART, 14 simple features, 8 ms p50 on 1 thread → “Is an exact product sold?”. Yes → results page, best match first → a person checks negation queries (“without”, “sin”). No → row “Similar items”: substitutes, labelled → separate row “Goes with”: complements.

  1. Ship LambdaMART on the 14 simple features as the re-ranker. It is the best method inside 200 ms: NDCG@10 0.707 against 0.682 for BM25. It takes about 8 ms (p50) and 11 ms (p95). Its gain over BM25 is clear in Spanish and not quite clear in English. Hybrid search is a close second (0.702) and needs no training: if the team cannot keep labels coming, ship hybrid search.
  2. Train only on the shop’s own candidates. Random negatives cost most of the gain.
  3. Keep the cross-encoder as the quality target. It is 0.011 better than LambdaMART, but takes 340 ms. Test it on LambdaMART’s top few candidates, or as a cached score for the most common queries.
  4. When no exact product is sold, say so. Show the substitutes in a row labelled “Similar items”, and complements in a separate row, “Goes with”. Do not mix them with exact results. The ranker cannot find this case: use the catalogue (a missing brand, model or size) to decide that the exact product is not sold.
  5. What a person still checks: a weekly sample of negation queries in both languages, before and after every change. Check the labels of that sample too.

Latency and cost per 1,000 queries for the recommended design: about 8 ms p50 and 11 ms p95 per search, including the query encoding. That is with one thread on the course team’s laptop. The cost is US$0.0004 per 1,000 searches at the assumed US$0.17 an hour. Measure both on your own server.

A new query has no candidate list from the shop’s search. So the notebook adds a stand-in first stage: the top 30 by RRF of BM25 and dense search over every product of the language. Then LambdaMART, trained on the 600 train queries, orders them. The same bi-encoder computes the query vector live.

for text, loc in [("wireless earbuds for running", "us"), ("phone case without magnet", "us"),
("auriculares inalambricos para correr", "es")]:
found, ms = search(text, loc)

The output (the first search includes loading the model):

"wireless earbuds for running" (us, 2985 ms)
1. Bose Sport Earbuds - Wireless Earphones - Bluetooth In Ear Headphones for Work
2. Bose Sport Open Earbuds — True Wireless Open Ear Headphones - Sweat Resistant
3. SAMSUNG Galaxy Buds Live True Wireless Earbuds US Version Active Noise Cancell
4. Wireless Earbuds, Upgraded Boltune Bluetooth V5.2 in-Ear Stereo Wireless Headp
5. FutureCharger for Airpods Pro Charger, Wireless Charger for Airpod 3/Airpods/A
"phone case without magnet" (us, 26 ms)
1. OCASE iPhone X Wallet Case, iPhone 10 Case [ Wireless Charging ] [ Card Slot ]
2. CravenSpeed Gemini Phone Mount for Mini [3rd Gen Models Only] (Magnet)
3. Dreem Fibonacci 2-in-1 Wallet-Case for Apple iPhone X & Xs - Luxury Vegan Leat
4. TJS Compatible with Samsung Galaxy A50 2019 Case, with Full Coverage Tempered
5. Vofolen 2-in-1 Case for iPhone 8 Plus Case 7 Plus Wallet Card Holder ID Slot D
"auriculares inalambricos para correr" (es, 38 ms)
1. Auriculares Inalambricos Deportivos, Auriculares Bluetooth 5.1 con Micrófono,
2. Auriculares Inalámbricos Bluetooth 5.0, Auriculares Deportivos En La Oreja, 60
3. Auriculares Bluetooth 5.0 Auriculares Inalambricos Bluetooth Deportivos,Auricu
4. LYCHL Auriculares Inalambricos Deportivos, Auriculares Bluetooth 5.0 Sport IP7
5. Auriculares Bluetooth, Auriculares Inalámbricos, Auriculares Internos, Auricul

What this tells us. Nobody judged these results, so read them yourself. The earbuds look right in both languages; the fifth English result is a charger, a complement. The negation query fails again: a car mount labelled “(Magnet)” comes second. The sample has only 32,759 products, which limits what can be found. Your times will differ.

Easier or harder than Larkfield? Easier: short lists, with 4 exact products in 10. Harder: real, messy queries in two languages, finer labels, and only hard negatives. Larkfield’s log is synthetic and its French is machine-translated; here the queries are real, but there are no clicks. So compare directions, not numbers. Hybrid search helped both. LambdaMART gained more over hybrid search on Larkfield (+0.047) than here (+0.005).

  • A sample. 400 test queries per language give intervals of about ±0.02. Smaller differences need the full test split (11,373 English and Spanish queries).
  • Re-ranking only. The candidates are fixed, so we cannot say how well a method finds products that the shop’s search missed.
  • Labels are noisy, as the solar panel example shows. Every method is measured against them.
  • The “no exact” case is constructed by hiding exact products.
  • Value is an assumption from the click model. It gives directions, not a forecast.
  • Times are from one laptop, one thread, models loaded.
  • The data has no customer data. Its brand names are real; a ranking is not a judgment of a brand.
  • Label substitutes honestly. A substitute shown as the product asked for misleads the shopper. Say “Similar items”, and show the difference that matters (no 3D, other size).
  • Do not let ranking replace availability. Pushing paid or own-brand substitutes above an exact product the shop sells is a business rule, not relevance. Measure it separately, as with Hana’s sale rule in Zero results and business rules.
  • Report both languages, not only the total.
  • Model licences. The IBM models are Apache-2.0; their cards (bi-encoder, cross-encoder) claim permissive training data, with no MS MARCO. ESCI’s own baseline used an MS MARCO cross-encoder (MS MARCO’s terms: non-commercial research only); we do not.

This study shows that you can rank with fine labels, compare methods fairly, and turn a measure into a business decision. Make it yours with one change. For example, add the Japanese queries, or a classifier that predicts “substitute” (ESCI’s third task) for the “Similar items” row.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • A results.csv with the comparison table, and the slices by language and by kind.
  • A short DECISION.md: the design, its settings, its latency and cost, and what a person checks.
  • The data source and licence: ESCI, Apache-2.0, with the NOTICE line and a list of your changes. Apache-2.0 allows you to include your sample, with the licence file.

A README template:

# Re-ranking shopping queries with hard negatives (ESCI, English and Spanish)
## Problem
A marketplace re-orders the 16-40 products its search finds for each query.
Most wrong answers look right. Budget: 200 ms per search on a CPU.
## Data and licence
Amazon Shopping Queries Dataset (ESCI), Apache-2.0, github.com/amazon-science/esci-data.
Sample: 400 test + 300 train queries per language (us, es), seed 38.
## Approach
NDCG@10 with ESCI gains (E 1, S 0.1, C 0.01, I 0). BM25 baseline; BM25 per field;
dense; hybrid RRF; cross-encoder (recorded scores); LambdaMART on 14 features
trained on the shop's candidates. Paired bootstrap intervals; slices by language
and by query kind.
## Results (800 test queries)
LambdaMART 0.707 vs BM25 0.682 (+0.025), 8 ms p50. Cross-encoder 0.718 but 340 ms.
Easy negatives lose 0.015. Negation queries are the worst slice for every method.
Without an exact product, every method is close to a random order.
## Limits
Sample; re-ranking only; noisy labels; value from an assumed click model.

How to describe it in an interview, in two sentences: “I compared five ranking methods on Amazon’s public shopping-queries data in English and Spanish. I chose LambdaMART: +0.025 NDCG@10 over BM25 at 8 ms, where the cross-encoder took 340 ms. I also showed that random negatives lose most of the gain, and that no ranker picks good substitutes when the exact product is missing.”

Go further:

  • Re-rank LambdaMART’s top 5 with the cross-encoder. Predict first: does it fit 200 ms, and how much of the gap does it close?
  • Add a feature for negations: a penalty when a title contains the word after “without” or “sin”. Measure the negation slice.
  • Train with the course’s gains instead of ESCI’s, and compare the share of substitutes in the top 3.

Check your understanding

A random order of the candidates scores 0.543 NDCG@10 here, but Larkfield's baseline BM25 scored only 0.429. Is ranking easier on this data? Explain.

Show the answer

The two numbers measure different tasks. Here, the shop’s search already chose about 16 candidates per query, and about 4 in 10 of them are exact. So even a random order puts good products in the top 10. Larkfield searched all 40,066 products, so a method had to find the few good ones first. The gains also differ (ESCI’s exact 1, substitute 0.1 against the course’s 3 and 1). Compare methods on the same queries, labels and gains, never across datasets.

Check your understanding

LambdaMART trained with random products as negatives scores 0.015 lower than the same model trained on the shop's own candidates. Why?

Show the answer

Random products are easy negatives: they share no words with the query, so the model learns that sharing words means relevant. In the real list, every candidate shares words with the query, because the search engine chose them. That rule cannot separate an exact product from a substitute or an irrelevant look-alike. The shop’s own candidates are hard negatives, and they teach the model the details that matter.

Check your understanding

Lucía asks: 'When we do not sell the exact product, can we let LambdaMART choose the best substitute?' What do you answer, with the numbers?

Show the answer

Not alone. With the exact products hidden, LambdaMART shows an irrelevant product first for about 16% of queries. A random order does it for 21%, and every method is close to 16%. The models learned what an exact match looks like, not what makes a good substitute. So the shop should use its catalogue to decide that the exact product is missing. Then it labels the row “Similar items” and keeps complements in a separate row. A model trained to recognise substitutes could improve the order.

Recap

  • ESCI’s labels are exact, substitute, complement and irrelevant. With its gains (1, 0.1, 0.01, 0), a substitute in first place costs 0.311 of NDCG.
  • A search engine showed every candidate, so every wrong answer is a hard negative. A random order scores 0.543; BM25 0.682.
  • Hybrid search 0.702, LambdaMART 0.707 (8 ms), the cross-encoder 0.718 (340 ms, over budget). The English cross-encoder helped Spanish here.
  • Easy negatives lose 0.015. Negations are the worst slice for every method.
  • A substitute earns 10% of an exact product and 40 times an irrelevant one. Without an exact product, no ranker beats a random order by much: label substitutes as “Similar items”.

Next: long questions over a corpus of documents, where the queries are arguments, not product names.

Next: Search a help centre or a scientific corpus