Product search for a furniture shop
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In eight modules, we built Larkfield's search on part of a real furniture catalogue. In this case study, we use the full WANDS dataset, compare five designs on NDCG and latency, and recommend one for a 200 ms budget.
Case study 1 of 3 · Search and Ranking: Find the Best MatchProduct search for a furniture shop
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Take the whole course to a real catalogue and make one decision from it. You compare five families of search designs on all 42,994 products and 480 real queries of WANDS. You measure quality, time and cost under one protocol, find where each design fails, and recommend one for a 200 ms budget.
The notebook runs every step of this page, plus four tasks with checks. Its setup cells install the packages and download the three WANDS files (about 96 MB) and the course team’s files (about 34 MB). It needs no account and no key. It ran in under one minute on our laptop once the files were downloaded.
Before you startWhat to know and to have ready
What you need to know
These lessons matter most:
- NDCG@10, recall@100 and the paired comparison with its interval, from Ranking measures and Build a test set that represents your users.
- BM25, from BM25.
- Normalising, spelling and synonyms, from Normalise text, Fix spelling and Synonyms and query rewriting.
- Dense search and reciprocal rank fusion, from Search by meaning and Hybrid search.
- Two stages, the cross-encoder and LambdaMART, from Two-stage ranking, Cross-encoder re-ranking and Learning to rank.
- Stage times and cost per 1,000 queries, from Latency and cost budgets.
What you need on your computer (only for the local route; the notebook in Colab needs none of it)
- Python 3.12 in a new virtual environment, and Jupyter or VS Code. The
larkfield-searchproject is not needed. - About 400 MB of free disk space: the data, and the bi-encoder (about 200 MB) for the last section.
- On macOS, Homebrew’s OpenMP library for LightGBM:
brew install libomp.
The notebook works in a new folder, wands-search; delete it to start again. Tested on macOS (Apple M4) with Python 3.12. Windows, Linux, Colab and Kaggle are not tested.
The situation
Section titled “The situation”Nadia leads search at an online furniture shop. Shoppers type queries such as “turquoise pillows” and “filaret outdoor sofa”. Today the search is plain BM25 over the product name and description. Her team has ideas: rules for words, a vector model, a model that re-reads the results, a model that learns from judgments. Each idea makes the search slower. The engineers set a hard limit: 200 ms for one search, on normal CPU servers, with no GPU.
Ben, who runs the search service, asks: “Which of these ideas is worth its milliseconds?” This case study answers with measurements.
The shop is imagined. The data is real: WANDS, which Wayfair published to test product search.
What you will build
Section titled “What you will build”You will build one comparison table and one recommendation. The table has ten rows, all on the same 480 queries and judgments. The measures are NDCG@10 with its 95% interval, recall@100 and MRR. Then the time per search (p50 and p95) and the cost per 1,000 searches. Then you find where each design fails, and choose one for the budget.
Every method comes from the course. What is new is the full catalogue and every real query. The course’s shop, Larkfield, used a filtered copy of this data, so you can also check how far the course’s results carry over.
The data
Section titled “The data”WANDS has three tab-separated files. product.csv has one row per product: name, product class, category path, description, features and ratings. query.csv has one row per query, typed by a real shopper, with the product class it asks for. label.csv has one row per judgment: a trained annotator’s label for one product and one query, Exact, Partial or Irrelevant.
The notebook loads the three files:
products = pd.read_csv("wands/product.csv", sep="\t")queries = pd.read_csv("wands/query.csv", sep="\t")labels = pd.read_csv("wands/label.csv", sep="\t")print(f"{len(products):,} products, {len(queries)} queries, {len(labels):,} judgments")print(labels["label"].value_counts().to_dict())The output (the column list is left out here):
42,994 products, 480 queries, 233,448 judgments{'Partial': 146633, 'Irrelevant': 61201, 'Exact': 25614}What this tells us. Most judgments are Partial: the product is related but not what the shopper asked for, such as a desk lamp for “desk”. Only 11% are Exact. NDCG gives an Exact product a gain of 3 and a Partial one 1, so the search must put the few Exact products first.
First checks
Section titled “First checks”First, check the judgments and the gaps:
repeated (query, product) pairs: 1575pairs with two different labels: 14products without a class: 2852products without a description: 6008queries with no exact product: 101judged products per query: median 216231,873 judgments kept; 1 queries have no exact or partial product (they have no NDCG)What this tells us. Four facts change later steps:
- Repeats. 1,575 pairs appear more than once, and 14 have two different labels. We keep the first label, as the course did. 14 pairs cannot move a result.
- Gaps in the text. 6,008 products have no description and 2,852 have no class. We fill in nothing.
- Queries that some measures skip. 101 queries have no Exact product, so recall and MRR skip them. One query has no Exact or Partial product, so NDCG skips it: NDCG averages over 479 queries.
- Judged pools. Each query has a pool of judged products (median 216 of 42,994). A product outside the pool counts as irrelevant. So a method that finds good products outside it, such as dense search, looks worse than it is.
How it differs from Larkfield’s catalogue
Section titled “How it differs from Larkfield’s catalogue”Larkfield’s catalogue is WANDS with three changes. The notebook measures them:
products in departments that Larkfield left out: 928{'School Furniture and Supplies': 455, 'Commercial Business Furniture': 177, 'Contractor': 160, 'Foodservice': 36, 'Shop Product Type': 33, 'Browse By Brand': 24, 'Reception Area': 17, 'Clips/Clamps': 14, 'Protection Plans': 12}products with the shop's own brand in the name: 172queries with an exact product that Larkfield left out: 57| Larkfield (the course) | WANDS, full (this case study) | |
|---|---|---|
| Products | 40,066 public (42,066 kept, 2,000 held back) | 42,994: also school, office, contractor and food-service products |
| Names | the shop’s brand renamed “larkfield”; new IDs | real names, with the real house brand “wayfair sleep” and other real brands |
| Queries | WANDS queries in splits, plus machine-translated French, planted misspellings, planted codes and planted broad queries | the 480 real queries only, no splits |
| Shop data | synthetic prices, stock, sale flags, query log and clicks | none: products, queries and judgments only |
So this data is harder in one way: more products compete, and no planted problems are easy to fix. It is easier in another: no French and no planted misspellings. The 57 queries with an Exact product in a dropped department are not harder. BM25 scores 0.695 on them, against 0.636 on the others.
Frame the problem
Section titled “Frame the problem”These are the decisions before any model:
- The unit: one query and its ranked list.
- The main measure: NDCG@10 with gains 3 (Exact) and 1 (Partial), the course’s measure from Ranking measures. Shoppers see the first ten.
- The second measures: recall@100 and MRR. Recall@100 counts the Exact products in the top 100, where a second stage can find them. MRR shows how high the first Exact product is.
- The budget: the p95 time of one search must be at most 200 ms. p95 is the time that 95% of searches beat. Users feel the slow searches, so the budget applies to p95.
- The protocol: every design runs on the same 480 queries. LambdaMART learns from judgments, so it uses cross-validation by query, explained below. Nothing is tuned on these queries. Every setting comes from the course: BM25’s k1 and b, the field weights, RRF’s k and LambdaMART’s settings. The course chose them on Larkfield’s training and validation queries.
One caveat: Larkfield’s queries are WANDS queries too. But the settings are few and general, and the comparison with Larkfield below shows no sign that this helped.
Explore what the decision needs
Section titled “Explore what the decision needs”Two questions matter. First, what kinds of queries does the shop get? The course’s labels from a language model sort them into three kinds. A broad query asks for a wide group (“dinosaur”), a specific one gives details (“turquoise pillows”), and a known-item query names one product (“hulmeville writing desk with hutch”). These labels were not checked by people.
{'specific': 293, 'known-item': 129, 'broad': 58}{'Furniture': 169, 'Décor & Pillows': 63, 'Home Improvement': 52, 'Outdoor': 35, '(none)': 25, 'Lighting': 25, 'Storage & Organization': 23, 'Rugs': 20}What this tells us. Most queries are specific, and a quarter name one product, where a failure is easy to notice. The second line gives the department of each query’s Exact products. Slices with fewer than 10 queries are left out: they are too noisy.
Second: how long is each step? The course team timed every stage on 40 queries (Apple M4, one thread unless the name says 4 threads). The notebook reads those times from wands_latency_ms.csv, and it times the stages that need no model on your computer.
Text description of the diagram
A flowchart from left to right. “A query” splits into “Query understanding · 0.2 ms” and “Encode the query · 4.8 ms”. They lead to “Lexical top 100” and “Dense top 100 · 1.3 ms”, which join in “Fusion: RRF · under 0.1 ms”. From the fusion, two branches: “Cross-encoder, top 10: 190 ms (1 thread), 142 ms (4 threads)”, drawn in red, and “LambdaMART · 3.7 ms with the hybrid stage”. Both lead to “The page, top 10”. A box “Baseline: BM25 · 0.1 ms” also leads to the page, with a dotted arrow.
The baseline: BM25
Section titled “The baseline: BM25”The baseline is today’s search: BM25 (k1 1.2, b 0.75) over the name and description as one field, with plain words.
def bm25(query_id, text, k=100): """The baseline: BM25 on the title and description as one field, plain words.""" return top(scores(IDX["all"], tokens, text), k)
evaluate("bm25", bm25);The output:
bm25 NDCG@10 0.643 (0.616 to 0.670) recall@100 0.601 MRR 0.669What this tells us. 0.643 is the number to beat. Larkfield’s test set gave BM25 only 0.429, but it had French and planted queries. On real English queries, BM25 is strong.
Here is a failure that the other designs fix. For “accent chairs living room”, BM25’s first three results are chairs whose descriptions repeat “living room”:
"accent chairs living room"1. warwickshire linen queen anne back parsons chair in beige [not judged]2. satchell linen upholstered dining chair [not judged]3. lawncrest 29.5 '' wide polyester armchair [not judged]None is judged, so each counts as irrelevant. BM25 cannot tell that “accent chairs” is the product and “living room” only the place.
The alternatives
Section titled “The alternatives”Query understanding
Section titled “Query understanding”Query understanding prepares the words before BM25, in three steps:
- Normalise: lower case, no accents, units written one way, words cut to their stem. Then BM25 per field, with the course’s field weights.
- Fix spelling: an unknown word becomes the most frequent word one or two edits away.
- Add synonyms: the course’s rules, such as “couch, sofa”. Added words count half. We left out the course’s planted bad rule “light, lamp”.
Predict. WANDS queries are typed by real people. Will spelling correction help them as much as it helped Larkfield’s planted misspellings?
bm25-norm NDCG@10 0.706 (0.681 to 0.731) recall@100 0.617 MRR 0.665bm25-norm+spell NDCG@10 0.704 (0.679 to 0.728) recall@100 0.617 MRR 0.666bm25-norm+spell+syn NDCG@10 0.708 (0.683 to 0.732) recall@100 0.617 MRR 0.670spelling changed 38 queries: 5 better, 12 worse worse: 'trinaic towel rod' -> 'trinsic towel rod' NDCG@10 0.936 -> 0.321 worse: 'filaret outdoor sofa' -> 'flared outdoor sofa' NDCG@10 0.931 -> 0.438 worse: 'togo chair' -> 'toro chair' NDCG@10 1.000 -> 0.641 worse: 'adjustable height artist stool' -> 'adjustable height artis stool' NDCG@10 1.000 -> 0.694 better: 'lowes tile' -> 'lower tile' NDCG@10 0.136 -> 0.546 better: 'foutains with brick look' -> 'fountains with brick look' NDCG@10 0.122 -> 0.510What this tells us. Normalising gives almost all the gain: +0.063 (95% interval +0.046 to +0.080) over BM25. Spelling correction is a small loss (−0.003, −0.007 to +0.002). Its dictionary comes from names, classes and attributes, so it does not know “artist”, and it turns real names such as “filaret” into other words. Real queries have few typos and many names. Synonyms add +0.004 (0.000 to +0.010): not clear.
So guard spelling instead of dropping it. The later designs keep the course’s full chain, so the comparison with Larkfield stays fair.
Dense and hybrid search
Section titled “Dense and hybrid search”Dense search compares the meaning of the query with the meaning of each product. IBM’s granite bi-encoder turns both into vectors of 384 numbers, and the closest products win. The course team encoded the products once (157 seconds on 4 threads); the notebook loads the vectors. Hybrid search fuses the lexical top 100 and the dense top 100 with reciprocal rank fusion (RRF). Each product gets the sum of 1 ÷ (60 + rank) over the two lists.
dense NDCG@10 0.639 (0.611 to 0.666) recall@100 0.542 MRR 0.607hybrid-rrf NDCG@10 0.721 (0.697 to 0.745) recall@100 0.624 MRR 0.696What this tells us. Dense search alone is no better than BM25 (−0.005, −0.027 to +0.018), partly because it finds unjudged products. Fused with query understanding, it adds +0.013 (+0.001 to +0.025): small but clear. Hybrid search is the best first stage, with three Exact chairs first for “accent chairs living room”. RRF with k = 10 scores 0.713, lower than with k = 60.
Re-ranking with a cross-encoder
Section titled “Re-ranking with a cross-encoder”A cross-encoder reads the query and one product together and gives one score. It reads better than any first stage, but it runs once per product, so it only re-orders the first n hybrid candidates. The course team scored the hybrid top 50 of every query with IBM’s granite reranker: 24,000 pairs in 347 seconds on 4 threads. The notebook replays those scores.
rerank-ce top 10 NDCG@10 0.730 (0.707 to 0.754) recall@100 0.624 MRR 0.758rerank-ce top 20 NDCG@10 0.746 (0.722 to 0.770) recall@100 0.624 MRR 0.772rerank-ce top 50 NDCG@10 0.748 (0.725 to 0.772) recall@100 0.624 MRR 0.771What this tells us. Re-ranking more candidates helps, up to 20. Top 20 adds +0.025 (+0.016 to +0.034) over hybrid search; top 50 adds almost nothing. Hybrid search’s top 10 holds 25% of the Exact products, and its top 20 holds 34%. Re-ranking can only reorder what is there. On Larkfield the cross-encoder was not clearly better, because it is English only and the course had French queries.
Learning to rank with LambdaMART
Section titled “Learning to rank with LambdaMART”LambdaMART learns how to order the hybrid top 100 from the judgments. Each candidate is one row of 15 features. Text: BM25 per field, the plain BM25 score, the dense similarity, the RRF score, and its ranks in the hybrid, lexical and dense lists. Product: the rating and the numbers of ratings and reviews. Query: the number of words, and whether it has a number. It is the model of Learning to rank without the shop features (price, stock, clicks).
LambdaMART needs training queries, but here every query is also a test query. The solution is cross-validation by query.
Cross-validation by query
Section titled “Cross-validation by query”Think of a teacher with one set of exam questions. She splits them into five groups, teaches with four and tests with the fifth, five times. Every question is tested once on a student who never practised it. That is cross-validation: train on part of the data, test on the rest, and rotate.
In search, the split must keep each query whole. A query’s candidates share the query’s words, and the model learns how good they are together. Suppose some candidates of “turquoise pillows” were in training and others in testing. Then the model has seen that query, and the score is too good. So the five folds are five groups of 96 whole queries. The machine learning course calls this a group split, in Train, validation and test sets.
Text description of the diagram
Five boxes “Fold 1” to “Fold 5, 96 queries” come from “480 queries, shuffled once”. Folds 2 to 5 lead to “Train LambdaMART on 4 folds”. That leads to “Rank fold 1: queries it never saw”, with a dotted line from Fold 1. The last box says “Repeat for folds 2 to 5: every query ranked once by a model that never saw it”.
The code puts each query in one fold, trains five models, and ranks each fold with the model that did not see it:
order = np.random.default_rng(0).permutation(queries["query_id"].to_numpy())FOLD = {int(q): i % 5 for i, q in enumerate(order)}table["fold"] = table["query_id"].map(FOLD)
def cross_validated(features, name): """Train on 4 folds, rank the 5th, five times: every query is ranked by a model that never saw it.""" out = table[["query_id", "product_id"]].copy() for k in range(5): model = fit(table[table["fold"] != k], features) test = table["fold"] == k out.loc[test, "score"] = model.predict(table.loc[test, features]) ...The output (48,000 candidate rows: 480 queries × 100 candidates):
48,000 candidate rows for 480 queries in 2 s; labels {0: 16993, 1: 21222, 2: 9785}ltr NDCG@10 0.739 (0.716 to 0.763) recall@100 0.624 MRR 0.715What this tells us. LambdaMART reaches 0.739: +0.018 (+0.009 to +0.028) over hybrid search, a clear gain. It is close to the cross-encoder on 20 candidates (−0.007, −0.017 to +0.004: not clear). The largest feature shares are the hybrid rank (30%), the RRF score (19%) and BM25 on the title (9%). The model learns when to trust hybrid search, and when a strong title match should move a product up. Without the two dense features, the score does not fall (0.742): the dense signal still arrives through the candidates and the hybrid rank.
Common mistake
Measuring the model on the queries it learned from
Train LambdaMART on all 480 queries, then measure it on the same 480 queries:
ltr on its own training queries NDCG@10 0.882 (0.864 to 0.900) recall@100 0.624 MRR 0.9120.882 is not a result: the model has seen the answers. Only the cross-validated 0.739 counts. The same mistake hides in smaller forms, such as tuning the number of trees or choosing features by looking at the test score.
One more design combines the two second stages. The cross-encoder’s score of the first 10 hybrid candidates becomes a 16th feature of LambdaMART (empty for the other 90). It reaches 0.751 (0.727 to 0.775), +0.012 (+0.006 to +0.017) over LambdaMART alone: a small, clear gain.
A fair comparison
Section titled “A fair comparison”Now all the designs go into one table. The time of one search is the sum of its stages, on the 40 timed queries. The cost assumes a cloud server with 4 vCPUs at US$0.17 an hour (the course’s assumed price: check your provider). It runs four one-thread searches at once, or one four-thread search.
method ndcg@10 low high recall@100 mrr p50 ms p95 ms US$ per 1,000 queries bm25 0.643 0.616 0.670 0.601 0.669 0.1 0.2 0.00000 bm25-norm+spell+syn 0.708 0.683 0.732 0.617 0.670 0.2 0.5 0.00000 dense 0.639 0.611 0.666 0.542 0.607 6.1 6.5 0.00007 hybrid-rrf 0.721 0.697 0.745 0.624 0.696 6.4 6.9 0.00008 rerank-ce top 10 0.730 0.707 0.754 0.624 0.758 196.9 226.7 0.00232rerank-ce top 10, 4 threads 0.730 0.707 0.754 0.624 0.758 148.8 174.4 0.00703 rerank-ce top 20 0.746 0.722 0.770 0.624 0.772 397.9 507.5 0.00470 rerank-ce top 50 0.748 0.725 0.772 0.624 0.771 970.9 1205.1 0.01146 ltr 0.739 0.716 0.763 0.624 0.715 8.5 9.6 0.00010 ltr + ce top 10, 4 threads 0.751 0.727 0.775 0.624 0.735 150.7 177.6 0.00712Text description of the chart
Two panels with the same nine designs. Quality, NDCG@10 with interval: BM25 0.643 (0.616 to 0.67); query understanding 0.708; dense only 0.639; hybrid 0.721; cross-encoder top 10, 4 threads 0.73; top 20 0.746; top 50 0.748; LambdaMART 0.739 (0.716 to 0.763); LambdaMART + cross-encoder top 10 0.751. Time, p95 milliseconds, with a line at the 200 ms budget: BM25 0.2; query understanding 0.5; dense 6.5; hybrid 6.9; cross-encoder top 10, 4 threads 174.4; top 20 507.5; top 50 1205.1; LambdaMART 9.6; LambdaMART + cross-encoder top 10, 4 threads 177.6.
What this tells us.
- Every family beats the baseline, except dense search alone. Against BM25: query understanding +0.065, hybrid +0.078 and LambdaMART +0.096. The cross-encoder on 20 candidates gives +0.102, and both together +0.107. All intervals are far above zero.
- Time splits the designs in two. Without the cross-encoder, p95 is under 10 ms. With it, at least 174 ms: only 10 candidates on 4 threads fit, with 22 to 26 ms to spare.
- Cost is not the problem; time is. The most expensive design costs about one cent per 1,000 searches.
Where each design fails
Section titled “Where each design fails”An average can hide a group of queries that gets worse. Here is each design minus BM25, by kind of query and by department:
Text description of the chart
Differences against BM25 with a line at zero. Broad queries (57): hybrid +0.075, cross-encoder top 20 +0.088, LambdaMART +0.080. Specific (293): +0.093, +0.122, +0.110. Known-item (129): understanding +0.008 (−0.023 to +0.038), hybrid +0.045, LambdaMART +0.071. Furniture: hybrid +0.091, LambdaMART +0.114. Décor and pillows: hybrid +0.086, LambdaMART +0.104. Rugs (20): understanding −0.022, hybrid −0.028 (−0.077 to +0.021), cross-encoder top 20 −0.004, LambdaMART +0.050 (+0.010 to +0.098).
What this tells us. The worst slice of each design:
- Query understanding: known-item queries (+0.008, not clear). These queries are names, such as “togo chair”. Stemming and spelling do little for a name, or harm it.
- Dense search: broad queries (−0.049) and the storage department (−0.056). For “dinosaur”, many products are close in meaning, and many are probably not judged.
- Hybrid search: rugs (−0.028, −0.077 to +0.021). Only 20 queries, so not clear. Rug queries are often about sizes, colours and patterns, which dense search reads loosely.
- The cross-encoder on 20: rugs again (−0.004). Of the three kinds, its smallest gain is on known-item queries (+0.064).
- LambdaMART: never below BM25 in any slice with 10 queries or more. Its worst department is storage (+0.035). On rugs it gains +0.050 (+0.010 to +0.098).
Averages also hide single queries. For “moen 5995 arbor one” (a kitchen faucet), LambdaMART falls from 0.538 to 0.130. Dense search brings garden arbors, and the model ranks them first:
"moen 5995 arbor one"1. bakersfield vinyl arbor [irrelevant]2. rosewood wood arbor [irrelevant]3. athens 39 '' w x 23 '' d vinyl arbor [irrelevant]The model number “5995” and the brand “moen” mean nothing to the bi-encoder. Larkfield solved model numbers with a code rule in Exact matches and rules. A real shop needs such a rule for brands and model numbers.
How the numbers compare with Larkfield’s
Section titled “How the numbers compare with Larkfield’s”The course’s test set has 120 real English WANDS queries. Here are the same 120 queries on both catalogues (NDCG@10, 95% interval):
| Design | Larkfield’s catalogue (the course) | Full WANDS (this case study) |
|---|---|---|
| BM25 | 0.650 (0.594 to 0.702) | 0.656 (0.602 to 0.707) |
| Query understanding | 0.703 (0.647 to 0.753) | 0.712 (0.661 to 0.762) |
| Dense | 0.632 (0.575 to 0.687) | 0.628 (0.572 to 0.683) |
| Hybrid (RRF) | 0.719 (0.667 to 0.767) | 0.720 (0.671 to 0.766) |
| Cross-encoder, top 50 | 0.754 (0.701 to 0.803) | 0.755 (0.705 to 0.802) |
| LambdaMART | 0.729 (0.677 to 0.776) | 0.737 (0.688 to 0.783) |
What this tells us. The numbers agree within 0.01 everywhere: the 928 extra products and the real names change almost nothing. The course’s LambdaMART learned from 260 training queries, with features from the synthetic log. This one learned from about 384 real queries per fold, without them. The course’s lessons hold on real data. Only the planted problems do not carry over: on real queries, spelling correction does not help.
The recommendation
Section titled “The recommendation”Nadia asks for one design. Ben asks for its settings and risks.
Use hybrid search with LambdaMART. On all 480 real queries, it scores 0.739 (0.716 to 0.763) NDCG@10, against 0.643 for today’s BM25. That is +0.096 (+0.080 to +0.113): better on 266 queries, worse on 91, and better in every slice. It needs 8.5 ms at p50 and 9.6 ms at p95 on one CPU thread on our laptop. It costs about US$0.0001 per 1,000 searches.
The settings:
- Stage one: query understanding and the granite bi-encoder, fused by RRF with k = 60 of the two top-100 lists. Query understanding normalises and stems, uses BM25 per field (title 1, description 1, category 0.5), and adds synonyms at half weight.
- Stage two: LambdaMART (LightGBM
lambdarank, 300 trees, learning rate 0.05, 31 leaves) on the 15 features, trained on all 480 queries for production. Cross-validation only measured; it chose nothing. - Spelling: guard it. Correct a word only when the query returns too few results, or protect known names. As it is, it costs 0.003.
Why not the most accurate design that fits? LambdaMART with the cross-encoder on 10 candidates (4 threads) scores 0.751, +0.012 more. But its p95 is 177.6 ms on an idle laptop, which leaves 22 ms for the network and the page. It needs a whole 4-vCPU server per search, so traffic peaks would push it over the budget. It also costs about 70 times as much. So it is the challenger: test it online against the recommendation, as in Test changes online.
What a person still checks:
- Known-item queries with brands and model numbers: add an exact-match rule for them, and review the 20 worst known-item queries each month.
- New judgments. When the catalogue changes, judge a new sample of queries and products, and retrain.
- The time under load. Measure p95 on the real servers at peak traffic before release.
End to end on new queries
Section titled “End to end on new queries”The notebook’s last section trains the final model on all 480 queries and runs three queries that are not in WANDS. The bi-encoder computes the query vector live.
"grey outdoor cushion for a bench" (3064 ms, with loading the model) 1. texture outdoor bench cushion Furniture Cushions 2. indoor & outdoor bench cushion Furniture Cushions 3. outdoor bench cushion Furniture Cushions 4. wicker indoor/outdoor bench cushion Furniture Cushions 5. damask indoor/outdoor bench cushion Furniture Cushions"dresser for a small kids room" (15 ms) 1. summer breeze kids 6 drawer double dresser Kids Dressers & Chests 2. monarch hill poppy 6 drawer double dresser Kids Dressers & Chests 3. 3 drawer dresser Kids Dressers & Chests 4. 4 drawer dresser Kids Dressers & Chests 5. monarch hill poppy 3 drawer dresser Kids Dressers & Chests"walnut desk with drawers" (13 ms) 1. computer desk , walnut Desks 2. stickland 4-drawer 60 '' writing desk aged walnut Desks 3. rhapsody writing desk Desks 4. lorello writing desk Desks 5. sadie secretary desk with hutch DesksWhat this tells us. Nobody judged these results: read them as a shopper. Every result is the right kind of product. Each search after the first takes about 15 ms, with 1 thread for the model. But no cushion says “grey”: the colour is in the product features, which the model gives little weight. For the walnut desk, results 3 to 5 do not say “walnut”. That suggests the next feature: match the query’s colour or material with the product’s attributes.
Insights and limits
Section titled “Insights and limits”- Most of the gain is cheap. Normalising gives +0.063 for 0.2 ms. Hybrid search adds +0.013 for about 6 ms, and LambdaMART +0.018 for about 2 ms more. The cross-encoder’s next +0.012 costs about 140 ms.
- Judged pools favour lexical search. Dense search’s numbers are probably a little too low.
- One laptop, no load. Servers differ.
- No shop data. A real shop would add prices, stock and clicks, with the selection bias of Learn from clicks.
- 480 queries are few. Read the rug result as a warning, not a fact.
- The query kinds are unchecked labels from a language model.
Responsible use
Section titled “Responsible use”- Brand names are real. Your portfolio may show them as data. Do not suggest that any company endorses your work, or present your search as Wayfair’s.
- Credit the data. MIT requires the copyright notice with copies. If you publish the data, include the licence text. If your notebook only downloads it, name WANDS, the authors and the licence in your README.
- The models. The granite models are Apache-2.0. Their model cards (bi-encoder, cross-encoder) say that their training data is permissively licensed and has no MS MARCO. We did not check this claim. Do not swap in a cross-encoder trained on MS MARCO for a commercial project without reading its terms.
- What a ranking does to people. Search decides which sellers’ products are seen. Check that groups such as small brands or new products do not vanish from the first page.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can compare search designs honestly and decide under a budget. Make it yours with one change. Add the colour-match feature, guard the spelling step, or use another open product dataset, such as the next case study’s. Make it easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
- The comparison table as a CSV, with the stage times and where they were measured.
- A short
DECISION.md: the design, its settings, what a person checks, and the challenger. - The data source and licence: WANDS, the authors, the commit, MIT, with a link and the checksums.
A README template:
# Choosing a product search design under a 200 ms budget
## ProblemA furniture shop's search must find the best products first, in under 200 ms (p95) on a CPU.Baseline: BM25 on name and description.
## Data and licenceWANDS (Wayfair, ECIR 2022): 42,994 products, 480 real queries, 233,448 judgments.MIT licence, https://github.com/wayfair/WANDS (commit 3b74dcf). Not changed.
## ApproachNine designs, one protocol: the same 480 queries and judgments, NDCG@10 with a bootstrapinterval, recall@100, MRR, p50/p95 time per stage, cost per 1,000 searches.LambdaMART measured with 5-fold cross-validation by query.
## ResultsBM25 0.643 -> hybrid + LambdaMART 0.739 (+0.096, 95% interval +0.080 to +0.113), p95 9.6 ms.Cross-encoder on 20 candidates: 0.746 but 508 ms p95.
## Where it failsKnown-item queries with model numbers ("moen 5995 arbor one"); rugs for hybrid search.
## LimitsJudged pools; one laptop with no load; no prices, stock or clicks; 480 queries.How to describe it in an interview, in two sentences: “I compared five families of product-search designs on a real catalogue of 43,000 products and 480 judged queries, with one protocol for quality, latency and cost. I recommended hybrid search with LambdaMART, which beat BM25 by 0.096 NDCG@10 in under 10 ms; the slower cross-encoder stays a challenger for an online test.”
Go further:
- Add a feature that matches colours and materials in the query with the product’s attributes. Predict first: which slice will gain most?
- Guard spelling (correct only when a query finds fewer than 10 products) and measure the 38 changed queries again.
- Fill the judged pools. Ask a language model to label the unjudged products in dense search’s top 10, as in Judgments from a language model. How much do the dense and hybrid numbers move?
- Time the recommended design with 4 searches running at once, and see whether p95 stays under 200 ms.
Knowledge check
Section titled “Knowledge check”Check your understanding
LambdaMART scores 0.882 when it is measured on the queries it learned from, and 0.739 with cross-validation by query. Which number do you report, and why must each fold hold whole queries?
Show the answer
Report 0.739. On the queries it learned from, the model has seen the answers. Cross-validation ranks every query with a model that never saw it, so it estimates how the model does on new queries. The folds must hold whole queries, because a query’s candidates share its words and the model learns them as a group. If a query had candidates in training and in testing, the model would partly know it, and the score would be too good.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
LambdaMART with the cross-encoder on 10 candidates scores 0.751 with a p95 of 177.6 ms on 4 threads. It fits the 200 ms budget. Why does the case study still recommend LambdaMART alone (0.739, 9.6 ms)?
Show the answer
The gain is small (+0.012) and the margin is thin. 177.6 ms was measured on an idle laptop and leaves about 22 ms for the network and the page. The design needs a whole 4-vCPU server per search, so at traffic peaks p95 goes over the budget. LambdaMART alone is far inside the budget and better than BM25 in every slice. The cross-encoder version stays a challenger: test it online.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On Larkfield's data, spelling correction was a clear gain. On the full WANDS queries it is a small loss. What is the difference, and what would you do?
Show the answer
Larkfield’s test set had planted misspellings, one edit away from a known word: exactly what the corrector fixes. The real WANDS queries have few typos and many product names, such as “filaret” or “togo”. The dictionary does not know them, or common words like “artist”, so the corrector turns real words into wrong ones. Guard it: correct only when the query finds too few products, protect known names, or build the dictionary from more text. Then measure the changed queries again.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On the full WANDS dataset (42,994 products, 480 real queries), BM25 scores 0.643 NDCG@10. Normalising gives most of the gain (+0.063); spelling correction hurts real names.
- Hybrid search (0.721) beats both of its parts. The best second stages are the cross-encoder on 20 candidates (0.746) and LambdaMART (0.739, cross-validated by query).
- Without the cross-encoder, every design runs in under 10 ms. The cross-encoder fits 200 ms only on 10 candidates with 4 threads.
- The recommendation: hybrid search with LambdaMART, +0.096 over BM25 in 9.6 ms at p95. The cross-encoder version is a challenger for an online test.
- On the same 120 test queries, every design agrees with Larkfield’s within 0.01.
Next, a bigger shopping dataset in two languages, where the hard part is telling a near miss from the right product.