Final assignment
- After the case studies
- about 3 hr
Where we are
Each module added one part of search and measured it alone: the test set, BM25, query understanding, hybrid search, re-ranking and the page. In this assignment, you use them all for Larkfield's new season.
What you will be able to do
Use every module of the course on one new situation, and deliver a search design pack that Omar, Amira and Hana can act on. The new season looks like the course’s catalogue, but its products have no clicks and its queries have no judgments. So you must measure each choice again, not copy it from the lessons.
Before you startWhat to know and to have ready
- You completed Modules 1 to 8. Each part below names the lessons that it uses.
- You do not need the case studies of Module 9 for this assignment. (The certificate asks for at least one of them.) To see five designs compared on all of WANDS first, open Product search for a furniture shop.
- Your folder
larkfield-searchworks at the end of Module 8:python -m pytest -qprints39 passed, as in Latency and cost budgets. The course data is inlarkfield-search/data, and the foldernextia-learning-labsis besidelarkfield-search, as in Meet the catalogue and the query log. - About 3 hours of work, plus the time to judge results. Judging is the longest part. Plan about 3 hours for about 1,000 grades, at about 10 seconds a grade. The computer work is short. In the author’s solution, the slowest design was the cross-encoder: about one second a query, because it reads 50 products for each query.
- Tested on macOS with Python 3.12.13 and the project’s pinned packages. The Windows commands are written, but they were not run.
The situation: a new season
Section titled “The situation: a new season”Larkfield adds a new season to its catalogue. Three things change at once:
- New products. 2,000 products arrive in March. Nobody has searched for them yet, so they have no clicks in the log. This is the cold start of New products and the cold start, but for a whole season.
- New queries. Customers will type queries that the course never used: English queries, French queries, and some practice problems we made on purpose.
- No judgments. Nobody has judged the new queries. Before you can say that a design is better, you need a test set of your own.
Omar, the product manager, wants the new products to be found from the first day. Amira, the developer, keeps her budget: 200 ms for one search. Hana, the merchandising lead, asks again for “sale items first” on the new season’s page. Tomás, the AI engineer, has an answer ready: “Use the best design of the course. LambdaMART won, so ship LambdaMART.”
Priya asks you to answer all of them with evidence. The picture shows the order of the work:
Text description of the diagram
Two grey boxes on the left: the course data (40,066 products, the log, the clicks and the judgments) and the new season (2,000 new products with no clicks, 120 queries with no judgments). Both go into make_data.py, which makes one data folder with 42,066 products. Then step 1, choose and judge: a rule picks the sample, you pool the top 10 results, judge them with guidelines, and measure agreement with a second judge. The judged queries split in two: a dev part that you measure as often as you like, and a held-out part, drawn in red, that stays locked. The dev part feeds steps 2 and 3, the designs and the query changes, then step 4, the choice by slices, latency and cost. After the choice, the held-out run happens once, in red, with no change after it. Step 5, the online test plan decided in advance, comes last.
Be honest about the data. The products, the queries and the people’s judgments come from WANDS (MIT licence), as in the course. The prices, stock, sale flags and dates are synthetic: our code made them. The query log is synthetic too. A language model machine-translated the French queries from English queries. So translating them back is easier than it would be with real French customers. Quote your numbers “on Larkfield’s new-season queries”, with their intervals, and do not promise them for a real shop.
The data
Section titled “The data”The data is in the labs repository, in C38/final/. The course data, C38/data/, stays as it was.
| File | Rows | What one row is |
|---|---|---|
catalogue_new.csv.gz |
2,000 | A new-season product, with the same 17 columns as the course catalogue. All were added between 2026-03-02 and 2026-03-15. None is in the query log, and none has a click. 403 have a French title (machine-translated). |
queries.csv |
120 | A query of the new season: query_id (N-001 to N-120), query and language (en or fr). No judgments. |
embeddings/products_new.npy, embeddings/queries.npy |
2,000 and 120 | The embeddings of the new products and of the queries, made with the course’s bi-encoder, in file order. |
make_data.py |
Makes one data folder that the search command can read: the course data, this folder and your judgments. |
The folder also has embeddings/model.json, README.md, dataset-card.md and SHA256SUMS, and it is about 2 MB. The IDs are new and in a random order: an ID says nothing about a query. The course team keeps the judgments of the 120 queries. Nobody chooses anything with them.
Check the files, make the data folder, and give it to the project. Run this in your larkfield-search folder, with .venv active:
git -C ../nextia-learning-labs pullF=../nextia-learning-labs/C38/final(cd "$F" && shasum -a 256 -c SHA256SUMS | grep -vc ': OK$')python "$F/make_data.py"export LARKFIELD_DATA=final-data LARKFIELD_OUTPUTS=outputs-finalsearch infosearch indexgit -C ..\nextia-learning-labs pull$F = "..\nextia-learning-labs\C38\final"Get-FileHash "$F\queries.csv" -Algorithm SHA256python "$F\make_data.py"$env:LARKFIELD_DATA = "final-data"; $env:LARKFIELD_OUTPUTS = "outputs-final"search infosearch indexCompare the hash with the line for queries.csv in SHA256SUMS. Do the same for catalogue_new.csv.gz.
The number that shasum prints is the number of wrong files. It must be 0. Then make_data.py, search info and search index print:
Wrote final-data/: 42,066 products (2,000 new), 120 final queries, 0 of your judgments.Splits: train 260, validation 121, course-test 243, test 120No judgments yet: search eval prints nan for the final queries until you add yours.Note: search check compares with the course data, so it fails on this folder. Use search info.Catalogue: 42,066 products in 551 product classes; 8,000 with a French title (machine-translated)In stock: 87.1%; on sale: 14.8% (synthetic)Queries with judgments: 744 (260 train, 121 validation, 120 test); English 583, French 161Judgments: 291,230 (exact, partial or irrelevant)Query log (synthetic): 20,000 searches, 1,903 different queries, 2025-09-01 to 2025-10-26Indexed 42,066 products in 2.5 s: SQLite FTS5 (title, description) and BM25 (title and description). Index size 55.5 MB.What happened:
make_data.pywrote a new folder,final-data/. Yourdata/folder did not change. The new folder has the 42,066 products of the new season, the course’s queries and judgments, and the 120 new queries.- The course’s test queries are now the split
course-test. The new queries are the splittest, sosearch evalandsearch comparemeasure them. They have no judgments yet, sosearch evalprintsnanfor now. search infocounts the 120 new queries as “queries with judgments”, and their judgments as 0: the 291,230 judgments are all the course’s.LARKFIELD_DATAgives the folder to everysearchcommand, andLARKFIELD_OUTPUTSkeeps the new index apart from the course’s index. Close the terminal, or unset both, to go back to the course data.search checkcompares with the course data, so it fails on this folder. That is expected.
search bench and search interleave read the course’s test queries, so run them with the course data (the two variables unset). Part 4 shows how to time a design on the new catalogue.
Make a folder search-pack/ in your larkfield-search folder. Put every file of the assignment in it. What to submit lists the files.
The five parts
Section titled “The five parts”The pack follows the course’s completion rules: one part for each required item.
Text description of the diagram
A box on top: the folder search-pack, with REPORT.md and the commands that make every number. Below it, five boxes. 1 Test set: GUIDELINES.md, splits.csv, pool.csv and judgments.csv, with the agreement with a second judge. 2 Designs: comparison.csv, with BM25 and three other families, overall and by slice, with intervals. 3 Query changes: one row per change, with the paired difference that kept or dropped it. 4 The choice: latency p50 and p95, cost per 1,000 queries, and OMAR.md, which says why this design. 5 Online plan: ONLINE.md, with interleaving or an A/B test, the metric, the guardrails, the size and the stopping rule.
Part 1: a test set of your own
Section titled “Part 1: a test set of your own”Uses Relevance judgments, Build a test set that represents your users and Judgments from a language model.
There are no judgments. NDCG needs them. So your first job is the one that a real team does with a new season: choose queries, look at the results, and grade them. You cannot judge all 120 queries and all 42,066 products. You judge a sample, and you judge it well.
-
Choose the queries before you run any design. Take at least 40 of the 120 queries. Choose them from the public columns only, with a rule that you write in
REPORT.md. For example: every query whose number ends in 1, 3, 6 or 8. Never choose queries because a design looks good or bad on them. -
Choose the held-out part now, with a second rule. For example: of your chosen queries, those whose number is a multiple of 3. Write both parts in
search-pack/splits.csv, with the columnsquery_idandpart(devorheldout). You will measure the held-out part once, at the end, as in Keep some queries away from your work. -
Label the slices from the text. Add a column
kindtosplits.csv: for exampleenglish,french,code(a product code) andmisspelt. Read each query to decide. Also add any slice that the course taught and that you can see, such as queries with a number. Write the size of each slice inREPORT.md. -
Write your guidelines. Start from the project’s
docs/judging-guidelines.md, and save your version assearch-pack/GUIDELINES.md. Add a rule for each new case that you meet: a French query and an English product, a product code, a misspelt brand name, a new product with almost no description. -
Pool the results. For each chosen query, take the top 10 of at least four designs from different families, for example
bm25,bm25-norm+spell+syn,hybrid-rrfandltr. Judge every product that one of them shows. This is called pooling: no design is then measured on products that nobody judged. A short script does it with the project’s functions:# search-pack/pool.py: run it with LARKFIELD_DATA=final-data (and after search ltr train)import pandas as pdfrom larkfield_search import data, enginechosen = set(pd.read_csv("search-pack/splits.csv")["query_id"])q = data.queries()q = q[q["query_id"].isin(chosen)]cat = data.catalogue().set_index("product_id")pool = {}for run in ["bm25", "bm25-norm+spell+syn", "hybrid-rrf", "ltr"]:ranked, _ = engine.run_queries(run, q, k=10)for qid, ids in ranked.items():for pid in ids:pool.setdefault((qid, pid), []).append(run)text = dict(zip(q["query_id"], q["query"]))rows = [{"query_id": qid, "query": text[qid], "product_id": pid, "title": cat.loc[pid, "title"],"product_class": cat.loc[pid, "product_class"], "found_by": " ".join(runs), "your_grade": ""}for (qid, pid), runs in sorted(pool.items())]pd.DataFrame(rows).to_csv("search-pack/pool.csv", index=False)print(len(rows), "pairs to judge")Run
search ltr trainonce before it: the LambdaMART model learns from the course’s training queries, as in Learning to rank. In the author’s run, these four designs gave about 24 products per query to judge: 1,139 pairs for 48 queries. With dense search and the cross-encoder as well, the pool grew to 1,521 pairs. -
Judge. Fill
your_gradewith 2 (exact), 1 (partial) or 0 (irrelevant), following your guidelines. Open the product’s description when the title is not enough. Do not look atfound_bywhile you judge: it tells you which design found the product, and it can change your grade. Save the file assearch-pack/judgments.csv. -
Measure agreement with a second judge. Ask a colleague to judge at least 100 of your pairs without seeing your grades. Or use a language model as a judge, as in Judgments from a language model. Give it your guidelines, the query and the product, and ask for one grade. Use your own key, and give the cost. Report the agreement, Cohen’s kappa and the 3 × 3 table, as
search judge agreedoes. Say which judge is stricter, and by language. -
Give the project your judgments:
Terminal window python "$F/make_data.py" --judgments search-pack/judgments.csv --splits search-pack/splits.csvsearch eval --run bm25 --by kindNow your dev queries are the split
test, and your held-out queries the splitheldout, which no command reads. The columnkindof your splits file is inqueries.csv, so--by kindworks.
Part 2: a baseline and three other families
Section titled “Part 2: a baseline and three other families”Uses BM25, Search by meaning, Hybrid search, Cross-encoder re-ranking and Learning to rank.
-
The baseline. Measure
bm25on your dev queries. Every other design is compared with it. -
At least three other families. For example: query understanding on BM25 (
bm25-norm+spell+syn), dense search (dense), hybrid search (hybrid-rrforhybrid-blend), the cross-encoder (rerank-ce) and LambdaMART (ltr). Two settings of one family count as one family.Terminal window search eval --run hybrid-rrf --by kindsearch compare bm25-norm+spell+syn ltr --by kindThe recorded cross-encoder scores cover only the course’s queries. On the new queries,
rerank-cedownloads the cross-encoder the first time (about 600 MB) and scores every pair on your computer. -
Report the course’s columns for each design: NDCG@10 with its 95% interval, recall@100, MRR, the zero-result rate, judged@10, and the latency. Write them in
search-pack/comparison.csv, one row per design, as the course’s comparison table does. -
Slice. Give NDCG@10 by
kindfor each design, with the size of each slice. Usesearch comparefor the differences that you claim: a paired difference with its interval. -
Look at judged@10. A design that finds products that your pool missed is measured unfairly, because an unjudged product counts as irrelevant. If judged@10 of a design is low, judge its missing products too, and say so.
-
Look at the new products. Count how often each design puts a new-season product in its top 10, and where your exact new products land. A design that learned from clicks has no clicks for them.
Part 3: query understanding, one change at a time
Section titled “Part 3: query understanding, one change at a time”Uses Normalise text, Fix spelling, Synonyms and query rewriting, Detect what the query asks for and Exact matches and rules.
-
Add one change at a time, and measure each one against the step before it, with
search compare: the code rule (bm25-fields+codes), normalising and stemming (bm25-norm), spelling correction (+spell), synonyms (+syn), and the class and price detector (+detect).Terminal window search compare bm25-norm bm25-norm+spell --by kind -
Keep or drop each change by its measurement. Write one row per change in
REPORT.md: the paired difference with its interval, the slices that gained, the slices that lost, and your decision. “It helped the misspelt slice and did not hurt the others” is a reason. “It is a good idea” is not. -
Find one query that a change hurts. Spelling correction changes real names into common words, as in A wrong correction is worse than no correction. Look for a query whose NDCG@10 falls. Say what you would do: a guard for words in the catalogue, a “did you mean” link, or nothing.
-
Try one change of your own, for example a new synonym rule in a copy of
synonyms.csv, a guard for the spelling corrector, or the code rule for codes written in a new way. Measure it like the others. -
The language-model rewrite (
llm-rewrite) has recorded answers only for the course’s queries. On the new queries, it searches with the query itself, so it measures nothing new. Leave it out, or rewrite the queries with your own key and give the cost and the time per query.
Part 4: choose a design, then run the held-out part
Section titled “Part 4: choose a design, then run the held-out part”Uses Latency and cost budgets, New products and the cold start and Zero results and business rules.
-
Time each candidate design on the new catalogue.
LARKFIELD_LIVE=1makes every run encode the query and score every pair now, as in production.LARKFIELD_THREADS=1keeps PyTorch on one thread, as the course’s table does:Terminal window LARKFIELD_LIVE=1 LARKFIELD_THREADS=1 search eval --run ltrOn PowerShell, set
$env:LARKFIELD_LIVE = "1"and$env:LARKFIELD_THREADS = "1"first, and remove both afterwards. The last line ofsearch evalgives p50 and p95 in milliseconds. Run each design twice and keep the second run: the first one loads the models. -
Turn the time into a cost per 1,000 queries, as in Turn milliseconds into money. Use the course’s server price: US$0.17 an hour for 4 vCPUs, an assumption in
costs.json. Say how many queries run at once. -
Check the budget and the business rules. Amira’s budget is 200 ms. If Hana’s “sale items first” is in your design, measure what it costs in NDCG@10 and in out-of-stock first results. If you add an exploration slot for new products (
ltr+explore), measure what it costs and what it gives. -
Choose, on the dev queries. Write the reason in
search-pack/OMAR.md: one page, no code. Say what the design does better, for which slices, at what latency and cost, and which slice gets worse, with your comment. A choice on the average alone is not accepted. -
Now run the held-out part, once. Make its data folder and measure your chosen design, the baseline and the other designs that you report, with no change:
Terminal window python "$F/make_data.py" --out final-heldout --judgments search-pack/judgments.csv --splits search-pack/splits.csv --heldoutLARKFIELD_DATA=final-heldout search eval --run ltr --by kindWrite the numbers in
REPORT.mdas they are. If the held-out result disagrees with the dev result, say so. Do not go back and change the design: that would turn the held-out part into a second dev part.
Part 5: an online test plan
Section titled “Part 5: an online test plan”Uses Test changes online.
Your offline test says which design is better on your judgments. People may disagree. Write the plan for the online test in search-pack/ONLINE.md, before any test starts, as in A plan, written before the test:
- The question, and interleaving or an A/B test. Say why. Interleaving needs fewer searches; an A/B test measures the add-to-cart rate of each design alone.
- The metric that decides: for example interleaving wins, or the add-to-cart rate.
- Guardrails: measures that must not get worse, for example the zero-result rate, the p95 latency against 200 ms, the share of out-of-stock first results, and the share of new products shown.
- The size, decided in advance. Size it with the course’s simulation: run
search interleavefor your two designs on the course data (the two variables unset). Then read How many searches a real difference needs. Turn the size into weeks of Larkfield’s traffic. - The stopping rule: look once, at the planned size. Stop early only if a guardrail breaks. Never stop because the new design “looks good”, as Peeking: winners that do not exist shows.
- Rollback: how to return to the old design in minutes.
What to submit
Section titled “What to submit”The folder search-pack/ in your larkfield-search folder:
search-pack/ GUIDELINES.md Part 1: your judging guidelines (the course's plus your rules) splits.csv Part 1: the chosen queries, dev or heldout, and your slice labels pool.csv Part 1: the pooled pairs and the designs that found them judgments.csv Part 1: your grades agreement.md Part 1: the second judge, the agreement, kappa and the 3 x 3 table comparison.csv Parts 2 and 4: one row per design, the course's columns OMAR.md Part 4: one page for Omar, no code ONLINE.md Part 5: the online test plan pool.py and any other script or notebook that you used REPORT.md Parts 1 to 5, with the commands that make every number requirements.txt exact versions (pip freeze)REPORT.md has these sections:
# Search design pack for the new season
## Data checks (files, checksums, what you see in the new season)## Test set (Part 1: the two rules, the slices and their sizes, the pool, the guidelines' new rules)## Agreement (Part 1: the second judge, agreement, kappa, who is stricter, by language)## Designs by slice (Part 2: the table, the slices, judged@10, the new products)## Query changes (Part 3: one row per change, the query that a change hurts, your own change)## The choice (Part 4: latency, cost per 1,000, the budget, the rules, the slice that got worse)## Held-out run (Part 4: the numbers of the one run, as they came)## Online test plan (Part 5: a summary of ONLINE.md)## Limits (what your test set cannot see, and what you would measure next)## How to reproduce (commands, versions, run times, a fresh environment)Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. The course is complete only when all five items are demonstrated. You can improve an item and send it again.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. Test set (Module 2) | At least 40 queries chosen by a written rule before any run, with a held-out part chosen by a second rule. Written guidelines with the new cases. A pool of the top 10 of at least four designs, all judged. Agreement with a second judge on at least 100 pairs: agreement, kappa, the table, and who is stricter. Slice sizes given. | Agreement on fewer than 100 pairs, or with no table. Or the pool comes from one or two designs. | No written guidelines. Or the queries chosen after seeing the results. |
| 2. Designs (Modules 3, 5 and 6) | BM25 and at least three other families, on the same dev queries and judgments. NDCG@10 with its interval, recall@100, the zero-result rate and the latency for each, overall and by slice with sizes. judged@10 reported and discussed. | No intervals, or no slices. Or the latency is missing for a design. | Fewer than three other families. Or each design measured on different queries. |
| 3. Query changes (Modules 3 and 4) | Each change measured against the step before it, with a paired interval and its slices. A decision per change, with its reason. One query that a change hurts, explained. | The changes are measured all at once, so their effects cannot be told apart. | A change kept “because it should help”, with no measurement. |
| 4. The choice (Modules 7 and 8) | A design chosen on the dev queries, with p50 and p95 measured on the new catalogue, the cost per 1,000 queries with the server price as an assumption, and the 200 ms budget checked. The slice that gets worse is named and discussed. The held-out part run once, after the choice, and reported as it came. | The latency copied from the course, not measured on the new catalogue. Or the cost has no assumption. | The held-out part used to tune or to choose. Or the choice made on the average alone, while a slice gets worse with no comment. |
| 5. Online plan (Module 8) | Interleaving or an A/B test with the reason, the deciding metric, at least two guardrails, a size decided in advance and sized with the course’s simulation, a stopping rule that looks once, and a rollback. | No size, or a size with no source. Or no guardrail on latency. | “Run it until it wins.” Or no stopping rule. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- The held-out queries used to tune the design: a design or a setting changed after the held-out run, or the held-out run repeated until it looked good. (Item 4.)
- A design chosen on the average alone, when a slice gets worse and the pack says nothing about it. The slices are the ones that the course taught: French, misspelt queries, codes, broad queries and new products. (Item 4.)
- Grades copied from another source. WANDS is public, and its judgments of these queries can be found. Looking them up is not allowed, and it does not help. The rubric grades your method, not your score: the rules, the pool, the guidelines, the agreement and the held-out run. (Item 1.)
What a good pack looks like
Section titled “What a good pack looks like”A good pack is honest before it is impressive. It has three things.
- Numbers with their sizes. Each NDCG has its interval and the number of queries behind it, and says whether it is from the dev part, the held-out part, or the course team’s score.
- A simple baseline in each comparison. BM25 for the designs; the step before for each query change. If your change does not beat it, the pack says so.
- A decision with its cost. “On the new season, this design finds the best product first more often for these slices, costs this much per 1,000 queries, and is worse for this slice, which we accept because …”
These ranges come from the author’s own solution. The author built it from the public files only, with the course’s code, but did not judge by hand. The course team’s judgments stood in for the grades that you write. chat-small graded the pooled pairs that the team never judged. Then the course team scored the designs on all 120 queries with every judgment that it keeps. Your methods and your sample can differ. Compare the method and the size of each result, not the exact number.
- Your test set is small, and its intervals show it. With 32 dev queries, the author’s intervals of NDCG@10 were 0.12 to 0.23 wide. A slice of 6 French queries said that LambdaMART was clearly worse for French. The course team’s score on all 30 French queries said that it was much better. Give the sizes, and trust no slice of a handful of queries.
- Your NDCG is higher than the team’s, and that is fine. The author’s dev numbers were about 0.15 to 0.2 higher than the team’s score of the same designs. A complete pool leaves almost no unjudged product in your top 10. The team’s judgments miss about a third of the top 10, and a missing judgment counts as irrelevant. Compare the differences between designs, not the levels.
- The model judge is strict. chat-small agreed with the people’s grades on about 3 pairs in 4 (kappa about 0.55). It gave the lower grade about four times as often as the higher one, as in the course.
- Query understanding is the biggest cheap gain. On the team’s score, normalising, spelling and synonyms raised NDCG@10 by about 0.15 over BM25, for under 2 ms a query. Spelling correction brought most misspelt queries back (about +0.25 on that slice) and cost a little on French. The synonym rules changed almost nothing. The class detector helped French and cost a little on English. On 32 dev queries the author dropped it; the full score would have kept it for French.
- Hybrid blending gave a small gain that held. The author chose
hybrid-blendon the dev queries: about +0.03 over the lexical design, with no slice worse. It takes under 10 ms a query and costs about US$0.0001 per 1,000 queries. The held-out run (+0.04, with a wide interval) and the team’s score (about +0.025, with an interval clear of zero) agreed. - LambdaMART was the best on average, and the worst for the new products. On the team’s score, it led the hybrid designs by about 0.03, mostly on French, at about 37 ms a query. But new products filled under 1% of its top 10, against about 7% for the other designs. (They are about 5% of the catalogue.) The model learned from clicks and from how long a product has been in the shop, and the new products have neither. The exploration slot raised the share to about 4%, for a small loss of NDCG. A pack that chooses LambdaMART must say this, and add the slot or another plan.
- The cross-encoder is too slow. It took about 1 s a query on one thread, five times Amira’s budget, for a gain smaller than LambdaMART’s.
- Dense search alone lost to the lexical design, by about 0.06, and by more than 0.2 on misspelt queries.
- The online test needs a size decided in advance. For the author’s change, interleaving checked every 100 searches looked “clear” after a median of 600 searches. The A/B test, checked the same way, named the old design as the winner in 2 of 5 runs. In Test changes online, a fair test looked at once needed about 5,000 searches.
Common mistakes
Section titled “Common mistakes”Common mistake
The held-out part used twice
Symptom: the held-out NDCG@10 of your chosen design is as high as on the dev queries. Your report says that you “fixed one thing” after the first held-out run.
Cause: after the held-out run, you changed the design and ran the held-out part again. From then on, the held-out part was a second dev part, and its number is too good.
Fix: run the held-out part once, after every choice. If it disagrees with the dev part, report both, and say what you would test next. A new held-out part needs new judged queries.
Consequence in AI work: the team reports a gain that the first real customers do not see.
Common mistake
A better average that hides a worse slice
Symptom: your chosen design has the best NDCG@10 overall. A colleague finds that French queries got worse.
Cause: the average mixes the slices. A large English gain can hide a French loss, as the cross-encoder did in Cross-encoder re-ranking: it reads English only.
Fix: give every slice with its size, and use search compare --by kind. Name the slice that loses, and say what you do about it: accept it, route it to another design, or fix it.
Consequence in AI work: a change that “won” is rolled back after complaints from one group of customers.
- The test set chosen after the results. Choosing queries where a design fails, or where it shines, makes the test set argue for that design. Write the rule first.
- A pool from one design. Every other design is then measured on products that nobody judged, and looks worse than it is. Pool at least four designs, and check judged@10.
- The course’s latency copied. The new catalogue is larger, and your computer is not the course team’s. Measure with
LARKFIELD_LIVE=1. - A model judge treated as people. In the course, the model judge was stricter than the WANDS judges. If it is your only judge, your NDCG is lower than people’s grades would make it, and French pairs were never checked against people.
- Spelling correction kept for every query. It helps misspelt queries and can break real names. Measure it by slice, and guard it.
- A test stopped when it “looks good”. With repeated looks, an A/A test shows a false winner in about half of the tests. Decide the size first, and look once.
Show the public check values
Open this only after you have your own numbers. These values come from the public files and the course data, so you can check them. They are not the answers to the assignment.
- Data: 2,000 new products, added from 2026-03-02 to 2026-03-15, 403 with a French title; 120 queries, 90 marked
enand 30 markedfr. The folder has 9 files, about 2.1 MB, and every file inSHA256SUMSis OK. None of the new products is in the course’slog/impressions.csv.gz. - The data folder:
make_data.pywrites 42,066 products in 551 product classes, with 8,000 French titles. New products are about 5% of the catalogue.search indextakes a few seconds and writes 55.5 MB. - LambdaMART:
search ltr trainonfinal-datalearns from the same 260 training queries as in the course, with 13,209 candidates with a label above 0. (The course data gives 13,763: some new products now take a place among the candidates, and nobody judged them for the training queries.) - Speed on the author’s laptop (Apple M4,
LARKFIELD_LIVE=1, one thread, p50): BM25 0.5 ms, BM25 with query understanding 0.8 ms, dense 6.8 ms, the hybrid designs about 8 ms, LambdaMART about 37 ms, LambdaMART with the exploration slot about 61 ms, the cross-encoder about 970 ms.
Recap
You can now take a search problem from a new catalogue to a decision. You build a test set of your own, with a rule, guidelines, a pool and a second judge. You keep a part of it away until the end. You compare BM25 with other families of designs by slice, with intervals. You keep each query change only when a measurement supports it. You choose a design by relevance, latency and cost, with the slice that loses in plain view. And you plan an online test with its size and its stopping rule decided in advance. The case studies of Module 9 use the same steps on other open data: Product search for a furniture shop, Shopping queries with hard negatives and Search a help centre or a scientific corpus.