New products and the cold start
- Lesson 26
- 30 min
- Module 7: The parts of search that people see
Where we are
Our boosts and click features help popular products. A new product has no clicks, so it never rises. In this lesson, we see the cold start problem, and we use content features and exploration slots to give new products a chance.
Lesson 26 of 30 · Search and Ranking: Find the Best MatchNew products and the cold start
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Give products with no history a fair chance to be seen, and measure what that chance costs.
NotebookBuild an exploration slot in a notebook
The notebook counts how many new products the synthetic log ever showed or clicked. Then it finds where the right new products rank in the recorded hybrid results. You write an exploration slot and measure its cost on the test queries. Its setup cells download the data, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The features of the ranker, including clicks and age, from Learning to rank.
- Why clicks only come from what was shown, from Learn from clicks.
What you need on your computer
- The project with the files of this module, copied in Autocomplete, and the trained ranker. This lesson explains the end of
business.py; it adds no file.
A new book on the back shelf
Section titled “A new book on the back shelf”A bookshop gets a new book. The staff put it on a shelf at the back. Nobody walks past that shelf, so nobody buys the book. After a month, the manager looks at the sales and says: “Nobody wants this book. Do not order it again.”
The book never had a chance. Good bookshops know this, so they have a table at the entrance: “New arrivals”. Every new book spends some time there. If people buy it, it earns its place. If not, it goes to the back.
The problem: no clicks, no rank, no clicks
Section titled “The problem: no clicks, no rank, no clicks”Larkfield added 600 products in the last two weeks of the synthetic log. Their “added” dates are synthetic, made by our code. How often did the log show them? Only 35 of the 600 were ever on a results page, and only 13 got a click. Across the whole catalogue, 1,678 products have a click.
The ranker of Module 6 uses clicks and carts from the log as features. You saw the feedback loop behind this in Learn from clicks: the log only has clicks on products that were shown. A new product is the extreme case. It has no clicks, so the ranker puts it low, and few people see it. For new items, this loop is called the cold start problem: the system knows nothing about a new item, and it does not create the behaviour it needs.
Text description of the diagram
Four boxes in a loop. “A new product: 600 added in the last two weeks” leads to “No clicks yet: 13 of 600 have one”, then to “The ranker puts it low (popularity features)”, then to “Few people see it: 35 of 600 were shown”, which leads back to “No clicks yet”. A fifth box with a thick border, “Exploration slot: position 4 for the best new product”, has a dotted arrow to “Few people see it”, labelled “breaks the loop”.
One product, measured
Section titled “One product, measured”Take the test query “kids chair”. The judges graded the new product “flint hills club kids chair” (LK-68994) as an exact match. It was added on 2025-10-20. The ranker ltr puts it at rank 65. Why? Look at its features next to the top 3. Type the lines after >>> into python, in the project folder. ltr.candidates gives the candidates of a query (the hybrid top 100) with their 19 features. ltr.booster("judgments") loads the ranker that you trained on judgments in Module 6:
>>> from larkfield_search import ltr>>> f = ltr.candidates("kids chair", "W-131")>>> f["score"] = ltr.booster("judgments").predict(f[ltr.FEATURES])>>> f = f.sort_values("score", ascending=False).reset_index(drop=True)>>> f["rank"] = f.index + 1>>> cols = ["product_id", "rank", "bm25_title", "dense", "clicks_log", "age_days", "score"]>>> f.loc[(f["rank"] <= 3) | (f["product_id"] == "LK-68994"), cols].round(2) product_id rank bm25_title dense clicks_log age_days score0 LK-17212 1 4.04 0.93 2.56 208 3.731 LK-32609 2 4.04 0.91 2.48 587 3.052 LK-75196 3 4.04 0.91 2.94 187 3.0064 LK-68994 65 3.77 0.92 0.00 6 -0.32The content features are almost the same. Content features describe the product’s own text. The BM25 score of the title is 3.77, against 4.04 for the top 3. The dense similarity is 0.92, against 0.91 to 0.93. The difference is the clicks. clicks_log is log(1 + clicks): 2.56 means 12 clicks, and 0.00 means none. And the product is 6 days old, against 187 to 587 days.
What if the new product had the clicks of a typical top-10 product? We can try it: copy the median clicks_log and carts_log of the top 10 into its row, and score it again.
>>> row = f[f["product_id"] == "LK-68994"].copy()>>> row[["clicks_log", "carts_log"]] = f.loc[f["rank"] <= 10, ["clicks_log", "carts_log"]].median().to_numpy()>>> new_score = ltr.booster("judgments").predict(row[ltr.FEATURES])[0]>>> round(new_score, 2), int((f["score"] > new_score).sum()) + 1(np.float64(1.49), 7)With about 7 clicks, the same product would be at rank 7 instead of 65. Its text is as good as the others. Only its history is missing.
Two ideas
Section titled “Two ideas”Idea 1: a fair prior from content
Section titled “Idea 1: a fair prior from content”A prior is a value you assume before you have data. A new product has no clicks yet. Zero is a bad guess: it says “nobody wants it”, when the truth is “nobody saw it”. A fairer guess is what a typical product of the same class gets.
We tried this as a measurement: each new product gets the median clicks and carts of the shown products of its class. The ranker stays the same. We ran this once for this lesson. It is not part of the project, so you see only our results below.
Idea 2: an exploration slot
Section titled “Idea 2: an exploration slot”An exploration slot is a fixed place on the results page for something the system is not sure about. Here, the best new product among the 100 candidates goes to position 4. It is the “New arrivals” table of the bookshop. People see the new product; some click; the clicks go into the log; the next ranker learns from them.
The word “exploration” comes from a general trade-off. Exploitation shows what you already know works. Exploration shows something uncertain, to learn about it. A system that only exploits never learns about new items.
A tiny worked example
Section titled “A tiny worked example”Here is explore on a list of six products, A to F. Only E is new. The slot is position 4.
| Position | Before | After |
|---|---|---|
| 1 | A | A |
| 2 | B | B |
| 3 | C | C |
| 4 | D | E (new) |
| 5 | E (new) | D |
| 6 | F | F |
The rule takes the best new product, E. If E is already at position 4 or higher, nothing changes. Here it is at position 5, so it moves to 4, and D moves down one place. Nothing else changes. If the list has no new product, nothing changes either.
The project’s code does exactly this:
def explore(results, slot=4): """An exploration slot: the best new product (added in the last two weeks) among the candidates goes to position `slot`, so new products get seen and can earn clicks.""" added = _col("added") new = [r for r in results if added[r[0]] >= NEW_FROM] if not new or new[0] in results[:slot]: return results rest = [r for r in results if r[0] != new[0][0]] return rest[: slot - 1] + [new[0]] + rest[slot - 1:]NEW_FROM is 2025-10-13, two weeks before the end of the log.
Larkfield: measure both ideas
Section titled “Larkfield: measure both ideas”First, the slot on “kids chair”:
search query "kids chair" --run ltr+explore -k 5ltr+explore: "kids chair" -> 5 results (query W-131: judged results are marked) 1. LK-17212 gearld modern kids chair Kids Chairs 412.99 [exact] 2. LK-32609 raziel kids club chair Kids Chairs 500.99 [exact] 3. LK-75196 pamrapo kids cotton chair Kids Chairs 513.99 [exact] 4. LK-68994 flint hills club kids chair Kids Chairs 586.99 [exact] 5. LK-58375 little modern kids chair 78.99 [exact]Text description of the diagram
Two lists of five. Left, ltr, where the new product is at rank 65: gearld modern kids chair, raziel kids club chair, pamrapo kids cotton chair, little modern kids chair, ruckus kids chair, all exact. Right, ltr with the exploration slot: the same first three, then flint hills club kids chair (exact, a new product added 2025-10-20) at rank 4, then little modern kids chair. Both lists score NDCG@10 1.000.

On this query the slot is free: the new product is exact. On all the test queries, it is not:
search compare ltr ltr+explore --by sourceltr+explore minus ltr on 241 test queries: ndcg@10 -0.016 (95% interval -0.020 to -0.012); better on 27, worse on 98, the same on 116
source queries ltr ltr+explore difference low high worse code 20 1.000 1.000 0.000 0.000 0.000 0 planted 3 0.571 0.550 -0.021 -0.073 0.010 1translated 59 0.474 0.454 -0.020 -0.028 -0.013 30 typo 40 0.615 0.604 -0.011 -0.022 -0.001 20 wands 119 0.729 0.711 -0.018 -0.025 -0.012 47The slot puts a new product at position 4, even when no new product is right for the query. On 98 queries that pushes a better product down. The cost is 0.016 NDCG@10, a clear loss. The code queries keep 1.000: the code rule puts the coded product first, and the slot only touches position 4.
Now both ideas side by side. For the exact new products in the test candidates, where do they rank? And on how many test queries does a new product reach the top 10?
Text description of the chart
A bar chart with four bars: the median rank of the 54 exact new products among the test queries’ candidates. hybrid-rrf 30.5; ltr 35.5; ltr with the class prior 34; ltr with the exploration slot 4.
| Design | Median rank of exact new products | Test queries with a new product in the top 10 | NDCG@10 against ltr |
|---|---|---|---|
hybrid-rrf (no click features) |
30.5 | 41 | (a different ranker) |
ltr |
35.5 | 23 | 0 |
ltr + class prior |
34 | 32 | 0.000 (−0.001 to +0.001) |
ltr+explore (slot at 4) |
4 | 193 | −0.016 (−0.020 to −0.012) |
Read the table from top to bottom:
- Hybrid search has no click features, so new products are not punished: median rank 30.5. The ranker
ltrpushes them down to 35.5, and shows a new product in the top 10 for only 23 queries instead of 41. This is the cold start, measured. - The class prior is fair and free: no change in NDCG@10, and 9 more queries show a new product. But it moves the median rank only from 35.5 to 34. Why so little? The median shown product of a class gets few clicks. The top 10 of a query are the popular ones: for “kids chair”, their median is about 7 clicks. A fair prior is not a boost.
- The slot brings the median to 4 and shows a new product on 193 queries. It costs 0.016 NDCG@10.
So the prior removes an unfair penalty, and the slot buys exposure. They solve two parts of the problem, and you can use both.
One honest limit of our data: new products keep their WANDS ratings and review counts, because WANDS has no dates. In a real shop, a new product has no reviews either, so its cold start is even colder.
When to use an exploration slot, and what it costs
Section titled “When to use an exploration slot, and what it costs”Use it when new products arrive often and a learned ranker uses popularity: shops, news, music, job ads. Without it, the catalogue’s new part stays invisible.
Limit it. The slot costs relevance on every page where the best new product is not right. Three common limits:
- Only for good matches. Use the slot only when the new product is already in the top 20 of stage one, for example. We did not measure this version: try it yourself.
- Only for a time. A product leaves the “new” pool after two weeks, or after a number of impressions.
- Only on some traffic. Show the slot to a part of the searches, and compare clicks and carts with the rest. Test changes online shows how.
Measure the right thing. Offline NDCG@10 can only show the cost of the slot. The gain comes later: clicks that teach the ranker about new products. Only an online test can show that gain.
Common mistake
New products that never rise
Symptom: month after month, the same products fill the top of every page. New products sell badly, and the buyers stop ordering new lines.
Cause: the click feedback loop of Learn from clicks, with nothing to stop it. With ltr, an exact new product is at rank 35.5 in the median.
Diagnosis: compare the median rank of exact new products under a ranker without click features (hybrid-rrf, 30.5) and with them (ltr, 35.5). Count the new products with any impression in the log: here 35 of 600.
Fix: give new products a fair prior for their missing history, and an exploration slot with clear limits. Then check that the clicks they earn reach the next training.
Consequence in AI work: a system that learns from its own outputs needs a planned way to try new items. Exploration is that way, and its price must be measured like any other change.
Try it yourself
Section titled “Try it yourself”The slot is at position 4. Before you run anything, predict: is position 2 more or less costly? Change the slot in business.py (the slot=4 default) to 2, and run:
search compare ltr ltr+explorePut the slot back to 4 afterwards.
Our result, and what it shows
ltr+explore minus ltr on 241 test queries: ndcg@10 -0.026 (95% interval -0.033 to -0.021); better on 26, worse on 103, the same on 112Position 2 costs 0.026, against 0.016 at position 4. A higher slot pushes a better product down from a place that matters more, and NDCG gives the top places the largest weight. It also gives the new product more attention. In the click model of the course, a product at position 2 is looked at twice as often as one at position 4. The right place is a business choice: how much relevance you pay for how much exposure.
Knowledge check
Section titled “Knowledge check”Check your understanding
Omar says: “New products get no clicks, so people do not want them. Let us drop them from search.” What does the log really say?
Show the answer
It says that people did not see them. Only 35 of the 600 new products were ever shown in the synthetic log. A product that is never shown cannot be clicked. The ranker put an exact new product, the flint hills club kids chair, at rank 65. The reason was its missing clicks, not its text. No clicks means no data, not a bad product.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The exploration slot costs 0.016 NDCG@10 on the test queries. Why can an offline test not show its benefit?
Show the answer
The benefit is future learning: new products get impressions and clicks, and the next ranker learns which of them are good. The offline test uses fixed judgments and a fixed ranker, so it can only see the cost of moving a product to position 4. An online test that runs over weeks and retrains the ranker is needed to see the gain.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why did giving new products the median clicks of their class change so little?
Show the answer
Most products of a class get few clicks, so the median is low. The top 10 of a query are the popular products: for “kids chair”, their median is about 7 clicks. A fair prior removes the penalty of “zero clicks”, but it does not make a new product look as popular as the best sellers. That is why the slot is still needed for exposure.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The cold start: a new product has no clicks, so a ranker that uses clicks puts it low, so it gets no clicks. In the synthetic log, only 35 of 600 new products were ever shown.
- Content features (BM25, dense) treat a new product fairly. The click features do not: an exact new kids chair is at rank 65 with
ltr, and at rank 7 with typical clicks. - A class prior for the missing history is fair and free (NDCG@10 change 0.000), but it moves new products little.
- An exploration slot at position 4 brings exact new products from a median rank of 35.5 to 4, for 0.016 NDCG@10. Limit it, and measure its benefit online.
- The search page now has autocomplete, facets, a fallback, business rules and a slot for new products.
Every part of this module adds work to each search: suggestions, counts, rules. In the next module, you measure what each stage of a search costs in time and money. Then you decide what fits Amira’s budget of 200 ms.
Next: Latency and cost budgets