Skip to content
Français

New products and the cold start

  • Lesson 26
  • 30 min
  • Module 7: The parts of search that people see

Where we are

Our boosts and click features help popular products. A new product has no clicks, so it never rises. In this lesson, we see the cold start problem, and we use content features and exploration slots to give new products a chance.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Give products with no history a fair chance to be seen, and measure what that chance costs.

NotebookBuild an exploration slot in a notebookColab · Kaggle · GitHub

The notebook counts how many new products the synthetic log ever showed or clicked. Then it finds where the right new products rank in the recorded hybrid results. You write an exploration slot and measure its cost on the test queries. Its setup cells download the data, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • The project with the files of this module, copied in Autocomplete, and the trained ranker. This lesson explains the end of business.py; it adds no file.

A bookshop gets a new book. The staff put it on a shelf at the back. Nobody walks past that shelf, so nobody buys the book. After a month, the manager looks at the sales and says: “Nobody wants this book. Do not order it again.”

The book never had a chance. Good bookshops know this, so they have a table at the entrance: “New arrivals”. Every new book spends some time there. If people buy it, it earns its place. If not, it goes to the back.

The problem: no clicks, no rank, no clicks

Section titled “The problem: no clicks, no rank, no clicks”

Larkfield added 600 products in the last two weeks of the synthetic log. Their “added” dates are synthetic, made by our code. How often did the log show them? Only 35 of the 600 were ever on a results page, and only 13 got a click. Across the whole catalogue, 1,678 products have a click.

The ranker of Module 6 uses clicks and carts from the log as features. You saw the feedback loop behind this in Learn from clicks: the log only has clicks on products that were shown. A new product is the extreme case. It has no clicks, so the ranker puts it low, and few people see it. For new items, this loop is called the cold start problem: the system knows nothing about a new item, and it does not create the behaviour it needs.

The cold start loop. A new product, 600 added in the last two weeks, has no clicks yet: 13 of 600 have one. The ranker puts it low because of its popularity features. Few people see it: 35 of 600 were shown. That leads back to no clicks. An exploration slot, position 4 for the best new product, breaks the loop at "few people see it".
The cold start loop, with the numbers of the synthetic log. An exploration slot breaks it by showing new products on purpose.
Text description of the diagram

Four boxes in a loop. “A new product: 600 added in the last two weeks” leads to “No clicks yet: 13 of 600 have one”, then to “The ranker puts it low (popularity features)”, then to “Few people see it: 35 of 600 were shown”, which leads back to “No clicks yet”. A fifth box with a thick border, “Exploration slot: position 4 for the best new product”, has a dotted arrow to “Few people see it”, labelled “breaks the loop”.

Take the test query “kids chair”. The judges graded the new product “flint hills club kids chair” (LK-68994) as an exact match. It was added on 2025-10-20. The ranker ltr puts it at rank 65. Why? Look at its features next to the top 3. Type the lines after >>> into python, in the project folder. ltr.candidates gives the candidates of a query (the hybrid top 100) with their 19 features. ltr.booster("judgments") loads the ranker that you trained on judgments in Module 6:

>>> from larkfield_search import ltr
>>> f = ltr.candidates("kids chair", "W-131")
>>> f["score"] = ltr.booster("judgments").predict(f[ltr.FEATURES])
>>> f = f.sort_values("score", ascending=False).reset_index(drop=True)
>>> f["rank"] = f.index + 1
>>> cols = ["product_id", "rank", "bm25_title", "dense", "clicks_log", "age_days", "score"]
>>> f.loc[(f["rank"] <= 3) | (f["product_id"] == "LK-68994"), cols].round(2)
product_id rank bm25_title dense clicks_log age_days score
0 LK-17212 1 4.04 0.93 2.56 208 3.73
1 LK-32609 2 4.04 0.91 2.48 587 3.05
2 LK-75196 3 4.04 0.91 2.94 187 3.00
64 LK-68994 65 3.77 0.92 0.00 6 -0.32

The content features are almost the same. Content features describe the product’s own text. The BM25 score of the title is 3.77, against 4.04 for the top 3. The dense similarity is 0.92, against 0.91 to 0.93. The difference is the clicks. clicks_log is log(1 + clicks): 2.56 means 12 clicks, and 0.00 means none. And the product is 6 days old, against 187 to 587 days.

What if the new product had the clicks of a typical top-10 product? We can try it: copy the median clicks_log and carts_log of the top 10 into its row, and score it again.

>>> row = f[f["product_id"] == "LK-68994"].copy()
>>> row[["clicks_log", "carts_log"]] = f.loc[f["rank"] <= 10, ["clicks_log", "carts_log"]].median().to_numpy()
>>> new_score = ltr.booster("judgments").predict(row[ltr.FEATURES])[0]
>>> round(new_score, 2), int((f["score"] > new_score).sum()) + 1
(np.float64(1.49), 7)

With about 7 clicks, the same product would be at rank 7 instead of 65. Its text is as good as the others. Only its history is missing.

A prior is a value you assume before you have data. A new product has no clicks yet. Zero is a bad guess: it says “nobody wants it”, when the truth is “nobody saw it”. A fairer guess is what a typical product of the same class gets.

We tried this as a measurement: each new product gets the median clicks and carts of the shown products of its class. The ranker stays the same. We ran this once for this lesson. It is not part of the project, so you see only our results below.

An exploration slot is a fixed place on the results page for something the system is not sure about. Here, the best new product among the 100 candidates goes to position 4. It is the “New arrivals” table of the bookshop. People see the new product; some click; the clicks go into the log; the next ranker learns from them.

The word “exploration” comes from a general trade-off. Exploitation shows what you already know works. Exploration shows something uncertain, to learn about it. A system that only exploits never learns about new items.

Here is explore on a list of six products, A to F. Only E is new. The slot is position 4.

Position Before After
1 A A
2 B B
3 C C
4 D E (new)
5 E (new) D
6 F F

The rule takes the best new product, E. If E is already at position 4 or higher, nothing changes. Here it is at position 5, so it moves to 4, and D moves down one place. Nothing else changes. If the list has no new product, nothing changes either.

The project’s code does exactly this:

def explore(results, slot=4):
"""An exploration slot: the best new product (added in the last two weeks) among the candidates goes to
position `slot`, so new products get seen and can earn clicks."""
added = _col("added")
new = [r for r in results if added[r[0]] >= NEW_FROM]
if not new or new[0] in results[:slot]:
return results
rest = [r for r in results if r[0] != new[0][0]]
return rest[: slot - 1] + [new[0]] + rest[slot - 1:]

NEW_FROM is 2025-10-13, two weeks before the end of the log.

First, the slot on “kids chair”:

Terminal window
search query "kids chair" --run ltr+explore -k 5
ltr+explore: "kids chair" -> 5 results (query W-131: judged results are marked)
1. LK-17212 gearld modern kids chair Kids Chairs 412.99 [exact]
2. LK-32609 raziel kids club chair Kids Chairs 500.99 [exact]
3. LK-75196 pamrapo kids cotton chair Kids Chairs 513.99 [exact]
4. LK-68994 flint hills club kids chair Kids Chairs 586.99 [exact]
5. LK-58375 little modern kids chair 78.99 [exact]
The top 5 for kids chair with ltr and with the exploration slot. With ltr, five exact kids chairs; the new product is at rank 65. With the slot, the new product flint hills club kids chair, exact, added 2025-10-20, is at rank 4; little modern kids chair moves to rank 5. Both lists score NDCG@10 1.000.
The exploration slot on "kids chair": an exact new product moves from rank 65 to rank 4. Here the slot costs nothing, because the new product is right.
Text description of the diagram

Two lists of five. Left, ltr, where the new product is at rank 65: gearld modern kids chair, raziel kids club chair, pamrapo kids cotton chair, little modern kids chair, ruckus kids chair, all exact. Right, ltr with the exploration slot: the same first three, then flint hills club kids chair (exact, a new product added 2025-10-20) at rank 4, then little modern kids chair. Both lists score NDCG@10 1.000.

The search page for kids chair with the run ltr+explore. The fourth result, flint hills club kids chair, LK-68994, Kids Chairs, 586.99 dollars, is in a blue box labelled Exploration slot: a new product.
The page with search serve --run ltr+explore. A shop would also mark the product as new, so people know why it is there. We added the blue box.

On this query the slot is free: the new product is exact. On all the test queries, it is not:

Terminal window
search compare ltr ltr+explore --by source
ltr+explore minus ltr on 241 test queries: ndcg@10 -0.016 (95% interval -0.020 to -0.012); better on 27, worse on 98, the same on 116
source queries ltr ltr+explore difference low high worse
code 20 1.000 1.000 0.000 0.000 0.000 0
planted 3 0.571 0.550 -0.021 -0.073 0.010 1
translated 59 0.474 0.454 -0.020 -0.028 -0.013 30
typo 40 0.615 0.604 -0.011 -0.022 -0.001 20
wands 119 0.729 0.711 -0.018 -0.025 -0.012 47

The slot puts a new product at position 4, even when no new product is right for the query. On 98 queries that pushes a better product down. The cost is 0.016 NDCG@10, a clear loss. The code queries keep 1.000: the code rule puts the coded product first, and the slot only touches position 4.

Now both ideas side by side. For the exact new products in the test candidates, where do they rank? And on how many test queries does a new product reach the top 10?

The 54 exact new products in the test candidates: their median rank under each design. Lower is better. The slot brings them to position 4; the prior hardly moves them.
Text description of the chart

A bar chart with four bars: the median rank of the 54 exact new products among the test queries’ candidates. hybrid-rrf 30.5; ltr 35.5; ltr with the class prior 34; ltr with the exploration slot 4.

Design Median rank of exact new products Test queries with a new product in the top 10 NDCG@10 against ltr
hybrid-rrf (no click features) 30.5 41 (a different ranker)
ltr 35.5 23 0
ltr + class prior 34 32 0.000 (−0.001 to +0.001)
ltr+explore (slot at 4) 4 193 −0.016 (−0.020 to −0.012)

Read the table from top to bottom:

  • Hybrid search has no click features, so new products are not punished: median rank 30.5. The ranker ltr pushes them down to 35.5, and shows a new product in the top 10 for only 23 queries instead of 41. This is the cold start, measured.
  • The class prior is fair and free: no change in NDCG@10, and 9 more queries show a new product. But it moves the median rank only from 35.5 to 34. Why so little? The median shown product of a class gets few clicks. The top 10 of a query are the popular ones: for “kids chair”, their median is about 7 clicks. A fair prior is not a boost.
  • The slot brings the median to 4 and shows a new product on 193 queries. It costs 0.016 NDCG@10.

So the prior removes an unfair penalty, and the slot buys exposure. They solve two parts of the problem, and you can use both.

One honest limit of our data: new products keep their WANDS ratings and review counts, because WANDS has no dates. In a real shop, a new product has no reviews either, so its cold start is even colder.

When to use an exploration slot, and what it costs

Section titled “When to use an exploration slot, and what it costs”

Use it when new products arrive often and a learned ranker uses popularity: shops, news, music, job ads. Without it, the catalogue’s new part stays invisible.

Limit it. The slot costs relevance on every page where the best new product is not right. Three common limits:

  • Only for good matches. Use the slot only when the new product is already in the top 20 of stage one, for example. We did not measure this version: try it yourself.
  • Only for a time. A product leaves the “new” pool after two weeks, or after a number of impressions.
  • Only on some traffic. Show the slot to a part of the searches, and compare clicks and carts with the rest. Test changes online shows how.

Measure the right thing. Offline NDCG@10 can only show the cost of the slot. The gain comes later: clicks that teach the ranker about new products. Only an online test can show that gain.

Common mistake

New products that never rise

Symptom: month after month, the same products fill the top of every page. New products sell badly, and the buyers stop ordering new lines.

Cause: the click feedback loop of Learn from clicks, with nothing to stop it. With ltr, an exact new product is at rank 35.5 in the median.

Diagnosis: compare the median rank of exact new products under a ranker without click features (hybrid-rrf, 30.5) and with them (ltr, 35.5). Count the new products with any impression in the log: here 35 of 600.

Fix: give new products a fair prior for their missing history, and an exploration slot with clear limits. Then check that the clicks they earn reach the next training.

Consequence in AI work: a system that learns from its own outputs needs a planned way to try new items. Exploration is that way, and its price must be measured like any other change.

The slot is at position 4. Before you run anything, predict: is position 2 more or less costly? Change the slot in business.py (the slot=4 default) to 2, and run:

Terminal window
search compare ltr ltr+explore

Put the slot back to 4 afterwards.

Our result, and what it shows
ltr+explore minus ltr on 241 test queries: ndcg@10 -0.026 (95% interval -0.033 to -0.021); better on 26, worse on 103, the same on 112

Position 2 costs 0.026, against 0.016 at position 4. A higher slot pushes a better product down from a place that matters more, and NDCG gives the top places the largest weight. It also gives the new product more attention. In the click model of the course, a product at position 2 is looked at twice as often as one at position 4. The right place is a business choice: how much relevance you pay for how much exposure.

Check your understanding

Omar says: “New products get no clicks, so people do not want them. Let us drop them from search.” What does the log really say?

Show the answer

It says that people did not see them. Only 35 of the 600 new products were ever shown in the synthetic log. A product that is never shown cannot be clicked. The ranker put an exact new product, the flint hills club kids chair, at rank 65. The reason was its missing clicks, not its text. No clicks means no data, not a bad product.

Check your understanding

The exploration slot costs 0.016 NDCG@10 on the test queries. Why can an offline test not show its benefit?

Show the answer

The benefit is future learning: new products get impressions and clicks, and the next ranker learns which of them are good. The offline test uses fixed judgments and a fixed ranker, so it can only see the cost of moving a product to position 4. An online test that runs over weeks and retrains the ranker is needed to see the gain.

Check your understanding

Why did giving new products the median clicks of their class change so little?

Show the answer

Most products of a class get few clicks, so the median is low. The top 10 of a query are the popular products: for “kids chair”, their median is about 7 clicks. A fair prior removes the penalty of “zero clicks”, but it does not make a new product look as popular as the best sellers. That is why the slot is still needed for exposure.

Recap

  • The cold start: a new product has no clicks, so a ranker that uses clicks puts it low, so it gets no clicks. In the synthetic log, only 35 of 600 new products were ever shown.
  • Content features (BM25, dense) treat a new product fairly. The click features do not: an exact new kids chair is at rank 65 with ltr, and at rank 7 with typical clicks.
  • A class prior for the missing history is fair and free (NDCG@10 change 0.000), but it moves new products little.
  • An exploration slot at position 4 brings exact new products from a median rank of 35.5 to 4, for 0.016 NDCG@10. Limit it, and measure its benefit online.
  • The search page now has autocomplete, facets, a fallback, business rules and a slot for new products.

Every part of this module adds work to each search: suggestions, counts, rules. In the next module, you measure what each stage of a search costs in time and money. Then you decide what fits Amira’s budget of 200 ms.

Next: Latency and cost budgets