Skip to content
Français

The baselines to beat

  • Lesson 7
  • 35 min
  • Module 2: Data, implicit feedback and an honest split

Where we are

We have an honest time split. Before we try clever methods, we need a score to beat. In this lesson, we build three popularity baselines and see what each one scores, and what each one gets wrong.

What you will be able to do

Set the scores that every clever method must beat, with three popularity baselines on the honest time split. See what each one gets right, and what each one gets wrong, so that a new method has to earn its place.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook scores popularity, shows its top 10 with the winter coat, draws the coat orders month by month, and asks you to build trending. Its setup cells install the packages and download the data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • The time split (train before 22 June 2026, test on the orders of the next two weeks), from Split by time.
  • Recall@10 and the popularity baseline, from Meet the shop log.

What you need on your computer

  • Your larkfield-recs project with recs matrix and recs leakage from the last two lessons, and the labs repository next to it.

After this lesson, your project matches the snapshot end-of-m02. The shop log is synthetic, and the winter coats are a practice problem we made on purpose. The course team measured the numbers on an Apple M4 laptop. Tested on macOS; the Windows commands are not tested.

On her first day, a new shop assistant knows none of the customers. Still, she can help. A customer asks for an idea, and she points to the table at the door: “These are our bestsellers.” It is not personal, but it is often right, because many people like the same things.

After a week, she has two better ideas. First: “This week, everyone buys garden tables.” Recent bestsellers follow the season. Second: a customer asks about cushions, so she shows the best-selling cushions, not the best-selling garden gloves. She uses the customer’s favourite kind of product.

These three ideas need no clever model. They are the baselines of this lesson.

The problem: a score with nothing to compare

Section titled “The problem: a score with nothing to compare”

Tomás shows his first personal model to Omar: “recall@10 is 0.12 on the time split.” Omar asks the right question: “Is that good?”

Nobody can say. A number alone means little. Random products score 0.004: almost nothing. If showing everyone the same bestsellers scores 0.106, then 0.12 is a small gain for a lot of work. If the bestsellers scored 0.02, then 0.12 would be a big step. So before any clever method, we measure the simplest methods that are still useful. A baseline is such a method. Its score is the bar that a new method must clear. A method that does not beat it is not worth its cost.

All three count customers, not order lines: a product’s popularity is the number of different customers who ordered it in the training data. So one customer who orders 40 bags of compost counts once.

Baseline The list Personal?
popular the 10 products that the most customers ordered, over the whole training log no: everyone gets the same list
trending the same, but only the orders of the 28 days before the period no: everyone gets the same list
popular-category the most ordered products of the customer’s favourite categories, in proportion to the customer’s events a little: one list per mix of categories
Section titled “A tiny example: one customer’s popular-category list”

Popular-category shares the 10 places between the customer’s categories. Here is a constructed customer with 20 points of events (with the weights view 1, basket 3, order 5):

Category Points Share Places of 10
Garden furniture 12 60% 6
Lighting 6 30% 3
Kitchen 2 10% 1

The list then has the 6 bestsellers of garden furniture, the 3 bestsellers of lighting and the bestseller of kitchen. A customer with no events gets the plain popular list.

Here are the three baselines and random suggestions on the test period, with their 95% intervals. An interval comes from scoring many resamples of the 2,118 customers (a bootstrap). Two methods whose intervals overlap a lot may not really differ.

recall@10 on the test period (2,118 customers, synthetic log), with 95% intervals (bootstrap over customers).
Text description of the chart

Four rows. random: 0.004 (0.002 to 0.007). popular: 0.106 (0.095 to 0.118). popular-category: 0.143 (0.129 to 0.157). trending: 0.156 (0.143 to 0.172). The intervals of popular-category and trending overlap a lot; popular is clearly below both, and random is close to zero.

Three things to see:

  1. Popularity is far better than random: 0.106 against 0.004. Many people want the same few products. This is the long tail of Meet the shop log at work.
  2. Recent beats all-time: trending (0.156) beats popular (0.106) clearly. The test period is late June, and the last 28 days know it.
  3. Popular-category (0.143) and trending (0.156) are close. Their intervals overlap, so we cannot say that one is better for all customers. The segments show where each one wins:
Segment (test customers) popular popular-category trending
new: no order before (952) 0.108 0.118 0.169
occasional: 1 or 2 orders (556) 0.110 0.161 0.162
loyal: 3 or more orders (610) 0.098 0.164 0.132

Popular-category helps the customers with a history: their favourite categories mean something. For new customers, it has little to work with, and the season matters more.

What popularity gets wrong: a winter coat in July

Section titled “What popularity gets wrong: a winter coat in July”

Here is the popular list on the test period. For each product, the table shows how many customers ordered it before 22 June 2026, and how many in the test period.

# Product Subcategory Ordered by (before) Ordered in the test period
1 Navy Nitrile-Grip Garden Gloves Gloves and kneelers 1,334 59
2 Terracotta Linen Roman-Style Curtains Curtains 1,143 48
3 Black Rustic Wooden Planter Box Pots and planters 1,013 17
4 Green Waterproof Chair Protector Continence care 983 34
5 Dockside Red Striped Cushion Cushions 949 43
6 Copper Corduroy Duffle Coat Winter coats 857 0
7 Terracotta Ankle Garden Clogs Garden boots 696 16
8 Oak Timber Swing Seat Hammocks and swings 643 10
9 Bramble Terracotta Mosaic Side Table Garden tables 534 106
10 Dockside Red Loft Bed Kids’ beds 525 16

Number 6 is a duffle coat. 857 customers ordered it, so it is one of the shop’s bestsellers of all time. In the test period, nobody ordered it. This is a practice problem we made on purpose. The chart shows why:

Order lines of all winter coats by month (synthetic log, planted). The coats sell from October to February; in June and July 2026, nobody ordered one.
Text description of the chart

A line with one point per month, from February 2025 to July 2026. Winter coats sell in winter. There are 170 order lines in February 2025, then almost none from April to August 2025 (1 to 3 a month). Then 150 in October, 181 in November and 159 in December 2025, 271 in January and 231 in February 2026. Then the sales fall: 36 in March, 12 in April and 1 in May. June and July 2026, the months of the test period, have 0.

An all-time count adds up two winters of coat orders. It does not know that the season is over. Trending counts only the last 28 days, so the coat drops out of its list. Its top product is a garden coffee table, ordered by 133 test customers.

Two more products in this list need care later:

  • The swing seat (number 8 here, and in the trending list too): its stock flag says out of stock from 15 June 2026. The 10 test-period orders in the table are back-orders in our synthetic log. A baseline does not check the stock. This is another practice problem we made on purpose; Business rules and re-ranking handles it.
  • Number 4 is a continence care product: a sensitive health product, bought again and again by the people who need it. That makes it popular. Even a home page that shows the same list to everyone would show it. Privacy and sensitive suggestions decides where such products may appear.

Common mistake

A baseline that is not run on the same split

Tomás reports that his model “beats popularity by 30%”. He scored his model on the time split, but he took the popularity score from a blog post about another shop. The comparison means nothing. A baseline must use the same split, the same customers and the same measure as the method it is compared with. In this course, every row of the comparison table is scored with recs evaluate on the same test period. The harness makes it hard to compare in any other way.

Here is a fact from the rest of this course, measured on the same test period. Trending (0.156) beats every single personal method of the course’s comparison table. Item-cf scores 0.088, user-user 0.145 and tuned ALS 0.149. Only two methods beat it clearly. One is a hybrid blend of several methods, with its weights tuned on the validation period (0.193). The other is the two-stage ranker of Why two stages (0.234).

This is not special to Larkfield. In many real shops, a recent bestseller list is a strong baseline, because most customers have very little history. More than half of Larkfield’s buyers ordered once. A personal method needs history to be personal.

So the team keeps two rules:

  1. Every new method is compared with trending, not only with popular.
  2. Trending is the fallback: the list for a customer that a method cannot serve, such as a new visitor with no events.

When to use popularity baselines, and what they cost

Section titled “When to use popularity baselines, and what they cost”
Use them Their limits
Cost almost nothing: a count, under a second none worth measuring
Strength strong when histories are short and seasons matter the same list for everyone: coverage is 0.3% of the products on sale
Weakness no discovery: the long tail never appears; the old season stays in an all-time list

Coverage is the share of the products on sale that appear in at least one customer’s list. popular and trending show 10 products to everyone: 0.3% coverage. popular-category shows 6.7%. A shop that only ever shows its 10 bestsellers sells them more, and learns nothing about the other 2,984 products. Beyond accuracy measures this.

The baselines are one new file of the snapshot end-of-m02. Methods are found by their files, like the commands.

  1. In your project folder, with the virtual environment active, copy the baselines, the tests of this module and the README from the snapshot:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m02
    for f in larkfield_recs/methods/baselines.py tests/test_m02.py README.md; do cp "$S/$f" "$f"; done
  2. Run the tests: python -m pytest -q. Expect 11 passed.

  3. List the methods: recs evaluate --list.

  4. Score the three baselines: recs evaluate popular, then recs evaluate trending, then recs evaluate popular-category.

  5. Open larkfield_recs/methods/baselines.py. Find where popular-category shares the 10 places between categories.

Show the expected result

Step 3 prints:

trending the most ordered products of the last 28 days
popular-category the most ordered products of each customer's favourite categories
popular the most ordered products over the whole training log
random random products on sale

Step 4 prints, for trending:

Method trending (the most ordered products of the last 28 days), test period
Customers scored: 2,118; trained on events before 2026-06-22
recall@10 0.156 (95% interval 0.143 to 0.172), NDCG@10 0.100
coverage 0.3% of the products on sale
time: fit 0.0 s, suggestions 0.0 s

For popular: recall@10 0.106 (95% interval 0.095 to 0.118), NDCG@10 0.061 and coverage 0.3%. For popular-category: recall@10 0.143 (95% interval 0.129 to 0.157), NDCG@10 0.085 and coverage 6.7%. Each run takes less than a second. NDCG@10 also gives points for the place of a right product in the list; Ranking metrics explains it.

Now recs leakage shows popular-category too (+59%), as in Split by time.

  1. Open the notebook of this lesson and run the setup cells.
  2. Run section 1. It scores popular and shows its top 10. Find number 6.
  3. Run section 2: the winter coat orders by month.
Show the expected result

Section 1 prints popular, recall@10 on the test period: 0.106 and the list of the table above, with the duffle coat P10167 at number 6. Section 2 draws the chart of this lesson: coat orders in winter, none in June and July 2026.

Part 1. In section 3 of the notebook, build trending: the top 10 of the orders of the 28 days before the test period. Predict first: higher or lower than popular? Then run the check.

Part 2. Popular-category counts popularity over the whole log. Combine it with the idea of trending: count only the last 28 days. In the project, run recs evaluate popular-category --set days=28. Predict first: better than trending, or not?

Show the answer

Part 1: recall@10 0.156, the same as recs evaluate trending. The check prints Check passed: trending_recall is 0.156. The coat is gone; a garden coffee table is at the top.

recent = orders[orders["day"] >= first_test_day - 28]
trending = top10(recent)
trending_recall = mean_recall(trending, truth)

Part 2 prints (the course team’s run):

Method popular-category (the most ordered products of each customer's favourite categories), test period, settings {'days': 28}
Customers scored: 2,118; trained on events before 2026-06-22
recall@10 0.173 (95% interval 0.159 to 0.188), NDCG@10 0.109
coverage 6.7% of the products on sale
time: fit 0.0 s, suggestions 0.0 s

0.173: better than trending (0.156), and better for all three segments (new 0.163, occasional 0.183, loyal 0.180). The customer’s favourite categories and the season together make a stronger baseline than each idea alone. The length of the window matters much less. Trending with 7, 14 or 56 days scores 0.158, 0.157 and 0.158. Try recs evaluate trending --set days=7 (the method’s description still says 28 days).

This mix is a hint of what later modules do with more care: combine several signals. Keep trending as the baseline of the comparison table, as the rest of the course does, and remember that a simple mix already reached 0.173.

Check your understanding

What is a baseline, and why must Larkfield measure one before trying a personal model? Use the numbers of this lesson.

Show the answer

A baseline is the simplest method that is still useful; its score is the bar a new method must clear. Without it, a score such as 0.12 means nothing. On the same test period, random suggestions score 0.004, popular 0.106 and trending 0.156. A personal model at 0.12 would be worse than trending, which costs almost nothing.

Check your understanding

Predict and explain: the duffle coat is number 6 of popular, but it is not in the trending list. Why? What does this tell you about all-time counts?

Show the answer

857 customers ordered the coat, but in winter: 0 orders in June and July 2026. An all-time count adds up past seasons and cannot see that the season is over. Trending counts only the 28 days before the test period, so the coat drops out. All-time counts lag behind seasons and trends; a recent window follows them.

Check your understanding

Popular-category scores 0.164 for loyal customers and trending 0.132. For new customers, it is the other way round (0.118 against 0.169). Explain the difference.

Show the answer

Popular-category uses the customer’s favourite categories. Loyal customers have long histories, so their categories mean something and the list fits them. New customers have no orders and few events, so their category mix is weak or empty. For them the season matters more, and trending’s recent bestsellers fit them better. This is why we look at segments, not only at one average.

Recap

  • A baseline is the simplest useful method. Every new method must beat it on the same split, customers and measure.
  • On the test period: random 0.004, popular 0.106, popular-category 0.143, trending 0.156 (recall@10).
  • popular shows a winter coat at number 6 in July: 857 buyers, 0 orders in the test period. All-time counts do not follow seasons.
  • Popular-category helps customers with a history (loyal 0.164); trending helps new customers (0.169).
  • Popularity is hard to beat. Trending beats every single personal method of the course’s comparison table. Only a tuned hybrid blend (0.193) and the two-stage ranker (0.234) beat it clearly. Trending is the comparison and the fallback.

Next, we look closer at how to score a ranked list: precision, recall, MAP and NDCG.

Next: Ranking metrics