The baselines to beat
- Lesson 7
- 35 min
- Module 2: Data, implicit feedback and an honest split
Where we are
We have an honest time split. Before we try clever methods, we need a score to beat. In this lesson, we build three popularity baselines and see what each one scores, and what each one gets wrong.
Lesson 7 of 29 · Recommendation and Personalisation: Show Each Person What MattersThe baselines to beat
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Set the scores that every clever method must beat, with three popularity baselines on the honest time split. See what each one gets right, and what each one gets wrong, so that a new method has to earn its place.
NotebookDo this lesson in a notebook
The notebook scores popularity, shows its top 10 with the winter coat, draws the coat orders month by month, and asks you to build trending. Its setup cells install the packages and download the data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The time split (train before 22 June 2026, test on the orders of the next two weeks), from Split by time.
- Recall@10 and the popularity baseline, from Meet the shop log.
What you need on your computer
- Your
larkfield-recsproject withrecs matrixandrecs leakagefrom the last two lessons, and the labs repository next to it.
After this lesson, your project matches the snapshot end-of-m02. The shop log is synthetic, and the winter coats are a practice problem we made on purpose. The course team measured the numbers on an Apple M4 laptop. Tested on macOS; the Windows commands are not tested.
The new assistant’s first day
Section titled “The new assistant’s first day”On her first day, a new shop assistant knows none of the customers. Still, she can help. A customer asks for an idea, and she points to the table at the door: “These are our bestsellers.” It is not personal, but it is often right, because many people like the same things.
After a week, she has two better ideas. First: “This week, everyone buys garden tables.” Recent bestsellers follow the season. Second: a customer asks about cushions, so she shows the best-selling cushions, not the best-selling garden gloves. She uses the customer’s favourite kind of product.
These three ideas need no clever model. They are the baselines of this lesson.
The problem: a score with nothing to compare
Section titled “The problem: a score with nothing to compare”Tomás shows his first personal model to Omar: “recall@10 is 0.12 on the time split.” Omar asks the right question: “Is that good?”
Nobody can say. A number alone means little. Random products score 0.004: almost nothing. If showing everyone the same bestsellers scores 0.106, then 0.12 is a small gain for a lot of work. If the bestsellers scored 0.02, then 0.12 would be a big step. So before any clever method, we measure the simplest methods that are still useful. A baseline is such a method. Its score is the bar that a new method must clear. A method that does not beat it is not worth its cost.
Three popularity baselines
Section titled “Three popularity baselines”All three count customers, not order lines: a product’s popularity is the number of different customers who ordered it in the training data. So one customer who orders 40 bags of compost counts once.
| Baseline | The list | Personal? |
|---|---|---|
popular |
the 10 products that the most customers ordered, over the whole training log | no: everyone gets the same list |
trending |
the same, but only the orders of the 28 days before the period | no: everyone gets the same list |
popular-category |
the most ordered products of the customer’s favourite categories, in proportion to the customer’s events | a little: one list per mix of categories |
A tiny example: one customer’s popular-category list
Section titled “A tiny example: one customer’s popular-category list”Popular-category shares the 10 places between the customer’s categories. Here is a constructed customer with 20 points of events (with the weights view 1, basket 3, order 5):
| Category | Points | Share | Places of 10 |
|---|---|---|---|
| Garden furniture | 12 | 60% | 6 |
| Lighting | 6 | 30% | 3 |
| Kitchen | 2 | 10% | 1 |
The list then has the 6 bestsellers of garden furniture, the 3 bestsellers of lighting and the bestseller of kitchen. A customer with no events gets the plain popular list.
The scores
Section titled “The scores”Here are the three baselines and random suggestions on the test period, with their 95% intervals. An interval comes from scoring many resamples of the 2,118 customers (a bootstrap). Two methods whose intervals overlap a lot may not really differ.
Text description of the chart
Four rows. random: 0.004 (0.002 to 0.007). popular: 0.106 (0.095 to 0.118). popular-category: 0.143 (0.129 to 0.157). trending: 0.156 (0.143 to 0.172). The intervals of popular-category and trending overlap a lot; popular is clearly below both, and random is close to zero.
Three things to see:
- Popularity is far better than random: 0.106 against 0.004. Many people want the same few products. This is the long tail of Meet the shop log at work.
- Recent beats all-time: trending (0.156) beats popular (0.106) clearly. The test period is late June, and the last 28 days know it.
- Popular-category (0.143) and trending (0.156) are close. Their intervals overlap, so we cannot say that one is better for all customers. The segments show where each one wins:
| Segment (test customers) | popular | popular-category | trending |
|---|---|---|---|
| new: no order before (952) | 0.108 | 0.118 | 0.169 |
| occasional: 1 or 2 orders (556) | 0.110 | 0.161 | 0.162 |
| loyal: 3 or more orders (610) | 0.098 | 0.164 | 0.132 |
Popular-category helps the customers with a history: their favourite categories mean something. For new customers, it has little to work with, and the season matters more.
What popularity gets wrong: a winter coat in July
Section titled “What popularity gets wrong: a winter coat in July”Here is the popular list on the test period. For each product, the table shows how many customers ordered it before 22 June 2026, and how many in the test period.
| # | Product | Subcategory | Ordered by (before) | Ordered in the test period |
|---|---|---|---|---|
| 1 | Navy Nitrile-Grip Garden Gloves | Gloves and kneelers | 1,334 | 59 |
| 2 | Terracotta Linen Roman-Style Curtains | Curtains | 1,143 | 48 |
| 3 | Black Rustic Wooden Planter Box | Pots and planters | 1,013 | 17 |
| 4 | Green Waterproof Chair Protector | Continence care | 983 | 34 |
| 5 | Dockside Red Striped Cushion | Cushions | 949 | 43 |
| 6 | Copper Corduroy Duffle Coat | Winter coats | 857 | 0 |
| 7 | Terracotta Ankle Garden Clogs | Garden boots | 696 | 16 |
| 8 | Oak Timber Swing Seat | Hammocks and swings | 643 | 10 |
| 9 | Bramble Terracotta Mosaic Side Table | Garden tables | 534 | 106 |
| 10 | Dockside Red Loft Bed | Kids’ beds | 525 | 16 |
Number 6 is a duffle coat. 857 customers ordered it, so it is one of the shop’s bestsellers of all time. In the test period, nobody ordered it. This is a practice problem we made on purpose. The chart shows why:
Text description of the chart
A line with one point per month, from February 2025 to July 2026. Winter coats sell in winter. There are 170 order lines in February 2025, then almost none from April to August 2025 (1 to 3 a month). Then 150 in October, 181 in November and 159 in December 2025, 271 in January and 231 in February 2026. Then the sales fall: 36 in March, 12 in April and 1 in May. June and July 2026, the months of the test period, have 0.
An all-time count adds up two winters of coat orders. It does not know that the season is over. Trending counts only the last 28 days, so the coat drops out of its list. Its top product is a garden coffee table, ordered by 133 test customers.
Two more products in this list need care later:
- The swing seat (number 8 here, and in the trending list too): its stock flag says out of stock from 15 June 2026. The 10 test-period orders in the table are back-orders in our synthetic log. A baseline does not check the stock. This is another practice problem we made on purpose; Business rules and re-ranking handles it.
- Number 4 is a continence care product: a sensitive health product, bought again and again by the people who need it. That makes it popular. Even a home page that shows the same list to everyone would show it. Privacy and sensitive suggestions decides where such products may appear.
Common mistake
A baseline that is not run on the same split
Tomás reports that his model “beats popularity by 30%”. He scored his model on the time split, but he took the popularity score from a blog post about another shop. The comparison means nothing. A baseline must use the same split, the same customers and the same measure as the method it is compared with. In this course, every row of the comparison table is scored with recs evaluate on the same test period. The harness makes it hard to compare in any other way.
Popularity is hard to beat
Section titled “Popularity is hard to beat”Here is a fact from the rest of this course, measured on the same test period. Trending (0.156) beats every single personal method of the course’s comparison table. Item-cf scores 0.088, user-user 0.145 and tuned ALS 0.149. Only two methods beat it clearly. One is a hybrid blend of several methods, with its weights tuned on the validation period (0.193). The other is the two-stage ranker of Why two stages (0.234).
This is not special to Larkfield. In many real shops, a recent bestseller list is a strong baseline, because most customers have very little history. More than half of Larkfield’s buyers ordered once. A personal method needs history to be personal.
So the team keeps two rules:
- Every new method is compared with trending, not only with
popular. - Trending is the fallback: the list for a customer that a method cannot serve, such as a new visitor with no events.
When to use popularity baselines, and what they cost
Section titled “When to use popularity baselines, and what they cost”| Use them | Their limits | |
|---|---|---|
| Cost | almost nothing: a count, under a second | none worth measuring |
| Strength | strong when histories are short and seasons matter | the same list for everyone: coverage is 0.3% of the products on sale |
| Weakness | no discovery: the long tail never appears; the old season stays in an all-time list |
Coverage is the share of the products on sale that appear in at least one customer’s list. popular and trending show 10 products to everyone: 0.3% coverage. popular-category shows 6.7%. A shop that only ever shows its 10 bestsellers sells them more, and learns nothing about the other 2,984 products. Beyond accuracy measures this.
The project: three baselines
Section titled “The project: three baselines”The baselines are one new file of the snapshot end-of-m02. Methods are found by their files, like the commands.
-
In your project folder, with the virtual environment active, copy the baselines, the tests of this module and the README from the snapshot:
Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m02for f in larkfield_recs/methods/baselines.py tests/test_m02.py README.md; do cp "$S/$f" "$f"; doneTerminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m02"foreach ($f in 'larkfield_recs\methods\baselines.py', 'tests\test_m02.py', 'README.md') { Copy-Item "$S\$f" $f } -
Run the tests:
python -m pytest -q. Expect11 passed. -
List the methods:
recs evaluate --list. -
Score the three baselines:
recs evaluate popular, thenrecs evaluate trending, thenrecs evaluate popular-category. -
Open
larkfield_recs/methods/baselines.py. Find wherepopular-categoryshares the 10 places between categories.
Show the expected result
Step 3 prints:
trending the most ordered products of the last 28 dayspopular-category the most ordered products of each customer's favourite categoriespopular the most ordered products over the whole training lograndom random products on saleStep 4 prints, for trending:
Method trending (the most ordered products of the last 28 days), test periodCustomers scored: 2,118; trained on events before 2026-06-22recall@10 0.156 (95% interval 0.143 to 0.172), NDCG@10 0.100coverage 0.3% of the products on saletime: fit 0.0 s, suggestions 0.0 sFor popular: recall@10 0.106 (95% interval 0.095 to 0.118), NDCG@10 0.061 and coverage 0.3%. For popular-category: recall@10 0.143 (95% interval 0.129 to 0.157), NDCG@10 0.085 and coverage 6.7%. Each run takes less than a second. NDCG@10 also gives points for the place of a right product in the list; Ranking metrics explains it.
Now recs leakage shows popular-category too (+59%), as in Split by time.
Guided practice: popular in the notebook
Section titled “Guided practice: popular in the notebook”- Open the notebook of this lesson and run the setup cells.
- Run section 1. It scores
popularand shows its top 10. Find number 6. - Run section 2: the winter coat orders by month.
Show the expected result
Section 1 prints popular, recall@10 on the test period: 0.106 and the list of the table above, with the duffle coat P10167 at number 6. Section 2 draws the chart of this lesson: coat orders in winter, none in June and July 2026.
Your turn: trending, then a better mix
Section titled “Your turn: trending, then a better mix”Part 1. In section 3 of the notebook, build trending: the top 10 of the orders of the 28 days before the test period. Predict first: higher or lower than popular? Then run the check.
Part 2. Popular-category counts popularity over the whole log. Combine it with the idea of trending: count only the last 28 days. In the project, run recs evaluate popular-category --set days=28. Predict first: better than trending, or not?
Show the answer
Part 1: recall@10 0.156, the same as recs evaluate trending. The check prints Check passed: trending_recall is 0.156. The coat is gone; a garden coffee table is at the top.
recent = orders[orders["day"] >= first_test_day - 28]trending = top10(recent)trending_recall = mean_recall(trending, truth)Part 2 prints (the course team’s run):
Method popular-category (the most ordered products of each customer's favourite categories), test period, settings {'days': 28}Customers scored: 2,118; trained on events before 2026-06-22recall@10 0.173 (95% interval 0.159 to 0.188), NDCG@10 0.109coverage 6.7% of the products on saletime: fit 0.0 s, suggestions 0.0 s0.173: better than trending (0.156), and better for all three segments (new 0.163, occasional 0.183, loyal 0.180). The customer’s favourite categories and the season together make a stronger baseline than each idea alone. The length of the window matters much less. Trending with 7, 14 or 56 days scores 0.158, 0.157 and 0.158. Try recs evaluate trending --set days=7 (the method’s description still says 28 days).
This mix is a hint of what later modules do with more care: combine several signals. Keep trending as the baseline of the comparison table, as the rest of the course does, and remember that a simple mix already reached 0.173.
Knowledge check
Section titled “Knowledge check”Check your understanding
What is a baseline, and why must Larkfield measure one before trying a personal model? Use the numbers of this lesson.
Show the answer
A baseline is the simplest method that is still useful; its score is the bar a new method must clear. Without it, a score such as 0.12 means nothing. On the same test period, random suggestions score 0.004, popular 0.106 and trending 0.156. A personal model at 0.12 would be worse than trending, which costs almost nothing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: the duffle coat is number 6 of popular, but it is not in the trending list. Why? What does this tell you about all-time counts?
Show the answer
857 customers ordered the coat, but in winter: 0 orders in June and July 2026. An all-time count adds up past seasons and cannot see that the season is over. Trending counts only the 28 days before the test period, so the coat drops out. All-time counts lag behind seasons and trends; a recent window follows them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Popular-category scores 0.164 for loyal customers and trending 0.132. For new customers, it is the other way round (0.118 against 0.169). Explain the difference.
Show the answer
Popular-category uses the customer’s favourite categories. Loyal customers have long histories, so their categories mean something and the list fits them. New customers have no orders and few events, so their category mix is weak or empty. For them the season matters more, and trending’s recent bestsellers fit them better. This is why we look at segments, not only at one average.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A baseline is the simplest useful method. Every new method must beat it on the same split, customers and measure.
- On the test period: random 0.004, popular 0.106, popular-category 0.143, trending 0.156 (recall@10).
popularshows a winter coat at number 6 in July: 857 buyers, 0 orders in the test period. All-time counts do not follow seasons.- Popular-category helps customers with a history (loyal 0.164); trending helps new customers (0.169).
- Popularity is hard to beat. Trending beats every single personal method of the course’s comparison table. Only a tuned hybrid blend (0.193) and the two-stage ranker (0.234) beat it clearly. Trending is the comparison and the fallback.
Next, we look closer at how to score a ranked list: precision, recall, MAP and NDCG.
Next: Ranking metrics