Evaluate by segment
- Lesson 10
- 35 min
- Module 3: Measure recommendations offline
Where we are
In the last two lessons, we measured how accurate a list is and how much of the shop it shows. Every one of those numbers is an average over 2,118 customers, and an average can hide a group that a method fails. In this lesson, we measure new, occasional and loyal customers apart, and new products apart, starting from a class whose average went up.
Lesson 10 of 29 · Recommendation and Personalisation: Show Each Person What MattersEvaluate by segment
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Find the customers and products that one average hides. Measure new, occasional and loyal customers apart, compare two methods on the same customers with an interval, and check new products on their own.
NotebookDo this lesson in a notebook
The notebook splits the test customers into the three segments, scores the popular and trending lists in each one, and computes bootstrap intervals. You compare the two lists on the same customers, segment by segment, and check the new products. Its setup cells download about 8 MB of practice data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Recall@10 and the bootstrap interval over customers, from Ranking metrics.
- New products in the lists, from Beyond accuracy.
- Why a random split makes the test customers heavier buyers, from Split by time.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject with the four files of the last lesson.recs evaluate popularprints novelty and diversity. - The practice files in
../nextia-learning-labs, beside the project folder.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.
A class whose average went up
Section titled “A class whose average went up”A school tries a new way of teaching mathematics in one class. Half of the students are strong, and half started the year behind. At the end of the year, the class average went up from 62.5% to 65%. The head teacher is happy.
Then a teacher looks at the students one group at a time. The strongest students gained a lot: their marks went from 80% to 90%. The students who started the year behind lost ground: from 45% to 40%. The average rose because the strong students gained more than the weak students lost. The new method helps the students who need help least, and the average says the opposite of the truth for the weak students.
A recommender has the same kind of groups. A loyal customer with 20 orders gives a method a lot to learn from. A new customer gives it nothing. One average over both can hide a method that fails every new customer.
The problem: “about the same” is not the whole story
Section titled “The problem: “about the same” is not the whole story”Look at two baselines from Module 2. On all 2,118 test customers, trending has recall@10 0.156 and popular-category has 0.143. The project can compare them on the same customers, which you learn in this lesson. It prints:
recall@10: trending 0.156, popular-category 0.143difference -0.014 (95% interval -0.030 to +0.002); better for 246 customers, worse for 279, the same for 1,593The difference is not clear: the interval includes 0.So the two baselines are “about the same”, and Tomás says that it does not matter which one Larkfield uses. Now split the same customers by how many orders they made before the test period:
Text description of the diagram
The diagram starts with the 2,118 test customers, the customers who ordered in the test period. Arrows split them by their orders before the period. With 0 orders, 952 new customers: trending has recall@10 0.169 and popular-category 0.118. With 1 or 2 orders, 556 occasional customers: trending 0.162 and popular-category 0.161. With 3 or more orders, 610 loyal customers: trending 0.132 and popular-category 0.164. The three groups lead to one box with a thick border, all customers together: trending 0.156 and popular-category 0.143.
For new customers, popular-category is clearly worse. For loyal customers, it is better. The two effects almost cancel in the average. Suppose Larkfield chose one method for everybody from the average. Then 952 new customers or 610 loyal ones would get a worse home page.
The idea: segments
Section titled “The idea: segments”A segment is a group of customers, or of products, that we measure apart because a method may work differently for it. You choose segments that the business cares about, before you look at the results.
This course uses three customer segments, by the number of orders before the period. An order is one visit with an order, even if it had several products.
| Segment | Orders before the period | Test customers | What a method can learn about them |
|---|---|---|---|
| New | 0 (they may have viewed products) | 952 | only their views, or nothing |
| Occasional | 1 or 2 | 556 | one or two baskets of products |
| Loyal | 3 or more | 610 | a real history |
Almost half of the test customers are new. A customer with no history is the cold-start problem: the method has nothing to warm up from. In Module 1 you saw that most Larkfield buyers order only once, so new customers are not a corner case here. They are the largest group.
We also measure one product segment: new products, added to the shop in the 60 days before the period. They have little history, so methods that learn from history rarely show them. That is the cold-start problem for products.
A tiny example
Section titled “A tiny example”Six customers and two lists, with the recall@10 of each customer (a tiny example we made):
| Customer | Segment | List A | List B |
|---|---|---|---|
| Ana | new | 0.5 | 0.0 |
| Ben | new | 0.5 | 0.0 |
| Chen | new | 0.0 | 0.0 |
| Dara | loyal | 0.0 | 1.0 |
| Eli | loyal | 0.0 | 0.5 |
| Femi | loyal | 0.5 | 1.0 |
On average, list B wins: 0.417 against 0.250. But for the new customers, list A has 0.333 and list B has 0. List B never finds anything for a new customer, and the average does not show it.
What each baseline does for each segment
Section titled “What each baseline does for each segment”The chart shows recall@10 of the three non-personal baselines in each segment, with the bootstrap interval of each one.
Text description of the chart
Three panels, one for each segment, each with three rows and a horizontal axis for recall@10 from 0 to 0.25. New customers (952): popular 0.108 (0.090 to 0.125), trending 0.169 (0.147 to 0.192), popular-category 0.118 (0.100 to 0.138). Occasional customers (556): popular 0.110, trending 0.162, popular-category 0.161; the three intervals overlap. Loyal customers (610): popular 0.098, trending 0.132 (0.111 to 0.156), popular-category 0.164 (0.139 to 0.192).
Popular-category gives each customer the bestsellers of her favourite categories. A loyal customer has clear favourites, so this works. A new customer has at most a few views, and often none, so her “favourite categories” say little. Trending works for everybody in the same way, and the new customers of late June buy what is selling in late June.
Note that the intervals are wider than on all customers. Trending has 0.143 to 0.172 on 2,118 customers, and 0.147 to 0.192 on the 952 new ones. A segment has fewer customers, so its average depends more on chance.
Compare on the same customers
Section titled “Compare on the same customers”Two separate intervals that overlap do not prove that two methods are the same. There is a more exact way, because both methods scored the same customers.
Think of two pairs of running shoes. You could give one pair to one group of runners and the other pair to another group, and compare the groups’ times. But the groups differ: some runners are just faster. It is better to let each runner try both pairs, and look at the difference for each runner. The speed of the runner cancels out.
The project does this. For each customer, it takes recall@10 of method B minus recall@10 of method A. Then it bootstraps these differences, as in Ranking metrics, and gives a 95% interval for the mean difference. This is the paired difference. If the interval does not include 0, the difference is clear. The evaluation course explains it in more depth in Uncertainty and slices, part of Evaluating and Testing AI Systems.
Text description of the chart
Two panels, each with a vertical line at zero. First panel, trending minus popular: all customers +0.051 (+0.041 to +0.061); new +0.061 (+0.045 to +0.077); occasional +0.052 (+0.033 to +0.073); loyal +0.034 (+0.018 to +0.049). Every interval is above zero. Second panel, popular-category minus trending: all customers −0.014 (−0.030 to +0.002), which includes zero; new −0.051 (−0.073 to −0.030), all below zero; occasional −0.001 (−0.037 to +0.035), around zero; loyal +0.032 (−0.001 to +0.065), which just touches zero.
The first panel is the simple case: trending beats popular for every segment, most of all for new customers. The second panel is the story of this lesson. For loyal customers, the interval only just reaches 0: popular-category is probably better, but 610 customers are not enough to be sure.
The project also counts the customers. Trending against popular is better for 199 customers, worse for 50, and the same for 1,869. For most customers, both lists miss every order, so both get 0. The differences come from a few hundred customers, which is normal for recommendations.
New products: the other cold start
Section titled “New products: the other cold start”The second segment is products. Of the orders in the test period, 647 were of new products. How many of those did each list find?
| Method | Recall@10 on the orders of new products |
|---|---|
popular |
0.000 |
trending |
0.337 |
popular-category |
0.040 |
random |
0.006 |
popular never finds a new product: they have not had time to collect orders over the whole log. Trending looks at the last 28 days, when the new summer range was selling, so it finds 218 of the 647 orders. That is a reason for trending, and a warning for every method that learns only from history: watch this number. Split by time showed a related trap. A random split puts more loyal customers in the test: 45% against 29% in the time split. So it flatters methods that need history.
Guided practice: segments in your project
Section titled “Guided practice: segments in your project”-
In your
larkfield-recsfolder, with the virtual environment active, copy thecomparecommand and the tests of this module:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m03cp $S/larkfield_recs/commands/compare.py larkfield_recs/commands/cp $S/tests/test_m03.py tests/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m03"Copy-Item "$S\larkfield_recs\commands\compare.py" larkfield_recs\commands\Copy-Item "$S\tests\test_m03.py" tests\ -
Score trending by segment:
Terminal window recs evaluate trending --by segmentMethod trending (the most ordered products of the last 28 days), test periodCustomers scored: 2,118; trained on events before 2026-06-22recall@10 0.156 (95% interval 0.143 to 0.172), NDCG@10 0.100, MAP@10 0.073, precision@10 0.024, hit rate 20.4%coverage 0.3% of the products on sale, novelty 5.10 bits, diversity 0.98, new products in the lists 30.0%time: fit 0.0 s, suggestions 0.0 ssegment customers recall@10 NDCG@10 95% interval (recall)new 952 0.169 0.110 0.147 to 0.192occasional 556 0.162 0.101 0.134 to 0.190loyal 610 0.132 0.084 0.111 to 0.156new products: 647 orders of new products in the period; recall@10 on them 0.337 -
Compare popular and trending on the same customers.
recs comparereads the runs thatrecs evaluatesaved inoutputs/runs/, so evaluate both methods first. You evaluatedpopularin the last lesson.Terminal window recs compare popular trending --by segmentrecall@10: popular 0.106, trending 0.156difference +0.051 (95% interval +0.041 to +0.061); better for 199 customers, worse for 50, the same for 1,869The difference is clear: the interval does not include 0.coverage 0.3% -> 0.3%; novelty 4.39 -> 5.10new 952 customers +0.061 (+0.045 to +0.077)occasional 556 customers +0.052 (+0.033 to +0.073)loyal 610 customers +0.034 (+0.018 to +0.049) -
Run the tests with
python -m pytest -q. You see14 passed. -
Your project now matches the snapshot
end-of-m03. Only itsREADME.mddiffers.
In the notebook, section 5 asks you to compute the same paired differences yourself. The check prints Check passed: diff_by_segment is right.
Your turn: the two “equal” baselines
Section titled “Your turn: the two “equal” baselines”Change one thing: the pair of methods. Compare trending (the baseline) with popular-category. Before you run it, predict the sign of the difference for new and for loyal customers. Then run:
recs compare trending popular-category --by segmentShow the answer
recall@10: trending 0.156, popular-category 0.143difference -0.014 (95% interval -0.030 to +0.002); better for 246 customers, worse for 279, the same for 1,593The difference is not clear: the interval includes 0.coverage 0.3% -> 6.7%; novelty 5.10 -> 5.36 new 952 customers -0.051 (-0.073 to -0.030) occasional 556 customers -0.001 (-0.037 to +0.035) loyal 610 customers +0.032 (-0.001 to +0.065)For new customers, popular-category is clearly worse: its interval is below 0. For loyal customers, it is probably better, but the interval just touches 0. With --metric ndcg, the loyal difference is +0.024 (+0.001 to +0.049): clear on NDCG@10, but small. A sensible design for Larkfield is one method for new customers and another for customers with a history. Module 6 builds that kind of switch.
When to use segments, and what they cost
Section titled “When to use segments, and what they cost”Report every method by segment, beside the average. It costs nothing to compute, and it shows who wins and who loses. Choose the segments before you look at the results, and keep them the same for every method, as the comparison table does.
Two limits:
- Small segments, wide intervals. The loyal difference above has an interval of 0.066 for 610 customers. Do not split into segments of 50 customers and trust each number.
- Many segments, false alarms. With 20 segments, one of them can look “clearly” different by chance. Keep a few segments that the business acts on: Larkfield acts on new, occasional and loyal customers, and on new products.
Common mistakes
Section titled “Common mistakes”Common mistake
'No saved run' from recs compare
Symptom: recs compare popular random --quick stops with:
No saved run random-test-quick. Run first: recs evaluate random --quickCause: recs compare does not score methods. It reads the runs that recs evaluate saved, with the same period and the same --quick option.
Fix: run the command that the message prints, then compare again.
Common mistake
Two runs with different list lengths
Symptom: after recs evaluate trending --k 5, recs compare trending popular-category stops with:
The two runs have lists of different lengths (k=5 and k=10): run both with the same --k.Cause: a run with --k 5 replaces the saved run of the same method and period. Comparing a list of 5 with a list of 10 would not be fair.
Fix: run recs evaluate trending again (k = 10), then compare.
Knowledge check
Section titled “Knowledge check”Check your understanding
Method B beats method A by +0.010 recall@10 on all customers. By segment: new −0.040 (interval −0.060 to −0.020), loyal +0.060 (+0.030 to +0.090). Should Larkfield replace A with B? Explain with the segments.
Show the answer
Not for everybody. B is clearly worse for new customers (the whole interval is below 0) and clearly better for loyal customers. The average is positive only because the loyal gain is larger than the new-customer loss. At Larkfield, new customers are the largest test segment (952 of 2,118). A better design keeps A for customers with no history and uses B for loyal customers, then checks the result again by segment.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why is a paired comparison of two methods more exact than comparing their two separate intervals?
Show the answer
Both methods score the same customers, so the differences between customers cancel out, like each runner trying both pairs of shoes. Some customers are easy for every method and some are impossible for every method. The bootstrap of the per-customer differences measures only how much the two methods differ. Separate intervals also contain the variation between customers, so they can overlap even when one method is clearly better. Trending and popular-category overlap for all customers. But their paired difference for new customers is −0.051 (−0.073 to −0.030).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In your project, which command prints recall@10 on the orders of new products, and what does it print for popular? What does that tell a method designer?
Show the answer
recs evaluate popular --by segment prints new products: 647 orders of new products in the period; recall@10 on them 0.000. The all-time popular list never shows a product added in the 60 days before the period. Such a product has had no time to collect orders. A method that learns only from history has this cold-start problem for products. Watch this number for every method, beside recall@10 and coverage.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- One average can hide a group that a method fails.
trendingandpopular-categorydiffer by −0.014 on average, which is not clear. But popular-category is clearly worse for new customers (−0.051) and probably better for loyal ones (+0.032). - Segments by orders before the period: new (0) 952, occasional (1–2) 556, loyal (3+) 610 test customers. New customers are the cold-start problem, and they are the largest group.
- A segment’s interval is wider, because it has fewer customers.
- Paired difference: compare two methods on the same customers and bootstrap the per-customer differences. Trending − popular: +0.051 (+0.041 to +0.061), better for 199 customers, worse for 50, the same for 1,869.
- New products are a segment too: popular finds 0 of their 647 orders, trending 0.337.
recs evaluate <method> --by segmentandrecs compare A B --by segmentdo all of this in your project, which now matches the end of Module 3.
You can now judge any method by accuracy, variety and segment. But no method so far uses what a customer bought to suggest the next product. Next, you count which products are bought together.
Next: People also bought