Skip to content
Français

Evaluate by segment

  • Lesson 10
  • 35 min
  • Module 3: Measure recommendations offline

Where we are

In the last two lessons, we measured how accurate a list is and how much of the shop it shows. Every one of those numbers is an average over 2,118 customers, and an average can hide a group that a method fails. In this lesson, we measure new, occasional and loyal customers apart, and new products apart, starting from a class whose average went up.

What you will be able to do

Find the customers and products that one average hides. Measure new, occasional and loyal customers apart, compare two methods on the same customers with an interval, and check new products on their own.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook splits the test customers into the three segments, scores the popular and trending lists in each one, and computes bootstrap intervals. You compare the two lists on the same customers, segment by segment, and check the new products. Its setup cells download about 8 MB of practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the four files of the last lesson. recs evaluate popular prints novelty and diversity.
  • The practice files in ../nextia-learning-labs, beside the project folder.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

A school tries a new way of teaching mathematics in one class. Half of the students are strong, and half started the year behind. At the end of the year, the class average went up from 62.5% to 65%. The head teacher is happy.

Then a teacher looks at the students one group at a time. The strongest students gained a lot: their marks went from 80% to 90%. The students who started the year behind lost ground: from 45% to 40%. The average rose because the strong students gained more than the weak students lost. The new method helps the students who need help least, and the average says the opposite of the truth for the weak students.

A recommender has the same kind of groups. A loyal customer with 20 orders gives a method a lot to learn from. A new customer gives it nothing. One average over both can hide a method that fails every new customer.

The problem: “about the same” is not the whole story

Section titled “The problem: “about the same” is not the whole story”

Look at two baselines from Module 2. On all 2,118 test customers, trending has recall@10 0.156 and popular-category has 0.143. The project can compare them on the same customers, which you learn in this lesson. It prints:

recall@10: trending 0.156, popular-category 0.143
difference -0.014 (95% interval -0.030 to +0.002); better for 246 customers, worse for 279, the same for 1,593
The difference is not clear: the interval includes 0.

So the two baselines are “about the same”, and Tomás says that it does not matter which one Larkfield uses. Now split the same customers by how many orders they made before the test period:

2,118 test customers split into three groups by their orders before the period. 0 orders before: new, 952 customers, trending 0.169, popular-category 0.118. 1 or 2 orders before: occasional, 556 customers, trending 0.162, popular-category 0.161. 3 or more orders before: loyal, 610 customers, trending 0.132, popular-category 0.164. All three lead to a box with a thick border: all customers together, trending 0.156, popular-category 0.143.
Recall@10 of two baselines in each segment of the test period. The average hides opposite results for new and loyal customers.
Text description of the diagram

The diagram starts with the 2,118 test customers, the customers who ordered in the test period. Arrows split them by their orders before the period. With 0 orders, 952 new customers: trending has recall@10 0.169 and popular-category 0.118. With 1 or 2 orders, 556 occasional customers: trending 0.162 and popular-category 0.161. With 3 or more orders, 610 loyal customers: trending 0.132 and popular-category 0.164. The three groups lead to one box with a thick border, all customers together: trending 0.156 and popular-category 0.143.

For new customers, popular-category is clearly worse. For loyal customers, it is better. The two effects almost cancel in the average. Suppose Larkfield chose one method for everybody from the average. Then 952 new customers or 610 loyal ones would get a worse home page.

A segment is a group of customers, or of products, that we measure apart because a method may work differently for it. You choose segments that the business cares about, before you look at the results.

This course uses three customer segments, by the number of orders before the period. An order is one visit with an order, even if it had several products.

Segment Orders before the period Test customers What a method can learn about them
New 0 (they may have viewed products) 952 only their views, or nothing
Occasional 1 or 2 556 one or two baskets of products
Loyal 3 or more 610 a real history

Almost half of the test customers are new. A customer with no history is the cold-start problem: the method has nothing to warm up from. In Module 1 you saw that most Larkfield buyers order only once, so new customers are not a corner case here. They are the largest group.

We also measure one product segment: new products, added to the shop in the 60 days before the period. They have little history, so methods that learn from history rarely show them. That is the cold-start problem for products.

Six customers and two lists, with the recall@10 of each customer (a tiny example we made):

Customer Segment List A List B
Ana new 0.5 0.0
Ben new 0.5 0.0
Chen new 0.0 0.0
Dara loyal 0.0 1.0
Eli loyal 0.0 0.5
Femi loyal 0.5 1.0

On average, list B wins: 0.417 against 0.250. But for the new customers, list A has 0.333 and list B has 0. List B never finds anything for a new customer, and the average does not show it.

The chart shows recall@10 of the three non-personal baselines in each segment, with the bootstrap interval of each one.

Recall@10 by segment on the test period, with 95% intervals. Trending is best for new customers; popular-category is best for loyal customers.
Text description of the chart

Three panels, one for each segment, each with three rows and a horizontal axis for recall@10 from 0 to 0.25. New customers (952): popular 0.108 (0.090 to 0.125), trending 0.169 (0.147 to 0.192), popular-category 0.118 (0.100 to 0.138). Occasional customers (556): popular 0.110, trending 0.162, popular-category 0.161; the three intervals overlap. Loyal customers (610): popular 0.098, trending 0.132 (0.111 to 0.156), popular-category 0.164 (0.139 to 0.192).

Popular-category gives each customer the bestsellers of her favourite categories. A loyal customer has clear favourites, so this works. A new customer has at most a few views, and often none, so her “favourite categories” say little. Trending works for everybody in the same way, and the new customers of late June buy what is selling in late June.

Note that the intervals are wider than on all customers. Trending has 0.143 to 0.172 on 2,118 customers, and 0.147 to 0.192 on the 952 new ones. A segment has fewer customers, so its average depends more on chance.

Two separate intervals that overlap do not prove that two methods are the same. There is a more exact way, because both methods scored the same customers.

Think of two pairs of running shoes. You could give one pair to one group of runners and the other pair to another group, and compare the groups’ times. But the groups differ: some runners are just faster. It is better to let each runner try both pairs, and look at the difference for each runner. The speed of the runner cancels out.

The project does this. For each customer, it takes recall@10 of method B minus recall@10 of method A. Then it bootstraps these differences, as in Ranking metrics, and gives a 95% interval for the mean difference. This is the paired difference. If the interval does not include 0, the difference is clear. The evaluation course explains it in more depth in Uncertainty and slices, part of Evaluating and Testing AI Systems.

Paired differences in recall@10 on the test period, with 95% intervals over customers. Trending beats popular in every segment. Popular-category against trending is unclear on average, clearly worse for new customers and close to better for loyal ones.
Text description of the chart

Two panels, each with a vertical line at zero. First panel, trending minus popular: all customers +0.051 (+0.041 to +0.061); new +0.061 (+0.045 to +0.077); occasional +0.052 (+0.033 to +0.073); loyal +0.034 (+0.018 to +0.049). Every interval is above zero. Second panel, popular-category minus trending: all customers −0.014 (−0.030 to +0.002), which includes zero; new −0.051 (−0.073 to −0.030), all below zero; occasional −0.001 (−0.037 to +0.035), around zero; loyal +0.032 (−0.001 to +0.065), which just touches zero.

The first panel is the simple case: trending beats popular for every segment, most of all for new customers. The second panel is the story of this lesson. For loyal customers, the interval only just reaches 0: popular-category is probably better, but 610 customers are not enough to be sure.

The project also counts the customers. Trending against popular is better for 199 customers, worse for 50, and the same for 1,869. For most customers, both lists miss every order, so both get 0. The differences come from a few hundred customers, which is normal for recommendations.

The second segment is products. Of the orders in the test period, 647 were of new products. How many of those did each list find?

Method Recall@10 on the orders of new products
popular 0.000
trending 0.337
popular-category 0.040
random 0.006

popular never finds a new product: they have not had time to collect orders over the whole log. Trending looks at the last 28 days, when the new summer range was selling, so it finds 218 of the 647 orders. That is a reason for trending, and a warning for every method that learns only from history: watch this number. Split by time showed a related trap. A random split puts more loyal customers in the test: 45% against 29% in the time split. So it flatters methods that need history.

  1. In your larkfield-recs folder, with the virtual environment active, copy the compare command and the tests of this module:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m03
    cp $S/larkfield_recs/commands/compare.py larkfield_recs/commands/
    cp $S/tests/test_m03.py tests/
  2. Score trending by segment:

    Terminal window
    recs evaluate trending --by segment
    Method trending (the most ordered products of the last 28 days), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.156 (95% interval 0.143 to 0.172), NDCG@10 0.100, MAP@10 0.073, precision@10 0.024, hit rate 20.4%
    coverage 0.3% of the products on sale, novelty 5.10 bits, diversity 0.98, new products in the lists 30.0%
    time: fit 0.0 s, suggestions 0.0 s
    segment customers recall@10 NDCG@10 95% interval (recall)
    new 952 0.169 0.110 0.147 to 0.192
    occasional 556 0.162 0.101 0.134 to 0.190
    loyal 610 0.132 0.084 0.111 to 0.156
    new products: 647 orders of new products in the period; recall@10 on them 0.337
  3. Compare popular and trending on the same customers. recs compare reads the runs that recs evaluate saved in outputs/runs/, so evaluate both methods first. You evaluated popular in the last lesson.

    Terminal window
    recs compare popular trending --by segment
    recall@10: popular 0.106, trending 0.156
    difference +0.051 (95% interval +0.041 to +0.061); better for 199 customers, worse for 50, the same for 1,869
    The difference is clear: the interval does not include 0.
    coverage 0.3% -> 0.3%; novelty 4.39 -> 5.10
    new 952 customers +0.061 (+0.045 to +0.077)
    occasional 556 customers +0.052 (+0.033 to +0.073)
    loyal 610 customers +0.034 (+0.018 to +0.049)
  4. Run the tests with python -m pytest -q. You see 14 passed.

  5. Your project now matches the snapshot end-of-m03. Only its README.md differs.

In the notebook, section 5 asks you to compute the same paired differences yourself. The check prints Check passed: diff_by_segment is right.

Change one thing: the pair of methods. Compare trending (the baseline) with popular-category. Before you run it, predict the sign of the difference for new and for loyal customers. Then run:

Terminal window
recs compare trending popular-category --by segment
Show the answer
recall@10: trending 0.156, popular-category 0.143
difference -0.014 (95% interval -0.030 to +0.002); better for 246 customers, worse for 279, the same for 1,593
The difference is not clear: the interval includes 0.
coverage 0.3% -> 6.7%; novelty 5.10 -> 5.36
new 952 customers -0.051 (-0.073 to -0.030)
occasional 556 customers -0.001 (-0.037 to +0.035)
loyal 610 customers +0.032 (-0.001 to +0.065)

For new customers, popular-category is clearly worse: its interval is below 0. For loyal customers, it is probably better, but the interval just touches 0. With --metric ndcg, the loyal difference is +0.024 (+0.001 to +0.049): clear on NDCG@10, but small. A sensible design for Larkfield is one method for new customers and another for customers with a history. Module 6 builds that kind of switch.

Report every method by segment, beside the average. It costs nothing to compute, and it shows who wins and who loses. Choose the segments before you look at the results, and keep them the same for every method, as the comparison table does.

Two limits:

  • Small segments, wide intervals. The loyal difference above has an interval of 0.066 for 610 customers. Do not split into segments of 50 customers and trust each number.
  • Many segments, false alarms. With 20 segments, one of them can look “clearly” different by chance. Keep a few segments that the business acts on: Larkfield acts on new, occasional and loyal customers, and on new products.

Common mistake

'No saved run' from recs compare

Symptom: recs compare popular random --quick stops with:

No saved run random-test-quick. Run first: recs evaluate random --quick

Cause: recs compare does not score methods. It reads the runs that recs evaluate saved, with the same period and the same --quick option.

Fix: run the command that the message prints, then compare again.

Common mistake

Two runs with different list lengths

Symptom: after recs evaluate trending --k 5, recs compare trending popular-category stops with:

The two runs have lists of different lengths (k=5 and k=10): run both with the same --k.

Cause: a run with --k 5 replaces the saved run of the same method and period. Comparing a list of 5 with a list of 10 would not be fair.

Fix: run recs evaluate trending again (k = 10), then compare.

Check your understanding

Method B beats method A by +0.010 recall@10 on all customers. By segment: new −0.040 (interval −0.060 to −0.020), loyal +0.060 (+0.030 to +0.090). Should Larkfield replace A with B? Explain with the segments.

Show the answer

Not for everybody. B is clearly worse for new customers (the whole interval is below 0) and clearly better for loyal customers. The average is positive only because the loyal gain is larger than the new-customer loss. At Larkfield, new customers are the largest test segment (952 of 2,118). A better design keeps A for customers with no history and uses B for loyal customers, then checks the result again by segment.

Check your understanding

Why is a paired comparison of two methods more exact than comparing their two separate intervals?

Show the answer

Both methods score the same customers, so the differences between customers cancel out, like each runner trying both pairs of shoes. Some customers are easy for every method and some are impossible for every method. The bootstrap of the per-customer differences measures only how much the two methods differ. Separate intervals also contain the variation between customers, so they can overlap even when one method is clearly better. Trending and popular-category overlap for all customers. But their paired difference for new customers is −0.051 (−0.073 to −0.030).

Check your understanding

In your project, which command prints recall@10 on the orders of new products, and what does it print for popular? What does that tell a method designer?

Show the answer

recs evaluate popular --by segment prints new products: 647 orders of new products in the period; recall@10 on them 0.000. The all-time popular list never shows a product added in the 60 days before the period. Such a product has had no time to collect orders. A method that learns only from history has this cold-start problem for products. Watch this number for every method, beside recall@10 and coverage.

Recap

  • One average can hide a group that a method fails. trending and popular-category differ by −0.014 on average, which is not clear. But popular-category is clearly worse for new customers (−0.051) and probably better for loyal ones (+0.032).
  • Segments by orders before the period: new (0) 952, occasional (1–2) 556, loyal (3+) 610 test customers. New customers are the cold-start problem, and they are the largest group.
  • A segment’s interval is wider, because it has fewer customers.
  • Paired difference: compare two methods on the same customers and bootstrap the per-customer differences. Trending − popular: +0.051 (+0.041 to +0.061), better for 199 customers, worse for 50, the same for 1,869.
  • New products are a segment too: popular finds 0 of their 647 orders, trending 0.337.
  • recs evaluate <method> --by segment and recs compare A B --by segment do all of this in your project, which now matches the end of Module 3.

You can now judge any method by accuracy, variety and segment. But no method so far uses what a customer bought to suggest the next product. Next, you count which products are bought together.

Next: People also bought