Skip to content
Français

Beyond accuracy

  • Lesson 9
  • 35 min
  • Module 3: Measure recommendations offline

Where we are

In the last lesson, we measured how many right products a list contains, and how sure we are of that number. But a list can be right and still useless to the shop: the same ten bestsellers for every customer. In this lesson, we measure coverage, novelty and diversity, starting from a bookshop window that never changes.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Measure what accuracy does not see: how much of the shop the suggestions show, how surprising they are, and how varied each list is. Then see what it costs in recall to show more of the shop.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes coverage, novelty and diversity on a tiny example, then on Larkfield’s popular and trending lists. You write the novelty of a list, compare random lists, and try a popularity penalty on four products. Its setup cells download about 8 MB of practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Recall@10, the test period of 2,118 customers and the 95% interval, from Ranking metrics.
  • The four baselines (popular, trending, popular-category, random), from The baselines to beat.

What you need on your computer (only for the project steps)

  • Your larkfield-recs project at the end of Module 2, with its virtual environment active. recs check prints Setup check passed.
  • The practice files in ../nextia-learning-labs, beside the project folder. You copy four files from its snapshot end-of-m03.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

Think of a bookshop with a big window. Every week, the window shows the same ten bestsellers. Many people buy them, so the window is often “right”. But a regular customer walks past and learns nothing new. The other 10,000 books in the shop never get a place in the window. So they sell less, and next week they are even less popular.

A good bookseller does something else. She knows that you liked one crime novel, so she shows you a new author in the same style. Some of her ideas are wrong. But you find books that you would never have found alone, and the whole shop gets a chance.

The window and the bookseller can have the same accuracy. They are still very different for the shop.

Larkfield’s popular baseline is the bookshop window. It shows the 10 most ordered products of the whole log to all 2,118 test customers. Its recall@10 is 0.106, not bad for a list with no personal information. But think about what the customers see:

  • All 2,118 customers get the same 10 products. The shop sells 2,994.
  • Those 10 products are the ones that everybody has already seen.
  • One of them is a winter coat, in late June: a practice problem we made on purpose, from The baselines to beat.

Recall@10 sees none of this. It only asks whether the ordered products were in the list. Compare two baselines that you know:

Two boxes side by side. List A, correct but boring: the same 10 bestsellers for every customer; at least one right product for 15.8% of customers (recall@10 0.106); shows 10 of 2,994 products (coverage 0.3%). List B, personal and varied: bestsellers of each customer's favourite categories; at least one right product for 19.8% of customers (recall@10 0.143); shows 200 of 2,994 products (coverage 6.7%).
Two baselines on the test period. Accuracy tells only part of the story: list A shows 10 products of 2,994 to everyone.
Text description of the diagram

The picture has two columns. On the left, list A, correct but boring: the popular baseline gives the same 10 bestsellers to every customer. It has at least one right product for 15.8% of the customers, and its recall@10 is 0.106. It shows 10 of the 2,994 products, a coverage of 0.3%. On the right, list B, personal and varied: the popular-category baseline gives each customer the bestsellers of their favourite categories. It has at least one right product for 19.8% of the customers, and its recall@10 is 0.143. It shows 200 of the 2,994 products, a coverage of 6.7%.

We need measures for what the window misses. There are three common ones, and a fourth that names the problem.

Coverage: how much of the shop do we show?

Section titled “Coverage: how much of the shop do we show?”

Coverage is the share of the products on sale that appear in at least one customer’s list. It is a measure of the whole system, not of one list.

A tiny example from the notebook: a shop has 8 products, A to H. Three customers get lists of three. Ana gets A, B, C; Ben gets A, C, D; Chen gets A, B, C. Together, the lists show A, B, C and D: 4 products of 8, so the coverage is 0.5. Products E to H are never shown, so they can never be bought from a suggestion.

For Larkfield, give the lists to the test customers one by one and count the products shown so far. The chart follows coverage as more customers get their lists:

  • random
  • popular-category
  • popular and trending
Coverage as more test customers get their lists. Popular and trending stay at 10 products (0.3%); popular-category stops at 200 products (6.7%); random lists reach almost every product.
Text description of the chart

A line chart. The horizontal axis is the number of test customers who got their list, from 1 to 2,118. The vertical axis is the share of the 2,994 products on sale shown at least once, from 0 to 100%. Popular and trending are one flat line at 0.3%: every customer gets the same 10 products. Popular-category rises to 5.1% after 50 customers and stops at 6.7%, 200 products, after about 750 customers. Random rises fast: 15.3% after 50 customers, 48.6% after 200, 96.6% after 1,000, and almost 100% after 2,118.

Coverage is cheap to compute and easy to explain to Hana: “our suggestions never show 93% of the products.” It does not say whether the products shown are good. Random lists reach almost 100%, with a recall@10 of 0.004.

A suggestion of a product that everybody has seen tells the customer nothing new. Novelty measures this. For each product, take the share of customers who touched it before the period (a view, a basket or an order). A product that 90% of customers touched is not news; one that 2% touched is.

We turn the share into bits: novelty = −log2(share). Each time the share halves, the novelty grows by one bit. The tiny example of the notebook, with 100 customers:

Product Customers who touched it Share Novelty
A 90 0.90 0.15 bits
B 60 0.60 0.74 bits
C 10 0.10 3.32 bits
D 2 0.02 5.64 bits

The novelty of a method is the mean over all the suggestions that it made. On the test period, popular has 4.39 bits and trending 5.10 bits: trending shows products that are popular now, and fewer customers have seen them. Random lists have 8.66 bits, because most products are rarely seen. High novelty alone is not good: random lists almost never contain what people buy.

Imagine a list of ten suggestions with ten armchairs. Each one may be a good guess, but the customer needs only one armchair. Diversity looks inside one list. Take every pair of products in the list, and count the share of pairs whose two products are of different kinds. At Larkfield, a kind is a subcategory, such as Armchairs or Outdoor plants.

In the tiny example, A and B are armchairs, C is a cushion, D is a lamp. Ana’s list A, B, C has 6 ordered pairs; 2 of them (A with B, and B with A) are the same kind. So her diversity is 1 − 2/6 = 0.667. Ben’s list A, C, D has three different kinds, so its diversity is 1.0, the highest possible value. A list of one kind has diversity 0.

Here is a surprise. popular has a diversity of 1.00: its 10 bestsellers are 10 different subcategories. trending has 0.98: one pair of its products has the same subcategory. So each list is perfectly varied, and yet the whole system shows only 10 products. Diversity inside one list and coverage across all lists are different things. You need both.

Popularity bias is the habit of a recommender to give most places to products that are already popular. It is the bookshop window in one number. The harness measures it as the share of all suggestions that go to the 1% most popular products. These are the 29 products on sale that most customers touched before the period.

Method recall@10 coverage novelty diversity top 1% share new products in the lists
popular 0.106 0.3% 4.39 1.00 100.0% 0.0%
trending 0.156 0.3% 5.10 0.98 70.0% 30.0%
popular-category 0.143 6.7% 5.36 0.92 53.2% 1.6%
random 0.004 100.0% 8.66 0.99 1.2% 8.5%

All ten places of popular go to the top 1%. Trending gives 70% of its places to them, and 30% to new products: products added in the 60 days before the period. They are the new summer range, which sells now. The last column will matter again in the next lesson.

Guided practice: every measure in the harness

Section titled “Guided practice: every measure in the harness”

Your project at the end of Module 2 prints recall, NDCG and coverage. The end-of-module-3 snapshot has a harness that prints every measure of this lesson. You copy four files from it.

  1. In your larkfield-recs folder, with the virtual environment active, copy the new measures (beyond.py), the segments for the next lesson (segments.py), the new harness and the new evaluate command:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m03
    cp $S/larkfield_recs/beyond.py $S/larkfield_recs/segments.py $S/larkfield_recs/harness.py larkfield_recs/
    cp $S/larkfield_recs/commands/evaluate.py larkfield_recs/commands/
  2. Open larkfield_recs/beyond.py. Its function measures computes novelty, diversity, the share of new products and the top 1% share, as this lesson did by hand.

  3. Score the popular baseline again:

    Terminal window
    recs evaluate popular
    Method popular (the most ordered products over the whole training log), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.106 (95% interval 0.095 to 0.118), NDCG@10 0.061, MAP@10 0.041, precision@10 0.016, hit rate 15.8%
    coverage 0.3% of the products on sale, novelty 4.39 bits, diversity 1.00, new products in the lists 0.0%
    time: fit 0.0 s, suggestions 0.0 s
  4. Score popular-category:

    Terminal window
    recs evaluate popular-category
    Method popular-category (the most ordered products of each customer's favourite categories), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.143 (95% interval 0.129 to 0.157), NDCG@10 0.085, MAP@10 0.059, precision@10 0.022, hit rate 19.8%
    coverage 6.7% of the products on sale, novelty 5.36 bits, diversity 0.92, new products in the lists 1.6%
    time: fit 0.0 s, suggestions 0.0 s
  5. Run the tests with python -m pytest -q. You see 11 passed.

The recall@10 and its interval did not change: the new harness measures the same lists, and only prints more. If recs evaluate fails with an ImportError, read the first common mistake below.

Before you run it, predict four numbers for recs evaluate random: recall@10, coverage, novelty and the share of new products. Use what you know: there are 2,994 products on sale, and 250 of them are new. Then run it.

Show the answer
recall@10 0.004 (95% interval 0.002 to 0.007), NDCG@10 0.002, MAP@10 0.001, precision@10 0.001, hit rate 0.6%
coverage 100.0% of the products on sale, novelty 8.66 bits, diversity 0.99, new products in the lists 8.5%

Coverage is almost 100%: 21,180 random places reach nearly every product. The share of new products, 8.5%, is close to their share of the shop, 250 / 2,994 = 8.4%, as expected by chance. Novelty is the highest of all methods. And recall@10 is almost zero. Variety alone is worth nothing: a good recommender needs accuracy and variety.

Can we have both? One simple idea: push popular products down a little. The recommender ranks each product by score − penalty × popularity, with both numbers scaled to the same range from 0 to 1. With a penalty of 0, nothing changes. With a large penalty, the least popular products win.

A tiny example from the notebook. A method gives four products a score for one customer:

Product Score Popularity Score − 0.5 × popularity
Lantern 0.90 1.00 0.40
Gloves 0.85 0.20 0.75
Side table 0.60 0.90 0.15
Bird bath 0.55 0.10 0.50

With no penalty, the order is lantern, gloves, side table, bird bath. With a penalty of 0.5, it becomes gloves, bird bath, lantern, side table. The bestseller lantern moves from place 1 to place 3.

The baselines have no personal score to keep, so the course team tried the penalty on a personal method: ALS with tuned settings. ALS learns a hidden taste for each customer from what they did, and you build it in Module 5. Here you need only one fact: it gives every product a score for every customer. Customers with no history get the trending list. The team measured nine penalties on the test period.

Explore

A popularity penalty on tuned ALS

Penalty
0.0
Recall@10
0.149
NDCG@10
0.097
Coverage
16.5%
Novelty (bits)
5.70
Top 1% share
38.8%
New products
10.0%

No penalty: tuned ALS as it is. Recall@10 is 0.149. 16.5% of the products on sale appear in at least one list, and the 1% most popular products take 38.8% of all places.

  • Chosen penalty
  • Level with no penalty (dashed)
0.000.100.200.00.10.20.30.40.50.60.70.80.91.0Recall@10level with no penaltyRecall@10Popularity penalty 0.1: 0.150Popularity penalty 0.2: 0.151Popularity penalty 0.3: 0.151Popularity penalty 0.4: 0.148Popularity penalty 0.5: 0.143Popularity penalty 0.6: 0.137Popularity penalty 0.8: 0.113Popularity penalty 1.0: 0.068Popularity penalty 0.0: 0.1490.1490%15%30%0.00.10.20.30.40.50.60.70.80.91.0Coverage of the products on saleCoverage of the products on salePopularity penalty 0.1: 17.4%Popularity penalty 0.2: 18.4%Popularity penalty 0.3: 20.2%Popularity penalty 0.4: 22.6%Popularity penalty 0.5: 24.7%Popularity penalty 0.6: 25.0%Popularity penalty 0.8: 20.9%Popularity penalty 1.0: 14.9%Popularity penalty 0.0: 16.5%16.5%56780.00.10.20.30.40.50.60.70.80.91.0Popularity penaltyNovelty (bits)Novelty (bits)Popularity penalty 0.1: 5.74Popularity penalty 0.2: 5.78Popularity penalty 0.3: 5.83Popularity penalty 0.4: 5.91Popularity penalty 0.5: 6.01Popularity penalty 0.6: 6.21Popularity penalty 0.8: 6.98Popularity penalty 1.0: 7.72Popularity penalty 0.0: 5.705.70
0.000.100.200.00.20.40.60.81.0Recall@10Recall@10Popularity penalty 0.1: 0.150Popularity penalty 0.2: 0.151Popularity penalty 0.3: 0.151Popularity penalty 0.4: 0.148Popularity penalty 0.5: 0.143Popularity penalty 0.6: 0.137Popularity penalty 0.8: 0.113Popularity penalty 1.0: 0.068Popularity penalty 0.0: 0.1490.1490%15%30%0.00.20.40.60.81.0Coverage of the products on saleCoverage of the products on salePopularity penalty 0.1: 17.4%Popularity penalty 0.2: 18.4%Popularity penalty 0.3: 20.2%Popularity penalty 0.4: 22.6%Popularity penalty 0.5: 24.7%Popularity penalty 0.6: 25.0%Popularity penalty 0.8: 20.9%Popularity penalty 1.0: 14.9%Popularity penalty 0.0: 16.5%16.5%56780.00.20.40.60.81.0Popularity penaltyNovelty (bits)Novelty (bits)Popularity penalty 0.1: 5.74Popularity penalty 0.2: 5.78Popularity penalty 0.3: 5.83Popularity penalty 0.4: 5.91Popularity penalty 0.5: 6.01Popularity penalty 0.6: 6.21Popularity penalty 0.8: 6.98Popularity penalty 1.0: 7.72Popularity penalty 0.0: 5.705.70
Tuned ALS on the test period (2,118 customers), with nine measured popularity penalties. Move the penalty and watch recall@10, coverage and novelty; the dashed line in each panel is the level with no penalty.
Text description of the explorer

The explorer shows measured results of tuned ALS with a popularity penalty, on the 2,118 test customers. A slider chooses one of nine penalties: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8 and 1 (0.7 and 0.9 were not measured). Three panels show recall@10, coverage and novelty for all penalties, with the chosen one marked and a dashed line at the value with no penalty. Readouts show recall@10, NDCG@10, coverage, novelty, the share of the 1% most popular products and the share of new products.

With no penalty: recall@10 0.149, coverage 16.5%, novelty 5.70 bits, top 1% share 38.8%. Penalty 0.1: recall 0.150, coverage 17.4%. Penalty 0.2: recall 0.151, coverage 18.4%. Penalty 0.3: recall 0.151, coverage 20.2%, novelty 5.83. Penalty 0.4: recall 0.148, coverage 22.6%. Penalty 0.5: recall 0.143, coverage 24.7%. Penalty 0.6: recall 0.137, coverage 25.0%, the highest coverage. Penalty 0.8: recall 0.113, coverage 20.9%. Penalty 1: recall 0.068, coverage 14.9%, novelty 7.72 bits, top 1% share 21.6%.

Three results stand out:

  • A small penalty is free here. At 0.3, recall@10 is 0.151 against 0.149, and coverage rises from 16.5% to 20.2%. The small rise in recall is within the noise of the interval; the gain in coverage is real.
  • A middle penalty costs a little. At 0.6, coverage is highest, 25.0%, and recall falls to 0.137: 8% of its value.
  • A large penalty breaks the list. At 1, recall falls to 0.068, less than half. Coverage falls to 14.9%: many customers now get the same rarely bought products. Novelty is highest, 7.72 bits, and useless.

Priya chooses a small penalty and checks it again on the validation period. Omar asks the real question: do customers buy more from a varied list? Offline measures cannot answer that. A simulated online test later in the course looks at it.

Use coverage, novelty, diversity and the top 1% share beside recall@10, never instead of it. They have no “right” value. They are guardrails. Suppose a new method gains 0.005 in recall but drops coverage from 30% to 1%. It needs a discussion with Hana before it ships.

They cost almost nothing: the harness computes them from the same lists in under a second. Their limits:

  • They do not know whether a varied list sells. Random lists win on every one of them.
  • Diversity depends on what you call a “kind”. With categories instead of subcategories, the numbers change.
  • Novelty depends on the period: a product that is new today is old next month.

The Machine Learning: From Problem to Reliable Model course judges one prediction at a time. Here, the unit is a list, and the shop sees all the lists together. That is why some measures look at the whole system.

Common mistake

ImportError: cannot import name 'segments'

Symptom: after the copy, every recs evaluate stops with:

ImportError: cannot import name 'segments' from 'larkfield_recs' (…/larkfield_recs/__init__.py)

Cause: the new harness.py uses beyond.py and segments.py. The course team got this error after copying only harness.py and beyond.py.

Fix: copy all four files of step 1, then run recs evaluate popular again. If something is still wrong, compare your folder with the snapshot: diff -r larkfield_recs ../nextia-learning-labs/C37/snapshots/end-of-m03/larkfield_recs on macOS and Linux.

Common mistake

'Diversity is 1.00, so our suggestions are varied'

Symptom: Tomás reads diversity 1.00 for popular and says that the home page is as varied as it can be.

Cause: diversity looks inside one list. All 2,118 customers get the same varied list, so the system shows 10 products of 2,994.

Fix: read coverage and the top 1% share beside diversity. For popular: coverage 0.3%, top 1% share 100%.

Check your understanding

A method gives every customer the 10 most popular products of 10 different subcategories. Predict its diversity, its coverage and its top 1% share on the test period, and say what they tell Hana.

Show the answer

Diversity is 1.00, because each list has 10 different subcategories. Coverage is 10 / 2,994 = 0.3%, because every customer gets the same 10 products. The top 1% share is 100% if all 10 are among the 29 most popular products, as for popular. For Hana: each list looks varied, but the suggestions never show 99.7% of the shop. New or less known products never get a place.

Check your understanding

Tuned ALS has recall@10 0.149 and coverage 16.5%. With a popularity penalty of 1, it has recall 0.068, coverage 14.9% and novelty 7.72 bits. Explain why a strong penalty can lower coverage.

Show the answer

A strong penalty makes popularity count almost as much as the customer’s own score. So, for many customers, the least popular products win, and those are often the same few products for everybody. The lists become less personal and more alike, so fewer different products are shown. Coverage falls from its best value (25.0% at 0.6) to 14.9%. Novelty is highest because those products are rarely seen, but recall falls by more than half.

Check your understanding

In your project, run recs evaluate trending. Which line shows the share of new products in the lists, and what is its value? Why is it not 0%, as for popular?

Show the answer

The second line of the result: new products in the lists 30.0%. Trending counts the orders of the last 28 days, when the new summer range was selling well. So 3 of its 10 products are new: added in the 60 days before the period. Popular counts orders over the whole log. There, older products have had much more time to collect orders, so no new product reaches its top 10.

Recap

  • A list can be correct but boring. popular has recall@10 0.106, and every customer sees the same 10 of 2,994 products.
  • Coverage: the share of the products on sale shown to at least one customer (popular 0.3%, popular-category 6.7%, random almost 100%). Novelty: −log2 of the share of customers who touched a product, in bits (popular 4.39, trending 5.10). Diversity: the share of pairs in one list of different subcategories (popular 1.00).
  • Popularity bias: popular gives 100% of its places to the 1% most popular products, trending 70%.
  • Variety alone is worthless: random lists have the best coverage and novelty, and recall@10 0.004.
  • A popularity penalty on tuned ALS: 0.3 raises coverage from 16.5% to 20.2%, with recall 0.151 against 0.149. A penalty of 1 halves recall and even lowers coverage.

You can now see whether a method shows the whole shop. But every number so far is an average over 2,118 customers, and a new customer and a loyal one need different things. Next, you measure groups of customers and products apart.

Next: Evaluate by segment