Skip to content
Français

Ranking metrics

  • Lesson 8
  • 40 min
  • Module 3: Measure recommendations offline

Where we are

In Module 2, we built an honest time split and three popularity baselines, and the project printed a recall@10 for each one. But what does that number count, and how sure can we be of it? In this lesson, we score one list of five suggestions by hand, the way you would judge a friend’s gift ideas, and then score every test customer.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn one list of suggestions, and then the lists of 2,118 test customers, into a few honest numbers. Choose the number that matches the surface where the list appears, and say how sure you are of it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook scores the list of five suggestions by hand and asks you to write average precision. Then it scores the trending list on every test customer. Its setup cells install the packages and download about 8 MB of practice data, so it needs no local setup. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps; the notebook needs none of it)

  • Your larkfield-recs project at the end of Module 2, with its virtual environment active. recs check prints Setup check passed.
  • The practice files in ../nextia-learning-labs, beside the project folder.

Tested on macOS with Python 3.12, NumPy 2.5.4 and pandas 3.1.1. The Windows commands were not run.

Your sister’s birthday is next week. You ask a friend for gift ideas, and he writes down five, best idea first. A week later, you bought three gifts. Two of them were on his list, at places 2 and 3. The third gift was not on his list at all.

Was he a good adviser? You can answer in several ways, and each answer counts something different:

  • Two of his five ideas were useful. That is two out of five.
  • He found two of the three gifts you bought. That is two out of three.
  • At least one idea was useful. Yes or no.
  • His best ideas were not at the top. His first idea was wrong, so you had to read further.

None of these answers is wrong. They answer different questions. A recommender has the same problem, with 2,994 products instead of gifts.

In the machine learning course, a classifier says yes or no for each case, and accuracy counts the right answers. Try that on a list of suggestions.

Larkfield sells 2,994 products. A home page shows 10 of them: for each product, the list says “yes, show it” or “no”. Suppose a customer then orders 2 products, and neither is in the list. The list made 12 mistakes: 10 products shown but not ordered, and 2 products ordered but not shown. The other 2,982 “no” answers are right. So the accuracy is 2,982 / 2,994 = 99.6%, for a list that helped nobody.

Accuracy fails because almost every product is a correct “no”. A useful number must look only at the few products near the top of the list, because the customer sees only those. It must also care about the order. Everybody sees a right product at place 1; few people see a right product at place 10.

The idea: count the right products in the top k

Section titled “The idea: count the right products in the top k”

A recommender gives each customer a ranked list: the products in order, best first. We look only at the first k places, because a surface shows only k products. The home page shows 10, so this course uses k = 10. We write “@10” after a measure: recall@10 is recall computed on the first 10 places.

A suggestion is right when the customer ordered that product in the test period. This is the only thing we know: we cannot ask a customer whether she liked a product that she did not buy. A suggestion is right or wrong, nothing between, so this is called binary relevance.

Here is a real list from the test period, the trending list for customer C106777. It is the same list for every customer, because trending is not personal. We look at its first five places:

Place Product Ordered in the test period?
1 Lumi Green Nordic Coffee Table no
2 Navy Nitrile-Grip Garden Gloves yes
3 Bramble Terracotta Mosaic Side Table yes
4 Oak Timber Swing Seat no
5 Black Rustic Wooden Planter Box no

C106777 ordered three products: the gloves, the side table and the Hartley Blue Wall Lantern, which is not in the list. Like the friend, the list has two right ideas at places 2 and 3, and it missed one order.

Precision@k is the share of the k suggestions that were right: 2 / 5 = 0.4. It answers the friend’s first question: how many of his ideas were useful?

Recall@k is the share of the customer’s orders that the list found: 2 / 3 = 0.667. It answers the second question: how many of the gifts that you bought did he find? This is the main measure of this course, as recall@10.

Hit@k is 1 if at least one suggestion is right, and 0 if none is. Here it is 1. Over many customers, the mean of hit@k is the hit rate: the share of customers who found at least one product that they bought.

Precision@5 does not care where the right products are. Average precision does. At each right place, compute the precision of the list up to that place, and then take the mean:

  • Place 2 is right. The first two suggestions have one right product: precision 1 / 2.
  • Place 3 is right. The first three have two right products: precision 2 / 3.

Add them and divide by the number of products that could be right: the customer’s orders, at most k. So AP@5 = (1/2 + 2/3) / 3 = 0.389. If the two right products were at places 1 and 2, AP@5 would be (1/1 + 2/2) / 3 = 0.667. The mean of AP over all customers is MAP, the mean average precision.

NDCG: a right product near the top is worth more

Section titled “NDCG: a right product near the top is worth more”

NDCG gives each place a weight. A right product at place 1 is worth 1. Lower places are worth less, because fewer customers look that far. The weight of a place is 1 / log2(place + 1). The chart shows the weights of the 10 places of the home page.

The weight of a right product at each place of a list of 10, in NDCG: 1 / log2(place + 1). Place 2 counts 0.63; place 10 counts 0.29.
Text description of the chart

A column chart with ten columns, one for each place of the list. The labels leave out the leading zero: .63 is 0.63. The weight is 1.00 at place 1, 0.63 at place 2, 0.50 at place 3, 0.43 at place 4, 0.39 at place 5, 0.36 at place 6, 0.33 at place 7, 0.32 at place 8, 0.30 at place 9 and 0.29 at place 10. The weight falls fast over the first three places and slowly after that.

Add the weights of the right places. This sum is the DCG, the discounted cumulative gain: “gain” for each right product, “discounted” by its place. The list of C106777 has right products at places 2 and 3:

DCG = 0.631 + 0.500 = 1.131

The best possible list for this customer puts the three ordered products at places 1, 2 and 3. Its DCG is the ideal DCG: 1 + 0.631 + 0.500 = 2.131. NDCG divides one by the other, so the best possible list always scores 1:

NDCG@5 = 1.131 / 2.131 = 0.531

The “N” means normalised: divided by the best possible value. The search course uses NDCG with grades of relevance, in Ranking measures. There, a search result can be an exact match or only a partial match. Here, a product was ordered or not, so every right product has the same gain.

The trending list for customer C106777, five boxes from top to bottom: 1, coffee table, not ordered; 2, garden gloves, ordered, in a thick border; 3, mosaic side table, ordered, in a thick border; 4, swing seat, not ordered; 5, planter box, not ordered. A box says that the customer ordered the gloves, the side table and a wall lantern, which is not in the list. Arrows lead from the list and from the orders to a box with five results: precision@5 = 2 / 5 = 0.4; recall@5 = 2 / 3 = 0.667; hit@5 = 1; AP@5 = (1/2 + 2/3) / 3 = 0.389; NDCG@5 = 1.131 / 2.131 = 0.531.
The same five suggestions, scored five ways. Each measure answers a different question about the list.
Text description of the diagram

The picture shows the five places of the list from top to bottom. The two right products, at places 2 and 3, have a thick border. A second box lists the three products that the customer ordered, with the wall lantern marked as not in the list. Both lead to the results: precision@5 is 2 / 5 = 0.4; recall@5 is 2 / 3 = 0.667; hit@5 is 1, because at least one product is right; AP@5 is (1/2 + 2/3) / 3 = 0.389; NDCG@5 is 1.131 / 2.131 = 0.531.

Larkfield shows suggestions in three places. Each place needs a slightly different question:

Surface What the customer sees The question Measure
Product page, “people also bought” a short row of small images Is a right product near the top? NDCG@k, hit rate
Home page 10 products in a row Of what the customer will buy, how much did we show? recall@10, NDCG@10
Weekly email a few products in one email, read once Did the email contain at least one thing worth a click? hit rate

Precision@k says the same as recall@k when every customer orders the same number of products, but it is hard to read here. Most test customers order only 1 or 2 products. So even a perfect list of 10 has a precision@10 of only 0.1 or 0.2. Recall@k does not have this problem: a perfect list scores 1.

This course compares every method with recall@10 in the comparison table, and shows NDCG@10 and MAP@10 beside it. Recall@10 is easy to explain to Omar: “of the products that customers bought in the test weeks, the home page showed this share.”

One customer is one story. A score for a method is the mean over all 2,118 test customers, the customers who ordered in the test period. Each customer counts once, whether she ordered 1 product or 8.

In the notebook, section 4 does this for trending:

trending on 2,118 customers: recall@10 0.156, NDCG@10 0.100

So, on average, the trending list contains 15.6% of the products that a test customer ordered. The project computes exactly the same thing in larkfield_recs/metrics.py, which you have had since Module 1. Open it: recall_at_k, average_precision_at_k and ndcg_at_k are the calculations of this lesson, one function each, and per_customer applies them to every customer.

How sure are we? An interval over customers

Section titled “How sure are we? An interval over customers”

Taste a big pot of soup. One spoonful tells you a lot, but a second spoonful tastes a little different. Ten spoonfuls tell you more than one. A score over customers is the same: a different group of 2,118 customers would give a slightly different mean.

We cannot get another group of real customers. The bootstrap makes new groups from the ones we have. It draws 2,118 customers at random from the 2,118, with replacement: some customers are drawn twice, and some not at all. It computes the mean recall of this new group, and it repeats this 1,000 times. The middle 95% of the 1,000 means is the 95% interval. The project uses a fixed seed (37), so you get the same interval each time.

A narrow interval means that the score does not depend much on which customers we happened to have. The chart shows the four baselines, and the popular list scored on only 1,000 customers (--quick).

Recall@10 of the baselines on the test period, with 95% intervals from the bootstrap over customers. The same popular list on 1,000 customers has a wider interval.
Text description of the chart

A plot with one row for each result and a horizontal axis for recall@10 from 0 to 0.2. Popular on all 2,118 customers: 0.106, interval 0.095 to 0.118. Popular on a quick sample of 1,000 customers: 0.106, interval 0.090 to 0.123, which is wider. Trending: 0.156, interval 0.143 to 0.172. Popular-category: 0.143, interval 0.129 to 0.157. Random: 0.004, interval 0.002 to 0.007, close to zero. The intervals of trending and popular-category overlap; the interval of popular does not overlap with either.

Read three things from the chart:

  • Popular and trending are clearly different. Their intervals do not overlap.
  • Fewer customers, wider interval. On 1,000 customers, the same popular list gives 0.090 to 0.123 instead of 0.095 to 0.118. Use --quick to try ideas fast, never for the final number.
  • Trending and popular-category overlap. From these two intervals alone, we cannot say which one is better. Evaluate by segment compares two methods on the same customers, which is more exact.

The interval says how much the score depends on which customers we had. It does not say whether the test period is like next month. A time split and a second period, such as the validation period, answer that question.

Guided practice: the measures in your project

Section titled “Guided practice: the measures in your project”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.

  2. Run the setup cells. The last one prints Ready: 3 files in larkfield-recs.

  3. Run section 1. You see right or wrong at each place: [0, 1, 1, 0, 0] and precision@5 0.4 recall@5 0.667 hit 1.

  4. In section 2, predict whether NDCG@5 goes up or down if the right products move to places 1 and 2. Then run the cell. You see NDCG@5 0.5307.

  5. In section 3, write average_precision. Run the check cell. It prints Check passed: ap is 0.3889.

  6. Run section 4. You see recall@10 0.156, NDCG@10 0.100 on 2,118 customers.

  7. In a terminal, go to your larkfield-recs folder and activate its virtual environment.

    Terminal window
    source .venv/bin/activate
  8. Score the popular baseline on all test customers:

    Terminal window
    recs evaluate popular
    Method popular (the most ordered products over the whole training log), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.106 (95% interval 0.095 to 0.118), NDCG@10 0.061
    coverage 0.3% of the products on sale
    time: fit 0.0 s, suggestions 0.0 s
  9. Score it on the quick sample of 1,000 customers:

    Terminal window
    recs evaluate popular --quick
    Method popular (the most ordered products over the whole training log), test period
    Customers scored: 1,000 (quick sample); trained on events before 2026-06-22
    recall@10 0.106 (95% interval 0.090 to 0.123), NDCG@10 0.061
    coverage 0.3% of the products on sale
    time: fit 0.0 s, suggestions 0.0 s
  10. Score trending. You see recall@10 0.156 (95% interval 0.143 to 0.172), NDCG@10 0.100, the same values as the notebook.

Show the answer to step 5
def average_precision(rec, ordered, k=5):
total, right = 0.0, 0
for i, p in enumerate(rec[:k], 1):
if p in ordered:
right += 1
total += right / i
return total / min(len(ordered), k)

At place 2, right is 1 and the precision is 1 / 2. At place 3, right is 2 and the precision is 2 / 3. The sum, 1.1667, divided by min(3, 5) = 3 gives 0.3889.

The product page shows fewer products than the home page. Change one thing: the length of the list. Before you run it, predict: will recall@5 of popular be about half of its recall@10 (0.106), more than half, or less than half? Then run recs evaluate popular --k 5.

Show the answer
recall@5 0.062 (95% interval 0.053 to 0.072), NDCG@5 0.046
coverage 0.2% of the products on sale

Recall@5 is 0.062, a little more than half of 0.106. The first five places of the popular list find more orders than places 6 to 10, because the list is sorted best first. The interval is about as wide as before, because the same 2,118 customers are scored. Coverage falls from 0.3% to 0.2%: five products of 2,994 instead of ten. The next lesson looks closer at coverage.

Common mistake

'Precision@10 is 0.024, so the recommender is 98% wrong'

Symptom: Tomás reads precision@10 for trending, which the harness prints from the next lesson on: 0.024. “Only 2.4% of our suggestions are right. This is terrible.”

Cause: a test customer orders 1.77 products on average in the test weeks. A customer who orders one product can have at most one right suggestion of ten. So her best possible precision@10 is 0.1. Precision@10 is low for every method, even for a perfect one.

Fix: report recall@10 (a perfect list scores 1) and the hit rate. Use precision@k only when every customer could fill the k places, for example a list of 3 for customers who order many products.

Common mistake

Two scores that cannot be compared

Symptom: a notebook says that a new method gets recall 0.15 and “beats trending”. Trending got 0.156 in the project, so the new method looks almost the same.

Cause: the two numbers were not measured the same way. The new method used --quick (1,000 customers), or k = 5, or another period. Each change moves the score: popular gives 0.062 at k = 5 and 0.106 at k = 10.

Diagnose: read the first lines of each result: the period, the number of customers scored, and @10 or @5.

Fix: score both methods with the same command options, on the same customers. The harness does this for you when you use the defaults.

Check your understanding

A list of 10 suggestions has right products at places 1 and 10. The customer ordered 2 products. Give recall@10 and precision@10. Then explain why NDCG@10 is lower than 1.

Show the answer

Recall@10 is 2 / 2 = 1: the list found both orders. Precision@10 is 2 / 10 = 0.2. NDCG@10 is below 1 because the second right product is at place 10, where its weight is 1 / log2(11) = 0.29 instead of 0.63 at place 2. DCG = 1 + 0.289 = 1.289, the ideal DCG = 1 + 0.631 = 1.631, so NDCG@10 = 0.790. Recall does not care about the order inside the 10 places; NDCG does.

Check your understanding

Rafael wants one number for the weekly email, which shows a few products. Omar wants one number for the home page, which shows 10. Which measure would you suggest for each, and why?

Show the answer

For the email, the hit rate, with k = the number of products in the email. It is the share of customers whose email contained at least one product that they then bought. A customer reads the email once, so one useful product is what matters. For the home page, recall@10: the share of the bought products that the page showed. Put NDCG@10 beside it, because customers see the start of the row more. Both are means over the test customers, reported with their 95% interval.

Check your understanding

Trending has recall@10 0.156 (0.143 to 0.172) and popular-category 0.143 (0.129 to 0.157). Tomás says: 'Trending is better: 0.156 is higher.' What do you answer?

Show the answer

The two intervals overlap a lot, so the two separate intervals do not show that trending is better. With another group of customers, the order could change. The 95% interval comes from the bootstrap: 1,000 groups of customers drawn with replacement. To compare two methods, compare them on the same customers and compute an interval for the difference. That is the next step of this module.

Recap

  • Accuracy is useless for a list of suggestions: a list that helps nobody still gets 99.6%. Ranking measures look only at the first k places, and a suggestion is right when the customer ordered it.
  • One list of five, for customer C106777: precision@5 = 2 / 5 = 0.4, recall@5 = 2 / 3 = 0.667, hit@5 = 1, AP@5 = 0.389, NDCG@5 = 1.131 / 2.131 = 0.531.
  • NDCG weights each place by 1 / log2(place + 1) and divides by the best possible list. MAP is the mean of average precision over customers.
  • Match the measure to the surface. Use recall@10 and NDCG@10 for the home page, NDCG or the hit rate for a short product-page row, and the hit rate for an email. Recall@10 is the main measure of the comparison table.
  • A score is a mean over the 2,118 test customers. The bootstrap gives its 95% interval: trending 0.156 (0.143 to 0.172). Fewer customers give a wider interval.

You can now say how many right products a list contains, and how sure you are. But a list can be right and still boring: the same ten bestsellers for everybody, and nothing new. Next, you measure what accuracy does not see.

Next: Beyond accuracy