What success means
- Lesson 2
- 30 min
- Module 1: The recommendation problem
Where we are
We know what Larkfield's recommender must decide. Now we must say what a good decision is. In this lesson, we compare clicks, orders, added revenue and returns, and we see why more clicks can be a bad target. First, think of how a shop owner judges an assistant.
Lesson 2 of 29 · Recommendation and Personalisation: Show Each Person What MattersWhat success means
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Choose what “a better list” means for Larkfield before you compare any methods. If you count the wrong thing, you can build a recommender that wins on your dashboard and loses money for the shop.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s three surfaces, the path from a suggestion to an order, and the popular list of 22 June, from A good shop assistant.
- A mean (an average) and a percentage. Essential Mathematics and Statistics for AI explains both if you need them.
What you need on your computer
Nothing. This lesson has no code.
How do you judge a shop assistant?
Section titled “How do you judge a shop assistant?”Think again of the garden shop and its assistants. The owner wants to know which assistant is better. How can she measure it?
She could count conversations: how often each assistant shows a customer a product. One assistant stops every customer at the door and shows them a cheap offer. He has many conversations. But most people say “no, thank you”, and some of them stop coming to the shop.
She could count sales: the money at the till after each assistant’s conversations. That is closer. But one regular customer always buys her compost on the first Saturday of the month. If the assistant says “do you need compost?”, the sale is not thanks to him.
She could count sales that stay sold. Some customers bring back what an assistant pushed on them. The shop gives back the money, and it pays someone to put the product back on the shelf.
So the owner’s real question is: how much extra money, kept, does each assistant bring, without making customers leave? A recommender needs the same question, and the same care.
The problem: more clicks, less money
Section titled “The problem: more clicks, less money”Tomás tries a new home page list. He calls it “Today’s deals”: cheap, eye-catching products with a big discount label. He compares it with a list that Priya made, “Goes with your table”: products that match what the customer bought before. These are constructed numbers, for the example. Each list is shown on 1,000 visits:
| Today’s deals | Goes with your table | |
|---|---|---|
| Visits | 1,000 | 1,000 |
| Clicks on a suggestion | 150 | 80 |
| Orders from a suggestion | 30, at 20 dollars each | 16, at 45 dollars each |
| Money from those orders | 600 dollars | 720 dollars |
| Orders the customer would have made anyway | 15 (300 dollars) | 2 (90 dollars) |
| Orders returned | 6 (120 dollars) | 1 (45 dollars) |
| Added money, kept | 600 − 300 − 120 = 180 dollars | 720 − 90 − 45 = 585 dollars |
| Added revenue per visit | 180 / 1,000 = 0.18 dollars | 585 / 1,000 = 0.59 dollars (0.585) |
| Return handling, at 12 dollars each | 6 × 12 = 72 dollars | 1 × 12 = 12 dollars |
Tomás’s list gets almost twice as many clicks. It also adds less than a third of the money, and it causes six times as many returns. Its cheap deals attract people who were going to buy anyway: they click on a discount for something already on their list. Its impulse products come back more often.
Text description of the chart
The figure has two panels. Each panel has one row for each list, with a dot at its value. In the first panel, clicks per 1,000 visits: Today’s deals 150, Goes with your table 80. In the second panel, added revenue per visit: Today’s deals 0.18 dollars, Goes with your table 0.59 dollars. Today’s deals is ahead in the first panel and far behind in the second.
A click is easy to count, and it happens at once. That is why clicks are tempting as a target. But a click is only the third step of the path from a suggestion to an order. Everything after it decides whether the shop earns money.
The idea: one business measure, one offline stand-in, a few guardrails
Section titled “The idea: one business measure, one offline stand-in, a few guardrails”Priya and Omar agree on three kinds of measures for each surface.
The business measure
Section titled “The business measure”The business measure is the number that the shop really cares about. For Larkfield’s home page, Omar chooses added revenue per visit. It is the money from suggested products that the customer kept and would not have spent anyway, divided by the number of visits. The word revenue means the money from sales, before costs.
It has one big problem. You can measure it only by showing a list to real customers for some weeks, and comparing it with another list. That is an online test, often called an A/B test. The shop splits customers at random into group A and group B, and each group sees one list. It is slow, and a bad list costs real money while it runs. Module 8 is about online tests.
The offline stand-in
Section titled “The offline stand-in”So, before any customer sees a list, Priya needs a number that she can compute from the shop log alone. A measure computed from past data, without real customers, is called offline. It stands in for the business measure: it estimates the business measure in advance, cheaply, and the same way for every method.
The course’s main offline measure is recall@10 (say “recall at ten”). It takes each customer’s list of 10 products, made with the events before a date. Then it looks at what that customer really ordered in the next two weeks. For one customer:
recall@10 = (ordered products that were in the list) / (all products that the customer ordered)
Then it takes the mean over all customers who ordered. Here is the popular list of 22 June, scored for three real customers of the synthetic log:
| Customer | Ordered from 22 June to 5 July | In the popular list? | recall@10 |
|---|---|---|---|
| C100434 | Bramble Terracotta Mosaic Side Table; Millbrook Blush Pink Folding Side Table; Lumi Green Nordic Coffee Table | the Bramble Terracotta Mosaic Side Table: yes; the others: no | 1 / 3 = 0.333 |
| C100185 | Lumi Oak Compact Round Coffee Table; Millbrook Cream Stoneware Mini Bud Vase | no, no | 0 / 2 = 0 |
| C100304 | Terracotta Linen Roman-Style Curtains | yes | 1 / 1 = 1 |
The mean for these three customers is (0.333 + 0 + 1) / 3 = 0.444. For all 2,118 customers who ordered in those two weeks, the sum of the recall values is 223.9. So the popular list’s recall@10 is 223.9 / 2,118 = 0.106. You run this score yourself in the next lesson.
The guardrails
Section titled “The guardrails”A guardrail is a measure that must not get worse while you improve the main one. The name comes from the barrier at the side of a mountain road. It does not take you anywhere, but it stops you from going over the edge. For the home page, there are four guardrails. They are the return rate, the share of lists with a product that the customer already bought, and sensitive products in the list. The fourth is coverage: the share of the catalogue that the shop shows to at least one customer.
Text description of the diagram
The home page shows 10 suggestions on each visit. Its business measure is added revenue per visit: the money from suggestions that the customer would not have spent anyway. You can measure it only online. A dotted line links it to its offline stand-in, recall@10 on the next two weeks. That is the share of the products that people ordered that were in their list. You can compute it from the shop log today. The home page also has guardrails that must not get worse. They are the return rate, already-bought products, sensitive products, and the share of the catalogue that the shop ever shows.
Long-term loyalty belongs here too. A customer who comes back for years is worth more than one extra sale today. Loyalty is slow to measure, so a team watches signs of it: unsubscribes, and customers who stop visiting. Rafael’s email has the clearest sign: one click on Unsubscribe.
Why the stand-in and the business measure can disagree
Section titled “Why the stand-in and the business measure can disagree”Recall@10 is useful, cheap and fair. But it is not the business measure, and the two can point in different directions:
- The log was made without your list. Customers ordered from the old site. So recall counts orders that the customer would have made anyway. Those orders add no money.
- All 10 places count the same. Online, most people look only at the first few places. Module 3 adds measures that care about the order.
- Returns, unsubscribes and trust are not in it. Recall counts an order as right even if it comes back.
- Customers react to the list. A list changes what people see, so it changes what they buy next. Module 8 shows how.
Larkfield’s own data shows this. In Module 8, the course compares three home page lists in an online test. The test is simulated: it runs on our model of customer behaviour, never on real people. Offline, the course’s best method (a two-stage ranker from Module 7) scores recall@10 0.215. A list of what sold best in the last four weeks (“trending”) scores 0.151. That is 42% more for the two-stage ranker. Online, the picture is different:
Text description of the chart
The figure has two panels with the same three lists. In the first panel, offline recall@10 with business rules: two-stage ranker 0.215, trending 0.151, ALS 0.136. The second panel shows simulated added revenue per visit, with a bar for the 95% interval. Two-stage ranker: 6.82 dollars (5.45 to 8.25). Trending: 6.64 dollars (5.33 to 8.01). ALS: 3.18 dollars (2.25 to 4.10). The two-stage ranker is far ahead offline. Online, its interval and trending’s interval overlap almost completely. ALS is clearly behind online.
Online, the two-stage ranker adds 0.19 dollars per visit more than trending. But the 95% interval of that difference goes from −1.75 to +2.05 dollars. The interval is the range where the true difference probably lies, given how much the result moves by chance. It includes zero, so the test cannot say which list is better. The ranker is not clearly better online. Trending also gets the most clicks: 0.123 clicks per visit, against 0.114 for the ranker and 0.061 for ALS. So the most clicks did not bring clearly more money either.
Tomás is surprised. “The method with the best offline score should win.” Priya answers: “Offline scores choose which methods are worth an online test. They do not replace it.” You learn why this happens, and what to do about it, in Module 8.
The cost of two mistakes
Section titled “The cost of two mistakes”Two mistakes matter most, because they cost more than a wasted place.
Already bought
Section titled “Already bought”A product that the customer just bought takes a place and earns nothing. Worse, it tells the customer that the shop does not know them. Some products are bought again, such as compost or pet food, so “already bought” is not always wrong. A rule must know the difference. In Module 7, even the course’s best method, without business rules, shows a product that the customer already bought in 35% of its lists.
Sensitive
Section titled “Sensitive”Larkfield sells three sensitive subcategories: mobility aids, continence care, and pregnancy and fertility. A sensitive product is not a bad product. The problem is who else sees the suggestion. Continence care is bought again and again, so it is popular: a chair protector is at place 4 of the popular list, which everyone sees. Without rules, 5.7% of the places in the best method’s lists are sensitive health products.
The cost is highest in the email, because 3,178 Larkfield customers share one email address with their family. Here is a worked cost, under the assumptions of the course’s behaviour model. A sensitive product in a shared inbox adds 4 percentage points to the chance that the customer unsubscribes. A subscriber is worth 18 dollars a year. Suppose one weekly email shows a continence product to all 3,178 shared inboxes, and each of them is a subscriber:
| Step | Value |
|---|---|
| Extra unsubscribes | 3,178 × 0.04 = about 127 customers |
| Email value lost | 127 × 18 dollars = about 2,290 dollars a year |
| Trust lost | not in the table, and not easy to win back |
That is from one email. The fix is a rule, not a better model: never put a sensitive product in an email to a shared inbox. Module 7 adds such rules and measures what each one costs.
Guided practice: the home page
Section titled “Guided practice: the home page”Priya and Omar fill the table for the home page. Read each line, and check it against the diagram above.
| Home page | |
|---|---|
| Business measure | added revenue per visit |
| Offline stand-in | recall@10 on the next two weeks |
| Guardrails | return rate; lists with an already-bought product; sensitive products; coverage |
| Cost of a bad suggestion | a wasted place out of 10; a return at 12 dollars; a customer who thinks the shop does not know them |
The business measure is in money, because the shop wants money. The offline stand-in is a number that Priya can compute today, for every method, the same way. The guardrails are the things that the business measure does not see quickly.
Your turn: the product page and the email
Section titled “Your turn: the product page and the email”Fill the same four lines for the product page and for the weekly email. Think about what each surface is for, from A good shop assistant.
Show an answer
| Product page | Weekly email | |
|---|---|---|
| Business measure | added revenue per product page view, or the added basket value | added revenue per email sent, over several weeks |
| Offline stand-in | for a product, the share of the products bought with it, or right after it, that are in its row | recall@10 on the week after the email |
| Guardrails | out-of-stock products; products from an unrelated category; return rate | unsubscribe rate; sensitive products in shared inboxes; customers who opted out of personal suggestions get no personal list |
| Cost of a bad suggestion | a wasted place and a missed bigger basket | a lost subscriber (18 dollars a year under our assumption), and lost trust |
For the email, the guardrail matters more than the business measure. One bad email can lose a subscriber for good, and no extra sale today pays for that. Other answers are right too, if each measure belongs to its surface and money comes from what the customer keeps.
Common mistake
The click dashboard
Symptom: Tomás puts “Today’s deals” on the home page for a week. The dashboard shows clicks on suggestions up from 80 to 150 per 1,000 visits. The team celebrates. A month later, Omar sees more returns and no rise in sales.
Cause: the team chose clicks as its target. Clicks are the third step of the path from a suggestion to an order. In the constructed example, the deals list adds 0.18 dollars per visit, against 0.59 for the list that it replaced. It also causes six returns instead of one.
Fix: before you compare lists, write down the business measure, the offline stand-in and the guardrails for the surface. Report clicks only next to them, never alone. Count orders that the customer would have made anyway as zero, and count returns as lost money.
Consequence in AI work: a recommender that learns from clicks learns to get clicks. If clicks are the target, the system becomes very good at the wrong thing, and it does so automatically, every day. Module 8 shows such a loop.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain, with an example, why a recommender that raises clicks can lower revenue.
Show the answer
A click is only one step on the way to money. The products that attract the most clicks are often cheap deals, which people would have bought anyway, or impulse products, which come back more often. In the constructed example, “Today’s deals” gets 150 clicks per 1,000 visits against 80. But 15 of its 30 orders would have happened anyway, and 6 are returned. It adds 0.18 dollars per visit against 0.59 for “Goes with your table”, and it causes 72 dollars of return handling against 12. So the business measure must be added revenue that is kept, with returns as a guardrail.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: a new method raises recall@10 on the test period from 0.151 to 0.215. Will it add more money on the home page?
Show the answer
Not necessarily. Recall@10 is an offline stand-in. It counts orders that the customer would have made anyway, it treats all 10 places the same, and it does not see returns or unsubscribes. The course’s simulated online test shows exactly this. The two-stage ranker (0.215) adds 0.19 dollars per visit more than trending (0.151). But the 95% interval goes from −1.75 to +2.05, so it is not clearly better online. A higher offline score is a reason to run an online test, not a result.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Rafael asks for the guardrails of the weekly email. Name three, and say why each one matters.
Show the answer
The unsubscribe rate, because a lost subscriber cannot get any email again (18 dollars a year under our assumption). No sensitive products in a shared family inbox, because others can see them. 3,178 Larkfield customers share an address. Under the behaviour model’s assumption, one such email could cost about 127 subscribers. No personal list for customers who opted out of personal suggestions, because they did not agree to it. The return rate is a good fourth.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Clicks are a weak target. In the constructed example, the list with almost twice the clicks adds less than a third of the money and causes six times the returns.
- For each surface, write three kinds of measure. The business measure is what the shop wants: for the home page, added revenue per visit, kept, without orders made anyway. The offline stand-in is what you can compute from the log: recall@10. The guardrails must not get worse: returns, already-bought products, sensitive products, coverage, unsubscribes.
- recall@10 is the share of a customer’s ordered products that were in their list of 10, averaged over customers. The popular list scores 0.106 on the test period.
- Offline and online can disagree. In the simulated online test, the offline winner is not clearly better online (+0.19 dollars per visit, interval −1.75 to +2.05).
- “Already bought” wastes a place. A sensitive product in a shared inbox can cost a subscriber, and the fix is a rule.
You now know what to measure. But you have only read numbers that someone else computed. Next, you set up the project, open the shop log yourself, and compute the popular list’s score with one command.
Next: Meet the shop log