Skip to content
Français

What success means

  • Lesson 2
  • 30 min
  • Module 1: The recommendation problem

Where we are

We know what Larkfield's recommender must decide. Now we must say what a good decision is. In this lesson, we compare clicks, orders, added revenue and returns, and we see why more clicks can be a bad target. First, think of how a shop owner judges an assistant.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Choose what “a better list” means for Larkfield before you compare any methods. If you count the wrong thing, you can build a recommender that wins on your dashboard and loses money for the shop.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code.

Think again of the garden shop and its assistants. The owner wants to know which assistant is better. How can she measure it?

She could count conversations: how often each assistant shows a customer a product. One assistant stops every customer at the door and shows them a cheap offer. He has many conversations. But most people say “no, thank you”, and some of them stop coming to the shop.

She could count sales: the money at the till after each assistant’s conversations. That is closer. But one regular customer always buys her compost on the first Saturday of the month. If the assistant says “do you need compost?”, the sale is not thanks to him.

She could count sales that stay sold. Some customers bring back what an assistant pushed on them. The shop gives back the money, and it pays someone to put the product back on the shelf.

So the owner’s real question is: how much extra money, kept, does each assistant bring, without making customers leave? A recommender needs the same question, and the same care.

Tomás tries a new home page list. He calls it “Today’s deals”: cheap, eye-catching products with a big discount label. He compares it with a list that Priya made, “Goes with your table”: products that match what the customer bought before. These are constructed numbers, for the example. Each list is shown on 1,000 visits:

Today’s deals Goes with your table
Visits 1,000 1,000
Clicks on a suggestion 150 80
Orders from a suggestion 30, at 20 dollars each 16, at 45 dollars each
Money from those orders 600 dollars 720 dollars
Orders the customer would have made anyway 15 (300 dollars) 2 (90 dollars)
Orders returned 6 (120 dollars) 1 (45 dollars)
Added money, kept 600 − 300 − 120 = 180 dollars 720 − 90 − 45 = 585 dollars
Added revenue per visit 180 / 1,000 = 0.18 dollars 585 / 1,000 = 0.59 dollars (0.585)
Return handling, at 12 dollars each 6 × 12 = 72 dollars 1 × 12 = 12 dollars

Tomás’s list gets almost twice as many clicks. It also adds less than a third of the money, and it causes six times as many returns. Its cheap deals attract people who were going to buy anyway: they click on a discount for something already on their list. Its impulse products come back more often.

The two constructed lists of the example. The list with more clicks adds less money. The order of the two lists flips from the first panel to the second.
Text description of the chart

The figure has two panels. Each panel has one row for each list, with a dot at its value. In the first panel, clicks per 1,000 visits: Today’s deals 150, Goes with your table 80. In the second panel, added revenue per visit: Today’s deals 0.18 dollars, Goes with your table 0.59 dollars. Today’s deals is ahead in the first panel and far behind in the second.

A click is easy to count, and it happens at once. That is why clicks are tempting as a target. But a click is only the third step of the path from a suggestion to an order. Everything after it decides whether the shop earns money.

The idea: one business measure, one offline stand-in, a few guardrails

Section titled “The idea: one business measure, one offline stand-in, a few guardrails”

Priya and Omar agree on three kinds of measures for each surface.

The business measure is the number that the shop really cares about. For Larkfield’s home page, Omar chooses added revenue per visit. It is the money from suggested products that the customer kept and would not have spent anyway, divided by the number of visits. The word revenue means the money from sales, before costs.

It has one big problem. You can measure it only by showing a list to real customers for some weeks, and comparing it with another list. That is an online test, often called an A/B test. The shop splits customers at random into group A and group B, and each group sees one list. It is slow, and a bad list costs real money while it runs. Module 8 is about online tests.

So, before any customer sees a list, Priya needs a number that she can compute from the shop log alone. A measure computed from past data, without real customers, is called offline. It stands in for the business measure: it estimates the business measure in advance, cheaply, and the same way for every method.

The course’s main offline measure is recall@10 (say “recall at ten”). It takes each customer’s list of 10 products, made with the events before a date. Then it looks at what that customer really ordered in the next two weeks. For one customer:

recall@10 = (ordered products that were in the list) / (all products that the customer ordered)

Then it takes the mean over all customers who ordered. Here is the popular list of 22 June, scored for three real customers of the synthetic log:

Customer Ordered from 22 June to 5 July In the popular list? recall@10
C100434 Bramble Terracotta Mosaic Side Table; Millbrook Blush Pink Folding Side Table; Lumi Green Nordic Coffee Table the Bramble Terracotta Mosaic Side Table: yes; the others: no 1 / 3 = 0.333
C100185 Lumi Oak Compact Round Coffee Table; Millbrook Cream Stoneware Mini Bud Vase no, no 0 / 2 = 0
C100304 Terracotta Linen Roman-Style Curtains yes 1 / 1 = 1

The mean for these three customers is (0.333 + 0 + 1) / 3 = 0.444. For all 2,118 customers who ordered in those two weeks, the sum of the recall values is 223.9. So the popular list’s recall@10 is 223.9 / 2,118 = 0.106. You run this score yourself in the next lesson.

A guardrail is a measure that must not get worse while you improve the main one. The name comes from the barrier at the side of a mountain road. It does not take you anywhere, but it stops you from going over the edge. For the home page, there are four guardrails. They are the return rate, the share of lists with a product that the customer already bought, and sensitive products in the list. The fourth is coverage: the share of the catalogue that the shop shows to at least one customer.

The home page, with 10 suggestions for each visit, has an arrow to its business measure, added revenue per visit, the money from suggestions that the customer would not have spent anyway. A dotted line says that the business measure is estimated in advance by the offline stand-in, recall@10 on the next two weeks, the share of the products that people ordered that were in their list. A second arrow goes from the home page to its guardrails, which must not get worse: return rate, already-bought products, sensitive products, products shown (coverage).
One surface, its measure and its guardrails. The offline stand-in is what you can compute today; the business measure is what the shop wants.
Text description of the diagram

The home page shows 10 suggestions on each visit. Its business measure is added revenue per visit: the money from suggestions that the customer would not have spent anyway. You can measure it only online. A dotted line links it to its offline stand-in, recall@10 on the next two weeks. That is the share of the products that people ordered that were in their list. You can compute it from the shop log today. The home page also has guardrails that must not get worse. They are the return rate, already-bought products, sensitive products, and the share of the catalogue that the shop ever shows.

Long-term loyalty belongs here too. A customer who comes back for years is worth more than one extra sale today. Loyalty is slow to measure, so a team watches signs of it: unsubscribes, and customers who stop visiting. Rafael’s email has the clearest sign: one click on Unsubscribe.

Why the stand-in and the business measure can disagree

Section titled “Why the stand-in and the business measure can disagree”

Recall@10 is useful, cheap and fair. But it is not the business measure, and the two can point in different directions:

  • The log was made without your list. Customers ordered from the old site. So recall counts orders that the customer would have made anyway. Those orders add no money.
  • All 10 places count the same. Online, most people look only at the first few places. Module 3 adds measures that care about the order.
  • Returns, unsubscribes and trust are not in it. Recall counts an order as right even if it comes back.
  • Customers react to the list. A list changes what people see, so it changes what they buy next. Module 8 shows how.

Larkfield’s own data shows this. In Module 8, the course compares three home page lists in an online test. The test is simulated: it runs on our model of customer behaviour, never on real people. Offline, the course’s best method (a two-stage ranker from Module 7) scores recall@10 0.215. A list of what sold best in the last four weeks (“trending”) scores 0.151. That is 42% more for the two-stage ranker. Online, the picture is different:

Three home page lists, offline and in the simulated online test of Module 8 (business rules applied). The clear offline winner is not clearly better online.
Text description of the chart

The figure has two panels with the same three lists. In the first panel, offline recall@10 with business rules: two-stage ranker 0.215, trending 0.151, ALS 0.136. The second panel shows simulated added revenue per visit, with a bar for the 95% interval. Two-stage ranker: 6.82 dollars (5.45 to 8.25). Trending: 6.64 dollars (5.33 to 8.01). ALS: 3.18 dollars (2.25 to 4.10). The two-stage ranker is far ahead offline. Online, its interval and trending’s interval overlap almost completely. ALS is clearly behind online.

Online, the two-stage ranker adds 0.19 dollars per visit more than trending. But the 95% interval of that difference goes from −1.75 to +2.05 dollars. The interval is the range where the true difference probably lies, given how much the result moves by chance. It includes zero, so the test cannot say which list is better. The ranker is not clearly better online. Trending also gets the most clicks: 0.123 clicks per visit, against 0.114 for the ranker and 0.061 for ALS. So the most clicks did not bring clearly more money either.

Tomás is surprised. “The method with the best offline score should win.” Priya answers: “Offline scores choose which methods are worth an online test. They do not replace it.” You learn why this happens, and what to do about it, in Module 8.

Two mistakes matter most, because they cost more than a wasted place.

A product that the customer just bought takes a place and earns nothing. Worse, it tells the customer that the shop does not know them. Some products are bought again, such as compost or pet food, so “already bought” is not always wrong. A rule must know the difference. In Module 7, even the course’s best method, without business rules, shows a product that the customer already bought in 35% of its lists.

Larkfield sells three sensitive subcategories: mobility aids, continence care, and pregnancy and fertility. A sensitive product is not a bad product. The problem is who else sees the suggestion. Continence care is bought again and again, so it is popular: a chair protector is at place 4 of the popular list, which everyone sees. Without rules, 5.7% of the places in the best method’s lists are sensitive health products.

The cost is highest in the email, because 3,178 Larkfield customers share one email address with their family. Here is a worked cost, under the assumptions of the course’s behaviour model. A sensitive product in a shared inbox adds 4 percentage points to the chance that the customer unsubscribes. A subscriber is worth 18 dollars a year. Suppose one weekly email shows a continence product to all 3,178 shared inboxes, and each of them is a subscriber:

Step Value
Extra unsubscribes 3,178 × 0.04 = about 127 customers
Email value lost 127 × 18 dollars = about 2,290 dollars a year
Trust lost not in the table, and not easy to win back

That is from one email. The fix is a rule, not a better model: never put a sensitive product in an email to a shared inbox. Module 7 adds such rules and measures what each one costs.

Priya and Omar fill the table for the home page. Read each line, and check it against the diagram above.

Home page
Business measure added revenue per visit
Offline stand-in recall@10 on the next two weeks
Guardrails return rate; lists with an already-bought product; sensitive products; coverage
Cost of a bad suggestion a wasted place out of 10; a return at 12 dollars; a customer who thinks the shop does not know them

The business measure is in money, because the shop wants money. The offline stand-in is a number that Priya can compute today, for every method, the same way. The guardrails are the things that the business measure does not see quickly.

Fill the same four lines for the product page and for the weekly email. Think about what each surface is for, from A good shop assistant.

Show an answer
Product page Weekly email
Business measure added revenue per product page view, or the added basket value added revenue per email sent, over several weeks
Offline stand-in for a product, the share of the products bought with it, or right after it, that are in its row recall@10 on the week after the email
Guardrails out-of-stock products; products from an unrelated category; return rate unsubscribe rate; sensitive products in shared inboxes; customers who opted out of personal suggestions get no personal list
Cost of a bad suggestion a wasted place and a missed bigger basket a lost subscriber (18 dollars a year under our assumption), and lost trust

For the email, the guardrail matters more than the business measure. One bad email can lose a subscriber for good, and no extra sale today pays for that. Other answers are right too, if each measure belongs to its surface and money comes from what the customer keeps.

Common mistake

The click dashboard

Symptom: Tomás puts “Today’s deals” on the home page for a week. The dashboard shows clicks on suggestions up from 80 to 150 per 1,000 visits. The team celebrates. A month later, Omar sees more returns and no rise in sales.

Cause: the team chose clicks as its target. Clicks are the third step of the path from a suggestion to an order. In the constructed example, the deals list adds 0.18 dollars per visit, against 0.59 for the list that it replaced. It also causes six returns instead of one.

Fix: before you compare lists, write down the business measure, the offline stand-in and the guardrails for the surface. Report clicks only next to them, never alone. Count orders that the customer would have made anyway as zero, and count returns as lost money.

Consequence in AI work: a recommender that learns from clicks learns to get clicks. If clicks are the target, the system becomes very good at the wrong thing, and it does so automatically, every day. Module 8 shows such a loop.

Check your understanding

Explain, with an example, why a recommender that raises clicks can lower revenue.

Show the answer

A click is only one step on the way to money. The products that attract the most clicks are often cheap deals, which people would have bought anyway, or impulse products, which come back more often. In the constructed example, “Today’s deals” gets 150 clicks per 1,000 visits against 80. But 15 of its 30 orders would have happened anyway, and 6 are returned. It adds 0.18 dollars per visit against 0.59 for “Goes with your table”, and it causes 72 dollars of return handling against 12. So the business measure must be added revenue that is kept, with returns as a guardrail.

Check your understanding

Predict and explain: a new method raises recall@10 on the test period from 0.151 to 0.215. Will it add more money on the home page?

Show the answer

Not necessarily. Recall@10 is an offline stand-in. It counts orders that the customer would have made anyway, it treats all 10 places the same, and it does not see returns or unsubscribes. The course’s simulated online test shows exactly this. The two-stage ranker (0.215) adds 0.19 dollars per visit more than trending (0.151). But the 95% interval goes from −1.75 to +2.05, so it is not clearly better online. A higher offline score is a reason to run an online test, not a result.

Check your understanding

Rafael asks for the guardrails of the weekly email. Name three, and say why each one matters.

Show the answer

The unsubscribe rate, because a lost subscriber cannot get any email again (18 dollars a year under our assumption). No sensitive products in a shared family inbox, because others can see them. 3,178 Larkfield customers share an address. Under the behaviour model’s assumption, one such email could cost about 127 subscribers. No personal list for customers who opted out of personal suggestions, because they did not agree to it. The return rate is a good fourth.

Recap

  • Clicks are a weak target. In the constructed example, the list with almost twice the clicks adds less than a third of the money and causes six times the returns.
  • For each surface, write three kinds of measure. The business measure is what the shop wants: for the home page, added revenue per visit, kept, without orders made anyway. The offline stand-in is what you can compute from the log: recall@10. The guardrails must not get worse: returns, already-bought products, sensitive products, coverage, unsubscribes.
  • recall@10 is the share of a customer’s ordered products that were in their list of 10, averaged over customers. The popular list scores 0.106 on the test period.
  • Offline and online can disagree. In the simulated online test, the offline winner is not clearly better online (+0.19 dollars per visit, interval −1.75 to +2.05).
  • “Already bought” wastes a place. A sensitive product in a shared inbox can cost a subscriber, and the fix is a rule.

You now know what to measure. But you have only read numbers that someone else computed. Next, you set up the project, open the shop log yourself, and compute the popular list’s score with one command.

Next: Meet the shop log