Skip to content
Français

Recommend for an online retailer

  • Case study 1
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we built Larkfield's recommender on synthetic data. In this case study, we use real transactions from an online retailer, compare four method families with a time split, and report by customer segment.

What you will be able to do

Build and compare recommenders on two years of real invoices from a UK online shop, cleaned and split by time, by segment. Then you decide what the shop should show and for whom, what it costs and what a person still checks. Last, you run the design once on days that no method has seen.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Cleaning purchase data
  • Time split
  • Segments
  • Popularity and buy-again baselines
  • Co-occurrence and item-cf
  • ALS
  • Two-stage ranker
  • Fair comparison
  • A design decision with costs

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is a notebook, not the larkfield-recs project. It has every code block of this page, in the same order, with three tasks and their checks. Its setup cells install the packages and download the data (15 MB). It runs in about 3 minutes on a laptop CPU. Read the explanations here, and run the code there.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)

  • Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas, SciPy, matplotlib, implicit and LightGBM. On a Mac, LightGBM also needs brew install libomp.
  • A new, empty folder. The setup cell downloads the data into a folder online-retailer-recs. To start again, delete it and run the notebook from the top.

Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4, pandas 3.1.1, SciPy 1.18.1, implicit 0.7.3 and LightGBM 4.7.0. Windows, Linux, Colab and Kaggle were not run.

A UK online shop sells gifts and homeware: tea-light holders, lunch bags, paper chains, cake tins. Many of its customers are not families. They are small gift shops that buy in boxes of 12 or 24 and sell again. A good customer orders every few weeks. You may know the shop from the machine learning course’s case study Segment online-shop customers.

Grace runs the shop’s website. She wants a block of ten products in two places:

  • The account page. A customer logs in to order again. The block should help them find what they need quickly.
  • The home page. Guests and new customers see it. Nobody is logged in, so it cannot be personal.

Jonah is the shop’s only developer. He will build and run what you recommend. He asks: “Which method, for whom? How long does it take every night? What do I do when it breaks?”

A weak block wastes the best place on the page. A complex model that is only a little better costs Jonah’s time every week.

Grace and Jonah are made up. The data is real.

A comparison of seven methods on the same customers and the same month. They are popularity (popular, trending), the customer’s own history (buy-again), neighbours (co-occurrence, item-cf), ALS (als) and a two-stage ranker. The result is a recommendation for Grace and Jonah: a design, its settings, its cost, and what a person still checks.

Online Retail II is a public dataset from the UCI Machine Learning Repository. It has every invoice line of the shop from 1 December 2009 to 9 December 2011.

Source UCI Machine Learning Repository, dataset 502: Online Retail II, DOI 10.24432/C5CG6D
Creator Daqing Chen
Licence Creative Commons Attribution 4.0 International (CC BY 4.0)
Attribution “Online Retail II” by Daqing Chen, UCI Machine Learning Repository, https://doi.org/10.24432/C5CG6D, licensed under CC BY 4.0. Changed by Nextia Learning: the two Excel sheets joined into one gzip CSV file.
File online_retail_ii.csv.gz (15 MB), the course’s fixed copy in the labs repository, with a dataset card. We checked it against the UCI file: the same 1,067,371 rows, in the same order.

One row is one product on one invoice: who bought it, when, how many, and at what price. There are no views, baskets or ratings, so a method can only learn from purchases.

The columns we use:

Column Meaning
Invoice The invoice number. A number that starts with C is a cancellation: the customer returned the products.
StockCode The product code, such as 85123A. Some codes are not products: postage, manual changes, bank fees.
Description The product’s name, in capitals, such as WHITE HANGING HEART T-LIGHT HOLDER.
Quantity Units on the line. Negative on a return.
InvoiceDate Date and time of the invoice. The time split uses it.
Price Price of one unit, in pounds sterling.
Customer ID A five-digit number. Missing on 22.8% of the lines.

Larkfield’s shop log is synthetic. This data is real behaviour of real (pseudonymous) customers. Three differences matter:

  • Customers come back. Of the 5,852 customers, 1,618 (28%) bought only once. At Larkfield, 53.7% of buyers ordered once.
  • Baskets are big. The median invoice has 15 different products. In Larkfield’s test period, a customer ordered 1.77 products on average, in two weeks.
  • There is less to learn from. No views, no categories, no stock flag and no product descriptions, only short names.

Raw invoice lines are not purchases yet. The first cells of the notebook clean them in four steps, and count each one:

raw = read("data/online_retail_ii.csv.gz")
lines, steps = clean(raw)
for step, n in steps.items():
print(f"{step:16} {n:>9,}")
lines 1,067,371
repeated lines 34,335
no customer ID 235,151
not a product 3,641
price 0 or less 61
kept 794,183
return lines 17,587
Four boxes from left to right. 1,067,371 invoice lines. Remove 34,335 repeated lines, 235,151 lines with no customer ID, 3,641 lines that are not a product (postage, manual changes, fees and vouchers) and 61 lines with a price of 0 or less. The result: 794,183 lines, 5,852 customers and 4,621 products. The 17,587 return lines, whose invoice starts with C, are kept apart and used to remove pairs that were returned in full.
From invoice lines to purchases: four cleaning steps, and returns kept apart.
Text description of the diagram

The cleaning removes 34,335 repeated lines and 235,151 lines with no customer ID. It also removes 3,641 lines that are not products and 61 lines with a price of 0 or less. 794,183 lines are left, from 5,852 customers and 4,621 products. Of these, 17,587 are return lines. They are kept apart.

What each step does, and why:

  • Repeated lines. The source has two Excel sheets, and both contain 1 to 9 December 2010. An exact copy of a line is not a second purchase.
  • No customer ID. Guests, or customers who did not log in: we cannot personalise for them. (They are the people who see the home page.)
  • Not a product. POST and DOT are postage, M is a manual change, BANK CHARGES and AMAZONFEE are fees, GIFT and gift_0001_… are vouchers. Nobody wants “postage” in a list of suggestions.
  • Returns. A line whose invoice starts with C is a return, not a sale. Sometimes a customer’s returns cancel all of a product they bought before the cut-off date. Then we do not learn from that pair: a product sent back is not a taste to repeat. Before November 2011, customers returned 2,994 pairs in full.
  • When are the lists made? Every night, from all invoices before that night.
  • What do we predict? For each customer who buys in the next period, the products they buy in it.
  • The main measure: recall@10, the share of the products a customer bought that are in the first 10 suggestions, averaged over customers. We also report NDCG@10, coverage, novelty, diversity, recall on products that are new to the customer, and recall by segment, as the course’s harness does.
  • The segments, as in Evaluate by segment, but counted in invoices before the period. New customers have none, occasional customers have 1 or 2, and loyal customers have 3 or more.

The future must not leak into the past, as in Split by time. We use calendar months, because these customers order every few weeks:

Three boxes from left to right. Choose, with many runs: validation on October 2011, with 12 ALS settings and every method; the ranker learns on September. The chosen settings go to the test, one run each: November 2011, with the history before 1 November; the ranker learns on October. The chosen design goes to the new input, once: 1 to 9 December 2011, with the history before 1 December; the ranker learns on November.
Choose on October, test once on November, then run the chosen design once on 1 to 9 December. The ranker always learns on the month before the one it predicts.
Text description of the diagram

Validation is October 2011: we try 12 ALS settings and every method, and keep the best. The test is November 2011: each method runs once, with everything known before 1 November. The new input is 1 to 9 December 2011: only the chosen design runs, once. The ranker always learns on the month before the one it predicts.

The test month has 1,656 customers: 190 new, 356 occasional and 1,110 loyal. Most of them are loyal. Larkfield’s test period is the opposite: 952 of its 2,118 customers are new.

Customers who bought in each month (after cleaning). The shop is busiest in October and November, when shops buy their Christmas stock. December 2011 has only 9 days.
Text description of the chart

The line shows the number of customers who bought in each month, from December 2009 to December 2011. Most months have 700 to 1,100 customers. Each year it rises in September and peaks in November: 1,606 customers in November 2010 and 1,660 in November 2011. December 2011 has 614, because the data ends on 9 December.

The test month, November 2011, is the busiest month in the data: shops buy their Christmas stock. The top three of trending before November are a Christmas paper chain kit, a hot water bottle and a jumbo bag. Four of the top ten have “Christmas” in their name. All-time sales would miss the season, as Larkfield’s winter coats showed in July.

In November, a customer bought 33.2 different products on average (median 21). A list of 10 can find at most 10 of them, so recall@10 looks small even for a good list. Compare the methods with each other, not with Larkfield’s numbers.

The second number decides this case study. 39.0% of the November (customer, product) pairs are products the customer had bought before. The notebook’s first task computes it. Larkfield’s rules remove products a customer already bought, because most are bought once. Here, buying again is normal: a gift shop that sells lunch bags orders more lunch bags.

What customers bought in November 2011 (54,914 customer and product pairs). Almost two in five were products the customer had bought before.
Text description of the chart

Three columns. Of the 54,914 (customer, product) pairs of November 2011, 39.0% are products the customer had bought before. 60.3% are products the shop sold before, but not to this customer. 0.8% are products the shop had never sold before 1 November.

Only 0.8% of the November pairs are products never sold before 1 November. No method that learns from the past can show them, but they are few in this month.

We use three baselines. popular ranks the products by the number of customers who ever bought them. trending does the same for the last 28 days before the period. Everyone gets the same list. They come from The baselines to beat:

def popular(period):
return np.argsort(-period.popularity(), kind="stable")
def trending(period, days=28):
return np.argsort(-period.popularity(days), kind="stable")

The third comes from the exploration above: if customers buy the same products again, the simplest personal list is their own history. buy-again shows the customer’s own products: the ones on the most invoices first, then the most recent. A new customer has no history, so the list is filled with trending.

def buy_again_scores(period, rows):
"""The customer's own products: more invoices first, then the most recent."""
X, D = period.X[rows].toarray(), period.D[rows].toarray()
recency = np.divide(1.0, D, out=np.zeros_like(D), where=D > 0)
return np.where(X > 0, X + recency, 0.0)

On the test month: popular 0.035, trending 0.047, and buy-again 0.139 (95% interval 0.130 to 0.150). Buy-again is three times better than trending. The paired difference over the same customers is +0.092 (+0.083 to +0.102).

At Larkfield, trending (0.156) beat every single personal method. Here, the customer’s own history wins by far, because of the customers, not the method.

Co-occurrence counts the customers who bought both products: “people who bought this also bought that”, as in People also bought. Item-cf divides that count by the size of each product (cosine), so that bestsellers do not win every pair. Both score a product for a customer by adding its similarity to each product the customer bought. Products bought on many invoices weigh more (weight 1 + log of the invoice count).

def neighbour_scores(period, S, rows):
"""Sum over the customer's products of (weight x similarity); weight = 1 + log(invoices)."""
W = period.X[rows].copy()
W.data = 1 + np.log(W.data)
return np.asarray(W @ S)

Co-occurrence gets 0.047 and item-cf 0.063. Co-occurrence is no better than trending: the bestsellers are bought with everything, so they top every list. Item-cf is better, but far below buy-again. Both methods suggest other products, and here the next order is mostly the same products again.

ALS learns a vector for each customer and each product, as in Hidden tastes. The implicit library fits it; the confidence of a pair is 1 + log of its invoice count.

ALS has settings. We tried 12 on the validation month (October), as in Tune without fooling yourself:

ALS on the validation month (October 2011): 12 settings. More factors and stronger regularisation are better; the best is 256 factors with regularisation 10.
Text description of the chart

Two lines, for regularisation 0.1 and 10, over 32, 64, 128 and 256 factors. The line for regularisation 1 lies on the line for 0.1, so it is not drawn. Recall@10 on the validation month rises with the number of factors: with regularisation 10, from 0.082 at 32 factors to 0.104 at 256. Regularisation 10 is the highest at every size.

The best setting is the largest model: 256 factors and regularisation 10. A large ALS model can remember each customer’s own products, so it moves towards buy-again. On the test month, ALS gets 0.122 (0.114 to 0.133). That is clearly better than item-cf, and clearly worse than plain buy-again: the paired difference ALS minus buy-again is −0.017 (−0.023 to −0.011).

The two-stage design of Why two stages finds a few hundred candidates with fast sources, then a learned ranker orders them. We changed Larkfield’s design in two ways. There is no content source, because there are no product descriptions. And the customer’s own products (up to 50) are a source: the strongest signal here.

Source Candidates Share of November purchases found (any place)
trending 100 20.4%
item-cf 100 20.0%
ALS 100 27.2%
own products up to 50 19.5%
all four about 215 per customer 43.4%

The four sources together find 43.4% of what customers bought in November. That is the ceiling: no ranker can find more.

The ranker is LightGBM with the objective lambdarank, as in A learned ranker. It describes each candidate with 18 features. They are its place and score in each source, and its popularity (all time, 28 days, 7 days, the same month a year ago). They also include its price and age, and the customer’s history with it (invoices, days since the customer last bought it). It learns on October: candidates and features from before 1 October, labels from what customers bought in October. Then it ranks November’s candidates with features from before 1 November. It never sees November.

The ranker learned from 1,305 customers, 146,321 candidate rows and 17,063 purchases. Its most useful features, by gain:

The six most useful features of the ranker, by share of gain. own: the place in the customer's own products; idle: days since the customer's last order; pop.: customers who ever bought it; since: days since this customer bought it; pop. 7d: customers in the last 7 days; ALS: the place in the ALS list.
Text description of the chart

Six columns. own rank (the product’s place in the customer’s own history) has 29.4% of the gain. Then user days since last 8.3%, pop all 7.0%, pair days since 6.4%, pop 7 5.8% and als rank 5.4%.

The ranker uses the customer’s history first, then timing and popularity. It learned something buy-again cannot do: when to suggest a product again, and when a product of the season beats an old favourite.

Every method ran on the same 1,656 customers of November 2011, with the same history and measures:

Method recall@10 (95% interval) NDCG@10 new to the customer coverage novelty (bits) diversity fit (s)
popular 0.035 (0.031 to 0.039) 0.091 0.018 0.3% 2.52 0.98 0.0
trending 0.047 (0.043 to 0.052) 0.124 0.032 0.3% 3.11 0.98 0.0
buy-again 0.139 (0.130 to 0.150) 0.273 0.006 66.8% 4.08 0.97 0.0
co-occurrence 0.047 (0.043 to 0.052) 0.114 0.024 5.1% 2.68 0.97 0.4
item-cf 0.063 (0.058 to 0.069) 0.147 0.026 21.1% 3.15 0.94 0.4
als 0.122 (0.114 to 0.133) 0.238 0.012 28.4% 3.47 0.94 7.7
two-stage 0.159 (0.149 to 0.170) 0.323 0.008 44.5% 3.81 0.96 30.0

“New to the customer” is recall on the products the customer had never bought. “Fit” is the training time on one thread (Apple M4); for the two-stage ranker, it includes its sources and its training month.

How to read it:

  • The two-stage ranker is best on recall and NDCG. Against buy-again, the paired difference is +0.020 (+0.015 to +0.024): small, but clear. It is better for 672 customers, worse for 299 and the same for 685.
  • Buy-again is a very strong baseline. It costs nothing to train and beats ALS, item-cf and co-occurrence.
  • Every method is bad at new products for the customer. The best “new to the customer” score in the table is 0.032, for trending. Finding a product a customer has never bought is much harder than reminding them of one.
  • Coverage is high for buy-again (66.8%), because every customer’s history is different. The popularity lists show only 10 products to everyone.

The average hides the segments. Here is recall@10 by segment, with 95% intervals:

Recall@10 by segment on the test month (November 2011), with 95% intervals. For known customers, the two-stage ranker is best. For new customers, every personal method shows trending, and the ranker is no better.
Text description of the chart

Three panels, one per segment, with five methods each. New customers (190): trending, item-cf, ALS and buy-again all show the trending list and get 0.044; the two-stage ranker gets 0.035. Occasional customers (356): trending 0.040, item-cf 0.056, ALS 0.124, buy-again 0.107, two-stage 0.135. Loyal customers (1,110): trending 0.050, item-cf 0.068, ALS 0.135, buy-again 0.166, two-stage 0.188.

  • Loyal customers gain the most from the ranker: 0.188 against 0.166 for buy-again (+0.022, interval +0.018 to +0.027).
  • Occasional customers have one or two invoices. For them, ALS (0.124) is a little better than buy-again (0.107; difference +0.018, interval +0.003 to +0.033). With little history, ALS fills the list with products that similar customers bought. The ranker is better again (0.135; against buy-again +0.028, interval +0.018 to +0.039).
  • New customers have no history. Every personal method falls back to trending. The ranker reorders trending’s 100 products with its features, and it scores lower than trending’s own order (0.035 against 0.044). It learned its order on known customers, whose features look different. The difference is not clear: its interval (−0.010, from −0.029 to +0.005) includes zero, because there are only 190 new customers. But the ranker is not better for them, and trending is simpler.

That last point gives the design: the ranker for known customers, and plain trending for new customers. We call it two-stage + trending for new. On the test month, it gets 0.160 (0.150 to 0.171). We chose it on the validation month, before we looked at November. It was also best there: 0.127, against 0.121 for buy-again and 0.126 for the ranker alone.

Common mistake

Removing products the customer already bought

Symptom: Jonah copies Larkfield’s business rules into the shop. Recall@10 falls from 0.159 to 0.030, below trending.

Cause: the rule removes every product the customer bought before. At Larkfield, that is right: most products are bought once, and an old purchase in the list is wasted space. In this shop, 39.0% of what customers buy next month, they bought before. The rule removes the best suggestions.

Diagnose: before you copy a rule, measure the share of next-period purchases that are repeats. Then compare the list with and without the rule on the validation period, by segment. Here, loyal customers fall from 0.188 to 0.025.

Fix: a rule is a business decision about one shop, so decide it for this shop. Keep the customer’s own products in the account-page block. If Grace wants discovery too, give it its own block. Without the rule, recall on products new to the customer is only 0.008; with it, 0.059. Discovery is a different job.

The two-stage ranker with and without Larkfield's already-bought rule, on the test month. The rule cuts recall@10 by four fifths and raises recall on products new to the customer.
Text description of the chart

Four columns. Recall@10 of the two-stage ranker without the rule is 0.159, and with the rule 0.030. Recall on products that are new to the customer is 0.008 without the rule and 0.059 with it.

A visitor arrives. The question: logged in, with at least one invoice before tonight? If no, a new customer or a guest, the home page shows trending, the bestsellers of the last 28 days. If yes, an occasional or loyal customer, the account page shows the two-stage lists, made every night. A person checks both lists each week: stock, the season, and one product in four colours.
The recommended design: trending for guests and new customers, the two-stage ranker for customers with history, and a weekly check by a person.
Text description of the diagram

A visitor with no invoice (a guest or a new customer) sees trending on the home page: the bestsellers of the last 28 days. A logged-in customer with at least one invoice sees the two-stage list on the account page, made every night. A person checks both lists each week.

What we tell Grace and Jonah:

  1. Home page, for guests and new customers: trending, the bestsellers of the last 28 days, by number of customers. It is the best we have for people with no history, and it follows the season. No personal method helps them, because there is nothing to learn from.
  2. Account page, for customers with at least one invoice: the two-stage ranker. Sources: trending 100, item-cf 100, ALS 100 (256 factors, regularisation 10) and the customer’s own products 50. LightGBM lambdarank, 300 trees, learning rate 0.05, 31 leaves, 100 sampled negatives per customer, trained each month on the month before. No already-bought rule.
  3. The fallback is buy-again. If the nightly job fails, show the customer’s own products. It costs nothing to compute and it is the second-best method (0.139). Jonah can ship it first, in a day, and add the ranker when it works.
  4. What a person checks each week: products that are out of stock, because the data has no stock flag and the lists cannot know. Products out of season, such as Christmas products after December. And lists that show one product in several colours (the example below). A person also reviews the ranker’s numbers on the last month before each monthly retrain.

Measured on one thread of an Apple M4. Costs in money are assumptions.

Item Two-stage ranker Buy-again
Training each night (sources and ranker) 30 s 0 s
Lists for all 5,616 known customers 11 s 0.1 s
Memory (the whole analysis, all methods, one process) 2.5 GB peak less
Server time, assumption: a 4 vCPU server at US$0.17 an hour, billed for 1 hour a night US$0.17 a night, about US$62 a year about the same
Jonah’s time, assumption a monthly check of the ranker, and a fix when the data changes almost none

The server is not the cost: under a minute a night is almost free. The real cost is Jonah’s time: one more model to retrain, check and repair. The ranker adds +0.021 recall@10 over buy-again (+0.017 to +0.025). So the top 10 finds about 15% more of what customers buy. Whether that is worth Jonah’s time depends on what a found product earns, and offline recall cannot say that.

So our advice: ship buy-again first, then add the ranker, and test it online against buy-again before you keep it. Offline wins are not online wins showed why, and Online tests and guardrails showed how.

The last nine days, 1 to 9 December 2011, were never used. We run the whole design once: the ranker learns on November, then ranks for December with everything known before 1 December. New customers get trending.

Method recall@10 (95% interval) new occasional loyal
trending 0.052 (0.045 to 0.059) 0.015 0.044 0.055
buy-again 0.173 (0.156 to 0.191) 0.015 0.129 0.191
two-stage + trending for new 0.207 (0.189 to 0.226) 0.015 0.156 0.228

610 customers bought in those nine days: 27 new, 101 occasional and 482 loyal. The order is the same as in November: the design first, buy-again second, trending far behind. The numbers are higher than in November because the period is shorter and fewer products are bought per customer. Only 27 customers are new, so their number says little.

Customer 12347 is loyal: 7 invoices before December. From 1 to 9 December, they bought 11 different products. Here is their list from the design (the account page):

PlaceProductInvoices beforeSourcesBought 1–9 Dec
1RABBIT NIGHT LIGHT2trending, item-cf, ALS, ownyes
2ALARM CLOCK BAKELIKE PINK3trending, item-cf, ALS, own
3ALARM CLOCK BAKELIKE GREEN3trending, item-cf, ALS, own
4ALARM CLOCK BAKELIKE RED5trending, item-cf, ALS, own
5ALARM CLOCK BAKELIKE CHOCOLATE4item-cf, ALS, own
6AIRLINE BAG VINTAGE TOKYO 784item-cf, ALS, own
7PINK REGENCY TEACUP AND SAUCER2item-cf, ALS, own
8RED TOADSTOOL LED NIGHT LIGHT4trending, ALS, ownyes
9WOODLAND CHARLOTTE BAG4ALS, ownyes
10ROSES REGENCY TEACUP AND SAUCER3item-cf, ALS, own

3 of the 10 were bought, so recall@10 for this customer is 3 of 11 = 0.273. Every product in the list is one the customer bought before, and most are found by three or four sources. Look at places 2 to 5: the same alarm clock in four colours. The customer bought each colour before, so the ranker likes each one. A person would show one colour and use the other places for something else. That is the check in point 4 of the recommendation.

What we learned:

  • Know your customers before you choose a method. In this shop, most buyers come back and buy the same products again. That one fact made buy-again a strong baseline, and the already-bought rule a mistake.
  • The ranker helps, but not much over a good baseline. +0.020 recall@10 over buy-again on the test month. Compare the ranker with the best simple method, not only with trending.
  • New customers need a different method. The ranker, trained on known customers, was not better than trending for them.
  • Discovery is hard here. No method found many products that were new to the customer. That needs another block, other data (views, descriptions) and an online test.

Against Larkfield’s synthetic shop: this real data is easier for personal methods, because customers buy the same products again. It is harder in other ways. Big baskets keep recall@10 low, and new customers have no views at all. There are no categories, stock or descriptions, so there is no content source and no stock rule. Compare the order of the methods, not the values. On Larkfield’s test period, the two-stage ranker was also best (0.234) and item-cf was also weak (0.088).

Limits:

  • One shop, 2009 to 2011. Online shops and their customers have changed since then. Use the method here, not the numbers.
  • One test month, and it is November, the Christmas peak. A ranker trained on October learns the start of the season. In a quiet month, the result may differ. Before you trust the design, run the same comparison on two or three other months.
  • Offline only. Recall says whether customers bought the products in the list. It does not say whether the list caused the purchase. A gift shop that orders lunch bags every month will order them anyway. Only an online test can measure what the block adds.
  • Guests are left out. 22.8% of the raw lines have no customer ID. Our measure does not see the home page’s guests.
  • The customers are real. Customer IDs are numbers only, with a country. Do not try to identify a customer, and do not join this data with other data about the shop’s customers.
  • A recommender decides what people see. Lists that repeat old purchases keep new products out of view. Watch coverage, and give new products a place on purpose, as in Cold start for new products and new customers.
  • Say what is measured. Every number here is offline, on past data. Do not promise Grace more sales.
  • Credit the data. CC BY 4.0 requires the attribution above wherever you use or share the data or your results.

This case study shows an employer that you can turn a vague request (“put some suggestions on the page”) into a design you can defend. It also shows that you can say no to a rule that does not fit.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • DECISION.md: the recommendation, the settings, the costs with their assumptions, and the weekly checks.
  • The data source, the licence and the attribution. Do not commit the data file: link the source and give its SHA-256.

A README template:

# Product suggestions for an online gift wholesaler
## Question
What should the shop show in a block of 10 products, for guests, new customers
and returning customers, and is a learned ranker worth running?
## Data and licence
Online Retail II (UCI, Daqing Chen, doi:10.24432/C5CG6D), CC BY 4.0.
Invoice lines 2009-12 to 2011-12. Cleaned: repeated lines, no customer ID,
non-product codes, price <= 0; returns used to drop pairs returned in full.
## Approach
Time split by month: choose on October 2011, test once on November 2011,
run the chosen design once on 1-9 December 2011. Methods: popular, trending,
buy-again, co-occurrence, item-cf, ALS (implicit), two-stage (LightGBM lambdarank).
Measures: recall@10 with bootstrap intervals, NDCG@10, coverage, novelty,
diversity, recall on new-to-customer products, by segment.
## Results
(the comparison table and the segment chart)
## Decision
Trending for guests and new customers; two-stage for known customers; buy-again
as the fallback. No already-bought rule (it cuts recall by four fifths here).
## Limits
One shop, one test month (the Christmas peak), offline only, guests not measured.

How to describe it in two sentences: “I compared seven recommendation methods on two years of real invoices, with a time split and customer segments. The customers’ own history was the strongest signal. I recommended a two-stage ranker for returning customers, trending for new ones, and buy-again as the fallback.”

Go further:

  • Run the comparison on two other months (for example March and June 2011). Does the order of the methods stay the same?
  • Add a diversity rule: at most one product per name group (the alarm clocks). Measure what it costs in recall.
  • Make a second block for discovery: item-cf with the customer’s own products removed. Measure “new to the customer” on its own.

Check your understanding

Larkfield's business rules remove products the customer already bought. Why is that rule wrong for this shop, and how would you check a rule before you copy it?

Show the answer

In this shop, 39.0% of the products customers buy next month are products they bought before: resellers order the same products again. The rule removes those, and recall@10 of the ranker falls from 0.159 to 0.030. Before you copy a rule, measure the share of repeats among next period’s purchases. Then compare the lists with and without the rule on the validation period, by segment.

Check your understanding

The two-stage ranker is best on average. Why do we still show trending to new customers?

Show the answer

New customers have no history, so the ranker only reorders trending’s 100 candidates. It learned its order on known customers. For new customers, it scored lower than trending’s own order (0.035 against 0.044). The difference is not clear, because there are only 190 new customers. So the ranker adds nothing for them, while trending is simpler and follows the season. We chose this on the validation month, before the test.

Check your understanding

Grace asks: the ranker finds more of what customers buy, so will it raise sales? What do you answer?

Show the answer

Not necessarily. Recall says that customers bought the products in the list, not that the list made them buy. Many of these products are regular orders that would happen anyway. The ranker adds +0.021 recall@10 over buy-again offline. Ship buy-again first, then test the ranker online against it, and keep it only if it adds sales that pay for Jonah’s time.

Recap

  • Real purchase data needs cleaning first: repeated lines, missing customer IDs, non-product codes, and returns (a pair returned in full is not a taste).
  • Split by time and choose on a validation month. Here: choose on October 2011, test once on November, run once on 1 to 9 December.
  • In this shop, 39.0% of next month’s purchases are repeats. So buy-again (0.139) beats trending (0.047), ALS (0.122) and item-cf (0.063).
  • The two-stage ranker is best for known customers (0.159), but it is no better than trending for new customers.
  • The design: trending for guests and new customers, the ranker for known customers, buy-again as the fallback, and a weekly check by a person. The server costs almost nothing; Jonah’s time is the cost. Test it online before you keep it.
  • A rule from another shop can be wrong: the already-bought rule cuts recall from 0.159 to 0.030 here.

Next, we leave purchases behind. In the next case study, we suggest the next product from real browsing sessions of anonymous visitors, where there is no customer history at all.

Next: Next item from browsing sessions