Recommend for an online retailer
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we built Larkfield's recommender on synthetic data. In this case study, we use real transactions from an online retailer, compare four method families with a time split, and report by customer segment.
Case study 1 of 4 · Recommendation and Personalisation: Show Each Person What MattersRecommend for an online retailer
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Build and compare recommenders on two years of real invoices from a UK online shop, cleaned and split by time, by segment. Then you decide what the shop should show and for whom, what it costs and what a person still checks. Last, you run the design once on days that no method has seen.
This case study is a notebook, not the larkfield-recs project. It has every code block of this page, in the same order, with three tasks and their checks. Its setup cells install the packages and download the data (15 MB). It runs in about 3 minutes on a laptop CPU. Read the explanations here, and run the code there.
Before you startWhat to know and to have ready
What you need to know
- The time split, from Split by time, and the baselines, from The baselines to beat.
- Recall@10 and NDCG@10, from Ranking metrics; coverage, novelty and diversity, from Beyond accuracy; segments and paired differences, from Evaluate by segment.
- Item-item collaborative filtering, from Item-item collaborative filtering, and ALS, from Hidden tastes.
- The two-stage ranker, from Why two stages and A learned ranker, and rules, from Business rules and re-ranking.
What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)
- Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas, SciPy, matplotlib, implicit and LightGBM. On a Mac, LightGBM also needs
brew install libomp. - A new, empty folder. The setup cell downloads the data into a folder
online-retailer-recs. To start again, delete it and run the notebook from the top.
Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4, pandas 3.1.1, SciPy 1.18.1, implicit 0.7.3 and LightGBM 4.7.0. Windows, Linux, Colab and Kaggle were not run.
The situation
Section titled “The situation”A UK online shop sells gifts and homeware: tea-light holders, lunch bags, paper chains, cake tins. Many of its customers are not families. They are small gift shops that buy in boxes of 12 or 24 and sell again. A good customer orders every few weeks. You may know the shop from the machine learning course’s case study Segment online-shop customers.
Grace runs the shop’s website. She wants a block of ten products in two places:
- The account page. A customer logs in to order again. The block should help them find what they need quickly.
- The home page. Guests and new customers see it. Nobody is logged in, so it cannot be personal.
Jonah is the shop’s only developer. He will build and run what you recommend. He asks: “Which method, for whom? How long does it take every night? What do I do when it breaks?”
A weak block wastes the best place on the page. A complex model that is only a little better costs Jonah’s time every week.
Grace and Jonah are made up. The data is real.
What you will build
Section titled “What you will build”A comparison of seven methods on the same customers and the same month. They are popularity (popular, trending), the customer’s own history (buy-again), neighbours (co-occurrence, item-cf), ALS (als) and a two-stage ranker. The result is a recommendation for Grace and Jonah: a design, its settings, its cost, and what a person still checks.
The data
Section titled “The data”Online Retail II is a public dataset from the UCI Machine Learning Repository. It has every invoice line of the shop from 1 December 2009 to 9 December 2011.
| Source | UCI Machine Learning Repository, dataset 502: Online Retail II, DOI 10.24432/C5CG6D |
| Creator | Daqing Chen |
| Licence | Creative Commons Attribution 4.0 International (CC BY 4.0) |
| Attribution | “Online Retail II” by Daqing Chen, UCI Machine Learning Repository, https://doi.org/10.24432/C5CG6D, licensed under CC BY 4.0. Changed by Nextia Learning: the two Excel sheets joined into one gzip CSV file. |
| File | online_retail_ii.csv.gz (15 MB), the course’s fixed copy in the labs repository, with a dataset card. We checked it against the UCI file: the same 1,067,371 rows, in the same order. |
One row is one product on one invoice: who bought it, when, how many, and at what price. There are no views, baskets or ratings, so a method can only learn from purchases.
The columns we use:
| Column | Meaning |
|---|---|
Invoice |
The invoice number. A number that starts with C is a cancellation: the customer returned the products. |
StockCode |
The product code, such as 85123A. Some codes are not products: postage, manual changes, bank fees. |
Description |
The product’s name, in capitals, such as WHITE HANGING HEART T-LIGHT HOLDER. |
Quantity |
Units on the line. Negative on a return. |
InvoiceDate |
Date and time of the invoice. The time split uses it. |
Price |
Price of one unit, in pounds sterling. |
Customer ID |
A five-digit number. Missing on 22.8% of the lines. |
How this data differs from Larkfield
Section titled “How this data differs from Larkfield”Larkfield’s shop log is synthetic. This data is real behaviour of real (pseudonymous) customers. Three differences matter:
- Customers come back. Of the 5,852 customers, 1,618 (28%) bought only once. At Larkfield, 53.7% of buyers ordered once.
- Baskets are big. The median invoice has 15 different products. In Larkfield’s test period, a customer ordered 1.77 products on average, in two weeks.
- There is less to learn from. No views, no categories, no stock flag and no product descriptions, only short names.
Clean the lines
Section titled “Clean the lines”Raw invoice lines are not purchases yet. The first cells of the notebook clean them in four steps, and count each one:
raw = read("data/online_retail_ii.csv.gz")lines, steps = clean(raw)for step, n in steps.items(): print(f"{step:16} {n:>9,}")lines 1,067,371repeated lines 34,335no customer ID 235,151not a product 3,641price 0 or less 61kept 794,183return lines 17,587Text description of the diagram
The cleaning removes 34,335 repeated lines and 235,151 lines with no customer ID. It also removes 3,641 lines that are not products and 61 lines with a price of 0 or less. 794,183 lines are left, from 5,852 customers and 4,621 products. Of these, 17,587 are return lines. They are kept apart.
What each step does, and why:
- Repeated lines. The source has two Excel sheets, and both contain 1 to 9 December 2010. An exact copy of a line is not a second purchase.
- No customer ID. Guests, or customers who did not log in: we cannot personalise for them. (They are the people who see the home page.)
- Not a product.
POSTandDOTare postage,Mis a manual change,BANK CHARGESandAMAZONFEEare fees,GIFTandgift_0001_…are vouchers. Nobody wants “postage” in a list of suggestions. - Returns. A line whose invoice starts with
Cis a return, not a sale. Sometimes a customer’s returns cancel all of a product they bought before the cut-off date. Then we do not learn from that pair: a product sent back is not a taste to repeat. Before November 2011, customers returned 2,994 pairs in full.
Frame the problem
Section titled “Frame the problem”- When are the lists made? Every night, from all invoices before that night.
- What do we predict? For each customer who buys in the next period, the products they buy in it.
- The main measure: recall@10, the share of the products a customer bought that are in the first 10 suggestions, averaged over customers. We also report NDCG@10, coverage, novelty, diversity, recall on products that are new to the customer, and recall by segment, as the course’s harness does.
- The segments, as in Evaluate by segment, but counted in invoices before the period. New customers have none, occasional customers have 1 or 2, and loyal customers have 3 or more.
The time split
Section titled “The time split”The future must not leak into the past, as in Split by time. We use calendar months, because these customers order every few weeks:
Text description of the diagram
Validation is October 2011: we try 12 ALS settings and every method, and keep the best. The test is November 2011: each method runs once, with everything known before 1 November. The new input is 1 to 9 December 2011: only the chosen design runs, once. The ranker always learns on the month before the one it predicts.
The test month has 1,656 customers: 190 new, 356 occasional and 1,110 loyal. Most of them are loyal. Larkfield’s test period is the opposite: 952 of its 2,118 customers are new.
Explore what the decision needs
Section titled “Explore what the decision needs”The season
Section titled “The season”Text description of the chart
The line shows the number of customers who bought in each month, from December 2009 to December 2011. Most months have 700 to 1,100 customers. Each year it rises in September and peaks in November: 1,606 customers in November 2010 and 1,660 in November 2011. December 2011 has 614, because the data ends on 9 December.
The test month, November 2011, is the busiest month in the data: shops buy their Christmas stock. The top three of trending before November are a Christmas paper chain kit, a hot water bottle and a jumbo bag. Four of the top ten have “Christmas” in their name. All-time sales would miss the season, as Larkfield’s winter coats showed in July.
Big baskets and repeat buying
Section titled “Big baskets and repeat buying”In November, a customer bought 33.2 different products on average (median 21). A list of 10 can find at most 10 of them, so recall@10 looks small even for a good list. Compare the methods with each other, not with Larkfield’s numbers.
The second number decides this case study. 39.0% of the November (customer, product) pairs are products the customer had bought before. The notebook’s first task computes it. Larkfield’s rules remove products a customer already bought, because most are bought once. Here, buying again is normal: a gift shop that sells lunch bags orders more lunch bags.
Text description of the chart
Three columns. Of the 54,914 (customer, product) pairs of November 2011, 39.0% are products the customer had bought before. 60.3% are products the shop sold before, but not to this customer. 0.8% are products the shop had never sold before 1 November.
Only 0.8% of the November pairs are products never sold before 1 November. No method that learns from the past can show them, but they are few in this month.
The baselines
Section titled “The baselines”We use three baselines. popular ranks the products by the number of customers who ever bought them. trending does the same for the last 28 days before the period. Everyone gets the same list. They come from The baselines to beat:
def popular(period): return np.argsort(-period.popularity(), kind="stable")
def trending(period, days=28): return np.argsort(-period.popularity(days), kind="stable")The third comes from the exploration above: if customers buy the same products again, the simplest personal list is their own history. buy-again shows the customer’s own products: the ones on the most invoices first, then the most recent. A new customer has no history, so the list is filled with trending.
def buy_again_scores(period, rows): """The customer's own products: more invoices first, then the most recent.""" X, D = period.X[rows].toarray(), period.D[rows].toarray() recency = np.divide(1.0, D, out=np.zeros_like(D), where=D > 0) return np.where(X > 0, X + recency, 0.0)On the test month: popular 0.035, trending 0.047, and buy-again 0.139 (95% interval 0.130 to 0.150). Buy-again is three times better than trending. The paired difference over the same customers is +0.092 (+0.083 to +0.102).
At Larkfield, trending (0.156) beat every single personal method. Here, the customer’s own history wins by far, because of the customers, not the method.
The alternatives
Section titled “The alternatives”Co-occurrence and item-cf
Section titled “Co-occurrence and item-cf”Co-occurrence counts the customers who bought both products: “people who bought this also bought that”, as in People also bought. Item-cf divides that count by the size of each product (cosine), so that bestsellers do not win every pair. Both score a product for a customer by adding its similarity to each product the customer bought. Products bought on many invoices weigh more (weight 1 + log of the invoice count).
def neighbour_scores(period, S, rows): """Sum over the customer's products of (weight x similarity); weight = 1 + log(invoices).""" W = period.X[rows].copy() W.data = 1 + np.log(W.data) return np.asarray(W @ S)Co-occurrence gets 0.047 and item-cf 0.063. Co-occurrence is no better than trending: the bestsellers are bought with everything, so they top every list. Item-cf is better, but far below buy-again. Both methods suggest other products, and here the next order is mostly the same products again.
ALS learns a vector for each customer and each product, as in Hidden tastes. The implicit library fits it; the confidence of a pair is 1 + log of its invoice count.
ALS has settings. We tried 12 on the validation month (October), as in Tune without fooling yourself:
Text description of the chart
Two lines, for regularisation 0.1 and 10, over 32, 64, 128 and 256 factors. The line for regularisation 1 lies on the line for 0.1, so it is not drawn. Recall@10 on the validation month rises with the number of factors: with regularisation 10, from 0.082 at 32 factors to 0.104 at 256. Regularisation 10 is the highest at every size.
The best setting is the largest model: 256 factors and regularisation 10. A large ALS model can remember each customer’s own products, so it moves towards buy-again. On the test month, ALS gets 0.122 (0.114 to 0.133). That is clearly better than item-cf, and clearly worse than plain buy-again: the paired difference ALS minus buy-again is −0.017 (−0.023 to −0.011).
The two-stage ranker
Section titled “The two-stage ranker”The two-stage design of Why two stages finds a few hundred candidates with fast sources, then a learned ranker orders them. We changed Larkfield’s design in two ways. There is no content source, because there are no product descriptions. And the customer’s own products (up to 50) are a source: the strongest signal here.
| Source | Candidates | Share of November purchases found (any place) |
|---|---|---|
| trending | 100 | 20.4% |
| item-cf | 100 | 20.0% |
| ALS | 100 | 27.2% |
| own products | up to 50 | 19.5% |
| all four | about 215 per customer | 43.4% |
The four sources together find 43.4% of what customers bought in November. That is the ceiling: no ranker can find more.
The ranker is LightGBM with the objective lambdarank, as in A learned ranker. It describes each candidate with 18 features. They are its place and score in each source, and its popularity (all time, 28 days, 7 days, the same month a year ago). They also include its price and age, and the customer’s history with it (invoices, days since the customer last bought it). It learns on October: candidates and features from before 1 October, labels from what customers bought in October. Then it ranks November’s candidates with features from before 1 November. It never sees November.
The ranker learned from 1,305 customers, 146,321 candidate rows and 17,063 purchases. Its most useful features, by gain:
Text description of the chart
Six columns. own rank (the product’s place in the customer’s own history) has 29.4% of the gain. Then user days since last 8.3%, pop all 7.0%, pair days since 6.4%, pop 7 5.8% and als rank 5.4%.
The ranker uses the customer’s history first, then timing and popularity. It learned something buy-again cannot do: when to suggest a product again, and when a product of the season beats an old favourite.
A fair comparison
Section titled “A fair comparison”Every method ran on the same 1,656 customers of November 2011, with the same history and measures:
| Method | recall@10 (95% interval) | NDCG@10 | new to the customer | coverage | novelty (bits) | diversity | fit (s) |
|---|---|---|---|---|---|---|---|
popular |
0.035 (0.031 to 0.039) | 0.091 | 0.018 | 0.3% | 2.52 | 0.98 | 0.0 |
trending |
0.047 (0.043 to 0.052) | 0.124 | 0.032 | 0.3% | 3.11 | 0.98 | 0.0 |
buy-again |
0.139 (0.130 to 0.150) | 0.273 | 0.006 | 66.8% | 4.08 | 0.97 | 0.0 |
co-occurrence |
0.047 (0.043 to 0.052) | 0.114 | 0.024 | 5.1% | 2.68 | 0.97 | 0.4 |
item-cf |
0.063 (0.058 to 0.069) | 0.147 | 0.026 | 21.1% | 3.15 | 0.94 | 0.4 |
als |
0.122 (0.114 to 0.133) | 0.238 | 0.012 | 28.4% | 3.47 | 0.94 | 7.7 |
two-stage |
0.159 (0.149 to 0.170) | 0.323 | 0.008 | 44.5% | 3.81 | 0.96 | 30.0 |
“New to the customer” is recall on the products the customer had never bought. “Fit” is the training time on one thread (Apple M4); for the two-stage ranker, it includes its sources and its training month.
How to read it:
- The two-stage ranker is best on recall and NDCG. Against buy-again, the paired difference is +0.020 (+0.015 to +0.024): small, but clear. It is better for 672 customers, worse for 299 and the same for 685.
- Buy-again is a very strong baseline. It costs nothing to train and beats ALS, item-cf and co-occurrence.
- Every method is bad at new products for the customer. The best “new to the customer” score in the table is 0.032, for trending. Finding a product a customer has never bought is much harder than reminding them of one.
- Coverage is high for buy-again (66.8%), because every customer’s history is different. The popularity lists show only 10 products to everyone.
By segment
Section titled “By segment”The average hides the segments. Here is recall@10 by segment, with 95% intervals:
Text description of the chart
Three panels, one per segment, with five methods each. New customers (190): trending, item-cf, ALS and buy-again all show the trending list and get 0.044; the two-stage ranker gets 0.035. Occasional customers (356): trending 0.040, item-cf 0.056, ALS 0.124, buy-again 0.107, two-stage 0.135. Loyal customers (1,110): trending 0.050, item-cf 0.068, ALS 0.135, buy-again 0.166, two-stage 0.188.
- Loyal customers gain the most from the ranker: 0.188 against 0.166 for buy-again (+0.022, interval +0.018 to +0.027).
- Occasional customers have one or two invoices. For them, ALS (0.124) is a little better than buy-again (0.107; difference +0.018, interval +0.003 to +0.033). With little history, ALS fills the list with products that similar customers bought. The ranker is better again (0.135; against buy-again +0.028, interval +0.018 to +0.039).
- New customers have no history. Every personal method falls back to trending. The ranker reorders trending’s 100 products with its features, and it scores lower than trending’s own order (0.035 against 0.044). It learned its order on known customers, whose features look different. The difference is not clear: its interval (−0.010, from −0.029 to +0.005) includes zero, because there are only 190 new customers. But the ranker is not better for them, and trending is simpler.
That last point gives the design: the ranker for known customers, and plain trending for new customers. We call it two-stage + trending for new. On the test month, it gets 0.160 (0.150 to 0.171). We chose it on the validation month, before we looked at November. It was also best there: 0.127, against 0.121 for buy-again and 0.126 for the ranker alone.
Failure case: a rule from another shop
Section titled “Failure case: a rule from another shop”Common mistake
Removing products the customer already bought
Symptom: Jonah copies Larkfield’s business rules into the shop. Recall@10 falls from 0.159 to 0.030, below trending.
Cause: the rule removes every product the customer bought before. At Larkfield, that is right: most products are bought once, and an old purchase in the list is wasted space. In this shop, 39.0% of what customers buy next month, they bought before. The rule removes the best suggestions.
Diagnose: before you copy a rule, measure the share of next-period purchases that are repeats. Then compare the list with and without the rule on the validation period, by segment. Here, loyal customers fall from 0.188 to 0.025.
Fix: a rule is a business decision about one shop, so decide it for this shop. Keep the customer’s own products in the account-page block. If Grace wants discovery too, give it its own block. Without the rule, recall on products new to the customer is only 0.008; with it, 0.059. Discovery is a different job.
Text description of the chart
Four columns. Recall@10 of the two-stage ranker without the rule is 0.159, and with the rule 0.030. Recall on products that are new to the customer is 0.008 without the rule and 0.059 with it.
The recommendation
Section titled “The recommendation”Text description of the diagram
A visitor with no invoice (a guest or a new customer) sees trending on the home page: the bestsellers of the last 28 days. A logged-in customer with at least one invoice sees the two-stage list on the account page, made every night. A person checks both lists each week.
What we tell Grace and Jonah:
- Home page, for guests and new customers:
trending, the bestsellers of the last 28 days, by number of customers. It is the best we have for people with no history, and it follows the season. No personal method helps them, because there is nothing to learn from. - Account page, for customers with at least one invoice: the two-stage ranker. Sources: trending 100, item-cf 100, ALS 100 (256 factors, regularisation 10) and the customer’s own products 50. LightGBM
lambdarank, 300 trees, learning rate 0.05, 31 leaves, 100 sampled negatives per customer, trained each month on the month before. No already-bought rule. - The fallback is buy-again. If the nightly job fails, show the customer’s own products. It costs nothing to compute and it is the second-best method (0.139). Jonah can ship it first, in a day, and add the ranker when it works.
- What a person checks each week: products that are out of stock, because the data has no stock flag and the lists cannot know. Products out of season, such as Christmas products after December. And lists that show one product in several colours (the example below). A person also reviews the ranker’s numbers on the last month before each monthly retrain.
What it costs
Section titled “What it costs”Measured on one thread of an Apple M4. Costs in money are assumptions.
| Item | Two-stage ranker | Buy-again |
|---|---|---|
| Training each night (sources and ranker) | 30 s | 0 s |
| Lists for all 5,616 known customers | 11 s | 0.1 s |
| Memory (the whole analysis, all methods, one process) | 2.5 GB peak | less |
| Server time, assumption: a 4 vCPU server at US$0.17 an hour, billed for 1 hour a night | US$0.17 a night, about US$62 a year | about the same |
| Jonah’s time, assumption | a monthly check of the ranker, and a fix when the data changes | almost none |
The server is not the cost: under a minute a night is almost free. The real cost is Jonah’s time: one more model to retrain, check and repair. The ranker adds +0.021 recall@10 over buy-again (+0.017 to +0.025). So the top 10 finds about 15% more of what customers buy. Whether that is worth Jonah’s time depends on what a found product earns, and offline recall cannot say that.
So our advice: ship buy-again first, then add the ranker, and test it online against buy-again before you keep it. Offline wins are not online wins showed why, and Online tests and guardrails showed how.
End to end on new input
Section titled “End to end on new input”The last nine days, 1 to 9 December 2011, were never used. We run the whole design once: the ranker learns on November, then ranks for December with everything known before 1 December. New customers get trending.
| Method | recall@10 (95% interval) | new | occasional | loyal |
|---|---|---|---|---|
trending |
0.052 (0.045 to 0.059) | 0.015 | 0.044 | 0.055 |
buy-again |
0.173 (0.156 to 0.191) | 0.015 | 0.129 | 0.191 |
| two-stage + trending for new | 0.207 (0.189 to 0.226) | 0.015 | 0.156 | 0.228 |
610 customers bought in those nine days: 27 new, 101 occasional and 482 loyal. The order is the same as in November: the design first, buy-again second, trending far behind. The numbers are higher than in November because the period is shorter and fewer products are bought per customer. Only 27 customers are new, so their number says little.
One customer’s list
Section titled “One customer’s list”Customer 12347 is loyal: 7 invoices before December. From 1 to 9 December, they bought 11 different products. Here is their list from the design (the account page):
| Place | Product | Invoices before | Sources | Bought 1–9 Dec |
|---|---|---|---|---|
| 1 | RABBIT NIGHT LIGHT | 2 | trending, item-cf, ALS, own | yes |
| 2 | ALARM CLOCK BAKELIKE PINK | 3 | trending, item-cf, ALS, own | |
| 3 | ALARM CLOCK BAKELIKE GREEN | 3 | trending, item-cf, ALS, own | |
| 4 | ALARM CLOCK BAKELIKE RED | 5 | trending, item-cf, ALS, own | |
| 5 | ALARM CLOCK BAKELIKE CHOCOLATE | 4 | item-cf, ALS, own | |
| 6 | AIRLINE BAG VINTAGE TOKYO 78 | 4 | item-cf, ALS, own | |
| 7 | PINK REGENCY TEACUP AND SAUCER | 2 | item-cf, ALS, own | |
| 8 | RED TOADSTOOL LED NIGHT LIGHT | 4 | trending, ALS, own | yes |
| 9 | WOODLAND CHARLOTTE BAG | 4 | ALS, own | yes |
| 10 | ROSES REGENCY TEACUP AND SAUCER | 3 | item-cf, ALS, own |
3 of the 10 were bought, so recall@10 for this customer is 3 of 11 = 0.273. Every product in the list is one the customer bought before, and most are found by three or four sources. Look at places 2 to 5: the same alarm clock in four colours. The customer bought each colour before, so the ranker likes each one. A person would show one colour and use the other places for something else. That is the check in point 4 of the recommendation.
Insights and limits
Section titled “Insights and limits”What we learned:
- Know your customers before you choose a method. In this shop, most buyers come back and buy the same products again. That one fact made buy-again a strong baseline, and the already-bought rule a mistake.
- The ranker helps, but not much over a good baseline. +0.020 recall@10 over buy-again on the test month. Compare the ranker with the best simple method, not only with trending.
- New customers need a different method. The ranker, trained on known customers, was not better than trending for them.
- Discovery is hard here. No method found many products that were new to the customer. That needs another block, other data (views, descriptions) and an online test.
Against Larkfield’s synthetic shop: this real data is easier for personal methods, because customers buy the same products again. It is harder in other ways. Big baskets keep recall@10 low, and new customers have no views at all. There are no categories, stock or descriptions, so there is no content source and no stock rule. Compare the order of the methods, not the values. On Larkfield’s test period, the two-stage ranker was also best (0.234) and item-cf was also weak (0.088).
Limits:
- One shop, 2009 to 2011. Online shops and their customers have changed since then. Use the method here, not the numbers.
- One test month, and it is November, the Christmas peak. A ranker trained on October learns the start of the season. In a quiet month, the result may differ. Before you trust the design, run the same comparison on two or three other months.
- Offline only. Recall says whether customers bought the products in the list. It does not say whether the list caused the purchase. A gift shop that orders lunch bags every month will order them anyway. Only an online test can measure what the block adds.
- Guests are left out. 22.8% of the raw lines have no customer ID. Our measure does not see the home page’s guests.
Responsible use
Section titled “Responsible use”- The customers are real. Customer IDs are numbers only, with a country. Do not try to identify a customer, and do not join this data with other data about the shop’s customers.
- A recommender decides what people see. Lists that repeat old purchases keep new products out of view. Watch coverage, and give new products a place on purpose, as in Cold start for new products and new customers.
- Say what is measured. Every number here is offline, on past data. Do not promise Grace more sales.
- Credit the data. CC BY 4.0 requires the attribution above wherever you use or share the data or your results.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can turn a vague request (“put some suggestions on the page”) into a design you can defend. It also shows that you can say no to a rule that does not fit.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
DECISION.md: the recommendation, the settings, the costs with their assumptions, and the weekly checks.- The data source, the licence and the attribution. Do not commit the data file: link the source and give its SHA-256.
A README template:
# Product suggestions for an online gift wholesaler
## QuestionWhat should the shop show in a block of 10 products, for guests, new customersand returning customers, and is a learned ranker worth running?
## Data and licenceOnline Retail II (UCI, Daqing Chen, doi:10.24432/C5CG6D), CC BY 4.0.Invoice lines 2009-12 to 2011-12. Cleaned: repeated lines, no customer ID,non-product codes, price <= 0; returns used to drop pairs returned in full.
## ApproachTime split by month: choose on October 2011, test once on November 2011,run the chosen design once on 1-9 December 2011. Methods: popular, trending,buy-again, co-occurrence, item-cf, ALS (implicit), two-stage (LightGBM lambdarank).Measures: recall@10 with bootstrap intervals, NDCG@10, coverage, novelty,diversity, recall on new-to-customer products, by segment.
## Results(the comparison table and the segment chart)
## DecisionTrending for guests and new customers; two-stage for known customers; buy-againas the fallback. No already-bought rule (it cuts recall by four fifths here).
## LimitsOne shop, one test month (the Christmas peak), offline only, guests not measured.How to describe it in two sentences: “I compared seven recommendation methods on two years of real invoices, with a time split and customer segments. The customers’ own history was the strongest signal. I recommended a two-stage ranker for returning customers, trending for new ones, and buy-again as the fallback.”
Go further:
- Run the comparison on two other months (for example March and June 2011). Does the order of the methods stay the same?
- Add a diversity rule: at most one product per name group (the alarm clocks). Measure what it costs in recall.
- Make a second block for discovery: item-cf with the customer’s own products removed. Measure “new to the customer” on its own.
Knowledge check
Section titled “Knowledge check”Check your understanding
Larkfield's business rules remove products the customer already bought. Why is that rule wrong for this shop, and how would you check a rule before you copy it?
Show the answer
In this shop, 39.0% of the products customers buy next month are products they bought before: resellers order the same products again. The rule removes those, and recall@10 of the ranker falls from 0.159 to 0.030. Before you copy a rule, measure the share of repeats among next period’s purchases. Then compare the lists with and without the rule on the validation period, by segment.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The two-stage ranker is best on average. Why do we still show trending to new customers?
Show the answer
New customers have no history, so the ranker only reorders trending’s 100 candidates. It learned its order on known customers. For new customers, it scored lower than trending’s own order (0.035 against 0.044). The difference is not clear, because there are only 190 new customers. So the ranker adds nothing for them, while trending is simpler and follows the season. We chose this on the validation month, before the test.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace asks: the ranker finds more of what customers buy, so will it raise sales? What do you answer?
Show the answer
Not necessarily. Recall says that customers bought the products in the list, not that the list made them buy. Many of these products are regular orders that would happen anyway. The ranker adds +0.021 recall@10 over buy-again offline. Ship buy-again first, then test the ranker online against it, and keep it only if it adds sales that pay for Jonah’s time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Real purchase data needs cleaning first: repeated lines, missing customer IDs, non-product codes, and returns (a pair returned in full is not a taste).
- Split by time and choose on a validation month. Here: choose on October 2011, test once on November, run once on 1 to 9 December.
- In this shop, 39.0% of next month’s purchases are repeats. So buy-again (0.139) beats trending (0.047), ALS (0.122) and item-cf (0.063).
- The two-stage ranker is best for known customers (0.159), but it is no better than trending for new customers.
- The design: trending for guests and new customers, the ranker for known customers, buy-again as the fallback, and a weekly check by a person. The server costs almost nothing; Jonah’s time is the cost. Test it online before you keep it.
- A rule from another shop can be wrong: the already-bought rule cuts recall from 0.159 to 0.030 here.
Next, we leave purchases behind. In the next case study, we suggest the next product from real browsing sessions of anonymous visitors, where there is no customer history at all.