Next item from browsing sessions
- Case study 2
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we knew each customer's history. In this case study, we suggest the next product from real browsing sessions of anonymous visitors, with clicks, carts and orders.
Case study 2 of 4 · Recommendation and Personalisation: Show Each Person What MattersNext item from browsing sessions
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Suggest the next product to visitors you know nothing about, from real browsing sessions of a large online shop. You split the visits by day, cut each one at a random point, and measure clicks, carts and orders separately. Then you decide what the block should show, what it costs and what a person still checks. Last, you run the design once on a day that no method has seen.
This case study is a notebook, not the larkfield-recs project. It has every code block of this page, in the same order, with two tasks and their checks. Its setup cells install the packages and download the data (20 MB). It runs in about 5 minutes on a laptop CPU. Read the explanations here, and run the code there.
Before you startWhat to know and to have ready
What you need to know
- The time split, from Split by time, and the baselines, from The baselines to beat.
- Recall@k, from Ranking metrics; coverage, from Beyond accuracy; segments and paired differences, from Evaluate by segment.
- Co-occurrence, from People also bought, and ALS, from Hidden tastes.
- Session suggestions and the Markov model, from Sessions, sequences and LLMs, briefly; rules, from Business rules and re-ranking.
What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)
- Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas, SciPy, matplotlib and implicit.
- A new, empty folder. The setup cell downloads the data into a folder
browsing-sessions. To start again, delete it and run the notebook from the top. - About 3 GB of free memory.
Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4, pandas 3.1.1, SciPy 1.18.1 and implicit 0.7.3. Windows, Linux, Colab and Kaggle were not run.
The situation
Section titled “The situation”A stranger walks into a shop. A good shop assistant knows nothing about them. But the assistant watches: the stranger picks up a tent, puts it down, picks up a sleeping mat, then goes back to the tent. The assistant does not suggest a sofa. They say: “The tent is a good choice. People who buy it often take this lamp too.”
An online shop has the same problem many thousands of times an hour. Most visitors are not logged in, so the shop only knows what they did in this visit. Recommending with no history is called cold start. Earlier, in Cold start for new products and new customers, Larkfield’s new customers got trending. Here, we try to do better with the visit itself.
Lena is the product manager of a large online shop. She wants a block of 20 products on each product page, for visitors who are not logged in. She asks: “What should the block show? And how much better is it than our bestsellers?”
Jonas is the engineer who will build it. His questions: “Can the list be ready while the page loads? How much work is it every night?”
Lena and Jonas are made up. They do not work for OTTO. The data is real.
What you will build
Section titled “What you will build”A comparison of eight lists on the same visits and the same day. They come from four families: popularity, the visit itself, neighbours (co-visitation and the last item) and a learned model (ALS). The result is a recommendation for Lena and Jonas: a design, its settings, its cost, and what a person still checks.
The data
Section titled “The data”OTTO is a large online shop in Germany. Its data team published the OTTO Recommender Systems Dataset: anonymised browsing sessions from the shop’s website and app, in 2022.
| Source | github.com/otto-de/recsys-dataset (the files are on Kaggle, which needs an account) |
| Creators | Philipp Normann, Sophie Baumeister and Timo Wilm (OTTO) |
| Licence | Creative Commons Attribution 4.0 International (CC BY 4.0) for the data; the code is MIT |
| Attribution | “OTTO Recommender Systems Dataset” by Philipp Normann, Sophie Baumeister and Timo Wilm (OTTO), licensed under CC BY 4.0. Changed by Nextia Learning: every 5th session of the test week, one row per event, timestamps in seconds. |
| File | otto-sample.csv.gz (20 MB), the course’s fixed sample in the labs repository, with a dataset card |
The full dataset has 12.9 million sessions in four weeks, plus a test week: about 12 GB. That is too large for a laptop and for a 5-minute notebook, and its download needs an account. So the course made a sample: every 5th session of the test week, Monday 29 August to Sunday 4 September 2022. Whole sessions are kept. The sample has 2,763,918 events in 334,361 sessions, on 491,129 products.
What one row means. One row is one event in one visit. An event is a click on a product page, a product put in the basket (a cart), or an order.
| Column | Meaning |
|---|---|
session |
The visit’s ID. In this data, a session is all the activity of one visitor in the week. |
aid |
The product’s ID. It is a number only: there are no names, prices or categories. |
ts |
The time of the event, in seconds (UTC). The shop’s days start at 22:00 UTC, which is midnight in Germany. |
type |
clicks, carts or orders. |
Of the events, 2,463,954 are clicks, 229,815 are carts and 70,149 are orders. Half of the visits have 4 events or fewer; 27% have exactly 2.
Other open session datasets were not allowed. Retailrocket is CC BY-NC-SA 4.0 (no commercial use). YOOCHOOSE and Diginetica have no licence that we could find.
How this data differs from Larkfield
Section titled “How this data differs from Larkfield”Larkfield’s shop log is synthetic: our code made it, with a model of how customers behave. This data is real behaviour. Three differences matter:
- The visitors are strangers. OTTO’s sessions are anonymous, and the test week’s visitors are not in the training weeks. Larkfield’s customers have IDs and months of history.
- The catalogue is huge. 491,129 products in a fifth of one week, against Larkfield’s 2,994. 55% of the products were touched in only one visit.
- Products are numbers. No titles, no categories, no descriptions, no stock. A content method from Content-based recommendations cannot work here.
Frame the problem
Section titled “Frame the problem”- When is the list made? At the moment a visitor opens a page, from the events of the visit so far. Tables that need the past (counts, models) are rebuilt every night.
- What do we predict? The rest of the visit: the next click, and every product the visitor carts or orders later in the visit.
- For whom? The visits that start on the test day. They have no history before it.
The random cut
Section titled “The random cut”In real use, the list is made in the middle of a visit. We do not know when the visitor will stop. So we copy that: we cut each test visit at a random point. The events before the cut are visible. We predict the events after it.
Text description of the diagram
The visit has six events. The first three are visible: click A, click B, click A. The cut is random. After it come click C (the next click), cart C and order C. A list is right for clicks if it contains C, and right for carts and orders if it contains C.
OTTO’s Kaggle competition cut its test sessions in the same way. A visit with 2 events is always cut after the first. On the test day, the methods see 4.0 events on average.
The measures
Section titled “The measures”We measure recall@20 three times, as OTTO’s competition did (we average over visits, as the course does):
- clicks: is the next click in the list of 20? (1 or 0, averaged over visits with a next click);
- carts: the share of the later carted products that are in the list;
- orders: the same for orders.
To compare methods with one number, we use OTTO’s weighted score: 0.1 × clicks + 0.3 × carts + 0.6 × orders. Orders weigh most, because they are what the shop sells. We also report MRR@20 for the next click (OTTO recommends it) and coverage. The course harness’s diversity needs categories, which this data does not have. Each recall gets a 95% interval from the bootstrap, as in Evaluate by segment.
The segments are the visit’s own groups: how many events we have seen so far. 1 event (we know almost nothing), 2 to 4 events, and 5 or more.
The time split
Section titled “The time split”The future must not leak into the past, as in Split by time. We have one week, so we split it by day:
Text description of the diagram
Monday to Thursday are history only. Friday is the validation day: we choose the ALS settings and the design. Saturday is the test day: every method runs once, with everything known before Saturday, and we report these numbers. Sunday is the new input: only the chosen design runs, once, with everything known before Sunday.
The test day has 41,273 visits: 18,689 with 1 event seen, 13,783 with 2 to 4 and 8,801 with 5 or more. 40,221 have a next click, 6,729 cart something later, and 3,362 order something later.
Explore what the decision needs
Section titled “Explore what the decision needs”One question decides which methods can work: do visitors come back to products they already saw? The notebook’s first task computes one part of it.
Text description of the chart
Three columns. Of the next clicks, 32.2% were on a product the visitor had already seen in the visible part of the visit. Of the products carted later, 32.6% had been seen. Of the products ordered later, 61.4% had been seen.
Three in five of the products ordered later were already in the visible part of the visit. People look at a product, look at others, and come back before they buy. So the visit’s own products should be a strong list for orders, and a good one for carts. For the next click, two in three visitors move on to a product they have not seen yet: here the methods must find new products. The next click is on the same product as the last event in 22.2% of the visits.
The second fact is the size of the catalogue. The 1% most visited products have only 23% of the visits to products. So the same 20 bestsellers for everyone will rarely contain what one visitor wants.
The baselines
Section titled “The baselines”We use two baselines.
popular ranks products by the number of visits that touched them in the 24 hours before the day. That is the “session window” of the shop: the products of the moment. Everyone gets the same list, as with trending in The baselines to beat.
def popular(split, hours=24): """Products ranked by the number of visits that touched them in the last `hours` before the day.""" h = split.history recent = h[h["ts"] >= split.start - hours * 3600].drop_duplicates(["session", "item"]) counts = np.bincount(recent["item"], minlength=split.n_items) return np.argsort(-counts, kind="stable")[:100] # 100, so that fill() never runs shortThe second baseline comes from the exploration above. visit so far shows the products of the visit. A cart counts 6 times a click and an order 3 times. Recent events count more: the first event of the visit has weight 0.1, the last has weight 1. A visit with fewer than 20 products is filled with popular products.
On the test day: popular gets a score of 0.014. visit so far gets 0.568 (0.556 to 0.579): clicks 0.327, carts 0.439, orders 0.672.
The 20 bestsellers almost never contain what a visitor wants: 0.010 for the next click. On Larkfield’s test period, trending found the next viewed product 2.7% of the time (hit@10). Here the catalogue is 160 times larger, and popularity is even weaker.
The alternatives
Section titled “The alternatives”Co-visitation
Section titled “Co-visitation”Think of the shop assistant again. Over many days, they noticed: people who pick up the tent often pick up the sleeping mat in the same visit. That memory is co-visitation: for each pair of products, count the past visits in which both were seen.
It is the co-occurrence of People also bought, with two changes. We count visits, not customers, because visitors are strangers. And we count only pairs seen within 24 hours of each other.
Here it is on three tiny visits (an example we made):
| Visit | Products, in order |
|---|---|
| 1 | tent, sleeping mat, lamp |
| 2 | tent, sleeping mat |
| 3 | tent, lamp, stove |
| Visitors who looked at the tent also looked at | Visits |
|---|---|
| lamp | 2 of 3 |
| sleeping mat | 2 of 3 |
| stove | 1 of 3 |
A visitor on the tent page gets the lamp and the sleeping mat first (a tie), then the stove. Each visit counts once, even if the visitor looked at the tent five times.
On the real data, we use the last 30 events of each past visit and count each pair once per visit. We keep the 20 best neighbours of each product. For a test visit, a product’s score is the sum of its counts with the visit’s products, weighted like visit so far:
def covisit_scores(split, C): """Score each product by its co-visitation counts with the products of the visit, weighted by session_weights (recent events and carts count more).""" return session_matrix(split, session_weights(split)) @ CBefore Saturday, the table has 4,162,629 pairs and takes 16 seconds to build. Alone, co-visitation gets a score of 0.109. That is far below visit so far. The reason is simple: a product is not its own neighbour. So co-visitation never shows the products the visitor comes back to. But for the next click on a new product, it is the best method: 0.181.
The last item (Markov)
Section titled “The last item (Markov)”Sessions, sequences and LLMs, briefly taught the Markov model: count which product came directly after each product in past visits. Suggest the products that most often followed the visitor’s last product. It ignores the rest of the visit.
Alone, it gets 0.062, and 0.133 for the next click on a new product. Below co-visitation (paired difference of the score +0.047 (+0.041 to +0.053)).
At Larkfield, Markov found the next viewed product 62.2% of the time (hit@10, 16,801 moves). The two numbers measure different things, but the gap is real. Larkfield’s synthetic visitors move between a few related products of 2,994. OTTO’s real visitors jump around a catalogue of hundreds of thousands of products. In our sample, most pairs of products appear in only one or two visits.
ALS learns a short vector for each product, as in Hidden tastes. Here each past visit is a row, like a customer. We keep only the 48,655 products that at least 5 past visits touched: a vector learned from one or two visits is noise.
A test visit is new, so it has no vector. We build one from its products: the weighted sum of their vectors. Then we suggest the products whose vectors point most in the same direction (cosine, as in item-cf). We chose the settings on Friday, as in Tune without fooling yourself:
| Factors | Regularisation | Friday score | Fit (s) |
|---|---|---|---|
| 64 | 1 | 0.236 | 35 |
| 128 | 1 | 0.252 | 69 |
| 128 | 10 | 0.248 | 58 |
The best setting is 128 factors with regularisation 1. On Saturday, ALS alone gets 0.275. That is higher than co-visitation, because a product’s vector points in its own direction: ALS can show the visit’s own products. For the next click on a new product, it is weaker (0.071 against 0.181). Fitting it took 73 seconds on one thread.
Combine with the visit
Section titled “Combine with the visit”Each family is good at one part of the job. The visit’s own products find carts and orders. The neighbours find new products for the next click. So we put the visit’s own products first, then fill the free places with a neighbour method, then with popular products.
def combine(first, then): """The products of `first` (in order), then those of `then` that are not in it yet.""" out = np.full_like(first, -1) for r in range(len(first)): seen, row = set(), [] for p in list(first[r]) + list(then[r]): if p >= 0 and p not in seen: seen.add(p) row.append(p) if len(row) == K: break out[r, :len(row)] = row return outOn Friday, before we looked at Saturday, the three combinations scored: visit + co-visitation 0.606, visit + ALS 0.579, visit + Markov 0.595. We chose visit + co-visitation. The notebook’s second task builds it.
A fair comparison
Section titled “A fair comparison”Every method ran on the same 41,273 visits of Saturday, with the same history and the same measures. This is the case study’s comparison table:
| Method | clicks | carts | orders | score (95% interval) | MRR@20 | new next click | coverage |
|---|---|---|---|---|---|---|---|
popular | 0.010 | 0.014 | 0.014 | 0.014 (0.011 to 0.016) | 0.002 | 0.008 | 0.00% |
visit so far | 0.327 | 0.439 | 0.672 | 0.568 (0.556 to 0.579) | 0.268 | 0.007 | 14.78% |
co-visitation | 0.159 | 0.110 | 0.101 | 0.109 (0.103 to 0.115) | 0.056 | 0.181 | 22.03% |
last item (Markov) | 0.111 | 0.062 | 0.055 | 0.062 (0.058 to 0.067) | 0.048 | 0.133 | 17.63% |
ALS | 0.210 | 0.227 | 0.309 | 0.275 (0.265 to 0.286) | 0.153 | 0.071 | 9.11% |
visit + Markov | 0.410 | 0.479 | 0.696 | 0.602 (0.592 to 0.613) | 0.287 | 0.129 | 26.20% |
visit + ALS | 0.369 | 0.460 | 0.685 | 0.586 (0.574 to 0.597) | 0.275 | 0.069 | 18.91% |
| visit + co-visitation | 0.440 | 0.494 | 0.706 | 0.616 (0.605 to 0.626) | 0.292 | 0.174 | 28.61% |
“New next click” is recall@20 of the next click, only on visits whose next click is a product they had not seen. Coverage is the share of the 491,129 products that appear in at least one list.
How to read it:
- Visit + co-visitation is best on the score and on every one of the three recalls. Against
visit so far, the paired difference of the score is +0.048 (+0.044 to +0.052). Most of the gain is on clicks: +0.113 (+0.111 to +0.117). - The visit’s own products do most of the work.
visit so faralone is 92% of the best score. - Co-visitation beats the other neighbours as a filler: against visit + Markov +0.013 (+0.009 to +0.016), against visit + ALS +0.030 (+0.026 to +0.033).
- Popularity is far behind every other method, and its coverage is 20 products.
By segment
Section titled “By segment”The average hides the segments. Here is the score by segment, with 95% intervals:
Text description of the chart
Three panels, one per segment, with six methods each. Visits with 1 event seen: popular 0.015, Markov 0.082, co-visitation 0.105, ALS 0.296, visit so far 0.403, visit + co-visitation 0.501. Visits with 2 to 4 events: popular 0.018, Markov 0.065, co-visitation 0.105, ALS 0.313, visit so far 0.591, visit + co-visitation 0.641. Visits with 5 or more events: popular 0.010, Markov 0.053, co-visitation 0.115, ALS 0.235, visit so far 0.599, visit + co-visitation 0.626.
- 1 event seen.
visit so farshows one product and 19 popular ones; the design shows it and 19 neighbours. Here the neighbours add most. - 5 or more. The visit’s own products fill most of the 20 places, so the neighbours add less.
Failure case: a rule from another shop
Section titled “Failure case: a rule from another shop”Common mistake
Hiding the products the visitor has already seen
Symptom: Jonas copies a rule from the shop’s email recommender: “never show a product the customer already has”. For visitors, he writes it as “never show a product the visitor already saw”. The score falls from 0.616 to 0.057.
Cause: in a visit, the products already seen are the best guess for a cart or an order. The visitor saw 32.6% of later carts and 61.4% of later orders before the cut. The rule removes exactly those.
Diagnose: before you copy a rule, measure how often the next actions are on products already seen. Then compare the lists with and without the rule on the validation day, for clicks, carts and orders separately. Here, the next click on a new product even rises a little (0.174 without the rule, 0.183 with it). But carts fall from 0.494 to 0.066, and orders from 0.706 to 0.041.
Fix: a rule is a decision about one page. On the product page, keep the visit’s own products: “still thinking about these?” is useful there. If Lena wants a block for discovery, give it its own place, with co-visitation only, and measure the next click on new products. The library implicit has the same trap: its recommend hides a user’s own items unless you pass filter_already_liked_items=False.
Text description of the chart
Six columns, in pairs. Clicks: 0.440 without the rule, 0.124 with it. Carts: 0.494 without, 0.066 with. Orders: 0.706 without, 0.041 with.
The recommendation
Section titled “The recommendation”Text description of the diagram
A visitor who is not logged in opens a product page. The service reads the visit’s events so far. It fills 20 places in order. First the visit’s own products by weight, then their co-visitation neighbours, then the popular products of the last 24 hours. The co-visitation table is rebuilt every night. A person checks the lists.
What we tell Lena and Jonas:
- For visitors who are not logged in: visit + co-visitation. In this order:
- the visit’s own products (a cart counts 6 clicks, an order 3, recent events more);
- their co-visitation neighbours (pairs within 24 hours, once per visit, the last 30 events of each visit, 20 neighbours per product);
- popular products of the last 24 hours.
- The fallback is popular. If the co-visitation table is missing, show the visit’s own products, then popular. That is
visit so far, the second-best method (0.568), and it needs no table at all. - Not now: ALS and Markov. They did not beat co-visitation as fillers, and ALS costs a model fit every night.
- What a person checks: the data has no stock, categories or prices, so the lists cannot know them. Before the block goes live, the shop must remove products that are out of stock. It must also decide what to do with adult and sensitive products, as Larkfield’s rules did in Business rules and re-ranking. Each week, a person reviews the three recalls on the last day and the share of lists filled by popular products.
What it costs
Section titled “What it costs”Measured on one thread of an Apple M4, on our sample (a fifth of the traffic). Costs in money are assumptions.
| Item | Visit + co-visitation | Popular |
|---|---|---|
| Nightly build | co-visitation table: 16 s | 0.3 s |
| Table size | 4,162,629 pairs, 35 MB | 100 products |
| One list, when the page opens | 1.8 ms (median), 2.4 ms (95th percentile), in Python | a copy |
| Memory (the whole analysis, all methods, one process) | 2.1 GB peak | less |
| Server, assumption: the full traffic is about 5 times our sample, and the build grows a little faster than the data; a 4 vCPU server at US$0.17 an hour for 1 hour a night | about US$62 a year | about the same |
| Engineering, assumption | a nightly job, a table in memory, a weekly check | almost none |
The real cost is not the server. It is a nightly job that must not fail, and a service that reads the table at each page view. The design adds +0.048 (+0.044 to +0.052) to the score over visit so far, mostly on clicks. Whether that pays depends on what a click on a suggestion earns. Offline recall cannot say that.
So our advice: ship visit so far first (no table), then add co-visitation, and test it online against visit so far before you keep it. Offline wins are not online wins showed why, and Online tests and guardrails showed how.
End to end on new input
Section titled “End to end on new input”Sunday 4 September was never used. We run the whole design once: the co-visitation table from every event before Sunday, then the lists for the visits that start on Sunday.
| Method | clicks | carts | orders | score (95% interval) |
|---|---|---|---|---|
popular | 0.011 | 0.011 | 0.009 | 0.010 (0.008 to 0.013) |
visit so far | 0.310 | 0.429 | 0.665 | 0.559 (0.550 to 0.569) |
| visit + co-visitation | 0.438 | 0.494 | 0.701 | 0.613 (0.604 to 0.621) |
48,554 visits started on Sunday. The order is the same as on Saturday: the design first, visit so far second, popular far behind. Sunday is the busiest day of the week in the sample, with 512,345 events.
One visit, as the service sees it
Section titled “One visit, as the service sees it”In a service, the list is made for one visit at a time, when the page opens. The notebook’s suggest_one does that, and gives the same list as the batch code. We show the first Sunday visit (by ID) with 3 to 6 events seen and both a cart and an order after the cut. We chose it so that all three kinds of events are visible. It is not a typical visit.
The visitor’s events so far: click on product 828562 (minute 0), cart on product 828562 (minute 0.1), click on product 1371545 (minute 0.7).
| Place | Product | Source | After the cut |
|---|---|---|---|
| 1 | 828562 | visit | |
| 2 | 1371545 | visit | carted |
| 3 | 499958 | co-visitation | |
| 4 | 1709866 | co-visitation | |
| 5 | 1517468 | co-visitation | |
| 6 | 1171070 | co-visitation | |
| 7 | 422075 | co-visitation | |
| 8 | 1211184 | co-visitation | |
| 9 | 1817953 | co-visitation | |
| 10 | 894360 | co-visitation | |
| 11 | 169432 | co-visitation | |
| 12 | 14100 | co-visitation | |
| 13 | 747429 | co-visitation | |
| 14 | 796168 | co-visitation | |
| 15 | 1322700 | co-visitation | |
| 16 | 1669315 | co-visitation | |
| 17 | 1812099 | co-visitation | |
| 18 | 884826 | co-visitation | |
| 19 | 1272673 | co-visitation | |
| 20 | 62121 | co-visitation |
After the cut, the visitor clicked product 1752297 next. They carted 6 products and ordered 7. The list contains 1 of the carted products: the one the visitor had already looked at. It contains none of the orders and not the next click. This visitor went on to buy products that were not among the co-visitation neighbours of the first two. In a fifth of one week, most pairs appear in one visit only. So the 18 neighbours here have small counts, and ties go to the product with more past visits.
The products are numbers only, so we cannot say what they are. A person checking the lists would need the shop’s catalogue for that.
Insights and limits
Section titled “Insights and limits”What we learned:
- Watch the visit before you predict. Visitors come back to products they saw. That one fact made
visit so fara strong baseline and the seen-products rule a mistake. - Each family has its own job. The visit’s own products find carts and orders; co-visitation finds the next new product; popularity only fills gaps.
- Simple counting beat the learned model here. ALS was slower and not better as a filler.
- Measure each event type. One average would hide that the design helps clicks most and orders least.
Against Larkfield’s synthetic shop: this real data is harder in two ways. The catalogue is huge, so popularity and neighbour methods find much less. And visitors are strangers, so there is no history at all. It is easier in one way: the visit itself is a strong clue for carts and orders, because people come back to what they saw. Compare the order of the methods, not the values. At Larkfield, Markov and item-cf neighbours were almost equal (62.2% and 62.0% hit@10). Here co-visitation is clearly ahead of Markov.
Limits:
- A sample of one week. We used a fifth of the test week. OTTO’s four training weeks have about 80 times more events than our sample. With them, co-visitation and ALS would learn more, and their numbers would be higher. Use the method here, not the numbers.
- One test day. Saturday may behave differently from a weekday. Before you trust the design, run the same comparison on two or three other days.
- Offline only. Recall says whether the visitor clicked, carted or ordered a product in the list. It does not say the list caused it. OTTO’s own FAQ says that only about 20% of its product page views came from its recommendations; most came from search and product lists.
- No product information. No stock, prices or categories, so the lists cannot respect them.
Responsible use
Section titled “Responsible use”- The visitors are real, and anonymous. The IDs are anonymised numbers. Do not try to identify a visitor or a product, and do not join this data with other data.
- Session suggestions follow people closely. A list that repeats what someone just looked at can feel like being watched. A shared computer can show one person’s browsing to another. Decide which products (health, adult) must never come back in a visible block.
- A recommender decides what people see. Co-visitation and popularity favour products that are already visited. Watch coverage, and give new products a place on purpose.
- Say what is measured. Every number here is offline, on a sample of past data. Do not promise Lena more sales.
- Credit the data. CC BY 4.0 requires the attribution above wherever you use or share the data or your results.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can work with real, messy session data. You chose a measure that fits the business (clicks, carts and orders), and you said no to a rule that does not fit.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
DECISION.md: the design, its settings, the costs with their assumptions, and what a person checks.- The data source, the licence and the attribution. Do not commit the data file: link the source and give the SHA-256 of the sample.
A README template:
# Next-product suggestions for anonymous shop visitors
## QuestionWhat should a product page suggest to visitors who are not logged in,and is it better than the bestsellers?
## Data and licenceOTTO Recommender Systems Dataset (Normann, Baumeister, Wilm; OTTO),CC BY 4.0. A fixed sample: every 5th session of the test week(29 Aug to 4 Sep 2022), 2.76M events (clicks, carts, orders).
## ApproachTime split by day: choose on Friday, test once on Saturday, run the chosendesign once on Sunday. Each test visit cut at a random point. Methods:popular (24 h), the visit so far, co-visitation, last item (Markov), ALS(implicit), and each neighbour method after the visit's own products.Measures: recall@20 for clicks, carts and orders, OTTO's weighted scorewith bootstrap intervals, MRR@20, coverage, by visit length.
## Results(the comparison table and the segment chart)
## DecisionThe visit's own products, then co-visitation, then popular. Do not hideproducts the visitor already saw on the product page.
## LimitsA fifth of one week, one test day, offline only, no product information.How to describe it in two sentences: “I compared session-based recommendation methods on real browsing data from a large online shop, for clicks, carts and orders, with a time split. The visitor’s own products were the strongest signal and co-visitation added the best new products, so I recommended that design over a learned model.”
Go further:
- Run the comparison on Friday as the test day (choose on Thursday). Does the order of the methods stay the same?
- Give carts and orders their own co-visitation table (pairs where the second product was carted or ordered). Does the score for orders rise?
- Add a learned ranker over the candidates, as in A learned ranker. Train it on Friday’s visits and test it on Saturday.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does co-visitation alone score much lower than the visit's own products, and why does it still belong in the design?
Show the answer
A product is not its own neighbour, so co-visitation never shows the products the visitor already saw. But 32.6% of later carts and 61.4% of later orders were products already seen. So it misses many carts and most orders: its score is 0.109 against 0.568. It still belongs in the design because it is the best method for the next click on a new product (0.181). After the visit’s own products, it fills the free places and raises the score to 0.616.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Jonas wants to hide every product the visitor has already seen. What happens, and what do you tell him?
Show the answer
The score falls from 0.616 to 0.057. Carts and orders fall most, because visitors often cart, and usually order, products they looked at earlier in the visit. Tell him that a rule is a decision about one page. Keep the seen products on the product page. If the shop wants discovery, give it its own block and measure the next click on new products.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why do we cut each test visit at a random point, and why do we report clicks, carts and orders separately?
Show the answer
In real use, the list is made in the middle of a visit, and we do not know when the visitor will stop. A random cut copies that: the methods see only the events before the cut. We report the three event types separately because they need different things. The next click is usually a new product, while most orders are products already seen. One average would hide that the design helps clicks most. The weighted score (0.1, 0.3, 0.6) then gives one number for the decision, with orders weighing most.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Anonymous visitors have no history, but the visit itself is a strong clue: 61.4% of later orders were products already seen in the visit.
- Split by day and cut each test visit at a random point. Choose on Friday, test once on Saturday, run once on Sunday.
- Measure clicks, carts and orders separately, then combine them in OTTO’s weighted score.
visit so far(0.568) beats popular (0.014) by far. Co-visitation alone scores lower (0.109), but it finds the best new products for the next click.- The design: the visit’s own products, then co-visitation, then popular (0.616). The fallback is
visit so far. Test it online before you keep it. - A rule from another page can be wrong: hiding seen products cuts the score from 0.616 to 0.057.
Next, we recommend books from six million real ratings, and we compare accuracy with novelty and coverage.
Next: Books for readers