Skip to content
Français

Next item from browsing sessions

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we knew each customer's history. In this case study, we suggest the next product from real browsing sessions of anonymous visitors, with clicks, carts and orders.

What you will be able to do

Suggest the next product to visitors you know nothing about, from real browsing sessions of a large online shop. You split the visits by day, cut each one at a random point, and measure clicks, carts and orders separately. Then you decide what the block should show, what it costs and what a person still checks. Last, you run the design once on a day that no method has seen.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Session data
  • Time split with a random cut
  • Cold start
  • Popularity baseline
  • Co-visitation
  • Markov transitions
  • ALS
  • Recall per event type
  • Fair comparison
  • A design decision with costs

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is a notebook, not the larkfield-recs project. It has every code block of this page, in the same order, with two tasks and their checks. Its setup cells install the packages and download the data (20 MB). It runs in about 5 minutes on a laptop CPU. Read the explanations here, and run the code there.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)

  • Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas, SciPy, matplotlib and implicit.
  • A new, empty folder. The setup cell downloads the data into a folder browsing-sessions. To start again, delete it and run the notebook from the top.
  • About 3 GB of free memory.

Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4, pandas 3.1.1, SciPy 1.18.1 and implicit 0.7.3. Windows, Linux, Colab and Kaggle were not run.

A stranger walks into a shop. A good shop assistant knows nothing about them. But the assistant watches: the stranger picks up a tent, puts it down, picks up a sleeping mat, then goes back to the tent. The assistant does not suggest a sofa. They say: “The tent is a good choice. People who buy it often take this lamp too.”

An online shop has the same problem many thousands of times an hour. Most visitors are not logged in, so the shop only knows what they did in this visit. Recommending with no history is called cold start. Earlier, in Cold start for new products and new customers, Larkfield’s new customers got trending. Here, we try to do better with the visit itself.

Lena is the product manager of a large online shop. She wants a block of 20 products on each product page, for visitors who are not logged in. She asks: “What should the block show? And how much better is it than our bestsellers?”

Jonas is the engineer who will build it. His questions: “Can the list be ready while the page loads? How much work is it every night?”

Lena and Jonas are made up. They do not work for OTTO. The data is real.

A comparison of eight lists on the same visits and the same day. They come from four families: popularity, the visit itself, neighbours (co-visitation and the last item) and a learned model (ALS). The result is a recommendation for Lena and Jonas: a design, its settings, its cost, and what a person still checks.

OTTO is a large online shop in Germany. Its data team published the OTTO Recommender Systems Dataset: anonymised browsing sessions from the shop’s website and app, in 2022.

Source github.com/otto-de/recsys-dataset (the files are on Kaggle, which needs an account)
Creators Philipp Normann, Sophie Baumeister and Timo Wilm (OTTO)
Licence Creative Commons Attribution 4.0 International (CC BY 4.0) for the data; the code is MIT
Attribution “OTTO Recommender Systems Dataset” by Philipp Normann, Sophie Baumeister and Timo Wilm (OTTO), licensed under CC BY 4.0. Changed by Nextia Learning: every 5th session of the test week, one row per event, timestamps in seconds.
File otto-sample.csv.gz (20 MB), the course’s fixed sample in the labs repository, with a dataset card

The full dataset has 12.9 million sessions in four weeks, plus a test week: about 12 GB. That is too large for a laptop and for a 5-minute notebook, and its download needs an account. So the course made a sample: every 5th session of the test week, Monday 29 August to Sunday 4 September 2022. Whole sessions are kept. The sample has 2,763,918 events in 334,361 sessions, on 491,129 products.

What one row means. One row is one event in one visit. An event is a click on a product page, a product put in the basket (a cart), or an order.

Column Meaning
session The visit’s ID. In this data, a session is all the activity of one visitor in the week.
aid The product’s ID. It is a number only: there are no names, prices or categories.
ts The time of the event, in seconds (UTC). The shop’s days start at 22:00 UTC, which is midnight in Germany.
type clicks, carts or orders.

Of the events, 2,463,954 are clicks, 229,815 are carts and 70,149 are orders. Half of the visits have 4 events or fewer; 27% have exactly 2.

Other open session datasets were not allowed. Retailrocket is CC BY-NC-SA 4.0 (no commercial use). YOOCHOOSE and Diginetica have no licence that we could find.

Larkfield’s shop log is synthetic: our code made it, with a model of how customers behave. This data is real behaviour. Three differences matter:

  • The visitors are strangers. OTTO’s sessions are anonymous, and the test week’s visitors are not in the training weeks. Larkfield’s customers have IDs and months of history.
  • The catalogue is huge. 491,129 products in a fifth of one week, against Larkfield’s 2,994. 55% of the products were touched in only one visit.
  • Products are numbers. No titles, no categories, no descriptions, no stock. A content method from Content-based recommendations cannot work here.
  • When is the list made? At the moment a visitor opens a page, from the events of the visit so far. Tables that need the past (counts, models) are rebuilt every night.
  • What do we predict? The rest of the visit: the next click, and every product the visitor carts or orders later in the visit.
  • For whom? The visits that start on the test day. They have no history before it.

In real use, the list is made in the middle of a visit. We do not know when the visitor will stop. So we copy that: we cut each test visit at a random point. The events before the cut are visible. We predict the events after it.

A visit of six events in a row. Seen, the visit so far: 1. click A, 2. click B, 3. click A. Then a random cut. To predict, the rest of the visit: 4. click C, which is the next click, 5. cart C, 6. order C.
One visit (an example we made), cut at a random point. The methods see events 1 to 3. They must guess the next click (C) and the products carted and ordered later (C).
Text description of the diagram

The visit has six events. The first three are visible: click A, click B, click A. The cut is random. After it come click C (the next click), cart C and order C. A list is right for clicks if it contains C, and right for carts and orders if it contains C.

OTTO’s Kaggle competition cut its test sessions in the same way. A visit with 2 events is always cut after the first. On the test day, the methods see 4.0 events on average.

We measure recall@20 three times, as OTTO’s competition did (we average over visits, as the course does):

  • clicks: is the next click in the list of 20? (1 or 0, averaged over visits with a next click);
  • carts: the share of the later carted products that are in the list;
  • orders: the same for orders.

To compare methods with one number, we use OTTO’s weighted score: 0.1 × clicks + 0.3 × carts + 0.6 × orders. Orders weigh most, because they are what the shop sells. We also report MRR@20 for the next click (OTTO recommends it) and coverage. The course harness’s diversity needs categories, which this data does not have. Each recall gets a 95% interval from the bootstrap, as in Evaluate by segment.

The segments are the visit’s own groups: how many events we have seen so far. 1 event (we know almost nothing), 2 to 4 events, and 5 or more.

The future must not leak into the past, as in Split by time. We have one week, so we split it by day:

Four boxes from left to right. Monday to Thursday: past visits only, the models learn. Friday 2 September: validation, choose the settings and the design. The chosen settings go to Saturday 3 September: the test, every method once, with the history before Saturday; these numbers are reported. The chosen design goes to Sunday 4 September: the new input, the design once, with the history before Sunday.
Choose on Friday, test once on Saturday, then run the chosen design once on Sunday. Each day's models learn only from the events before that day.
Text description of the diagram

Monday to Thursday are history only. Friday is the validation day: we choose the ALS settings and the design. Saturday is the test day: every method runs once, with everything known before Saturday, and we report these numbers. Sunday is the new input: only the chosen design runs, once, with everything known before Sunday.

The test day has 41,273 visits: 18,689 with 1 event seen, 13,783 with 2 to 4 and 8,801 with 5 or more. 40,221 have a next click, 6,729 cart something later, and 3,362 order something later.

One question decides which methods can work: do visitors come back to products they already saw? The notebook’s first task computes one part of it.

What visitors did after the cut on the test day: the share of products they had already seen before the cut.
Text description of the chart

Three columns. Of the next clicks, 32.2% were on a product the visitor had already seen in the visible part of the visit. Of the products carted later, 32.6% had been seen. Of the products ordered later, 61.4% had been seen.

Three in five of the products ordered later were already in the visible part of the visit. People look at a product, look at others, and come back before they buy. So the visit’s own products should be a strong list for orders, and a good one for carts. For the next click, two in three visitors move on to a product they have not seen yet: here the methods must find new products. The next click is on the same product as the last event in 22.2% of the visits.

The second fact is the size of the catalogue. The 1% most visited products have only 23% of the visits to products. So the same 20 bestsellers for everyone will rarely contain what one visitor wants.

We use two baselines.

popular ranks products by the number of visits that touched them in the 24 hours before the day. That is the “session window” of the shop: the products of the moment. Everyone gets the same list, as with trending in The baselines to beat.

def popular(split, hours=24):
"""Products ranked by the number of visits that touched them in the last `hours` before the day."""
h = split.history
recent = h[h["ts"] >= split.start - hours * 3600].drop_duplicates(["session", "item"])
counts = np.bincount(recent["item"], minlength=split.n_items)
return np.argsort(-counts, kind="stable")[:100] # 100, so that fill() never runs short

The second baseline comes from the exploration above. visit so far shows the products of the visit. A cart counts 6 times a click and an order 3 times. Recent events count more: the first event of the visit has weight 0.1, the last has weight 1. A visit with fewer than 20 products is filled with popular products.

On the test day: popular gets a score of 0.014. visit so far gets 0.568 (0.556 to 0.579): clicks 0.327, carts 0.439, orders 0.672.

The 20 bestsellers almost never contain what a visitor wants: 0.010 for the next click. On Larkfield’s test period, trending found the next viewed product 2.7% of the time (hit@10). Here the catalogue is 160 times larger, and popularity is even weaker.

Think of the shop assistant again. Over many days, they noticed: people who pick up the tent often pick up the sleeping mat in the same visit. That memory is co-visitation: for each pair of products, count the past visits in which both were seen.

It is the co-occurrence of People also bought, with two changes. We count visits, not customers, because visitors are strangers. And we count only pairs seen within 24 hours of each other.

Here it is on three tiny visits (an example we made):

Visit Products, in order
1 tent, sleeping mat, lamp
2 tent, sleeping mat
3 tent, lamp, stove
Visitors who looked at the tent also looked at Visits
lamp 2 of 3
sleeping mat 2 of 3
stove 1 of 3

A visitor on the tent page gets the lamp and the sleeping mat first (a tie), then the stove. Each visit counts once, even if the visitor looked at the tent five times.

On the real data, we use the last 30 events of each past visit and count each pair once per visit. We keep the 20 best neighbours of each product. For a test visit, a product’s score is the sum of its counts with the visit’s products, weighted like visit so far:

def covisit_scores(split, C):
"""Score each product by its co-visitation counts with the products of the visit, weighted by
session_weights (recent events and carts count more)."""
return session_matrix(split, session_weights(split)) @ C

Before Saturday, the table has 4,162,629 pairs and takes 16 seconds to build. Alone, co-visitation gets a score of 0.109. That is far below visit so far. The reason is simple: a product is not its own neighbour. So co-visitation never shows the products the visitor comes back to. But for the next click on a new product, it is the best method: 0.181.

Sessions, sequences and LLMs, briefly taught the Markov model: count which product came directly after each product in past visits. Suggest the products that most often followed the visitor’s last product. It ignores the rest of the visit.

Alone, it gets 0.062, and 0.133 for the next click on a new product. Below co-visitation (paired difference of the score +0.047 (+0.041 to +0.053)).

At Larkfield, Markov found the next viewed product 62.2% of the time (hit@10, 16,801 moves). The two numbers measure different things, but the gap is real. Larkfield’s synthetic visitors move between a few related products of 2,994. OTTO’s real visitors jump around a catalogue of hundreds of thousands of products. In our sample, most pairs of products appear in only one or two visits.

ALS learns a short vector for each product, as in Hidden tastes. Here each past visit is a row, like a customer. We keep only the 48,655 products that at least 5 past visits touched: a vector learned from one or two visits is noise.

A test visit is new, so it has no vector. We build one from its products: the weighted sum of their vectors. Then we suggest the products whose vectors point most in the same direction (cosine, as in item-cf). We chose the settings on Friday, as in Tune without fooling yourself:

FactorsRegularisationFriday scoreFit (s)
6410.23635
12810.25269
128100.24858

The best setting is 128 factors with regularisation 1. On Saturday, ALS alone gets 0.275. That is higher than co-visitation, because a product’s vector points in its own direction: ALS can show the visit’s own products. For the next click on a new product, it is weaker (0.071 against 0.181). Fitting it took 73 seconds on one thread.

Each family is good at one part of the job. The visit’s own products find carts and orders. The neighbours find new products for the next click. So we put the visit’s own products first, then fill the free places with a neighbour method, then with popular products.

def combine(first, then):
"""The products of `first` (in order), then those of `then` that are not in it yet."""
out = np.full_like(first, -1)
for r in range(len(first)):
seen, row = set(), []
for p in list(first[r]) + list(then[r]):
if p >= 0 and p not in seen:
seen.add(p)
row.append(p)
if len(row) == K:
break
out[r, :len(row)] = row
return out

On Friday, before we looked at Saturday, the three combinations scored: visit + co-visitation 0.606, visit + ALS 0.579, visit + Markov 0.595. We chose visit + co-visitation. The notebook’s second task builds it.

Every method ran on the same 41,273 visits of Saturday, with the same history and the same measures. This is the case study’s comparison table:

Methodclickscartsordersscore (95% interval)MRR@20new next clickcoverage
popular0.0100.0140.0140.014 (0.011 to 0.016)0.0020.0080.00%
visit so far0.3270.4390.6720.568 (0.556 to 0.579)0.2680.00714.78%
co-visitation0.1590.1100.1010.109 (0.103 to 0.115)0.0560.18122.03%
last item (Markov)0.1110.0620.0550.062 (0.058 to 0.067)0.0480.13317.63%
ALS0.2100.2270.3090.275 (0.265 to 0.286)0.1530.0719.11%
visit + Markov0.4100.4790.6960.602 (0.592 to 0.613)0.2870.12926.20%
visit + ALS0.3690.4600.6850.586 (0.574 to 0.597)0.2750.06918.91%
visit + co-visitation0.4400.4940.7060.616 (0.605 to 0.626)0.2920.17428.61%

“New next click” is recall@20 of the next click, only on visits whose next click is a product they had not seen. Coverage is the share of the 491,129 products that appear in at least one list.

How to read it:

  • Visit + co-visitation is best on the score and on every one of the three recalls. Against visit so far, the paired difference of the score is +0.048 (+0.044 to +0.052). Most of the gain is on clicks: +0.113 (+0.111 to +0.117).
  • The visit’s own products do most of the work. visit so far alone is 92% of the best score.
  • Co-visitation beats the other neighbours as a filler: against visit + Markov +0.013 (+0.009 to +0.016), against visit + ALS +0.030 (+0.026 to +0.033).
  • Popularity is far behind every other method, and its coverage is 20 products.

The average hides the segments. Here is the score by segment, with 95% intervals:

OTTO's weighted score by segment on the test day (Saturday 3 September), with 95% intervals. The longer the visit, the more its own products help. Visit + co-visitation is first in every segment.
Text description of the chart

Three panels, one per segment, with six methods each. Visits with 1 event seen: popular 0.015, Markov 0.082, co-visitation 0.105, ALS 0.296, visit so far 0.403, visit + co-visitation 0.501. Visits with 2 to 4 events: popular 0.018, Markov 0.065, co-visitation 0.105, ALS 0.313, visit so far 0.591, visit + co-visitation 0.641. Visits with 5 or more events: popular 0.010, Markov 0.053, co-visitation 0.115, ALS 0.235, visit so far 0.599, visit + co-visitation 0.626.

  • 1 event seen. visit so far shows one product and 19 popular ones; the design shows it and 19 neighbours. Here the neighbours add most.
  • 5 or more. The visit’s own products fill most of the 20 places, so the neighbours add less.

Common mistake

Hiding the products the visitor has already seen

Symptom: Jonas copies a rule from the shop’s email recommender: “never show a product the customer already has”. For visitors, he writes it as “never show a product the visitor already saw”. The score falls from 0.616 to 0.057.

Cause: in a visit, the products already seen are the best guess for a cart or an order. The visitor saw 32.6% of later carts and 61.4% of later orders before the cut. The rule removes exactly those.

Diagnose: before you copy a rule, measure how often the next actions are on products already seen. Then compare the lists with and without the rule on the validation day, for clicks, carts and orders separately. Here, the next click on a new product even rises a little (0.174 without the rule, 0.183 with it). But carts fall from 0.494 to 0.066, and orders from 0.706 to 0.041.

Fix: a rule is a decision about one page. On the product page, keep the visit’s own products: “still thinking about these?” is useful there. If Lena wants a block for discovery, give it its own place, with co-visitation only, and measure the next click on new products. The library implicit has the same trap: its recommend hides a user’s own items unless you pass filter_already_liked_items=False.

Each pair: visit + co-visitation, then the same lists with the rule 'never show a product the visitor already saw' ('rule'), on the test day. The rule hurts carts and orders most.
Text description of the chart

Six columns, in pairs. Clicks: 0.440 without the rule, 0.124 with it. Carts: 0.494 without, 0.066 with. Orders: 0.706 without, 0.041 with.

A visitor who is not logged in opens a product page. The service reads the visit so far: clicks, carts and orders. It makes 20 suggestions in three steps: 1. the visit's own products, by weight; 2. their co-visitation neighbours, from a table rebuilt each night; 3. popular products of the last 24 hours. A person checks stock, adult and sensitive products, and the weekly numbers.
The recommended design for anonymous visitors: the visit's own products, then co-visitation, then popular products. The table is rebuilt every night; the list is made when the page opens.
Text description of the diagram

A visitor who is not logged in opens a product page. The service reads the visit’s events so far. It fills 20 places in order. First the visit’s own products by weight, then their co-visitation neighbours, then the popular products of the last 24 hours. The co-visitation table is rebuilt every night. A person checks the lists.

What we tell Lena and Jonas:

  1. For visitors who are not logged in: visit + co-visitation. In this order:
    • the visit’s own products (a cart counts 6 clicks, an order 3, recent events more);
    • their co-visitation neighbours (pairs within 24 hours, once per visit, the last 30 events of each visit, 20 neighbours per product);
    • popular products of the last 24 hours.
  2. The fallback is popular. If the co-visitation table is missing, show the visit’s own products, then popular. That is visit so far, the second-best method (0.568), and it needs no table at all.
  3. Not now: ALS and Markov. They did not beat co-visitation as fillers, and ALS costs a model fit every night.
  4. What a person checks: the data has no stock, categories or prices, so the lists cannot know them. Before the block goes live, the shop must remove products that are out of stock. It must also decide what to do with adult and sensitive products, as Larkfield’s rules did in Business rules and re-ranking. Each week, a person reviews the three recalls on the last day and the share of lists filled by popular products.

Measured on one thread of an Apple M4, on our sample (a fifth of the traffic). Costs in money are assumptions.

Item Visit + co-visitation Popular
Nightly build co-visitation table: 16 s 0.3 s
Table size 4,162,629 pairs, 35 MB 100 products
One list, when the page opens 1.8 ms (median), 2.4 ms (95th percentile), in Python a copy
Memory (the whole analysis, all methods, one process) 2.1 GB peak less
Server, assumption: the full traffic is about 5 times our sample, and the build grows a little faster than the data; a 4 vCPU server at US$0.17 an hour for 1 hour a night about US$62 a year about the same
Engineering, assumption a nightly job, a table in memory, a weekly check almost none

The real cost is not the server. It is a nightly job that must not fail, and a service that reads the table at each page view. The design adds +0.048 (+0.044 to +0.052) to the score over visit so far, mostly on clicks. Whether that pays depends on what a click on a suggestion earns. Offline recall cannot say that.

So our advice: ship visit so far first (no table), then add co-visitation, and test it online against visit so far before you keep it. Offline wins are not online wins showed why, and Online tests and guardrails showed how.

Sunday 4 September was never used. We run the whole design once: the co-visitation table from every event before Sunday, then the lists for the visits that start on Sunday.

Methodclickscartsordersscore (95% interval)
popular0.0110.0110.0090.010 (0.008 to 0.013)
visit so far0.3100.4290.6650.559 (0.550 to 0.569)
visit + co-visitation0.4380.4940.7010.613 (0.604 to 0.621)

48,554 visits started on Sunday. The order is the same as on Saturday: the design first, visit so far second, popular far behind. Sunday is the busiest day of the week in the sample, with 512,345 events.

In a service, the list is made for one visit at a time, when the page opens. The notebook’s suggest_one does that, and gives the same list as the batch code. We show the first Sunday visit (by ID) with 3 to 6 events seen and both a cart and an order after the cut. We chose it so that all three kinds of events are visible. It is not a typical visit.

The visitor’s events so far: click on product 828562 (minute 0), cart on product 828562 (minute 0.1), click on product 1371545 (minute 0.7).

PlaceProductSourceAfter the cut
1828562visit
21371545visitcarted
3499958co-visitation
41709866co-visitation
51517468co-visitation
61171070co-visitation
7422075co-visitation
81211184co-visitation
91817953co-visitation
10894360co-visitation
11169432co-visitation
1214100co-visitation
13747429co-visitation
14796168co-visitation
151322700co-visitation
161669315co-visitation
171812099co-visitation
18884826co-visitation
191272673co-visitation
2062121co-visitation

After the cut, the visitor clicked product 1752297 next. They carted 6 products and ordered 7. The list contains 1 of the carted products: the one the visitor had already looked at. It contains none of the orders and not the next click. This visitor went on to buy products that were not among the co-visitation neighbours of the first two. In a fifth of one week, most pairs appear in one visit only. So the 18 neighbours here have small counts, and ties go to the product with more past visits.

The products are numbers only, so we cannot say what they are. A person checking the lists would need the shop’s catalogue for that.

What we learned:

  • Watch the visit before you predict. Visitors come back to products they saw. That one fact made visit so far a strong baseline and the seen-products rule a mistake.
  • Each family has its own job. The visit’s own products find carts and orders; co-visitation finds the next new product; popularity only fills gaps.
  • Simple counting beat the learned model here. ALS was slower and not better as a filler.
  • Measure each event type. One average would hide that the design helps clicks most and orders least.

Against Larkfield’s synthetic shop: this real data is harder in two ways. The catalogue is huge, so popularity and neighbour methods find much less. And visitors are strangers, so there is no history at all. It is easier in one way: the visit itself is a strong clue for carts and orders, because people come back to what they saw. Compare the order of the methods, not the values. At Larkfield, Markov and item-cf neighbours were almost equal (62.2% and 62.0% hit@10). Here co-visitation is clearly ahead of Markov.

Limits:

  • A sample of one week. We used a fifth of the test week. OTTO’s four training weeks have about 80 times more events than our sample. With them, co-visitation and ALS would learn more, and their numbers would be higher. Use the method here, not the numbers.
  • One test day. Saturday may behave differently from a weekday. Before you trust the design, run the same comparison on two or three other days.
  • Offline only. Recall says whether the visitor clicked, carted or ordered a product in the list. It does not say the list caused it. OTTO’s own FAQ says that only about 20% of its product page views came from its recommendations; most came from search and product lists.
  • No product information. No stock, prices or categories, so the lists cannot respect them.
  • The visitors are real, and anonymous. The IDs are anonymised numbers. Do not try to identify a visitor or a product, and do not join this data with other data.
  • Session suggestions follow people closely. A list that repeats what someone just looked at can feel like being watched. A shared computer can show one person’s browsing to another. Decide which products (health, adult) must never come back in a visible block.
  • A recommender decides what people see. Co-visitation and popularity favour products that are already visited. Watch coverage, and give new products a place on purpose.
  • Say what is measured. Every number here is offline, on a sample of past data. Do not promise Lena more sales.
  • Credit the data. CC BY 4.0 requires the attribution above wherever you use or share the data or your results.

This case study shows an employer that you can work with real, messy session data. You chose a measure that fits the business (clicks, carts and orders), and you said no to a rule that does not fit.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • DECISION.md: the design, its settings, the costs with their assumptions, and what a person checks.
  • The data source, the licence and the attribution. Do not commit the data file: link the source and give the SHA-256 of the sample.

A README template:

# Next-product suggestions for anonymous shop visitors
## Question
What should a product page suggest to visitors who are not logged in,
and is it better than the bestsellers?
## Data and licence
OTTO Recommender Systems Dataset (Normann, Baumeister, Wilm; OTTO),
CC BY 4.0. A fixed sample: every 5th session of the test week
(29 Aug to 4 Sep 2022), 2.76M events (clicks, carts, orders).
## Approach
Time split by day: choose on Friday, test once on Saturday, run the chosen
design once on Sunday. Each test visit cut at a random point. Methods:
popular (24 h), the visit so far, co-visitation, last item (Markov), ALS
(implicit), and each neighbour method after the visit's own products.
Measures: recall@20 for clicks, carts and orders, OTTO's weighted score
with bootstrap intervals, MRR@20, coverage, by visit length.
## Results
(the comparison table and the segment chart)
## Decision
The visit's own products, then co-visitation, then popular. Do not hide
products the visitor already saw on the product page.
## Limits
A fifth of one week, one test day, offline only, no product information.

How to describe it in two sentences: “I compared session-based recommendation methods on real browsing data from a large online shop, for clicks, carts and orders, with a time split. The visitor’s own products were the strongest signal and co-visitation added the best new products, so I recommended that design over a learned model.”

Go further:

  • Run the comparison on Friday as the test day (choose on Thursday). Does the order of the methods stay the same?
  • Give carts and orders their own co-visitation table (pairs where the second product was carted or ordered). Does the score for orders rise?
  • Add a learned ranker over the candidates, as in A learned ranker. Train it on Friday’s visits and test it on Saturday.

Check your understanding

Why does co-visitation alone score much lower than the visit's own products, and why does it still belong in the design?

Show the answer

A product is not its own neighbour, so co-visitation never shows the products the visitor already saw. But 32.6% of later carts and 61.4% of later orders were products already seen. So it misses many carts and most orders: its score is 0.109 against 0.568. It still belongs in the design because it is the best method for the next click on a new product (0.181). After the visit’s own products, it fills the free places and raises the score to 0.616.

Check your understanding

Jonas wants to hide every product the visitor has already seen. What happens, and what do you tell him?

Show the answer

The score falls from 0.616 to 0.057. Carts and orders fall most, because visitors often cart, and usually order, products they looked at earlier in the visit. Tell him that a rule is a decision about one page. Keep the seen products on the product page. If the shop wants discovery, give it its own block and measure the next click on new products.

Check your understanding

Why do we cut each test visit at a random point, and why do we report clicks, carts and orders separately?

Show the answer

In real use, the list is made in the middle of a visit, and we do not know when the visitor will stop. A random cut copies that: the methods see only the events before the cut. We report the three event types separately because they need different things. The next click is usually a new product, while most orders are products already seen. One average would hide that the design helps clicks most. The weighted score (0.1, 0.3, 0.6) then gives one number for the decision, with orders weighing most.

Recap

  • Anonymous visitors have no history, but the visit itself is a strong clue: 61.4% of later orders were products already seen in the visit.
  • Split by day and cut each test visit at a random point. Choose on Friday, test once on Saturday, run once on Sunday.
  • Measure clicks, carts and orders separately, then combine them in OTTO’s weighted score.
  • visit so far (0.568) beats popular (0.014) by far. Co-visitation alone scores lower (0.109), but it finds the best new products for the next click.
  • The design: the visit’s own products, then co-visitation, then popular (0.616). The fallback is visit so far. Test it online before you keep it.
  • A rule from another page can be wrong: hiding seen products cuts the score from 0.616 to 0.057.

Next, we recommend books from six million real ratings, and we compare accuracy with novelty and coverage.

Next: Books for readers