Sessions, sequences and LLMs, briefly
- Lesson 23
- 35 min
- Module 7: Two stages: candidates and ranking
Where we are
Our system uses a customer's whole history. Sometimes we only know the current visit. In this lesson, we try simple session suggestions, and we see what an LLM adds for explanations and re-ranking, and what it costs.
Lesson 23 of 29 · Recommendation and Personalisation: Show Each Person What MattersSessions, sequences and LLMs, briefly
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Suggest the next product from the current visit alone, with a simple Markov baseline. Then look at what a language model adds to a recommender, from recorded answers. You will see made-up products, a re-ranking that does not help, slow and paid calls, explanations, and refused requests.
NotebookDo this lesson in a notebook
The notebook counts which product people view next, in all visits before the test period. It scores these Markov suggestions on the test period’s visits. You score the trending list on the same visits. Then it reads the recorded language-model answers and checks which product names really exist. No key is needed. Its setup cells download about 7 MB of practice data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Item-item neighbours on a product page, from Item-item collaborative filtering.
- The ranker and its candidates, from A learned ranker.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject with the files of the last lesson:recs rules two-stage --quickworks. - No model key. The project replays recorded answers.
Tested on macOS with Python 3.12. The Windows (PowerShell) and Linux commands were not run.
The assistant and the stranger
Section titled “The assistant and the stranger”A stranger walks into the shop. The assistant has never seen her. She knows no name and no past orders. But she watches. The stranger looks at a rattan armchair, then picks up a side table.
The assistant has seen this many times. People who look at that armchair often look at a seat cushion next. So she says: “This cushion fits that chair.” She uses only this visit.
This lesson has two parts. First, suggestions from the current visit. Then, a short look at language models in a recommender.
Part 1: suggestions from the current visit
Section titled “Part 1: suggestions from the current visit”The problem: no history
Section titled “The problem: no history”Many visitors are strangers to the recommender. They do not log in, or they are new. Of Larkfield’s 952 new test customers, 633 touched nothing before the period. For them, a home-page ranker can only show what is popular.
But during the visit, each click is a clue. A popular list ignores it. On the test period’s visits, the trending list contained the next product a visitor viewed only 2.7% of the time.
The idea: count what people view next
Section titled “The idea: count what people view next”A session is one visit: the views, basket adds and orders of one visitor, close together in time. In all past sessions, we count how often product B was viewed right after product A. When a visitor is on A, we suggest the products with the highest counts.
This is a Markov model, named after the mathematician Andrey Markov. It assumes that the next step depends only on the current step, not on the steps before.
Text description of the diagram
A visitor is on the page of the Rattan Armchair. In past visits, people on that page viewed the Rattan Side Table next 6 times, the Seat Cushion 4 times and the Garden Parasol once. So the suggestions are, in order: the side table, the cushion, the parasol.
To score it, we use the test period’s visits. For each move from one viewed product to the next, we ask two questions:
- hit@10: is the next product among the 10 suggestions?
- MRR@10 (mean reciprocal rank): 1 divided by its place, or 0 if it is missing. Place 1 gives 1, place 2 gives 0.5, place 4 gives 0.25. MRR is the average.
Larkfield: what people view next
Section titled “Larkfield: what people view next”Text description of the chart
A column chart with four columns, on 16,801 moves from one viewed product to the next in the test period. Markov transitions find the next product among 10 suggestions 62.2% of the time (MRR@10 0.238). Item-cf neighbours of the current product: 62.0% (MRR 0.235). Markov with trending to break ties: 62.8% (MRR 0.241). Trending: 2.7% (MRR 0.011).
The current product is a very strong clue. Markov finds the next product 62.2% of the time; trending finds it 2.7% of the time.
Markov and item-cf are almost equal. Both find products related to the current one. Often these are alternatives that people compare: in the notebook, people on the Driftwood armchair mostly viewed other armchairs next. Item-cf counts customers who touched both, at any time. Markov counts views in order, inside one visit. Here the order adds little. Adding trending to break ties helps a little more: 62.8%.
Sequence models, in one paragraph
Section titled “Sequence models, in one paragraph”Markov looks only at the last product. Neural sequence models read the whole visit. GRU4Rec (2016) uses a recurrent neural network; SASRec (2018) uses self-attention, the idea behind transformers. They can learn that “armchair, then side table, then cushion” means a patio set. They need much more data and longer training. We did not build one for Larkfield, so we do not know whether it would beat 0.622. Start with Markov: it is a strong baseline, and you can explain every suggestion with one count. The case study Next item from browsing sessions works with real sessions.
Part 2: language models, briefly
Section titled “Part 2: language models, briefly”Tomás’s idea: “just ask an LLM”
Section titled “Tomás’s idea: “just ask an LLM””Tomás has a simple plan. Send the customer’s recent products to a large language model (an LLM) and ask what they will buy next. LLMs know a lot about homes and gardens.
Think of a friend who knows a lot about furniture, but has never seen Larkfield’s catalogue. She can give good general advice. Can she name the products on Larkfield’s shelves?
We tested it once, with recorded answers. The calls went to a small chat model on Azure; the course team’s name for it is chat-small. Each answer was recorded once and is replayed by recs llm, so you need no key.
Without a list: made-up products
Section titled “Without a list: made-up products”For 30 customers, the model got the customer’s 10 most recent products and this instruction:
You help Larkfield, an online home and garden shop, choose product suggestions for one customer. You get thecustomer's recent products (newest first, with what they did). Name 10 products from Larkfield's shop that thiscustomer is likely to buy next, best first, with their exact product titles. Answer with JSON:{"products": ["<title>", ...]}.Text description of the chart
A column chart with three columns: invented, copied from the history, and real and new. It counts the 290 product names in the recorded answers for 29 customers. 210 names do not exist in Larkfield’s catalogue. 77 are real products copied from the customer’s own recent products. Only 3 are real products that the customer had not touched. One named product was ordered in the test period.
The results, from recorded answers of chat-small:
- 210 of 290 names do not exist. The model writes titles that sound like Larkfield’s products. A customer who viewed pet bowls got ten cat products, such as “Stainless Steel Cat Water Fountain” and “Wall-Mounted Cat Perch”. Larkfield sells none of them. A made-up answer that sounds right is called a hallucination.
- 77 of the 80 real names are copied from the history. For one customer, the answer was the customer’s ten recent products in a new order. A single-malt whisky and a coffee liqueur were among them.
- Only 1 named product was ordered in the test period.
The friend gives general advice, not Larkfield’s shelves. An LLM cannot know a catalogue it was not given.
With a list: re-ranking the ranker’s top 20
Section titled “With a list: re-ranking the ranker’s top 20”A better use: give the model a list. For 100 customers, the model got the recent products and the ranker’s top 20 candidates, with IDs and titles. It had to choose and order 10, using only IDs from the list, and give a short reason for the first three.
| The ranker alone | Ranker + LLM re-ranking | |
|---|---|---|
| recall@10 on these customers | 0.211 | 0.194 |
| Difference (95% interval) | -0.017 (-0.088 to +0.052): not clear | |
| Made-up IDs | – | 0 |
| Time per customer | about 10 ms | 4.8 s (median), 6.8 s (95th percentile) |
| Cost per 1,000 customers | a few seconds of CPU | US$0.36 |
| Requests refused | 0 | 2 of 100 |
With a list, the model made up no ID. But it did not improve the order: recall fell by 0.017, and the interval includes zero, so we cannot even say it is worse. It is about 470 times slower than the ranker (4.8 seconds against 10 milliseconds), and each call costs money. For a weekly email to all 40,000 customers, the calls would cost about 40 × US$0.36 = US$14.40 a week under this price. Prices change: check the provider’s current page.
Explanations: they sound right, but are they true?
Section titled “Explanations: they sound right, but are they true?”The re-ranking answers include reasons. For customer C100159, who added garden boots to the basket and viewed sun hats and globe light bulbs:
P10448 Sage Green Globe Light Bulb: You recently viewed this sage green globe bulb.P10590 Navy Tilting Patio Parasol: A patio parasol could complement your garden-ready footwear and sun hats.P11781 Oak Fringe-Trimmed Garden Parasol: This garden parasol pairs well with your recent outdoor and gardening interests.The first reason is true. The second is a guess that sounds kind. The third says almost nothing. A written reason is not the real reason the list was chosen. The ranker’s real reasons are its features, such as “viewed 2 days ago”. A template from data, “Because you viewed …”, is always true and costs nothing.
Refused requests
Section titled “Refused requests”The provider’s content filter refused 3 requests: 2 of the 100 re-ranking requests and 1 of the 30 free-form requests. In two of them, the customer’s history had continence or pregnancy products. In the third, a continence product was among the candidates. The provider did not say why.
So an LLM step can fail for reasons you do not control. It needs a fallback: show the ranker’s own list. And sending a customer’s health history to an outside provider is a privacy question in itself. Privacy and sensitive suggestions returns to it.
When to use them, when not, and what they cost
Section titled “When to use them, when not, and what they cost”Use Markov or item-cf neighbours for the product page and for visitors without history. They are cheap and strong: about 62.2% hit@10 on Larkfield’s visits. Use an LLM, if at all, on a short list that your system made. For example, it can write a friendly sentence for an email that a person checks.
Do not use an LLM to name products from memory: 210 of 290 names did not exist. Do not add it to a home page that must answer in 100 milliseconds: one call takes seconds. Do not use it to re-rank before you measure it: here it did not help.
Cost: Markov transitions take about 2 seconds to count and score in the project. LLM re-ranking took 4.8 s a call and US$0.36 per 1,000 customers, with refusals that need a fallback.
Guided practice: score sessions and replay the LLM
Section titled “Guided practice: score sessions and replay the LLM”-
Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.
-
Run section 1. It counts the next views in all visits before the test period, and shows the top next products for one product.
-
Run section 2. It scores the Markov suggestions on the test period’s visits.
-
In section 3, score the trending list on the same visits. The check prints
Check passed. -
Run section 4. It reads the recorded free-form answers and checks each product name against the catalogue.
-
In a terminal, in your
larkfield-recsfolder with the virtual environment active, copy the last files of this module:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m07cp $S/larkfield_recs/session.py larkfield_recs/cp $S/larkfield_recs/commands/session.py larkfield_recs/commands/cp $S/larkfield_recs/commands/llm.py larkfield_recs/commands/cp $S/tests/test_m07.py tests/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m07"Copy-Item "$S\larkfield_recs\session.py" larkfield_recs\Copy-Item "$S\larkfield_recs\commands\session.py" larkfield_recs\commands\Copy-Item "$S\larkfield_recs\commands\llm.py" larkfield_recs\commands\Copy-Item "$S\tests\test_m07.py" tests\ -
Score the next-product methods:
Terminal window recs sessionTest period: 16,801 moves from one viewed product to the nextmethod hit@10 MRR@10markov 0.622 0.238item-cf 0.620 0.235markov+trending 0.628 0.241trending 0.027 0.011 -
Replay and score the recorded re-ranking answers:
Terminal window recs llmRecorded answers from chat-small: 100 customers; refused by the provider's content filter: 2Time per call p50 4.8 s, p95 6.8 s; US$0.36 per 1,000 customersrecall@10 on these customers: ranker 0.211, ranker + LLM re-ranking 0.194; difference -0.017 (95% interval -0.088 to +0.052)IDs not in the candidate list (made up or changed): 0Example reasons for C100159:P10448 Sage Green Globe Light Bulb: You recently viewed this sage green globe bulb.P10590 Navy Tilting Patio Parasol: A patio parasol could complement your garden-ready footwear and sun hats.P11781 Oak Fringe-Trimmed Garden Parasol: This garden parasol pairs well with your recent outdoor and gardening interests. -
Run the tests of the whole module:
Terminal window python -m pytest -qThe last line is
26 passed. Your project now has every file of theend-of-m07snapshot.
Your turn: ask without a list
Section titled “Your turn: ask without a list”Before you run it, predict: of the products the model names without a list, what share exist in Larkfield’s catalogue?
recs llm freeformShow the answer
Recorded answers from chat-small: 30 customers; refused by the provider's content filter: 1Time per call p50 3.6 s, p95 6.6 s; US$0.23 per 1,000 customersProducts named: 290; exact titles that exist in the catalogue: 80 (28%), 77 of them copied from the customer's own history; the other 210 do not existNamed products that the customer ordered in the test period: 1Example: Birka Green Pine Shelf with Drawers; Classic Walnut Low Media Shelf; Cove Brass Corner Display Shelf; Saltmarsh Charcoal Pine Ladder ShelfOnly 28% exist, and almost all of those are the customer’s own recent products. The example is one of them: its four names are the customer’s four most recent products, in a new order. Without a list, the model either repeats the history or invents.
Common mistakes
Section titled “Common mistakes”Common mistake
Product names from an LLM shown as they are
Symptom: Tomás’s prototype email shows “Wall-Mounted Cat Perch”. A customer clicks it and gets an error page.
Cause: the model wrote a plausible title. Larkfield does not sell it. 210 of 290 names in our recorded answers did not exist.
Diagnose: match every name or ID against the catalogue before you show it, and count the misses.
Fix: never let the model name products freely. Give it a list of real candidates, and accept only IDs from that list. Fall back to the ranker’s list when the answer is wrong, slow or refused.
Common mistake
An LLM step with no fallback
Symptom: in a test of the weekly email, some customers get no email at all.
Cause: the provider refused their requests. In our recordings, 3 requests with health products were refused. The code had no plan for an error.
Diagnose: log every failed call with the customer and the error, and count them.
Fix: on any error or timeout, send the ranker’s own list. Better still, do not send sensitive histories to an outside model at all.
Knowledge check
Section titled “Knowledge check”Check your understanding
In past visits, people on a lamp page viewed a shade next 8 times, a bulb 5 times and a rug 2 times. A visitor on the lamp page then views the bulb. What is the Markov list's reciprocal rank for this move?
Show the answer
The Markov list is shade (8), bulb (5), rug (2). The bulb is at place 2, so the reciprocal rank is 1 / 2 = 0.5. The move is also a hit@10, because the bulb is in the list.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Asked to name products, the model invented 210 of 290. Asked to re-rank 20 candidates, it invented none. Why the difference?
Show the answer
Without a list, the model does not know Larkfield’s catalogue, so it writes titles that sound right. With a list, it only has to choose among real IDs that our system found, and the instruction says to use only those IDs. The list turns an open question into a choice.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks: should the home page re-rank with an LLM? Give two numbers from this lesson for your answer.
Show the answer
No, not as tested. Recall@10 changed by −0.017 (interval −0.088 to +0.052): no clear gain. One call took 4.8 seconds (median), far over the 100-millisecond budget, and cost US$0.36 per 1,000 customers. Two of 100 requests were refused and would need the fallback anyway.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A session is one visit. A Markov model suggests what people viewed next after the current product. On Larkfield’s visits it finds the next product 62.2% of the time (MRR 0.238); item-cf 62.0%; trending 2.7%.
- Sequence models such as GRU4Rec and SASRec read the whole visit. We did not build one; Markov is the baseline to beat.
- Without a list, an LLM named 290 products: 210 do not exist, and 77 of the 80 real ones are copied from the history.
- With a list, it invented no ID, but re-ranking did not help: 0.211 → 0.194, not clear. It takes 4.8 s a call and US$0.36 per 1,000 customers.
- Written reasons can sound right without being true. The content filter refused 3 requests with health products: an LLM step needs a fallback.
Module 7 built the best offline recommender of the course: 0.234 without rules, 0.215 with them. But offline data only shows what the old system chose to show. Next, we see why an offline win may not be an online win.