Skip to content
Français

Sessions, sequences and LLMs, briefly

  • Lesson 23
  • 35 min
  • Module 7: Two stages: candidates and ranking

Where we are

Our system uses a customer's whole history. Sometimes we only know the current visit. In this lesson, we try simple session suggestions, and we see what an LLM adds for explanations and re-ranking, and what it costs.

What you will be able to do

Suggest the next product from the current visit alone, with a simple Markov baseline. Then look at what a language model adds to a recommender, from recorded answers. You will see made-up products, a re-ranking that does not help, slow and paid calls, explanations, and refused requests.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook counts which product people view next, in all visits before the test period. It scores these Markov suggestions on the test period’s visits. You score the trending list on the same visits. Then it reads the recorded language-model answers and checks which product names really exist. No key is needed. Its setup cells download about 7 MB of practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the files of the last lesson: recs rules two-stage --quick works.
  • No model key. The project replays recorded answers.

Tested on macOS with Python 3.12. The Windows (PowerShell) and Linux commands were not run.

A stranger walks into the shop. The assistant has never seen her. She knows no name and no past orders. But she watches. The stranger looks at a rattan armchair, then picks up a side table.

The assistant has seen this many times. People who look at that armchair often look at a seat cushion next. So she says: “This cushion fits that chair.” She uses only this visit.

This lesson has two parts. First, suggestions from the current visit. Then, a short look at language models in a recommender.

Part 1: suggestions from the current visit

Section titled “Part 1: suggestions from the current visit”

Many visitors are strangers to the recommender. They do not log in, or they are new. Of Larkfield’s 952 new test customers, 633 touched nothing before the period. For them, a home-page ranker can only show what is popular.

But during the visit, each click is a clue. A popular list ignores it. On the test period’s visits, the trending list contained the next product a visitor viewed only 2.7% of the time.

A session is one visit: the views, basket adds and orders of one visitor, close together in time. In all past sessions, we count how often product B was viewed right after product A. When a visitor is on A, we suggest the products with the highest counts.

This is a Markov model, named after the mathematician Andrey Markov. It assumes that the next step depends only on the current step, not on the steps before.

A visitor is on the Rattan Armchair page. Arrows show what people viewed next in past visits: the Rattan Side Table 6 times, the Seat Cushion 4 times, the Garden Parasol 1 time. The suggestions, in order: 1. Rattan Side Table, 2. Seat Cushion, 3. Garden Parasol.
A Markov suggestion (a tiny example we made): rank the products by how often people viewed them right after the current product.
Text description of the diagram

A visitor is on the page of the Rattan Armchair. In past visits, people on that page viewed the Rattan Side Table next 6 times, the Seat Cushion 4 times and the Garden Parasol once. So the suggestions are, in order: the side table, the cushion, the parasol.

To score it, we use the test period’s visits. For each move from one viewed product to the next, we ask two questions:

  • hit@10: is the next product among the 10 suggestions?
  • MRR@10 (mean reciprocal rank): 1 divided by its place, or 0 if it is missing. Place 1 gives 1, place 2 gives 0.5, place 4 gives 0.25. MRR is the average.
Next-product suggestions on the test period's visits (16,801 moves from one viewed product to the next). Markov and item-cf neighbours each find the next product about 62% of the time; trending 2.7%. Markov+ is Markov with trending to break ties.
Text description of the chart

A column chart with four columns, on 16,801 moves from one viewed product to the next in the test period. Markov transitions find the next product among 10 suggestions 62.2% of the time (MRR@10 0.238). Item-cf neighbours of the current product: 62.0% (MRR 0.235). Markov with trending to break ties: 62.8% (MRR 0.241). Trending: 2.7% (MRR 0.011).

The current product is a very strong clue. Markov finds the next product 62.2% of the time; trending finds it 2.7% of the time.

Markov and item-cf are almost equal. Both find products related to the current one. Often these are alternatives that people compare: in the notebook, people on the Driftwood armchair mostly viewed other armchairs next. Item-cf counts customers who touched both, at any time. Markov counts views in order, inside one visit. Here the order adds little. Adding trending to break ties helps a little more: 62.8%.

Markov looks only at the last product. Neural sequence models read the whole visit. GRU4Rec (2016) uses a recurrent neural network; SASRec (2018) uses self-attention, the idea behind transformers. They can learn that “armchair, then side table, then cushion” means a patio set. They need much more data and longer training. We did not build one for Larkfield, so we do not know whether it would beat 0.622. Start with Markov: it is a strong baseline, and you can explain every suggestion with one count. The case study Next item from browsing sessions works with real sessions.

Tomás has a simple plan. Send the customer’s recent products to a large language model (an LLM) and ask what they will buy next. LLMs know a lot about homes and gardens.

Think of a friend who knows a lot about furniture, but has never seen Larkfield’s catalogue. She can give good general advice. Can she name the products on Larkfield’s shelves?

We tested it once, with recorded answers. The calls went to a small chat model on Azure; the course team’s name for it is chat-small. Each answer was recorded once and is replayed by recs llm, so you need no key.

For 30 customers, the model got the customer’s 10 most recent products and this instruction:

You help Larkfield, an online home and garden shop, choose product suggestions for one customer. You get the
customer's recent products (newest first, with what they did). Name 10 products from Larkfield's shop that this
customer is likely to buy next, best first, with their exact product titles. Answer with JSON:
{"products": ["<title>", ...]}.
The 290 product names in chat-small's recorded answers (29 customers; one request was refused). 210 names are not in the catalogue; 77 of the 80 real ones are copied from the customer's own history.
Text description of the chart

A column chart with three columns: invented, copied from the history, and real and new. It counts the 290 product names in the recorded answers for 29 customers. 210 names do not exist in Larkfield’s catalogue. 77 are real products copied from the customer’s own recent products. Only 3 are real products that the customer had not touched. One named product was ordered in the test period.

The results, from recorded answers of chat-small:

  • 210 of 290 names do not exist. The model writes titles that sound like Larkfield’s products. A customer who viewed pet bowls got ten cat products, such as “Stainless Steel Cat Water Fountain” and “Wall-Mounted Cat Perch”. Larkfield sells none of them. A made-up answer that sounds right is called a hallucination.
  • 77 of the 80 real names are copied from the history. For one customer, the answer was the customer’s ten recent products in a new order. A single-malt whisky and a coffee liqueur were among them.
  • Only 1 named product was ordered in the test period.

The friend gives general advice, not Larkfield’s shelves. An LLM cannot know a catalogue it was not given.

With a list: re-ranking the ranker’s top 20

Section titled “With a list: re-ranking the ranker’s top 20”

A better use: give the model a list. For 100 customers, the model got the recent products and the ranker’s top 20 candidates, with IDs and titles. It had to choose and order 10, using only IDs from the list, and give a short reason for the first three.

The ranker alone Ranker + LLM re-ranking
recall@10 on these customers 0.211 0.194
Difference (95% interval) -0.017 (-0.088 to +0.052): not clear
Made-up IDs – 0
Time per customer about 10 ms 4.8 s (median), 6.8 s (95th percentile)
Cost per 1,000 customers a few seconds of CPU US$0.36
Requests refused 0 2 of 100

With a list, the model made up no ID. But it did not improve the order: recall fell by 0.017, and the interval includes zero, so we cannot even say it is worse. It is about 470 times slower than the ranker (4.8 seconds against 10 milliseconds), and each call costs money. For a weekly email to all 40,000 customers, the calls would cost about 40 × US$0.36 = US$14.40 a week under this price. Prices change: check the provider’s current page.

Explanations: they sound right, but are they true?

Section titled “Explanations: they sound right, but are they true?”

The re-ranking answers include reasons. For customer C100159, who added garden boots to the basket and viewed sun hats and globe light bulbs:

P10448 Sage Green Globe Light Bulb: You recently viewed this sage green globe bulb.
P10590 Navy Tilting Patio Parasol: A patio parasol could complement your garden-ready footwear and sun hats.
P11781 Oak Fringe-Trimmed Garden Parasol: This garden parasol pairs well with your recent outdoor and gardening interests.

The first reason is true. The second is a guess that sounds kind. The third says almost nothing. A written reason is not the real reason the list was chosen. The ranker’s real reasons are its features, such as “viewed 2 days ago”. A template from data, “Because you viewed …”, is always true and costs nothing.

The provider’s content filter refused 3 requests: 2 of the 100 re-ranking requests and 1 of the 30 free-form requests. In two of them, the customer’s history had continence or pregnancy products. In the third, a continence product was among the candidates. The provider did not say why.

So an LLM step can fail for reasons you do not control. It needs a fallback: show the ranker’s own list. And sending a customer’s health history to an outside provider is a privacy question in itself. Privacy and sensitive suggestions returns to it.

When to use them, when not, and what they cost

Section titled “When to use them, when not, and what they cost”

Use Markov or item-cf neighbours for the product page and for visitors without history. They are cheap and strong: about 62.2% hit@10 on Larkfield’s visits. Use an LLM, if at all, on a short list that your system made. For example, it can write a friendly sentence for an email that a person checks.

Do not use an LLM to name products from memory: 210 of 290 names did not exist. Do not add it to a home page that must answer in 100 milliseconds: one call takes seconds. Do not use it to re-rank before you measure it: here it did not help.

Cost: Markov transitions take about 2 seconds to count and score in the project. LLM re-ranking took 4.8 s a call and US$0.36 per 1,000 customers, with refusals that need a fallback.

Guided practice: score sessions and replay the LLM

Section titled “Guided practice: score sessions and replay the LLM”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.

  2. Run section 1. It counts the next views in all visits before the test period, and shows the top next products for one product.

  3. Run section 2. It scores the Markov suggestions on the test period’s visits.

  4. In section 3, score the trending list on the same visits. The check prints Check passed.

  5. Run section 4. It reads the recorded free-form answers and checks each product name against the catalogue.

  6. In a terminal, in your larkfield-recs folder with the virtual environment active, copy the last files of this module:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m07
    cp $S/larkfield_recs/session.py larkfield_recs/
    cp $S/larkfield_recs/commands/session.py larkfield_recs/commands/
    cp $S/larkfield_recs/commands/llm.py larkfield_recs/commands/
    cp $S/tests/test_m07.py tests/
  7. Score the next-product methods:

    Terminal window
    recs session
    Test period: 16,801 moves from one viewed product to the next
    method hit@10 MRR@10
    markov 0.622 0.238
    item-cf 0.620 0.235
    markov+trending 0.628 0.241
    trending 0.027 0.011
  8. Replay and score the recorded re-ranking answers:

    Terminal window
    recs llm
    Recorded answers from chat-small: 100 customers; refused by the provider's content filter: 2
    Time per call p50 4.8 s, p95 6.8 s; US$0.36 per 1,000 customers
    recall@10 on these customers: ranker 0.211, ranker + LLM re-ranking 0.194; difference -0.017 (95% interval -0.088 to +0.052)
    IDs not in the candidate list (made up or changed): 0
    Example reasons for C100159:
    P10448 Sage Green Globe Light Bulb: You recently viewed this sage green globe bulb.
    P10590 Navy Tilting Patio Parasol: A patio parasol could complement your garden-ready footwear and sun hats.
    P11781 Oak Fringe-Trimmed Garden Parasol: This garden parasol pairs well with your recent outdoor and gardening interests.
  9. Run the tests of the whole module:

    Terminal window
    python -m pytest -q

    The last line is 26 passed. Your project now has every file of the end-of-m07 snapshot.

Before you run it, predict: of the products the model names without a list, what share exist in Larkfield’s catalogue?

Terminal window
recs llm freeform
Show the answer
Recorded answers from chat-small: 30 customers; refused by the provider's content filter: 1
Time per call p50 3.6 s, p95 6.6 s; US$0.23 per 1,000 customers
Products named: 290; exact titles that exist in the catalogue: 80 (28%), 77 of them copied from the customer's own history; the other 210 do not exist
Named products that the customer ordered in the test period: 1
Example: Birka Green Pine Shelf with Drawers; Classic Walnut Low Media Shelf; Cove Brass Corner Display Shelf; Saltmarsh Charcoal Pine Ladder Shelf

Only 28% exist, and almost all of those are the customer’s own recent products. The example is one of them: its four names are the customer’s four most recent products, in a new order. Without a list, the model either repeats the history or invents.

Common mistake

Product names from an LLM shown as they are

Symptom: Tomás’s prototype email shows “Wall-Mounted Cat Perch”. A customer clicks it and gets an error page.

Cause: the model wrote a plausible title. Larkfield does not sell it. 210 of 290 names in our recorded answers did not exist.

Diagnose: match every name or ID against the catalogue before you show it, and count the misses.

Fix: never let the model name products freely. Give it a list of real candidates, and accept only IDs from that list. Fall back to the ranker’s list when the answer is wrong, slow or refused.

Common mistake

An LLM step with no fallback

Symptom: in a test of the weekly email, some customers get no email at all.

Cause: the provider refused their requests. In our recordings, 3 requests with health products were refused. The code had no plan for an error.

Diagnose: log every failed call with the customer and the error, and count them.

Fix: on any error or timeout, send the ranker’s own list. Better still, do not send sensitive histories to an outside model at all.

Check your understanding

In past visits, people on a lamp page viewed a shade next 8 times, a bulb 5 times and a rug 2 times. A visitor on the lamp page then views the bulb. What is the Markov list's reciprocal rank for this move?

Show the answer

The Markov list is shade (8), bulb (5), rug (2). The bulb is at place 2, so the reciprocal rank is 1 / 2 = 0.5. The move is also a hit@10, because the bulb is in the list.

Check your understanding

Asked to name products, the model invented 210 of 290. Asked to re-rank 20 candidates, it invented none. Why the difference?

Show the answer

Without a list, the model does not know Larkfield’s catalogue, so it writes titles that sound right. With a list, it only has to choose among real IDs that our system found, and the instruction says to use only those IDs. The list turns an open question into a choice.

Check your understanding

Omar asks: should the home page re-rank with an LLM? Give two numbers from this lesson for your answer.

Show the answer

No, not as tested. Recall@10 changed by −0.017 (interval −0.088 to +0.052): no clear gain. One call took 4.8 seconds (median), far over the 100-millisecond budget, and cost US$0.36 per 1,000 customers. Two of 100 requests were refused and would need the fallback anyway.

Recap

  • A session is one visit. A Markov model suggests what people viewed next after the current product. On Larkfield’s visits it finds the next product 62.2% of the time (MRR 0.238); item-cf 62.0%; trending 2.7%.
  • Sequence models such as GRU4Rec and SASRec read the whole visit. We did not build one; Markov is the baseline to beat.
  • Without a list, an LLM named 290 products: 210 do not exist, and 77 of the 80 real ones are copied from the history.
  • With a list, it invented no ID, but re-ranking did not help: 0.211 → 0.194, not clear. It takes 4.8 s a call and US$0.36 per 1,000 customers.
  • Written reasons can sound right without being true. The content filter refused 3 requests with health products: an LLM step needs a fallback.

Module 7 built the best offline recommender of the course: 0.234 without rules, 0.215 with them. But offline data only shows what the old system chose to show. Next, we see why an offline win may not be an online win.

Next: Offline wins are not online wins