Skip to content
Français

User-user and its limits

  • Lesson 13
  • 30 min
  • Module 4: Neighbours and collaborative filtering

Where we are

In the last lesson, item-item collaborative filtering added up the neighbours of every product a customer touched. It was easy to explain, but weak on Larkfield's home page. We can turn the idea around and look for similar customers instead of similar products. In this lesson, we try user-user collaborative filtering, and we measure what it costs and how fragile its neighbours are when most customers buy once.

What you will be able to do

Find customers who touched the same products, and suggest what they touched. Measure how well that works for Larkfield, what it costs, and why its neighbours are fragile in a shop where most customers buy once.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook finds the customers most similar to a tiny example’s customer, then to real Larkfield customers. You count how many customers tie at the 50th place for one customer. Then it times the method. Its setup cells download about 8 MB of practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the three files of this module so far. recs explain C100123 prints a score of 4.353.
  • The practice files in ../nextia-learning-labs, beside the project folder.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

You and a colleague like the same films. Every film she loved, you loved too. So when she says “watch this one”, you listen. You do not need to know why the film is good. You trust her because her past choices matched yours.

That is user-user collaborative filtering. Instead of products that go together, we look for customers who go together: customers who touched the same products. Then we suggest the products they touched that our customer has not.

User-user collaborative filtering works in two steps:

  1. Find the customers most similar to our customer. The project uses cosine over who touched what: shared products divided by the square root of the two customers’ product counts. It keeps the 50 most similar customers.
  2. Score each product by the sum of (similarity of the neighbour × the neighbour’s weight for that product). The weights are the same as item-cf’s: view 1, basket 3, order 5, halved every 60 days.

The products with the highest scores make the list.

Take the eight customers of the camping shop from People also bought, and Kim from the last lesson. To keep the numbers simple, we use only her tent and leave out the lantern that she viewed. Each customer’s similarity to Kim is 1 / √(1 × the customer’s number of products), if the customer bought a tent. We count each purchase as 1 to keep the numbers simple.

Customer Bought Similarity to Kim
Chen Tent, Gloves 1 / √2 = 0.707
Ana Tent, Sleeping bag, Gloves 1 / √3 = 0.577
Ben Tent, Sleeping bag, Lantern 1 / √3 = 0.577
Gus Tent, Gloves, Lantern 1 / √3 = 0.577
Dara, Eli, Femi, Ivo no tent 0
Kim, who bought a tent, with arrows to four similar customers: Chen (tent, gloves) with similarity 0.707, and Ana (tent, sleeping bag, gloves), Ben (tent, sleeping bag, lantern) and Gus (tent, gloves, lantern) with 0.577 each, drawn with dashed borders because they tie. All four lead to Kim's scores: gloves 0.707 + 0.577 + 0.577 = 1.86, sleeping bag 0.577 + 0.577 = 1.15, lantern 0.577 + 0.577 = 1.15.
User-user for a customer with one product: one clear neighbour, then a three-way tie. The scores add the similarities of the neighbours who bought each product.
Text description of the diagram

Kim bought a tent. Arrows lead to the four customers who also bought a tent. Chen, who bought a tent and gloves, has similarity 0.707. Ana, Ben and Gus bought three products each, including the tent, and have similarity 0.577 each; their boxes have dashed borders because they tie. All four lead to Kim’s scores. Gloves: 0.707 + 0.577 + 0.577 = 1.86. Sleeping bag: 0.577 + 0.577 = 1.15. Lantern: 0.577 + 0.577 = 1.15.

Kim’s first suggestion is the gloves, the shop’s bestseller. Similar customers bring their popular products with them.

Now keep only the 2 most similar customers instead of 50. Chen is first. Second place is a tie of three customers at 0.577. The code must pick one of them, by whatever order it happens to use.

  • If it picks Ana, Kim’s list is gloves, then sleeping bag.
  • If it picks Ben, the list is gloves, then sleeping bag and lantern with equal scores.
  • If it picks Gus, the list is gloves, then lantern.

The second suggestion depends on an arbitrary choice. Kim told us one thing about herself: one tent. That is not enough to say which of three customers is “most like” her. This is the problem of user-user in a shop where most customers buy once.

The surprise first. On Larkfield’s test period, user-user gets 0.145 (95% interval 0.132 to 0.159). That is the best method of this module, and close to trending.

Recall@10 of the trending baseline and the four neighbour methods on Larkfield's test period, with 95% intervals over 2,118 customers. User-user is the best neighbour method; its interval overlaps with trending's.
Text description of the chart

A plot with one row per method and a horizontal axis for recall@10 from 0 to 0.2. Trending 0.156 (0.143 to 0.172). Cooc 0.122 (0.109 to 0.136). Lift 0.059 (0.049 to 0.069). Item-cf 0.088 (0.077 to 0.101). User-user 0.145 (0.132 to 0.159). The intervals of user-user and trending overlap.

On the same customers, recs compare says:

  • User-user is clearly better than item-cf: +0.057 (+0.047 to +0.068), and better in every segment. The largest gain is for occasional customers: +0.080.
  • User-user is not clearly different from trending: −0.011 (−0.027 to +0.005).

Why does it beat item-cf? Item-cf looks at the neighbours of the customer’s own old products. User-user looks at what similar customers touched, with recent events weighted more. Many of those customers were active recently, so their products include this season’s products. User-user puts more new products in its lists than item-cf (13.3% against 11.5%). It also finds more orders of new products: 0.162 against 0.128.

So user-user works here, offline. Its problems are elsewhere: cost and stability.

Limit 1: it is slow, and it grows with the square

Section titled “Limit 1: it is slow, and it grows with the square”

Item-cf keeps a small table: 50 neighbours for each of 2,994 products. To make a list, it looks up the customer’s products in the table.

User-user has no such table. To find a customer’s neighbours, it compares that customer with every customer of the shop: 40,000 similarities per customer. For all customers, that is 40,000 × 40,000 = 1.6 billion comparisons. Most of them are zero, which the sparse matrix skips, but the work is still large.

item-cf user-user
Lists for the 2,118 test customers 0.1 s 0.5 s
Lists for all 40,000 customers 0.8 s 10.1 s
Peak memory 609 MB 866 MB

These are measured on one thread of the reference laptop. Your times will differ. User-user is about 13 times slower for all customers, and uses the most memory of the module.

The bigger problem is the growth. Double the customers, and each customer has twice as many others to compare with, and there are twice as many customers. The work grows about four times. Item-cf’s table grows with the number of products, which changes much more slowly. For a request at page-load time, Amira’s budget is 100 ms (an assumption of this course). User-user must search 40,000 customers within that budget; item-cf only reads a few rows of its table.

A product touched by a hundred customers has stable neighbours. A customer who touched a handful of products does not. Their similarities to others are few distinct values, like Kim’s 0.707 and 0.577, so many customers tie.

The notebook shows an extreme case. Test customer C100306 touched one product. 103 other customers touched only that same product, so all 103 have a perfect similarity of 1. The method keeps 50 of them, and the order of the rows in memory decides which 50.

The project measures this for every test customer with a history. For each one, we find the 50th highest similarity. Then we ask: do more customers share that value than the places left? If so, the code must choose some of them arbitrarily.

Share of test customers whose 50 most similar customers end in a tie, so part of the 50 is an arbitrary choice. Only customers with at least one event before the period are counted.
Text description of the chart

A column chart with three columns. New customers with at least one view (319 customers): 80% have a tie at the 50th place. Occasional customers (556): 78%. Loyal customers (610): 60%. Even with longer histories, most cuts fall in a tie.

For 78% of the occasional customers, the 50 “most similar customers” end in a tie. Part of the group is chosen by the order of the rows in memory, not by the data. Even for loyal customers, it is 60%.

A second test: train the model one week earlier, on events before 15 June instead of 22 June. Then make lists for the same test customers. How many of the 10 products stay?

Share of the 10 products in a customer's list that stay the same when the model is trained one week earlier, by segment. User-user changes the most, especially for occasional customers.
Text description of the chart

Three panels, one per segment, with a horizontal axis from 70% to 95%. New customers: cooc 86%, item-cf 86%, user-user 85%. Occasional customers: cooc 84%, item-cf 81%, user-user 77%. Loyal customers: cooc 91%, item-cf 84%, user-user 84%.

User-user changes the most, but the difference is not dramatic: 82% of its products stay, against 84% for item-cf. It is largest for occasional customers (77% against 81%). Some change is good: the week brought new orders. But a customer who did nothing that week can still see a different home page. Other customers’ actions moved people in or out of their 50.

633 of the 952 new test customers touched nothing. User-user cannot find a similar customer for them, so the trending list fills their places, as for item-cf. For the 319 new customers with a few views, user-user does better than item-cf: 0.135 against 0.046. Trending still gives them 0.172.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use user-user when the community is small or medium and the histories are rich. Examples: a book club where members rate dozens of books, or a shop where each business client orders every month. Use it offline, as one source of candidates in a nightly batch. Its “customers like you” reason is easy to tell, though harder to show than “because you bought X”.

Do not use it at request time for a large shop: the search over all customers grows with the square of their number. Do not trust its neighbours for customers with one or two products: the cut falls in a tie. And do not judge it by recall alone: at Larkfield it matches trending offline, with much more work.

Cost: 10.1 s for 40,000 customers and 866 MB at peak here, against 0.8 s and 609 MB for item-cf.

The next module keeps the good idea, “customers like you”, and removes the cost. Matrix factorisation describes every customer and every product with a few numbers, learned from the whole log. Then similar customers need no search over everybody.

Guided practice: user-user in your project

Section titled “Guided practice: user-user in your project”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.

  2. In section 1, predict how many customers are “most similar” to Kim. Then run the cell.

  3. Run sections 2 and 3. Section 3 shows a test customer with one product, and how many customers share a similarity with them.

  4. In section 4, find C100123’s 50th highest similarity and count the ties. The check prints Check passed: (fiftieth, tied) is right.

  5. Run section 5. It times user-user on 500 customers. Your times will differ from ours.

  6. In a terminal, in your larkfield-recs folder with the virtual environment active, copy the tests of this module:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m04
    cp $S/tests/test_m04.py tests/
  7. Score user-user by segment. The method is already in neighbours.py, which you copied in the first lesson of this module.

    Terminal window
    recs evaluate user-user --by segment
    Method user-user (user-user collaborative filtering (cosine, 50 similar customers)), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.145 (95% interval 0.132 to 0.159), NDCG@10 0.094, MAP@10 0.070, precision@10 0.022, hit rate 18.7%
    coverage 54.2% of the products on sale, novelty 6.17 bits, diversity 0.66, new products in the lists 13.3%
    time: fit 0.0 s, suggestions 0.5 s
    segment customers recall@10 NDCG@10 95% interval (recall)
    new 952 0.157 0.104 0.136 to 0.180
    occasional 556 0.125 0.079 0.100 to 0.151
    loyal 610 0.147 0.092 0.122 to 0.173
    new products: 647 orders of new products in the period; recall@10 on them 0.162
  8. Compare it with item-cf on the same customers:

    Terminal window
    recs compare item-cf user-user --by segment
    recall@10: item-cf 0.088, user-user 0.145
    difference +0.057 (95% interval +0.047 to +0.068); better for 189 customers, worse for 38, the same for 1,891
    The difference is clear: the interval does not include 0.
    coverage 53.2% -> 54.2%; novelty 6.45 -> 6.17
    new 952 customers +0.030 (+0.020 to +0.041)
    occasional 556 customers +0.080 (+0.056 to +0.105)
    loyal 610 customers +0.079 (+0.057 to +0.103)
  9. Run the tests with python -m pytest -q. You see 17 passed.

  10. Your project now matches the snapshot end-of-m04. Only its README.md differs.

Show the answer to step 4
u = cpos["C100123"]
s = (unit[u] @ unit.T).toarray().ravel()
s[u] = 0
fiftieth = np.sort(s)[::-1][49]
tied = int((s == fiftieth).sum())

unit has each customer’s row divided by its length, so one matrix product gives the cosine with every customer. Sort the similarities from high to low and take place 50 (index 49). Then count the customers with exactly that value. For C100123, the 50th highest similarity is 0.354, and 15 customers share it. Only some of them fit in the 50, so the last places are an arbitrary choice.

Change one thing: the method you compare with. Before you run it, predict: is user-user better than trending for loyal customers? Then run recs compare trending user-user --by segment.

Show the answer
recall@10: trending 0.156, user-user 0.145
difference -0.011 (95% interval -0.027 to +0.005); better for 199 customers, worse for 235, the same for 1,684
The difference is not clear: the interval includes 0.
coverage 0.3% -> 54.2%; novelty 5.10 -> 6.17
new 952 customers -0.012 (-0.029 to +0.004)
occasional 556 customers -0.037 (-0.075 to +0.002)
loyal 610 customers +0.015 (-0.018 to +0.046)

For loyal customers, user-user is a little better (+0.015), but the interval includes 0, so it is not clear. Overall, no difference is clear in any segment. User-user shows far more of the shop (coverage 54.2% against 0.3%) for the same recall. That is a real gain for the long tail, but it costs about 13 times the work of item-cf.

Common mistake

A customer's home page changed, but they did nothing

Symptom: a customer writes to Larkfield: “Yesterday you suggested a lantern, today a sleeping bag. I have not been on your site since.” Amira checks: the customer’s events did not change.

Cause: the customer has a short history, so their 50 most similar customers end in a tie. New orders from other customers moved some of them in or out of the 50, and the list changed.

Diagnose: compute the customer’s similarities and count the ties at the 50th place, as in the notebook.

Fix: accept some change, or break ties by a stable rule (for example, the more recent customer first). Better: use item-cf or matrix factorisation for customers with short histories.

Common mistake

Fast in the test, slow in the nightly batch

Symptom: recs evaluate user-user says suggestions 0.5 s. Amira plans the nightly batch for all customers and expects a few seconds. It takes much longer.

Cause: the evaluation scores only the 2,118 test customers. The batch scores all 40,000, and each one is compared with all 40,000. Here it took 10.1 s against 0.8 s for item-cf, and the gap grows with the square of the customers.

Fix: time a method on the real number of customers before you choose it. Keep request-time work small: look up a table, do not search all customers.

Check your understanding

Kim bought only a tent. Why can user-user not say which customer is most like her after Chen, and why does that matter?

Show the answer

Ana, Ben and Gus each bought the tent and two other products, so each has similarity 1 / √3 = 0.577 to Kim. They tie. If the method keeps only a few neighbours, it must choose among them arbitrarily, and the choice changes Kim’s second suggestion. With one product, Kim’s history is too short to separate them. At Larkfield, most customers buy once, so this happens a lot: for 78% of the occasional test customers, the 50th neighbour is in a tie.

Check your understanding

User-user has a higher recall than item-cf at Larkfield (0.145 against 0.088). Give two reasons why Amira might still prefer item-cf for the live home page.

Show the answer

Cost: user-user compares each customer with all 40,000 customers, so it is about 13 times slower for all customers here and uses more memory. Its work grows with the square of the number of customers. Item-cf reads a small table of 50 neighbours per product. Stability and explanation: user-user’s neighbours often end in ties and change when other customers act. Item-cf’s neighbours come from many customers per product, and each suggestion can be explained with the customer’s own products.

Check your understanding

If Larkfield doubles its number of customers, roughly how much more work does user-user need to make lists for everybody? And item-cf?

Show the answer

User-user: about four times more. There are twice as many customers to score, and each one is compared with twice as many customers. Item-cf: its neighbour table depends on the products, not the customers, so the table costs about the same. Making the lists grows only with the number of customers, so about twice the work.

Recap

  • User-user collaborative filtering finds the 50 customers most similar to a customer (cosine over who touched what) and adds up their weighted products.
  • At Larkfield it is the best neighbour method: 0.145 on the test period. It is clearly better than item-cf (+0.057) and not clearly different from trending (−0.011, −0.027 to +0.005).
  • Slow: it compares each customer with all 40,000. Lists for everybody take 10.1 s against 0.8 s for item-cf, and the work grows with the square of the customers.
  • Fragile: with short histories, many customers tie. The 50th neighbour is in a tie for 78% of occasional customers. Trained one week earlier, 82% of its list stays, against 84% for item-cf.
  • The module’s rows in the comparison table: cooc 0.122, lift 0.059, item-cf 0.088, user-user 0.145. None beats trending (0.156).

Neighbour methods compare raw histories, product by product or customer by customer. Next, we learn a few hidden numbers for every customer and product. They describe taste, and they make “customers like you” cheap.

Next: Hidden tastes