Hidden tastes
- Lesson 14
- 35 min
- Module 5: Matrix factorisation for implicit feedback
Where we are
In Module 4, neighbour methods compared products by the customers they share. They cannot see a deeper taste, such as “likes natural wood”. In this lesson, we factorise the matrix into customer and product vectors, first by hand.
Lesson 14 of 29 · Recommendation and Personalisation: Show Each Person What MattersHidden tastes
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Understand matrix factorisation: give every customer and every product a short list of numbers, so that one multiplication scores any product for any customer. Work through a tiny example by hand, then see the tastes that the method finds in Larkfield’s shop log.
NotebookDo this lesson in a notebook
The notebook factorises the tiny matrix of this lesson with ten lines of NumPy. You compute one score yourself. Then it trains the same idea on the whole shop log with the implicit library, and draws the product map of this lesson. Its setup cells download about 8 MB of practice data and install implicit. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The user-item matrix: one row per customer, one column per product, mostly empty. From The user-item matrix.
- Item-item collaborative filtering and cosine similarity, from Item-item collaborative filtering.
- How to multiply two lists of numbers and add the results (a dot product). This lesson explains it again. Vector operations in Essential Mathematics and Statistics for AI has more.
What you need on your computer
Nothing for the lesson. The notebook is optional. Your larkfield-recs project gets the new method in the next lesson.
A shop assistant who knows your taste
Section titled “A shop assistant who knows your taste”Think of a good shop assistant in a home and garden shop. A regular customer comes in. The assistant does not remember every product that this customer ever bought. She remembers a few things about him: “he likes natural wood”, “he has a big garden”, “he does not care about bright colours”.
She also knows a few things about each product. A new oak lamp is “natural wood, indoor, calm colour”. So she can suggest the lamp, although this customer never bought a lamp. She matches his tastes with the product’s character.
That is the idea of this lesson. A few hidden tastes describe each customer, and the same few tastes describe each product. The difference from the assistant: nobody tells the method what the tastes are. It finds them in the shop log.
The problem: neighbours need a shared customer
Section titled “The problem: neighbours need a shared customer”In Module 4, item-item collaborative filtering compared two products by the customers who touched both. That works well when many customers link two products. But a link needs at least one shared customer.
Here is a tiny example we made. Ana bought an oak bench and an oak shelf. Ben bought the oak shelf and an oak lamp. Chen bought only the oak bench. What should Chen see?
Text description of the diagram
Three products in a row: oak bench, oak shelf, oak lamp. A line joins the bench and the shelf: Ana bought both. A line joins the shelf and the lamp: Ben bought both. A dotted line joins the bench and the lamp: no customer bought both, so their cosine similarity is 0. An arrow from Chen points to the bench, the only product he bought. The lamp has a thick border.
Item-item gives Chen the shelf, because the shelf shares a customer with the bench. It never gives him the lamp. The bench and the lamp have no customer in common, so their similarity is exactly 0. A person sees the pattern at once: all three products are natural oak, and the three customers like oak. The method sees only direct links.
This is not a rare case in Larkfield’s shop log. Count every pair of products and check if at least one customer touched both, with any event before the test period. Only 52.7% of the pairs share a customer. For the other 47.3%, item-item similarity is 0.
Our measurements in Module 4 showed the cost. On the test period, item-cf has recall@10 0.088, and only 0.045 for occasional customers. Their one or two old orders point at old products, and the method can only follow direct links from those products.
The idea: hidden tastes
Section titled “The idea: hidden tastes”We want a method that sees “likes oak” without being told. Here is the plan:
- Give every customer a short list of numbers, for example 2 numbers. One list per customer.
- Give every product a list of the same length.
- To score a product for a customer, multiply their two lists number by number, and add the results.
A list of numbers like this is a vector. The multiply-and-add in step 3 is the dot product. A large dot product means “this customer’s tastes match this product”. A small one means they do not match.
A worked example: two tastes
Section titled “A worked example: two tastes”Suppose, only for this example, that the two numbers mean “likes garden things” and “likes bedroom things”. We made these numbers up to show the arithmetic.
| Garden | Bedroom | |
|---|---|---|
| Customer Ana | 0.9 | 0.1 |
| Product: garden bench | 1.0 | 0.0 |
| Product: duvet | 0.0 | 1.0 |
| Product: picnic blanket | 0.6 | 0.4 |
The scores for Ana:
| Product | Dot product | Score |
|---|---|---|
| Garden bench | 0.9 × 1.0 + 0.1 × 0.0 | 0.90 |
| Picnic blanket | 0.9 × 0.6 + 0.1 × 0.4 | 0.58 |
| Duvet | 0.9 × 0.0 + 0.1 × 1.0 | 0.10 |
So the list for Ana starts with the bench, then the picnic blanket. The duvet comes last. Each score needs only two multiplications and one addition.
Why it is called matrix factorisation
Section titled “Why it is called matrix factorisation”To factorise a number is to write it as a product of smaller numbers: 12 = 3 × 4. Here we do the same with the user-item matrix. We write the big matrix as the product of two thin tables: the customer vectors and the product vectors.
Text description of the diagram
Four boxes from left to right. The matrix: 40,000 customers by 2,994 products, 119.8 million cells, only 467,160 filled (0.39%). An arrow labelled “is close to” leads to the customer vectors: 40,000 rows of 32 numbers. A line labelled “times” leads to the product vectors: 2,994 rows of 32 numbers. The last box, with a thick border: the score of any customer for any product is the dot product of their two rows; every cell gets a score, also the empty ones.
The two thin tables have 32 × (40,000 + 2,994) = about 1.4 million numbers. The full matrix has 119.8 million cells. So the vectors must summarise: they cannot copy every cell. To fit the filled cells with so few numbers, the method has to find patterns that many customers share. Those patterns are the hidden tastes. Their technical name is latent factors: “latent” means hidden.
Two facts make this useful:
- The method learns the vectors from the filled cells only.
- Then it can score the empty cells. An empty cell with a high score is a product that the customer has not touched but probably likes. That is a suggestion.
How the method finds the vectors: a tiny example
Section titled “How the method finds the vectors: a tiny example”Now the real method, on a matrix small enough to read. Five customers and four products. A 1 means “bought”, a 0 means “no event”. Products A and B are garden products. Products C and D are bedroom products. Customer 4 bought from both.
| A garden bench | B parasol | C duvet | D pillow | |
|---|---|---|---|---|
| Customer 0 | 1 | 1 | 0 | 0 |
| Customer 1 | 1 | 0 | 0 | 0 |
| Customer 2 | 0 | 0 | 1 | 1 |
| Customer 3 | 0 | 0 | 1 | 0 |
| Customer 4 | 1 | 1 | 1 | 0 |
The question of this example: customer 1 bought only the garden bench. Should we suggest the parasol (B) or the duvet (C)?
We do not know the vectors. The method finds them in rounds:
- Start with small random numbers, 2 numbers per customer and per product.
- Keep the product vectors fixed. For each customer, find the 2 numbers that make their scores closest to their row of the matrix. “Closest” means the smallest sum of squared differences. This is a least squares problem, and it has an exact answer.
- Keep the customer vectors fixed. For each product, find the 2 numbers that best fit its column, in the same way.
- Repeat steps 2 and 3. Each round, the scores move closer to the matrix.
The method takes turns: customers, then products, then customers again. So its name is alternating least squares, or ALS. One more detail: a small penalty for large numbers, the regularisation, keeps the vectors from fitting every cell exactly. The notebook uses 0.1. The next lesson explains this setting.
After 20 rounds, the notebook prints these vectors:
| Customer | Number 1 | Number 2 |
|---|---|---|
| 0 | -0.84 | 0.44 |
| 1 | -0.47 | 0.28 |
| 2 | -0.3 | -0.94 |
| 3 | -0.27 | -0.59 |
| 4 | -1.11 | -0.15 |
| Product | Number 1 | Number 2 |
|---|---|---|
| A garden bench | -1.02 | 0.42 |
| B parasol | -0.82 | 0.23 |
| C duvet | -0.65 | -0.99 |
| D pillow | -0.1 | -0.57 |
And the scores, the dot product of every customer with every product:
| A garden bench | B parasol | C duvet | D pillow | |
|---|---|---|---|---|
| Customer 0 | 1.04 | 0.79 | 0.11 | -0.17 |
| Customer 1 | 0.59 | 0.45 | 0.03 | -0.11 |
| Customer 2 | -0.09 | 0.03 | 1.13 | 0.57 |
| Customer 3 | 0.03 | 0.08 | 0.76 | 0.36 |
| Customer 4 | 1.07 | 0.87 | 0.87 | 0.2 |
Check one score by hand. Customer 1 and the parasol: -0.47 × -0.82 + 0.28 × 0.23 = 0.385 + 0.064 = 0.45.
Customer 1 never bought the parasol, but its score is 0.45. The duvet gets 0.03 and the pillow −0.11. So the answer is the parasol. Customer 1 looks like customers 0 and 4, who bought the bench and the parasol. The vectors carry this pattern from them to customer 1.
Note also that the scores are not exactly 0 and 1. Customer 0 gets 1.04 for the bench, close to its 1. The bought cells are close to 1, but not equal. With 2 numbers per vector, the method cannot copy the matrix. It can only summarise it, and the summary is what fills the empty cells.
The vectors as a picture
Section titled “The vectors as a picture”With 2 numbers per vector, we can draw each vector as an arrow on a page. In the next picture, we changed the sign of number 1 for every customer and every product. The scores stay the same, because −a × −b = a × b. The picture is easier to read with most arrows pointing right.
- products (arrows)
- customers 0 to 4 (dots)
Text description of the chart
Products are arrows from the origin. A, the garden bench, ends at 1.02, 0.42. B, the parasol, ends at 0.82, 0.23. C, the duvet, ends at 0.65, -0.99. D, the pillow, ends at 0.10, -0.57. Customers are dots. Customer 0 at 0.84, 0.44 and customer 1 at 0.47, 0.28 are up and right, near A and B. Customer 2 at 0.30, -0.94 and customer 3 at 0.27, -0.59 are down, near C and D. Customer 4 at 1.11, -0.15 is far to the right, between the two groups.
Read the picture like this: a customer and a product that point in the same direction give a high score. They point in different directions: a low or negative score. Customer 1 is a short dot in the garden direction, so the parasol scores well and the pillow badly.
The method did not name the two directions. We see “garden” and “bedroom” because we know the products. The numbers have no names. In a real model with 32 or 64 numbers, most directions mix several tastes, and no one can name them.
The tastes ALS finds in Larkfield’s shop log
Section titled “The tastes ALS finds in Larkfield’s shop log”Section 3 of the notebook trains ALS on every event before the test period, with 32 numbers per vector. It uses the 38,377 customers and 2,988 products with at least one event. A view counts 1, a basket add 3 and an order 5, as in Interactions, not ratings. The model never sees a product’s category or title.
Do products of one category get similar vectors? We can measure it. Take the cosine similarity of two product vectors: 1 when they point in the same direction, 0 when they have nothing in common. You used cosine in Module 4 on columns of the matrix. Here we use it on the 32 learned numbers.
| Pairs of products | Mean cosine similarity of their vectors |
|---|---|
| Same category | 0.627 |
| Different categories | 0.01 |
Products of the same category point in clearly similar directions. Products of different categories are almost unrelated. The model found the categories from who touched what.
A picture shows it better. We took the products of four categories. We scaled each vector to length 1, so that only its direction counts. Then we kept the two directions in which these vectors differ most (a method called principal component analysis) and drew 75 products of each category.
- Garden furniture
- Bedding and linen
- Pets
- Outdoor clothing
Text description of the chart
A scatter plot of 300 products, 75 from each of four categories, on two axes, direction 1 and direction 2. Each category forms its own cloud. Outdoor clothing is on the right, around 0.67, 0.25. Bedding and linen is at the top left, around -0.49, 0.5. Garden furniture is at the bottom middle, around -0.01, -0.38. Pets is at the bottom left, around -0.21, -0.48, close to garden furniture. In this picture, 97.5% of the products of these four categories are nearest to the centre of their own category.
The four categories form four clouds. In the picture, 97.5% of their products are closer to the centre of their own category than to any other centre. Pets and garden furniture are close to each other. Perhaps the same customers buy both: people with a garden often have a dog. The model learned that too.
Be careful with one point. The shop log is synthetic: our behaviour model gives each customer a liking for some categories. So clear clouds are what we expect here. In a real shop, the clouds overlap more. The method works the same way.
Your turn
Section titled “Your turn”The notebook prints the vectors of the tiny example. Before you run the task cell, predict: is customer 1’s parasol score closer to their bench score or to their duvet score?
Then compute it yourself, in section 2 of the notebook or on paper with the vectors above.
Show the answer
score_1_B = float(U[1] @ V[1]) gives 0.447. The check prints Check passed: score_1_B is 0.447. With the rounded numbers of the table, you get 0.45.
It is closer to the bench score, 0.59, than to the duvet score, 0.03. Customer 1 bought only the bench, but the vectors place them with the garden customers.
When to use matrix factorisation, and what it costs
Section titled “When to use matrix factorisation, and what it costs”Use it when many customers have a few interactions each, as at Larkfield. It links products through shared tastes, not only through shared customers. It gives a score for every product for every known customer, with one dot product each.
Do not expect it to:
- Serve a customer or product with no events. No events, no vector. The project’s
alsgives such a customer the trending list instead. Module 6 is about this cold-start problem. - Explain itself easily. Item-item can say “because you bought the oak bench”. A factorisation model can only say “your 32 numbers match this product’s 32 numbers”. The numbers have no names.
- Work with no settings. How many numbers per vector? How strong a penalty? The next two lessons choose these with care.
What it costs: one vector per customer and per product, about 1.4 million numbers here. Training ALS with the project’s settings took 3.8 seconds and 496 MB of memory on our laptop, with one thread (measured). The next lesson trains it.
Common mistakes
Section titled “Common mistakes”Common mistake
Reading a number of the vector as a named taste
Symptom: someone looks at number 7 of the product vectors and says “number 7 is the garden taste”.
Cause: the method does not name its numbers. Train it again with another random start, and the same tastes can come out in other numbers, with other signs. In the tiny example, we changed the sign of number 1, and every score stayed the same.
Fix: use the vectors through their dot products and similarities. If you want to know what the model learned, look at which products sit close together, as in the product map.
Common mistake
'No module named implicit' in the notebook
Symptom: the cell that trains ALS on the shop log stops with ModuleNotFoundError: No module named 'implicit'.
Cause: the first cell of the notebook installs implicit 0.7.3, and it was not run, or the kernel started before the install.
Fix: run the install cell. If it prints implicit 0.7.3 and the error stays, restart the kernel and run all the cells again.
Knowledge check
Section titled “Knowledge check”Check your understanding
A customer vector is (0.5, 0.8). Product X is (1.0, 0.0) and product Y is (0.2, 0.9). Which product comes first in the customer's list? Show the two dot products.
Show the answer
Y comes first. X: 0.5 × 1.0 + 0.8 × 0.0 = 0.50. Y: 0.5 × 0.2 + 0.8 × 0.9 = 0.10 + 0.72 = 0.82. The customer’s vector points mostly in the second direction, and so does Y.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the oak example, item-item never suggests the oak lamp to Chen. Why can matrix factorisation suggest it?
Show the answer
Item-item needs a customer who touched both the bench and the lamp, and there is none, so their similarity is 0. Matrix factorisation gives each product a vector that must fit all its customers. The shelf links the bench (through Ana) and the lamp (through Ben), so all three get similar vectors. Chen’s vector, which fits the bench, then also gives the lamp a high score. In Larkfield’s shop log, only 52.7% of product pairs share a customer, so this matters.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'The model found that number 3 means outdoor clothing.' What is right and what is wrong in this sentence?
Show the answer
Right: the vectors do carry category information. Products of one category have a mean cosine similarity of 0.627, against 0.010 for different categories. The categories also form clouds in the product map. Wrong: no single number is a category. A taste is a direction, usually a mix of several numbers, and another training run can put it in other numbers with other signs. Use the vectors through similarities, not one number at a time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Item-item needs a shared customer. In Larkfield’s shop log, only 52.7% of product pairs have one; for the rest, item-item similarity is 0.
- Matrix factorisation gives every customer and every product a short vector of hidden tastes (latent factors). The score is their dot product.
- The vectors are learned from the filled cells and fill the empty ones: an empty cell with a high score is a suggestion.
- ALS (alternating least squares) finds them in rounds: fix the products and fit the customers, then fix the customers and fit the products.
- In the tiny example, customer 1 bought only the bench and gets 0.45 for the parasol, against 0.03 for the duvet.
- In the shop log, products of one category get similar vectors (mean cosine 0.627 against 0.010), although the model never saw a category. The numbers themselves have no names.
You did ALS by hand on 20 cells. Next, the implicit library trains it on the whole shop log in seconds, and a second method, BPR, learns the order of products instead.