Skip to content
Français

Hidden tastes

  • Lesson 14
  • 35 min
  • Module 5: Matrix factorisation for implicit feedback

Where we are

In Module 4, neighbour methods compared products by the customers they share. They cannot see a deeper taste, such as “likes natural wood”. In this lesson, we factorise the matrix into customer and product vectors, first by hand.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Understand matrix factorisation: give every customer and every product a short list of numbers, so that one multiplication scores any product for any customer. Work through a tiny example by hand, then see the tastes that the method finds in Larkfield’s shop log.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook factorises the tiny matrix of this lesson with ten lines of NumPy. You compute one score yourself. Then it trains the same idea on the whole shop log with the implicit library, and draws the product map of this lesson. Its setup cells download about 8 MB of practice data and install implicit. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing for the lesson. The notebook is optional. Your larkfield-recs project gets the new method in the next lesson.

Think of a good shop assistant in a home and garden shop. A regular customer comes in. The assistant does not remember every product that this customer ever bought. She remembers a few things about him: “he likes natural wood”, “he has a big garden”, “he does not care about bright colours”.

She also knows a few things about each product. A new oak lamp is “natural wood, indoor, calm colour”. So she can suggest the lamp, although this customer never bought a lamp. She matches his tastes with the product’s character.

That is the idea of this lesson. A few hidden tastes describe each customer, and the same few tastes describe each product. The difference from the assistant: nobody tells the method what the tastes are. It finds them in the shop log.

The problem: neighbours need a shared customer

Section titled “The problem: neighbours need a shared customer”

In Module 4, item-item collaborative filtering compared two products by the customers who touched both. That works well when many customers link two products. But a link needs at least one shared customer.

Here is a tiny example we made. Ana bought an oak bench and an oak shelf. Ben bought the oak shelf and an oak lamp. Chen bought only the oak bench. What should Chen see?

Three products. The oak bench and the oak shelf are linked, because Ana bought both. The oak shelf and the oak lamp are linked, because Ben bought both. A dotted line between the oak bench and the oak lamp says that no customer bought both, so their similarity is 0. Chen bought only the bench.
A constructed example. Item-item similarity links only products with a shared customer, so the bench never leads to the lamp.
Text description of the diagram

Three products in a row: oak bench, oak shelf, oak lamp. A line joins the bench and the shelf: Ana bought both. A line joins the shelf and the lamp: Ben bought both. A dotted line joins the bench and the lamp: no customer bought both, so their cosine similarity is 0. An arrow from Chen points to the bench, the only product he bought. The lamp has a thick border.

Item-item gives Chen the shelf, because the shelf shares a customer with the bench. It never gives him the lamp. The bench and the lamp have no customer in common, so their similarity is exactly 0. A person sees the pattern at once: all three products are natural oak, and the three customers like oak. The method sees only direct links.

This is not a rare case in Larkfield’s shop log. Count every pair of products and check if at least one customer touched both, with any event before the test period. Only 52.7% of the pairs share a customer. For the other 47.3%, item-item similarity is 0.

Our measurements in Module 4 showed the cost. On the test period, item-cf has recall@10 0.088, and only 0.045 for occasional customers. Their one or two old orders point at old products, and the method can only follow direct links from those products.

We want a method that sees “likes oak” without being told. Here is the plan:

  1. Give every customer a short list of numbers, for example 2 numbers. One list per customer.
  2. Give every product a list of the same length.
  3. To score a product for a customer, multiply their two lists number by number, and add the results.

A list of numbers like this is a vector. The multiply-and-add in step 3 is the dot product. A large dot product means “this customer’s tastes match this product”. A small one means they do not match.

Suppose, only for this example, that the two numbers mean “likes garden things” and “likes bedroom things”. We made these numbers up to show the arithmetic.

Garden Bedroom
Customer Ana 0.9 0.1
Product: garden bench 1.0 0.0
Product: duvet 0.0 1.0
Product: picnic blanket 0.6 0.4

The scores for Ana:

Product Dot product Score
Garden bench 0.9 × 1.0 + 0.1 × 0.0 0.90
Picnic blanket 0.9 × 0.6 + 0.1 × 0.4 0.58
Duvet 0.9 × 0.0 + 0.1 × 1.0 0.10

So the list for Ana starts with the bench, then the picnic blanket. The duvet comes last. Each score needs only two multiplications and one addition.

To factorise a number is to write it as a product of smaller numbers: 12 = 3 × 4. Here we do the same with the user-item matrix. We write the big matrix as the product of two thin tables: the customer vectors and the product vectors.

The matrix of 40,000 customers by 2,994 products has 119.8 million cells, and only 467,160 are filled (0.39%). It is close to a table of customer vectors, 40,000 rows of 32 numbers, times a table of product vectors, 2,994 rows of 32 numbers. The score of any customer for any product is the dot product of their two rows, so every cell gets a score, also the empty ones.
Matrix factorisation of Larkfield's matrix before the test period, with 32 numbers per vector.
Text description of the diagram

Four boxes from left to right. The matrix: 40,000 customers by 2,994 products, 119.8 million cells, only 467,160 filled (0.39%). An arrow labelled “is close to” leads to the customer vectors: 40,000 rows of 32 numbers. A line labelled “times” leads to the product vectors: 2,994 rows of 32 numbers. The last box, with a thick border: the score of any customer for any product is the dot product of their two rows; every cell gets a score, also the empty ones.

The two thin tables have 32 × (40,000 + 2,994) = about 1.4 million numbers. The full matrix has 119.8 million cells. So the vectors must summarise: they cannot copy every cell. To fit the filled cells with so few numbers, the method has to find patterns that many customers share. Those patterns are the hidden tastes. Their technical name is latent factors: “latent” means hidden.

Two facts make this useful:

  • The method learns the vectors from the filled cells only.
  • Then it can score the empty cells. An empty cell with a high score is a product that the customer has not touched but probably likes. That is a suggestion.

How the method finds the vectors: a tiny example

Section titled “How the method finds the vectors: a tiny example”

Now the real method, on a matrix small enough to read. Five customers and four products. A 1 means “bought”, a 0 means “no event”. Products A and B are garden products. Products C and D are bedroom products. Customer 4 bought from both.

A garden bench B parasol C duvet D pillow
Customer 0 1 1 0 0
Customer 1 1 0 0 0
Customer 2 0 0 1 1
Customer 3 0 0 1 0
Customer 4 1 1 1 0

The question of this example: customer 1 bought only the garden bench. Should we suggest the parasol (B) or the duvet (C)?

We do not know the vectors. The method finds them in rounds:

  1. Start with small random numbers, 2 numbers per customer and per product.
  2. Keep the product vectors fixed. For each customer, find the 2 numbers that make their scores closest to their row of the matrix. “Closest” means the smallest sum of squared differences. This is a least squares problem, and it has an exact answer.
  3. Keep the customer vectors fixed. For each product, find the 2 numbers that best fit its column, in the same way.
  4. Repeat steps 2 and 3. Each round, the scores move closer to the matrix.

The method takes turns: customers, then products, then customers again. So its name is alternating least squares, or ALS. One more detail: a small penalty for large numbers, the regularisation, keeps the vectors from fitting every cell exactly. The notebook uses 0.1. The next lesson explains this setting.

After 20 rounds, the notebook prints these vectors:

Customer Number 1 Number 2
0 -0.84 0.44
1 -0.47 0.28
2 -0.3 -0.94
3 -0.27 -0.59
4 -1.11 -0.15
Product Number 1 Number 2
A garden bench -1.02 0.42
B parasol -0.82 0.23
C duvet -0.65 -0.99
D pillow -0.1 -0.57

And the scores, the dot product of every customer with every product:

A garden bench B parasol C duvet D pillow
Customer 0 1.04 0.79 0.11 -0.17
Customer 1 0.59 0.45 0.03 -0.11
Customer 2 -0.09 0.03 1.13 0.57
Customer 3 0.03 0.08 0.76 0.36
Customer 4 1.07 0.87 0.87 0.2

Check one score by hand. Customer 1 and the parasol: -0.47 × -0.82 + 0.28 × 0.23 = 0.385 + 0.064 = 0.45.

Customer 1 never bought the parasol, but its score is 0.45. The duvet gets 0.03 and the pillow −0.11. So the answer is the parasol. Customer 1 looks like customers 0 and 4, who bought the bench and the parasol. The vectors carry this pattern from them to customer 1.

Note also that the scores are not exactly 0 and 1. Customer 0 gets 1.04 for the bench, close to its 1. The bought cells are close to 1, but not equal. With 2 numbers per vector, the method cannot copy the matrix. It can only summarise it, and the summary is what fills the empty cells.

With 2 numbers per vector, we can draw each vector as an arrow on a page. In the next picture, we changed the sign of number 1 for every customer and every product. The scores stay the same, because −a × −b = a × b. The picture is easier to read with most arrows pointing right.

  • products (arrows)
  • customers 0 to 4 (dots)
The tiny example after 20 rounds of ALS. Garden products point up and right, bedroom products point down. Each customer sits near the products they bought.
Text description of the chart

Products are arrows from the origin. A, the garden bench, ends at 1.02, 0.42. B, the parasol, ends at 0.82, 0.23. C, the duvet, ends at 0.65, -0.99. D, the pillow, ends at 0.10, -0.57. Customers are dots. Customer 0 at 0.84, 0.44 and customer 1 at 0.47, 0.28 are up and right, near A and B. Customer 2 at 0.30, -0.94 and customer 3 at 0.27, -0.59 are down, near C and D. Customer 4 at 1.11, -0.15 is far to the right, between the two groups.

Read the picture like this: a customer and a product that point in the same direction give a high score. They point in different directions: a low or negative score. Customer 1 is a short dot in the garden direction, so the parasol scores well and the pillow badly.

The method did not name the two directions. We see “garden” and “bedroom” because we know the products. The numbers have no names. In a real model with 32 or 64 numbers, most directions mix several tastes, and no one can name them.

The tastes ALS finds in Larkfield’s shop log

Section titled “The tastes ALS finds in Larkfield’s shop log”

Section 3 of the notebook trains ALS on every event before the test period, with 32 numbers per vector. It uses the 38,377 customers and 2,988 products with at least one event. A view counts 1, a basket add 3 and an order 5, as in Interactions, not ratings. The model never sees a product’s category or title.

Do products of one category get similar vectors? We can measure it. Take the cosine similarity of two product vectors: 1 when they point in the same direction, 0 when they have nothing in common. You used cosine in Module 4 on columns of the matrix. Here we use it on the 32 learned numbers.

Pairs of products Mean cosine similarity of their vectors
Same category 0.627
Different categories 0.01

Products of the same category point in clearly similar directions. Products of different categories are almost unrelated. The model found the categories from who touched what.

A picture shows it better. We took the products of four categories. We scaled each vector to length 1, so that only its direction counts. Then we kept the two directions in which these vectors differ most (a method called principal component analysis) and drew 75 products of each category.

  • Garden furniture
  • Bedding and linen
  • Pets
  • Outdoor clothing
Products placed by their learned ALS vectors (32 numbers, shown in two directions), coloured by category. The model never saw the categories.
Text description of the chart

A scatter plot of 300 products, 75 from each of four categories, on two axes, direction 1 and direction 2. Each category forms its own cloud. Outdoor clothing is on the right, around 0.67, 0.25. Bedding and linen is at the top left, around -0.49, 0.5. Garden furniture is at the bottom middle, around -0.01, -0.38. Pets is at the bottom left, around -0.21, -0.48, close to garden furniture. In this picture, 97.5% of the products of these four categories are nearest to the centre of their own category.

The four categories form four clouds. In the picture, 97.5% of their products are closer to the centre of their own category than to any other centre. Pets and garden furniture are close to each other. Perhaps the same customers buy both: people with a garden often have a dog. The model learned that too.

Be careful with one point. The shop log is synthetic: our behaviour model gives each customer a liking for some categories. So clear clouds are what we expect here. In a real shop, the clouds overlap more. The method works the same way.

The notebook prints the vectors of the tiny example. Before you run the task cell, predict: is customer 1’s parasol score closer to their bench score or to their duvet score?

Then compute it yourself, in section 2 of the notebook or on paper with the vectors above.

Show the answer

score_1_B = float(U[1] @ V[1]) gives 0.447. The check prints Check passed: score_1_B is 0.447. With the rounded numbers of the table, you get 0.45.

It is closer to the bench score, 0.59, than to the duvet score, 0.03. Customer 1 bought only the bench, but the vectors place them with the garden customers.

When to use matrix factorisation, and what it costs

Section titled “When to use matrix factorisation, and what it costs”

Use it when many customers have a few interactions each, as at Larkfield. It links products through shared tastes, not only through shared customers. It gives a score for every product for every known customer, with one dot product each.

Do not expect it to:

  • Serve a customer or product with no events. No events, no vector. The project’s als gives such a customer the trending list instead. Module 6 is about this cold-start problem.
  • Explain itself easily. Item-item can say “because you bought the oak bench”. A factorisation model can only say “your 32 numbers match this product’s 32 numbers”. The numbers have no names.
  • Work with no settings. How many numbers per vector? How strong a penalty? The next two lessons choose these with care.

What it costs: one vector per customer and per product, about 1.4 million numbers here. Training ALS with the project’s settings took 3.8 seconds and 496 MB of memory on our laptop, with one thread (measured). The next lesson trains it.

Common mistake

Reading a number of the vector as a named taste

Symptom: someone looks at number 7 of the product vectors and says “number 7 is the garden taste”.

Cause: the method does not name its numbers. Train it again with another random start, and the same tastes can come out in other numbers, with other signs. In the tiny example, we changed the sign of number 1, and every score stayed the same.

Fix: use the vectors through their dot products and similarities. If you want to know what the model learned, look at which products sit close together, as in the product map.

Common mistake

'No module named implicit' in the notebook

Symptom: the cell that trains ALS on the shop log stops with ModuleNotFoundError: No module named 'implicit'.

Cause: the first cell of the notebook installs implicit 0.7.3, and it was not run, or the kernel started before the install.

Fix: run the install cell. If it prints implicit 0.7.3 and the error stays, restart the kernel and run all the cells again.

Check your understanding

A customer vector is (0.5, 0.8). Product X is (1.0, 0.0) and product Y is (0.2, 0.9). Which product comes first in the customer's list? Show the two dot products.

Show the answer

Y comes first. X: 0.5 × 1.0 + 0.8 × 0.0 = 0.50. Y: 0.5 × 0.2 + 0.8 × 0.9 = 0.10 + 0.72 = 0.82. The customer’s vector points mostly in the second direction, and so does Y.

Check your understanding

In the oak example, item-item never suggests the oak lamp to Chen. Why can matrix factorisation suggest it?

Show the answer

Item-item needs a customer who touched both the bench and the lamp, and there is none, so their similarity is 0. Matrix factorisation gives each product a vector that must fit all its customers. The shelf links the bench (through Ana) and the lamp (through Ben), so all three get similar vectors. Chen’s vector, which fits the bench, then also gives the lamp a high score. In Larkfield’s shop log, only 52.7% of product pairs share a customer, so this matters.

Check your understanding

Tomás says: 'The model found that number 3 means outdoor clothing.' What is right and what is wrong in this sentence?

Show the answer

Right: the vectors do carry category information. Products of one category have a mean cosine similarity of 0.627, against 0.010 for different categories. The categories also form clouds in the product map. Wrong: no single number is a category. A taste is a direction, usually a mix of several numbers, and another training run can put it in other numbers with other signs. Use the vectors through similarities, not one number at a time.

Recap

  • Item-item needs a shared customer. In Larkfield’s shop log, only 52.7% of product pairs have one; for the rest, item-item similarity is 0.
  • Matrix factorisation gives every customer and every product a short vector of hidden tastes (latent factors). The score is their dot product.
  • The vectors are learned from the filled cells and fill the empty ones: an empty cell with a high score is a suggestion.
  • ALS (alternating least squares) finds them in rounds: fix the products and fit the customers, then fix the customers and fit the products.
  • In the tiny example, customer 1 bought only the bench and gets 0.45 for the parasol, against 0.03 for the duvet.
  • In the shop log, products of one category get similar vectors (mean cosine 0.627 against 0.010), although the model never saw a category. The numbers themselves have no names.

You did ALS by hand on 20 cells. Next, the implicit library trains it on the whole shop log in seconds, and a second method, BPR, learns the order of products instead.

Next: ALS and BPR with implicit