Skip to content
Français

The user-item matrix

  • Lesson 5
  • 35 min
  • Module 2: Data, implicit feedback and an honest split

Where we are

We turned events into weighted interactions. Most methods need them as one table of customers and products. In this lesson, we build the user-item matrix as a sparse matrix and measure how empty it is.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Put every customer’s interactions into one table, the user-item matrix, and store it in a way that fits in memory. See how empty it is, and why that emptiness is the main difficulty of recommendation.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds the matrix of this lesson with SciPy, measures its memory, shows one customer’s row and counts the orders per customer. Your task: build a matrix of orders only. Its setup cells install the packages and download the data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Implicit feedback, the weights view 1, basket 3, order 5, and why an empty cell is unknown, from Interactions, not ratings.

What you need on your computer

  • Your larkfield-recs project from Meet the shop log, with its virtual environment active. recs check prints Setup check passed.
  • Your copy of the labs repository next to the project, up to date (git pull in nextia-learning-labs).

After this lesson, your project has the recs matrix command of the snapshot end-of-m02. The shop log is synthetic. The course team ran every command on an Apple M4 laptop. Tested on macOS; the Windows commands are not tested.

Imagine that our shop assistant keeps a notebook. Each line is one customer. Each column is one product in the shop. When a customer looks at a product, she writes a small mark in that box. When the customer buys it, she writes a big mark.

For a small shop with 30 regular customers and 50 products, this works well. One page shows who likes what. She can see that two customers have marks on the same garden chairs, and suggest to one of them what the other bought.

Now give her Larkfield: 40,000 customers and 2,994 products. The notebook has 119,760,000 boxes. She would need thousands of pages, and almost every box would be blank.

The table in the notebook has a name: the user-item matrix. “User” is the customer, “item” is the product, and a matrix is a table of numbers with rows and columns. In Larkfield’s matrix:

  • each row is one customer (40,000 rows);
  • each column is one product (2,994 columns);
  • each cell holds the customer’s interaction with the product: the sum of the weights of their events, from Interactions, not ratings.

Customer C100134 of the last lesson has a row with three filled cells. The floor lamp has 13, and each of the two other lamps has 1. The other 2,991 cells of the row are empty.

Almost every method of this course reads this matrix. Item-item methods compare its columns. User-user methods compare its rows. Matrix factorisation squeezes the whole table into a few hidden tastes. So we build it once, carefully.

Predict first: what share of the 119,760,000 cells do you think is filled?

Here is a real corner of the matrix: 30 customers and 80 products, both picked at random from the events before the test period.

A grid of 30 rows (customers) and 80 columns (products). Almost every cell is white. 13 cells are coloured blue; darker blue means a stronger signal. Under the grid: 13 filled cells of 2,400 (0.54%); the whole matrix is 0.390% filled.
A random corner of Larkfield's user-item matrix (30 customers × 80 products, seed 37). 13 of 2,400 cells are filled. On a phone, the picture turns on its side: products are the rows.
Text description of the diagram

The picture is a grid of 30 rows and 80 columns. Almost all cells are white: no event. 13 cells are blue. Light blue cells are weak signals, such as one view (weight 1). Dark blue cells are strong signals, with a weight of 13 or more (several views, basket adds and an order). One customer has four filled cells; most customers in the corner have none. The note under the grid says: 13 filled cells of 2,400 (0.54%); the whole matrix is 0.390% filled.

The whole matrix has 467,160 filled cells of 119,760,000: 0.390%. Fewer than 4 cells in 1,000 have any signal. A matrix with almost all cells empty is called sparse. The opposite, a matrix with most cells filled, is dense.

The problem: the dense table wastes memory

Section titled “The problem: the dense table wastes memory”

Tomás tries the simple way. He makes a normal table, a pandas DataFrame with a column for each product, and fills in the numbers. The table stores every cell, empty or not. With 4 bytes per number:

40,000 × 2,994 × 4 bytes = 479 MB

That is 479 MB to store 467,160 useful numbers. With pandas’s default 8-byte numbers, it is twice as much. And every computation on the table reads all the empty cells too: a sum over a row adds 2,994 numbers to find three.

A sparse matrix keeps only the filled cells. For each one, it keeps three numbers: the row, the column and the value. The empty cells cost nothing. Here is a tiny constructed example with 4 customers and 5 products:

On the left, a 4 by 5 matrix with customers c0 to c3 as rows and products p0 to p4 as columns. Four cells are filled: c0 p1 = 5, c1 p3 = 1, c1 p4 = 3, c3 p0 = 1. On the right, three lists of four numbers: row 0 1 1 3, column 1 3 4 0, value 5 1 3 1.
A constructed 4 × 5 example. The dense table keeps 20 numbers; the sparse matrix keeps three short lists.
Text description of the diagram

On the left is a matrix of 4 customers (c0 to c3) and 5 products (p0 to p4). Four cells have a value. c0 ordered p1 (5). c1 viewed p3 (1) and added p4 to the basket (3). c3 viewed p0 (1). The other 16 cells are empty. On the right are the three lists that a sparse matrix keeps. The rows are 0, 1, 1, 3. The columns are 1, 3, 4, 0. The values are 5, 1, 3, 1. The empty cells cost nothing.

Read the first column of the lists: row 0, column 1, value 5. That is customer c0 and product p1, with weight 5. Each filled cell is one column of the three lists.

SciPy, the scientific library for Python, has several sparse formats. The project uses CSR (“compressed sparse row”). CSR keeps the columns and the values in row order. Instead of one row number per cell, it keeps where each row starts in the lists. So it can read one customer’s row very fast. That is what a recommender does all the time: “give me this customer’s products”.

With CSR and 4-byte numbers, Larkfield’s matrix needs 3.9 MB instead of 479 MB.

The matrix is not only empty. Its rows are empty in a very uneven way. Here are the orders that each customer made before the test period:

Orders per customer before 22 June 2026 (synthetic log), for the 22,857 customers who ordered. More than half ordered once.
Text description of the chart

Six bars. 1 order: 12,281 customers. 2 orders: 4,633. 3 orders: 2,216. 4 orders: 1,248. 5 orders: 734. 6 or more orders: 1,745. In total, 22,857 customers ordered at least once before the test period; 17,143 of the 40,000 never ordered.

More than half of the customers who ordered did it once. A customer with one order and a few views has a row with a handful of filled cells. The median customer with any event has 7 filled cells: 7 products of 2,994.

The columns are uneven too, in the other direction. The 1% most touched products have 7.8% of all filled cells, and 6 products have no filled cell at all. This is the long tail of Meet the shop log, seen in the matrix.

Why does this matter? Every method in this course fills the empty cells with a guess. For a customer with 40 filled cells, there is a lot to learn from. For a customer with one order, there is almost nothing. That is the real difficulty of recommendation, and the reason why later lessons score each segment of customers separately: new, occasional and loyal.

Common mistake

Calling toarray() "just to look at it"

Tomás has the sparse matrix M and wants to see it as a table, so he calls M.toarray(). Python now builds the dense table: 479 MB with 4-byte numbers, about 958 MB with pandas’s default 8-byte numbers. On a laptop with other programs open, this can slow everything down. And a bigger shop makes it impossible: a shop with 1 million customers and 50,000 products would need 200 GB. Look at small parts instead: M[:5, :10].toarray() for a corner, or M.getrow(i) for one customer. Keep the full matrix sparse.

The command recs matrix builds the matrix from the events before the test period and prints how empty it is. It is a new file of the snapshot end-of-m02.

  1. In your project folder, with the virtual environment active, copy the command from the snapshot:

    Terminal window
    cp ../nextia-learning-labs/C37/snapshots/end-of-m02/larkfield_recs/commands/matrix.py larkfield_recs/commands/matrix.py
  2. Open larkfield_recs/commands/matrix.py and read run(). It loads the shop, takes the test period, and calls p.matrix(w) with your weights. p.matrix() is in larkfield_recs/data.py (function interaction_matrix). Read it too.

  3. Build the matrix with the default weights: recs matrix.

  4. Build it with orders only: recs matrix --weights view=0,basket=0,order=1.

The recs command finds its subcommands by their files, so the new file is enough: no other file changes. The core of interaction_matrix is three lines:

w = events.event.map(weights).astype(float).fillna(0).to_numpy()
m = sp.csr_matrix((w, (events.c.to_numpy(), events.p.to_numpy())), shape=(n_customers, n_products))
m.sum_duplicates()

The first line gives each event its weight. The second line makes a CSR matrix from three lists: the weights, the customer numbers (rows) and the product numbers (columns). A customer with five events on one product gives five entries for the same cell. The third line adds them into one value: the sum of the weights.

Show the expected result

Step 3 prints (the course team’s run):

Events before 2026-06-22: 770,140; weights {'view': 1.0, 'basket': 3.0, 'order': 5.0}
Matrix: 40,000 customers x 2,994 products, 467,160 filled cells (0.390%); memory 3.9 MB sparse, 479 MB if it were dense
Customers with a filled cell: 38,377; products per customer: median 7, mean 12.2
Orders per customer (customers who ordered): 1: 12,281, 2: 4,633, 3: 2,216, 4: 1,248, 5: 734, 6 or more: 1,745
Products: the 1% most touched products have 7.8% of the filled cells; 6 products have none

Step 4 prints:

Events before 2026-06-22: 770,140; weights {'view': 0.0, 'basket': 0.0, 'order': 1.0}
Matrix: 40,000 customers x 2,994 products, 73,593 filled cells (0.061%); memory 0.7 MB sparse, 479 MB if it were dense
Customers with a filled cell: 22,857; products per customer: median 2, mean 3.2
Orders per customer (customers who ordered): 1: 12,281, 2: 4,633, 3: 2,216, 4: 1,248, 5: 734, 6 or more: 1,745
Products: the 1% most touched products have 20.8% of the filled cells; 430 products have none

The data is the same for everyone, so your numbers are the same. Each command takes less than a second.

Compare the two runs. With orders only, the matrix keeps 73,593 cells instead of 467,160. It is more than six times emptier. 15,520 customers lose their whole row (38,377 against 22,857), and 430 products have no signal at all. The weak signals of the last lesson fill most of the matrix.

When to use a sparse matrix, and what it costs

Section titled “When to use a sparse matrix, and what it costs”
Sparse matrix (CSR) Dense table
Memory about 8 bytes for each filled cell: 3.9 MB here 4 bytes for every cell: 479 MB here
Read one customer’s row fast fast
Change cells one by one slow: build it again from the lists fast
Good for the whole matrix, every method in this course a small part you want to print or plot

A rule of thumb: when fewer than about 10% of the cells are filled, keep the matrix sparse. Recommendation data is almost always far below that. Libraries for implicit feedback, such as the implicit library of Hidden tastes, take a CSR matrix as their input.

The cost is small: CSR is slow to change cell by cell. So the project builds the matrix once from the events, in one step, and never edits it.

Guided practice: the matrix in the notebook

Section titled “Guided practice: the matrix in the notebook”
  1. Open the notebook of this lesson and run the setup cells. Wait for Ready: 4 files in larkfield-recs.
  2. Run section 1. It gives each event a row number (its customer) and a column number (its product).
  3. Predict the share of filled cells. Then run section 2.
  4. Run the next cell: the row of customer C100134.
  5. Run section 3: the orders per customer, as a bar chart.
Show the expected result

Section 2 prints:

40,000 x 2,994 = 119,760,000 cells; 467,160 filled (0.390%)
memory: 3.9 MB sparse, 479 MB as a dense table

The row of C100134 has 3 filled cells: the three lamps, with the weights 13, 1 and 1. Section 3 shows the same counts as recs matrix: 12,281 customers with 1 order, down to 1,745 with 6 or more.

Build a second matrix with the orders only: 1 in a cell if the customer ordered the product at least once. Predict first: how many cells are filled? Then write the cell of section 4 and run the check.

Show the answer

73,593 filled cells (0.061%), the same as recs matrix --weights view=0,basket=0,order=1. The check prints Check passed: filled_orders is 73593. The solution:

o = train[train["event"] == "order"]
O = sp.csr_matrix((np.ones(len(o)), (pd.Categorical(o["customer_id"], categories=cust).codes,
pd.Categorical(o["product_id"], categories=prod).codes)), shape=M.shape)
O.sum_duplicates()
filled_orders = O.nnz

sum_duplicates() turns two orders of the same product into one cell with the value 2. So nnz, the number of stored cells, counts each customer-product pair once.

Check your understanding

Explain the user-item matrix in your own words: what is a row, a column and a cell? Why do we call Larkfield's matrix sparse?

Show the answer

A row is one customer, and a column is one product. A cell holds the customer’s interaction with the product: the sum of the weights of their events. Larkfield’s matrix is sparse because almost every cell is empty. Only 467,160 of 119,760,000 cells have a value: 0.390%.

Check your understanding

A shop has 200,000 customers and 10,000 products, and 0.2% of the cells are filled. Estimate the memory of a dense table with 4-byte numbers, and of a sparse matrix at about 8 bytes per filled cell.

Show the answer

Dense: 200,000 × 10,000 × 4 bytes = 8,000,000,000 bytes, about 8 GB. Sparse: 0.2% of 2,000,000,000 cells is 4,000,000 filled cells. At about 8 bytes each, that is about 32 MB, plus a little for where each row starts. The sparse matrix is about 250 times smaller.

Check your understanding

More than half of Larkfield's buyers ordered only once. Why does this make recommendation hard, even with a perfect method?

Show the answer

A customer with one order and a few views has a row with only a handful of filled cells: very little to learn from. Any method must guess most of that customer’s interests from what other customers did. So the scores of later lessons are lower for customers with little history. To see it, we look at the segments (new, occasional, loyal) separately.

Recap

  • The user-item matrix has one row per customer, one column per product, and the interaction (the sum of the event weights) in each cell. Almost every method of this course reads it.
  • Larkfield’s matrix has 467,160 filled cells of 119,760,000: 0.390%. It is sparse.
  • A sparse matrix keeps only the filled cells: row, column and value. In CSR format it needs 3.9 MB instead of 479 MB. Never call toarray() on the whole matrix.
  • Rows are very uneven: more than half of the buyers ordered once, and the median customer has 7 filled cells.
  • recs matrix builds it; --weights view=0,basket=0,order=1 keeps only the orders: 73,593 cells, more than six times emptier.

Next, we hide some cells and ask a method to guess them, and see why the way we hide them changes the score.

Next: Split by time