The user-item matrix
- Lesson 5
- 35 min
- Module 2: Data, implicit feedback and an honest split
Where we are
We turned events into weighted interactions. Most methods need them as one table of customers and products. In this lesson, we build the user-item matrix as a sparse matrix and measure how empty it is.
Lesson 5 of 29 · Recommendation and Personalisation: Show Each Person What MattersThe user-item matrix
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Put every customer’s interactions into one table, the user-item matrix, and store it in a way that fits in memory. See how empty it is, and why that emptiness is the main difficulty of recommendation.
NotebookDo this lesson in a notebook
The notebook builds the matrix of this lesson with SciPy, measures its memory, shows one customer’s row and counts the orders per customer. Your task: build a matrix of orders only. Its setup cells install the packages and download the data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Implicit feedback, the weights view 1, basket 3, order 5, and why an empty cell is unknown, from Interactions, not ratings.
What you need on your computer
- Your
larkfield-recsproject from Meet the shop log, with its virtual environment active.recs checkprintsSetup check passed. - Your copy of the labs repository next to the project, up to date (
git pullinnextia-learning-labs).
After this lesson, your project has the recs matrix command of the snapshot end-of-m02. The shop log is synthetic. The course team ran every command on an Apple M4 laptop. Tested on macOS; the Windows commands are not tested.
A notebook with a page for everyone
Section titled “A notebook with a page for everyone”Imagine that our shop assistant keeps a notebook. Each line is one customer. Each column is one product in the shop. When a customer looks at a product, she writes a small mark in that box. When the customer buys it, she writes a big mark.
For a small shop with 30 regular customers and 50 products, this works well. One page shows who likes what. She can see that two customers have marks on the same garden chairs, and suggest to one of them what the other bought.
Now give her Larkfield: 40,000 customers and 2,994 products. The notebook has 119,760,000 boxes. She would need thousands of pages, and almost every box would be blank.
The user-item matrix
Section titled “The user-item matrix”The table in the notebook has a name: the user-item matrix. “User” is the customer, “item” is the product, and a matrix is a table of numbers with rows and columns. In Larkfield’s matrix:
- each row is one customer (40,000 rows);
- each column is one product (2,994 columns);
- each cell holds the customer’s interaction with the product: the sum of the weights of their events, from Interactions, not ratings.
Customer C100134 of the last lesson has a row with three filled cells. The floor lamp has 13, and each of the two other lamps has 1. The other 2,991 cells of the row are empty.
Almost every method of this course reads this matrix. Item-item methods compare its columns. User-user methods compare its rows. Matrix factorisation squeezes the whole table into a few hidden tastes. So we build it once, carefully.
How empty is it?
Section titled “How empty is it?”Predict first: what share of the 119,760,000 cells do you think is filled?
Here is a real corner of the matrix: 30 customers and 80 products, both picked at random from the events before the test period.
Text description of the diagram
The picture is a grid of 30 rows and 80 columns. Almost all cells are white: no event. 13 cells are blue. Light blue cells are weak signals, such as one view (weight 1). Dark blue cells are strong signals, with a weight of 13 or more (several views, basket adds and an order). One customer has four filled cells; most customers in the corner have none. The note under the grid says: 13 filled cells of 2,400 (0.54%); the whole matrix is 0.390% filled.
The whole matrix has 467,160 filled cells of 119,760,000: 0.390%. Fewer than 4 cells in 1,000 have any signal. A matrix with almost all cells empty is called sparse. The opposite, a matrix with most cells filled, is dense.
The problem: the dense table wastes memory
Section titled “The problem: the dense table wastes memory”Tomás tries the simple way. He makes a normal table, a pandas DataFrame with a column for each product, and fills in the numbers. The table stores every cell, empty or not. With 4 bytes per number:
40,000 × 2,994 × 4 bytes = 479 MB
That is 479 MB to store 467,160 useful numbers. With pandas’s default 8-byte numbers, it is twice as much. And every computation on the table reads all the empty cells too: a sum over a row adds 2,994 numbers to find three.
The idea: store only the filled cells
Section titled “The idea: store only the filled cells”A sparse matrix keeps only the filled cells. For each one, it keeps three numbers: the row, the column and the value. The empty cells cost nothing. Here is a tiny constructed example with 4 customers and 5 products:
Text description of the diagram
On the left is a matrix of 4 customers (c0 to c3) and 5 products (p0 to p4). Four cells have a value. c0 ordered p1 (5). c1 viewed p3 (1) and added p4 to the basket (3). c3 viewed p0 (1). The other 16 cells are empty. On the right are the three lists that a sparse matrix keeps. The rows are 0, 1, 1, 3. The columns are 1, 3, 4, 0. The values are 5, 1, 3, 1. The empty cells cost nothing.
Read the first column of the lists: row 0, column 1, value 5. That is customer c0 and product p1, with weight 5. Each filled cell is one column of the three lists.
SciPy, the scientific library for Python, has several sparse formats. The project uses CSR (“compressed sparse row”). CSR keeps the columns and the values in row order. Instead of one row number per cell, it keeps where each row starts in the lists. So it can read one customer’s row very fast. That is what a recommender does all the time: “give me this customer’s products”.
With CSR and 4-byte numbers, Larkfield’s matrix needs 3.9 MB instead of 479 MB.
Most customers have one or two orders
Section titled “Most customers have one or two orders”The matrix is not only empty. Its rows are empty in a very uneven way. Here are the orders that each customer made before the test period:
Text description of the chart
Six bars. 1 order: 12,281 customers. 2 orders: 4,633. 3 orders: 2,216. 4 orders: 1,248. 5 orders: 734. 6 or more orders: 1,745. In total, 22,857 customers ordered at least once before the test period; 17,143 of the 40,000 never ordered.
More than half of the customers who ordered did it once. A customer with one order and a few views has a row with a handful of filled cells. The median customer with any event has 7 filled cells: 7 products of 2,994.
The columns are uneven too, in the other direction. The 1% most touched products have 7.8% of all filled cells, and 6 products have no filled cell at all. This is the long tail of Meet the shop log, seen in the matrix.
Why does this matter? Every method in this course fills the empty cells with a guess. For a customer with 40 filled cells, there is a lot to learn from. For a customer with one order, there is almost nothing. That is the real difficulty of recommendation, and the reason why later lessons score each segment of customers separately: new, occasional and loyal.
Common mistake
Calling toarray() "just to look at it"
Tomás has the sparse matrix M and wants to see it as a table, so he calls M.toarray(). Python now builds the dense table: 479 MB with 4-byte numbers, about 958 MB with pandas’s default 8-byte numbers. On a laptop with other programs open, this can slow everything down. And a bigger shop makes it impossible: a shop with 1 million customers and 50,000 products would need 200 GB. Look at small parts instead: M[:5, :10].toarray() for a corner, or M.getrow(i) for one customer. Keep the full matrix sparse.
Build it in the project
Section titled “Build it in the project”The command recs matrix builds the matrix from the events before the test period and prints how empty it is. It is a new file of the snapshot end-of-m02.
-
In your project folder, with the virtual environment active, copy the command from the snapshot:
Terminal window cp ../nextia-learning-labs/C37/snapshots/end-of-m02/larkfield_recs/commands/matrix.py larkfield_recs/commands/matrix.pyTerminal window Copy-Item ..\nextia-learning-labs\C37\snapshots\end-of-m02\larkfield_recs\commands\matrix.py larkfield_recs\commands\matrix.py -
Open
larkfield_recs/commands/matrix.pyand readrun(). It loads the shop, takes the test period, and callsp.matrix(w)with your weights.p.matrix()is inlarkfield_recs/data.py(functioninteraction_matrix). Read it too. -
Build the matrix with the default weights:
recs matrix. -
Build it with orders only:
recs matrix --weights view=0,basket=0,order=1.
The recs command finds its subcommands by their files, so the new file is enough: no other file changes. The core of interaction_matrix is three lines:
w = events.event.map(weights).astype(float).fillna(0).to_numpy()m = sp.csr_matrix((w, (events.c.to_numpy(), events.p.to_numpy())), shape=(n_customers, n_products))m.sum_duplicates()The first line gives each event its weight. The second line makes a CSR matrix from three lists: the weights, the customer numbers (rows) and the product numbers (columns). A customer with five events on one product gives five entries for the same cell. The third line adds them into one value: the sum of the weights.
Show the expected result
Step 3 prints (the course team’s run):
Events before 2026-06-22: 770,140; weights {'view': 1.0, 'basket': 3.0, 'order': 5.0}Matrix: 40,000 customers x 2,994 products, 467,160 filled cells (0.390%); memory 3.9 MB sparse, 479 MB if it were denseCustomers with a filled cell: 38,377; products per customer: median 7, mean 12.2Orders per customer (customers who ordered): 1: 12,281, 2: 4,633, 3: 2,216, 4: 1,248, 5: 734, 6 or more: 1,745Products: the 1% most touched products have 7.8% of the filled cells; 6 products have noneStep 4 prints:
Events before 2026-06-22: 770,140; weights {'view': 0.0, 'basket': 0.0, 'order': 1.0}Matrix: 40,000 customers x 2,994 products, 73,593 filled cells (0.061%); memory 0.7 MB sparse, 479 MB if it were denseCustomers with a filled cell: 22,857; products per customer: median 2, mean 3.2Orders per customer (customers who ordered): 1: 12,281, 2: 4,633, 3: 2,216, 4: 1,248, 5: 734, 6 or more: 1,745Products: the 1% most touched products have 20.8% of the filled cells; 430 products have noneThe data is the same for everyone, so your numbers are the same. Each command takes less than a second.
Compare the two runs. With orders only, the matrix keeps 73,593 cells instead of 467,160. It is more than six times emptier. 15,520 customers lose their whole row (38,377 against 22,857), and 430 products have no signal at all. The weak signals of the last lesson fill most of the matrix.
When to use a sparse matrix, and what it costs
Section titled “When to use a sparse matrix, and what it costs”| Sparse matrix (CSR) | Dense table | |
|---|---|---|
| Memory | about 8 bytes for each filled cell: 3.9 MB here | 4 bytes for every cell: 479 MB here |
| Read one customer’s row | fast | fast |
| Change cells one by one | slow: build it again from the lists | fast |
| Good for | the whole matrix, every method in this course | a small part you want to print or plot |
A rule of thumb: when fewer than about 10% of the cells are filled, keep the matrix sparse. Recommendation data is almost always far below that. Libraries for implicit feedback, such as the implicit library of Hidden tastes, take a CSR matrix as their input.
The cost is small: CSR is slow to change cell by cell. So the project builds the matrix once from the events, in one step, and never edits it.
Guided practice: the matrix in the notebook
Section titled “Guided practice: the matrix in the notebook”- Open the notebook of this lesson and run the setup cells. Wait for
Ready: 4 files in larkfield-recs. - Run section 1. It gives each event a row number (its customer) and a column number (its product).
- Predict the share of filled cells. Then run section 2.
- Run the next cell: the row of customer C100134.
- Run section 3: the orders per customer, as a bar chart.
Show the expected result
Section 2 prints:
40,000 x 2,994 = 119,760,000 cells; 467,160 filled (0.390%)memory: 3.9 MB sparse, 479 MB as a dense tableThe row of C100134 has 3 filled cells: the three lamps, with the weights 13, 1 and 1. Section 3 shows the same counts as recs matrix: 12,281 customers with 1 order, down to 1,745 with 6 or more.
Your turn: a matrix of orders only
Section titled “Your turn: a matrix of orders only”Build a second matrix with the orders only: 1 in a cell if the customer ordered the product at least once. Predict first: how many cells are filled? Then write the cell of section 4 and run the check.
Show the answer
73,593 filled cells (0.061%), the same as recs matrix --weights view=0,basket=0,order=1. The check prints Check passed: filled_orders is 73593. The solution:
o = train[train["event"] == "order"]O = sp.csr_matrix((np.ones(len(o)), (pd.Categorical(o["customer_id"], categories=cust).codes, pd.Categorical(o["product_id"], categories=prod).codes)), shape=M.shape)O.sum_duplicates()filled_orders = O.nnzsum_duplicates() turns two orders of the same product into one cell with the value 2. So nnz, the number of stored cells, counts each customer-product pair once.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain the user-item matrix in your own words: what is a row, a column and a cell? Why do we call Larkfield's matrix sparse?
Show the answer
A row is one customer, and a column is one product. A cell holds the customer’s interaction with the product: the sum of the weights of their events. Larkfield’s matrix is sparse because almost every cell is empty. Only 467,160 of 119,760,000 cells have a value: 0.390%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A shop has 200,000 customers and 10,000 products, and 0.2% of the cells are filled. Estimate the memory of a dense table with 4-byte numbers, and of a sparse matrix at about 8 bytes per filled cell.
Show the answer
Dense: 200,000 × 10,000 × 4 bytes = 8,000,000,000 bytes, about 8 GB. Sparse: 0.2% of 2,000,000,000 cells is 4,000,000 filled cells. At about 8 bytes each, that is about 32 MB, plus a little for where each row starts. The sparse matrix is about 250 times smaller.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
More than half of Larkfield's buyers ordered only once. Why does this make recommendation hard, even with a perfect method?
Show the answer
A customer with one order and a few views has a row with only a handful of filled cells: very little to learn from. Any method must guess most of that customer’s interests from what other customers did. So the scores of later lessons are lower for customers with little history. To see it, we look at the segments (new, occasional, loyal) separately.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The user-item matrix has one row per customer, one column per product, and the interaction (the sum of the event weights) in each cell. Almost every method of this course reads it.
- Larkfield’s matrix has 467,160 filled cells of 119,760,000: 0.390%. It is sparse.
- A sparse matrix keeps only the filled cells: row, column and value. In CSR format it needs 3.9 MB instead of 479 MB. Never call
toarray()on the whole matrix. - Rows are very uneven: more than half of the buyers ordered once, and the median customer has 7 filled cells.
recs matrixbuilds it;--weights view=0,basket=0,order=1keeps only the orders: 73,593 cells, more than six times emptier.
Next, we hide some cells and ask a method to guess them, and see why the way we hide them changes the score.
Next: Split by time