Skip to content
Français

Tune without fooling yourself

  • Lesson 16
  • 35 min
  • Module 5: Matrix factorisation for implicit feedback

Where we are

ALS and BPR have settings that change their scores. It is easy to tune on the test period by mistake. In this lesson, we tune on the validation period, test once, and compare with the baselines by segment.

What you will be able to do

Choose the settings of ALS and BPR without fooling yourself. Tune on the validation period, and test the chosen setting once on the test period. Then compare it with the baselines and with item-item collaborative filtering, segment by segment.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook first shows, with a small simulation, why the best of many tries looks too good. Then it tunes ALS on the validation period with four settings, and you test the chosen one once. Its setup cells download about 8 MB of practice data and install implicit. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with mf.py and implicit from the last lesson. recs evaluate als prints recall@10 0.142.
  • The practice files in ../nextia-learning-labs, beside the project folder.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

A teacher prepares a class for a final exam. The students do practice tests. After each one, the teacher changes the lessons a little: more time on fractions, less on graphs. The practice marks go up.

Now imagine that the practice tests used the questions of the final exam. The marks would go up fast. The students learn those exact questions. On exam day, the class gets a top mark, and the head teacher thinks the teaching is excellent. Next year, with new questions, the marks fall. The top mark measured the questions, not the teaching.

Tuning a recommender has the same trap. You try many settings and keep the one with the best score. If you score them on the test period, the test period stops being a fair exam.

The problem: Tomás tunes on the test period

Section titled “The problem: Tomás tunes on the test period”

Tomás wants ALS to beat trending. He tries the same 18 settings that you will try in this lesson, and he scores each one on the test period. The best one, with 32 numbers per vector, regularisation 0.1 and alpha 2, gets recall@10 0.155. Trending has 0.157. “ALS ties with trending,” he says.

But he chose that setting because it had the best test score. Some of its score is real skill, and some is luck: a few customers that this setting happened to get right. The test number no longer tells how the setting will do on new weeks.

We ran this mistake on purpose, to measure it (a new run, logged with the course’s runs). The 18 settings score between 0.137 and 0.155 on the test period. The honest process, which you follow below, chooses another setting on the validation period, and that setting scores 0.149 on the test period. Tomás’s number is too high, and his conclusion “ALS ties with trending” is not supported.

How big can the luck be? A small simulation

Section titled “How big can the luck be? A small simulation”

The notebook makes the effect visible with a constructed case. Take 18 settings that are all equally good: each has true recall 0.150. Each measurement on about 2,000 customers adds chance. The bootstrap intervals of the last lessons are about ±0.015 wide, so we add chance with a standard deviation of 0.007.

Recall
True recall of every setting 0.150
Best of the 18 on the first measurement 0.160
The same “best” setting, measured again on new customers 0.151
Mean of the best of 18, over 1,000 repeats of the experiment 0.163

None of the 18 settings is better than the others. Yet the winner of the first measurement looks 0.010 better. Measured again, it falls back to 0.151. This effect has a name: the winner’s curse. When you pick the best of many noisy numbers, the number you picked is usually too high.

The idea: choose on one period, report on another

Section titled “The idea: choose on one period, report on another”

The cure is to keep two periods apart:

  • The validation period (8 to 21 June 2026) is for choosing. Try as many settings as you like there.
  • The test period (22 June to 5 July 2026) is for reporting. Score the chosen setting once, and report that number, whatever it is.

Settings that you choose, rather than settings the model learns from data, are called hyperparameters: factors, regularisation, alpha, iterations, learning rate. The machine learning course explains the same rule in Train, validation and test sets and Tune hyperparameters. Here the sets are periods of time.

Tuning, with 18 settings and many runs: learn from every event before 8 June 2026, score on the validation period from 8 to 21 June, and keep the best setting. Only the chosen setting goes on to the final test, one run: learn again from every event before 22 June 2026, with the validation weeks included, and score once on the test period, 22 June to 5 July. Report this number.
Tune on the validation period, then learn again with the validation weeks included and score the test period once.
Text description of the diagram

Two groups of boxes. The first group, tuning with 18 settings and many runs: learn from every event before 8 June 2026, then score on the validation period, 8 to 21 June, and keep the best setting. An arrow labelled “the chosen setting only” leads to the second group, the final test, with one run. It learns again from every event before 22 June 2026, with the validation weeks included. Then it scores once on the test period, 22 June to 5 July. That last box has a thick border: report this number.

Note the second training. For the final test, the model learns again from every event before 22 June. The two validation weeks are the most recent data, and a real shop would use them. The setting stays fixed; only the data grows.

The project has a command for this: recs tune. It runs every combination of the values you give, always on the validation period. It has no option to tune on the test period.

  1. Copy the command and the tests of this module:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m05
    cp $S/larkfield_recs/commands/tune.py larkfield_recs/commands/
    cp $S/tests/test_m05.py tests/
  2. Run the tests with python -m pytest -q. You see 20 passed. One new test checks that recs tune uses only the validation period.

  3. Tune ALS: 3 values of factors × 2 of regularisation × 3 of alpha = 18 settings. This takes about a minute.

    Terminal window
    recs tune als --grid factors=32,64,128 --grid regularization=0.01,0.1 --grid alpha=1,2,5
    {"factors": 32, "regularization": 0.01, "alpha": 1} recall@10 0.166 NDCG@10 0.101 fit 1.0 s
    {"factors": 32, "regularization": 0.01, "alpha": 2} recall@10 0.161 NDCG@10 0.098 fit 0.8 s
    {"factors": 32, "regularization": 0.01, "alpha": 5} recall@10 0.156 NDCG@10 0.094 fit 0.8 s
    {"factors": 32, "regularization": 0.1, "alpha": 1} recall@10 0.169 NDCG@10 0.103 fit 1.0 s
    {"factors": 32, "regularization": 0.1, "alpha": 2} recall@10 0.162 NDCG@10 0.098 fit 0.8 s
    {"factors": 32, "regularization": 0.1, "alpha": 5} recall@10 0.155 NDCG@10 0.094 fit 0.8 s
    {"factors": 64, "regularization": 0.01, "alpha": 1} recall@10 0.153 NDCG@10 0.097 fit 3.8 s
    {"factors": 64, "regularization": 0.01, "alpha": 2} recall@10 0.155 NDCG@10 0.096 fit 3.6 s
    {"factors": 64, "regularization": 0.01, "alpha": 5} recall@10 0.147 NDCG@10 0.091 fit 3.6 s
    {"factors": 64, "regularization": 0.1, "alpha": 1} recall@10 0.151 NDCG@10 0.096 fit 3.8 s
    {"factors": 64, "regularization": 0.1, "alpha": 2} recall@10 0.155 NDCG@10 0.096 fit 3.6 s
    {"factors": 64, "regularization": 0.1, "alpha": 5} recall@10 0.148 NDCG@10 0.091 fit 3.6 s
    {"factors": 128, "regularization": 0.01, "alpha": 1} recall@10 0.154 NDCG@10 0.095 fit 6.3 s
    {"factors": 128, "regularization": 0.01, "alpha": 2} recall@10 0.152 NDCG@10 0.094 fit 6.4 s
    {"factors": 128, "regularization": 0.01, "alpha": 5} recall@10 0.151 NDCG@10 0.093 fit 6.4 s
    {"factors": 128, "regularization": 0.1, "alpha": 1} recall@10 0.154 NDCG@10 0.095 fit 6.1 s
    {"factors": 128, "regularization": 0.1, "alpha": 2} recall@10 0.153 NDCG@10 0.094 fit 6.3 s
    {"factors": 128, "regularization": 0.1, "alpha": 5} recall@10 0.152 NDCG@10 0.093 fit 6.4 s
    Best on validation: {"factors": 32, "regularization": 0.1, "alpha": 1} (recall@10 0.169); saved to outputs/best-als.json
    Now test it once: recs evaluate als --set factors=32 --set regularization=0.1 --set alpha=1
  4. Test the chosen setting once, by segment:

    Terminal window
    recs evaluate als --set factors=32 --set regularization=0.1 --set alpha=1 --by segment
    Method als (matrix factorisation, alternating least squares (implicit)), test period, settings {'factors': 32, 'regularization': 0.1, 'alpha': 1}
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.149 (95% interval 0.135 to 0.164), NDCG@10 0.097, MAP@10 0.072, precision@10 0.023, hit rate 19.6%
    coverage 16.5% of the products on sale, novelty 5.70 bits, diversity 0.81, new products in the lists 10.0%
    time: fit 1.1 s, suggestions 0.1 s
    segment customers recall@10 NDCG@10 95% interval (recall)
    new 952 0.154 0.101 0.132 to 0.175
    occasional 556 0.140 0.086 0.115 to 0.167
    loyal 610 0.150 0.100 0.126 to 0.176
    new products: 647 orders of new products in the period; recall@10 on them 0.131
  5. Your project now matches the snapshot end-of-m05. Only its README.md differs.

For a first try with a new grid, add --quick: it scores 1,000 validation customers instead of all of them. recs tune als --grid factors=32,64 --grid alpha=1,2 --quick took 9 seconds and chose the same setting.

The chart shows the validation recall of the six settings with regularisation 0.1. The six settings with 0.01 look almost the same.

  • alpha 1
  • alpha 2
  • alpha 5
ALS on the validation period, regularisation 0.1. Fewer numbers per vector and a low alpha work best on this shop log.
Text description of the chart

Three lines, one for each alpha, over factors 32, 64 and 128. Alpha 1: 0.169, 0.151, 0.154. Alpha 2: 0.162, 0.155, 0.153. Alpha 5: 0.155, 0.147, 0.152. The highest point is 32 factors with alpha 1, recall 0.169, the chosen setting.

Two lessons from the grid:

  • More numbers per vector are not better here. 32 beats 64 and 128 for every alpha. Most Larkfield customers have very few events, and more numbers let the model fit noise in those few events. Fewer numbers also train three to six times faster.
  • A low alpha is best. Trusting the filled cells more (alpha 5) lowers recall, as alpha 20 did in your last task.
ALS Validation recall@10 Test recall@10
Default settings (64, 0.05, 2) 0.155 0.142
Chosen on validation (32, 0.1, 1) 0.169 0.149 (0.135 to 0.164)
For comparison: trending (not tuned) 0.152 0.157

The chosen setting fell from 0.169 on validation to 0.149 on test. Two reasons add up. First, the winner’s curse: it was the best of 18. Second, the two periods are different weeks. The default ALS also falls (0.155 to 0.142), while trending rises (0.152 to 0.156). The late-June weeks of the test period are harder for ALS and easier for a list that follows the season.

Tuning helped a little: 0.149 against 0.142 for the default. On the same customers, recs compare gives +0.007 (−0.001 to +0.015): not clear. Tuning also halved the coverage, from 33.0% to 16.5%: the tuned lists show fewer different products.

And BPR? We tuned it the same way: 6 settings of factors and learning rate. The best on validation (64 factors, learning rate 0.01) had validation recall 0.117. On the test period it scored 0.106, a little below the default’s 0.109. Tuning does not always help. You report the number anyway: that is what makes the test period worth trusting.

Your turn: does tuned ALS beat item-item, segment by segment?

Section titled “Your turn: does tuned ALS beat item-item, segment by segment?”

This is the question of the module. In Module 4, item-cf was weak for occasional and loyal customers. Does tuned ALS do better for each segment, clearly?

Before you run it, predict: in which segment will the gain be largest, and is any segment not clear?

The run names contain the settings in alphabetical order, so put the name in quotes:

Terminal window
recs compare item-cf "als[alpha=1,factors=32,regularization=0.1]" --by segment
Show the answer
recall@10: item-cf 0.088, als[alpha=1,factors=32,regularization=0.1] 0.149
difference +0.061 (95% interval +0.051 to +0.073); better for 210 customers, worse for 48, the same for 1,860
The difference is clear: the interval does not include 0.
coverage 53.2% -> 16.5%; novelty 6.45 -> 5.70
new 952 customers +0.027 (+0.016 to +0.039)
occasional 556 customers +0.095 (+0.068 to +0.124)
loyal 610 customers +0.083 (+0.060 to +0.105)

Yes, in every segment, and every interval is above 0. The gain is largest for occasional customers: 0.140 against 0.045, three times as much. Their one or two old orders point item-cf at alternatives that they no longer need, as Module 4 showed. ALS places them by taste, and their taste also fits what they buy next. For new customers, the gain is small (+0.027): they have only views, so both methods know little about them.

Beating item-cf is not enough. The baseline to beat is trending. The next chart shows recall@10 by segment for trending, tuned ALS and item-cf, with their intervals.

Recall@10 by segment on the test period, with 95% intervals. Tuned ALS is far above item-cf for occasional and loyal customers, and close to trending everywhere.
Text description of the chart

Three panels, one per segment, each with three rows. New customers (952): trending 0.169 (0.147 to 0.192), tuned ALS 0.154 (0.132 to 0.175), item-cf 0.127 (0.107 to 0.148). Occasional customers (556): trending 0.162 (0.134 to 0.190), tuned ALS 0.140 (0.115 to 0.167), item-cf 0.045 (0.030 to 0.061). Loyal customers (610): trending 0.132 (0.111 to 0.157), tuned ALS 0.150 (0.126 to 0.176), item-cf 0.067 (0.050 to 0.086).

On the same customers:

Terminal window
recs compare trending "als[alpha=1,factors=32,regularization=0.1]" --by segment
recall@10: trending 0.156, als[alpha=1,factors=32,regularization=0.1] 0.149
difference -0.007 (95% interval -0.022 to +0.009); better for 221 customers, worse for 246, the same for 1,651
The difference is not clear: the interval includes 0.
coverage 0.3% -> 16.5%; novelty 5.10 -> 5.70
new 952 customers -0.015 (-0.033 to +0.002)
occasional 556 customers -0.022 (-0.061 to +0.017)
loyal 610 customers +0.018 (-0.016 to +0.051)

So, plainly: tuned ALS (0.149) is still below trending (0.156) on Larkfield’s test period. The difference is not clear, on all customers or in any segment. ALS is a little better for loyal customers and a little worse for the others, and none of these differences is clear. After a whole module, the best single personal method is level with a list of last month’s bestsellers.

That is not a failure of the lesson. It is a common result in real shops: a strong non-personal baseline is hard to beat with one personal method. ALS does bring something that trending does not. Its lists differ between customers (coverage 16.5% against 0.3%), and they follow each customer’s taste. The next modules use that. Module 6 combines methods, and Module 7 lets a ranker learn when to trust which one.

Tune when a method has settings that change its result, which is almost always. Decide the grid and the measure before you look at any test number.

Rules that keep you honest:

  • Choose on the validation period. Score the test period once, for the final setting.
  • Report the test number even when it is disappointing, as for BPR here.
  • If you change the grid after a test result, the test period is used up for that decision. In practice, keep a later period hidden, as the course’s final assignment does.

What it costs: every setting is one training. 18 settings of ALS took about a minute on our laptop (one thread). The grid grows fast: add 3 values of iterations, and 18 becomes 54. Start with --quick and a small grid, then refine around the best value.

Common mistake

'no matches found' in zsh

Symptom: on macOS, recs compare item-cf als[alpha=1,factors=32,regularization=0.1] stops before it runs:

zsh: no matches found: als[alpha=1,factors=32,regularization=0.1]

Cause: zsh, the default shell on macOS, reads square brackets as a file pattern.

Fix: put the run name in quotes: "als[alpha=1,factors=32,regularization=0.1]". Quotes work in bash and PowerShell too.

Common mistake

'No saved run' for a tuned method

Symptom: recs compare item-cf "als[factors=32,regularization=0.1,alpha=1]" stops with:

No saved run als[factors=32,regularization=0.1,alpha=1]-test. Run first: recs evaluate als --set factors=32 --set regularization=0.1 --set alpha=1

Cause: the project saves a run with its settings in alphabetical order: alpha, factors, regularization.

Fix: write the settings in that order, or look up the exact name in outputs/runs/.

Check your understanding

Tomás tried 18 settings on the test period and reports the best one, 0.155. Priya chose on the validation period and reports 0.149 on the test period. Whose number should Omar use to decide, and why?

Show the answer

Priya’s. Tomás chose his setting because it had the best test score, so his 0.155 contains the luck of being the best of 18: the winner’s curse. Priya chose on the validation period and scored the test period once. So her 0.149 estimates how the setting does on weeks it has not seen. In the simulation, the best of 18 equal settings looked about 0.013 too good on average.

Check your understanding

Tuned ALS has validation recall 0.169 and test recall 0.149. Give two reasons for the fall. How do the untuned methods help you tell them apart?

Show the answer

First, the winner’s curse: 0.169 is the best of 18, so it is probably too high. Second, the periods differ: the test weeks are other weeks, with other customers and a later season. The untuned methods show the second effect without the first. Default ALS also falls (0.155 to 0.142), and trending rises (0.152 to 0.156). So the test period is harder for ALS anyway, and part of the fall has nothing to do with tuning.

Check your understanding

Recommend one method for the home page of loyal customers, from the methods so far. State the evidence.

Show the answer

For loyal customers (610 test customers), recall@10 is: popular-category 0.164 (0.139 to 0.192), tuned ALS 0.150 (0.126 to 0.176), user-user 0.147, trending 0.132, item-cf 0.067. Tuned ALS is clearly better than item-cf (+0.083, +0.060 to +0.105). But it is not clearly different from popular-category (−0.014, −0.036 to +0.007) or trending (+0.018, −0.016 to +0.051). A sound answer: popular-category or tuned ALS, and say that the data do not separate them. Popular-category is simpler and has the highest number. Tuned ALS gives more varied, more personal lists (coverage 16.5% against 6.7%). Whichever you choose, say that the difference is not clear on 610 customers.

Recap

  • If you choose settings on the test period, the test number is too high: the winner’s curse. Tomás’s best of 18 on the test period was 0.155; the honest number is 0.149.
  • Choose hyperparameters on the validation period, then learn again with the validation weeks included and score the test period once.
  • recs tune als --grid ... tried 18 settings in about a minute. The winner: 32 factors, regularisation 0.1, alpha 1, with validation recall 0.169. On the test period: 0.149 (0.135 to 0.164), against 0.142 with the defaults.
  • Tuned BPR scored 0.106, below its default 0.109. Report it anyway.
  • Tuned ALS clearly beats item-cf in every segment: new +0.027, occasional +0.095, loyal +0.083.
  • Tuned ALS is still below trending (0.156) on Larkfield’s test period. The difference (−0.007) is not clear in any segment.
  • Your project now matches the end of Module 5.

Every method so far needs history, and a product with no events gets no vector. Next, you describe products by their text instead, so that even a brand-new product can be suggested.

Next: Content-based recommendations