Skip to content
Français

Split by time

  • Lesson 6
  • 40 min
  • Module 2: Data, implicit feedback and an honest split

Where we are

We have the user-item matrix. To test a method, we must hide some interactions and predict them. In this lesson, we split by time, and we measure how much a random split makes a method look better than it is.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Test a recommender the way the shop will use it: learn from the past, guess the next weeks. Measure how much a random split makes methods look better, and explain why, so that you never trust such a score.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook makes the time split and the project’s random split with pandas. It counts the training events from the future and scores popularity on both. Your task: measure who each split tests. Its setup cells install the packages and download the data. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • Your larkfield-recs project with recs matrix from The user-item matrix, and the labs repository next to it.

After this lesson, your project has the recs leakage command of the snapshot end-of-m02. The shop log is synthetic. The course team measured the numbers on an Apple M4 laptop. Tested on macOS; the Windows commands are not tested.

Larkfield wants to know if a new shop assistant is good at suggestions. Here is a fair test. She studies the till receipts of the last year. Then, for the next two weeks, she writes down what she thinks each regular customer will buy. At the end of the two weeks, we compare her guesses with what people really bought.

Here is an unfair test. We take the whole year of receipts, cut out 5% of the lines at random, and give her the rest. She must guess the missing lines. But the receipts she studies include next month. She sees that Ana bought a tent in May and a camping lamp in July. So she easily guesses the sleeping bag that Ana bought in June. She did not predict anything. She read the answer from the future.

The second test is easier, so she gets a better mark. But the mark says nothing about the job she will do: guessing next week, with no receipts from next week.

Tomás builds his first model with a tutorial. The tutorial hides 5% of the orders at random, trains on the rest and measures recall@10. His matrix factorisation model (ALS, which you will build in ALS and BPR with implicit) gets 0.336. Popularity gets 0.113. He is ready to tell Omar, the product manager, that the model is three times better than popularity.

Priya scores the same model on the time split. It trains on every event before 22 June 2026 and is tested on the orders of 22 June to 5 July. The model gets 0.142. Most of Tomás’s score came from the way he hid the orders, not from the model.

This mistake is called leakage: information from the test (here, the future) leaks into the training data. The model looks good in the test and is worse in the shop.

A time split cuts the log at one date. Everything before the cut is training data. The orders after the cut are the test. The model never sees an event from after the cut. This is exactly the situation of the shop: tonight, we train on everything up to today, and tomorrow customers come.

Two timelines from February 2025 to July 2026, with a dot for orders. Random split: orange test dots in every month, mixed with blue training dots, also after 22 June 2026 (a dashed line). Time split: all dots before a red line at 22 June 2026 are blue (training), the dots after it are orange (the test, 22 June to 5 July).
The random split hides orders from every month, and trains on the rest, the future included. The time split trains only on the past. Dots follow the real orders per month; the share of orange dots in the random split is not to scale.
Text description of the diagram

Two timelines run from February 2025 to July 2026. Each dot stands for some of the shop’s orders, more dots in busier months. In the random split, orange dots (orders hidden for the test) appear in every month, mixed with blue dots (training). Blue dots also appear after the hidden orders and after 22 June 2026, so the model trains on the future. In the time split, every dot before a red line at 22 June 2026 is blue. Only the dots after it, the two weeks to 5 July, are orange. Nothing after the cut is used for training.

Here is a constructed example with one customer and three orders:

Date Ana ordered
10 May a tent
3 June a sleeping bag
20 July a camping lamp

Random split. The split hides the sleeping bag. The training data has the tent (May) and the camping lamp (July). Many other customers bought a tent, a sleeping bag and a lamp together. So any method that looks at what goes together finds the sleeping bag easily. It used July to guess June.

Time split, cut on 1 June. The training data has only the tent. The test is everything Ana ordered after 1 June: the sleeping bag and the lamp. The method must guess them from the tent alone, as the shop would on 1 June.

The time split is harder. It is also the only one of the two that answers the shop’s question.

Measured: how much does a random split inflate?

Section titled “Measured: how much does a random split inflate?”

Priya scores five methods on both splits, with the same 2,118 test customers on each side. You already know popularity. Trending (the most ordered products of the last 28 days) and popular-category come in the next lesson. Item-cf and ALS are personal methods of later modules.

Before you explore, predict: which method gains the most from the random split, and does any method lose?

Explore

The same five methods on a random split and on the time split

Split
Time split
Best method on this split
trending (0.156)
Loyal customers among those scored
28.8%

Train before 22 June 2026, test on the orders of the next two weeks. The time split: the honest score. trending is the best of these methods with 0.156. Loyal customers are 45.2% of the random split's customers and 28.8% of the time split's.

  • Time split
  • 95% interval
recall@10 on the test customersdashed mark: the random split0.00.10.20.30.4popularpopular, Time split: recall@10 0.1060.106trendingtrending, Time split: recall@10 0.1560.156popular-categorypopular-category, Time split: recall@10 0.1430.143item-cfitem-cf, Time split: recall@10 0.0880.088alsals, Time split: recall@10 0.1420.142
recall@10 on the test customersdashed mark: the random split0.00.10.20.30.4popularpopular, Time split: recall@10 0.1060.106trendingtrending, Time split: recall@10 0.1560.156popular-categorypopular-category, Time split: recall@10 0.1430.143item-cfitem-cf, Time split: recall@10 0.0880.088alsals, Time split: recall@10 0.1420.142
Who the time split scores (share of its customers)new: 45%new 45%occasional: 26.3%occasional 26.3%loyal: 28.8%loyal 28.8%
Who the time split scores (share of its customers)new: 45%45%newoccasional: 26.3%26%occasionalloyal: 28.8%29%loyal
Show both splits side by side
recall@10 of each method on both splits (2,118 customers each)
MethodRandom splitTime splitRandom against time
popular0.1130.106+7%
trending0.0930.156−40%
popular-category0.2270.143+59%
item-cf0.3100.088+251%
als0.3360.142+136%
recall@10 of each method with its 95% interval (bootstrap over customers), measured on the synthetic shop log. Below the bars: who each split scores.
Text description of the explorer

The explorer shows recall@10 of five methods. It starts on the time split. There, popular scores 0.106, trending 0.156, popular-category 0.143, item-cf 0.088 and ALS 0.142. Trending is the best. On the random split, popular scores 0.113 (+7%) and trending 0.093 (−40%). Popular-category scores 0.227 (+59%), item-cf 0.310 (+251%) and ALS 0.336 (+136%). ALS is then the best and item-cf second. A dashed mark on each row shows the score on the other split. Under the bars, a stacked bar shows the customers that the split scores. The time split has 45.0% new, 26.3% occasional and 28.8% loyal customers. The random split has 15.2% new, 39.6% occasional and 45.2% loyal customers. A closed section shows both splits side by side.

The answer: item-cf gains the most (0.088 → 0.310, +251%), then ALS (+136%) and popular-category (+59%). Popularity barely moves (+7%). And trending loses (0.156 → 0.093, −40%). The ranking turns around: the best method of the time split is the worst of the five on the random split.

There is no fixed size for this inflation. It depends on the method. A personal method that learns what goes together gains the most, because the future tells it exactly what goes together for this customer.

1. The training data contains the future. The random split trains on 34,813 events that happened on or after 22 June 2026, the first day of the test period. Some are the test customers’ own later views, baskets and orders. Item-cf finds “people who bought X also bought Y”. With the future in the training data, the Y is often an order the customer made after the hidden one, like Ana’s lamp.

2. The random split tests different customers. It picks orders at random, not customers. A customer with 20 orders has 20 chances to be picked; a customer with one order has one. So the random split tests heavier buyers. 45.2% of its customers are loyal (3 or more orders before), against 28.8% on the time split. Loyal customers have long histories, so personal methods guess them more easily. The time split has 45.0% new customers, who have no order at all.

And why does trending lose? Trending is built for “now”: the bestsellers of the 28 days before 22 June, mostly summer products. The random split’s hidden orders come from every month of the log, the winter too. A summer list is a poor guess for a December order. On the time split, “now” is exactly what the test asks for.

Common mistake

Choosing a method on a random split

Tomás compares methods on his random split and picks ALS (0.336) over trending (0.093). On the time split, which is the shop’s real situation, trending is better: 0.156 against 0.142. A random split does not only make every score too high. It can change which method wins. Every comparison in this course uses the same time split. If a paper or a tutorial reports a recommendation score, check how it split the data before you compare.

Many papers use a third way, leave-last-out: for each customer, the last interaction is the test and everything else is training. It looks like a time split, because the test is always “the latest”. But the cut is a different date for every customer.

Three timelines for Ana, Ben and Chen. Each customer's last order is orange (test), the others blue (training). Ana's last order is early. A dashed red line at Ana's test day shows that Ben and Chen have training orders after it.
A constructed example. Each customer's last order is the test, but on different days. Ana's test is early, so the training data has Ben's and Chen's later orders: for Ana, that is the future.
Text description of the diagram

Three customers, each on a line from left (early) to right (late). Ana has three orders, and her last one, the test, is early. Ben has two orders; his test is late. Chen has four orders; his test is the latest. A dashed red line at Ana’s test day crosses Ben’s and Chen’s lines: Chen has two training orders after it. When the model guesses Ana’s last order, it has learned from orders that came later.

Leave-last-out has two limits:

  1. The future of other customers leaks in. Ana’s test day is in spring, but the model trains on Chen’s orders from the summer. A trend that started after Ana’s test day helps to guess her order.
  2. It tests one order per customer, on many different days. The shop never works like that. It trains once, tonight, and suggests to everyone tomorrow.

Leave-last-out is not always wrong. It is simple, and it is fine for a first look at a method that ignores time. But for a decision about a real shop, use one global cut date for everyone, as Larkfield does.

Larkfield keeps three periods in data/splits.json, all cut by time:

Period Dates Used for
validation 8 to 21 June 2026 choosing settings (tuning), many times
test 22 June to 5 July 2026 the final comparison of methods
launch 1 to 14 May 2026 the new summer range, for cold start

For each period, the training data is every event before its first day. The test customers are the 2,118 customers who ordered in the test period. Every method of this course is scored on the same split. That is what makes the comparison table at the end of the course fair.

The project has the random split from the start (recs evaluate popular --period random), so that you can see the mistake. recs leakage scores methods on both splits side by side.

  1. In your project folder, with the virtual environment active, copy the command from the snapshot:

    Terminal window
    cp ../nextia-learning-labs/C37/snapshots/end-of-m02/larkfield_recs/commands/leakage.py larkfield_recs/commands/leakage.py
  2. Score popularity on the random split: recs evaluate popular --period random.

  3. Compare the two splits: recs leakage.

  4. Open larkfield_recs/data.py and find _random: the random split. Find the line that removes the hidden orders from the training data. Everything else, the future included, stays.

Show the expected result

Step 2 prints:

Method popular (the most ordered products over the whole training log), random split (leaky)
Customers scored: 2,118; trained on every event except the held-out orders
recall@10 0.113 (95% interval 0.099 to 0.125), NDCG@10 0.056
coverage 0.3% of the products on sale
time: fit 0.0 s, suggestions 0.0 s

The second line says what the model learned from: the whole log, the future included, except the hidden orders. The third line is the inflated score: 0.113 against 0.106 on the time split.

Step 3 prints (at this stage, the project has popularity, and the next lesson adds popular-category and trending):

Random split: 2,118 customers, orders held out at random from the whole log (the training data contains the future)
Time split: 2,118 customers, the orders of 2026-06-22 to 2026-07-05
method random split time split random / time (recall@10)
popular 0.113 0.106 +7%

If you copy methods/baselines.py first (the next lesson), recs leakage also shows popular-category 0.227 0.143 +59%. Item-cf and ALS join the table when their modules add them. Trending is not in the default list, but you can name it: recs leakage trending prints trending 0.093 0.156 -40%.

Guided practice: two splits in the notebook

Section titled “Guided practice: two splits in the notebook”
  1. Open the notebook of this lesson and run the setup cells.
  2. Run section 1, the time split.
  3. Run section 2. It makes the same random split as the project.
  4. Predict whether the random split helps popularity. Then run section 3.
Show the expected result
time split: 770,140 training events, the last on 2026-06-21
test: 2,118 customers ordered from 2026-06-22 to 2026-07-05
random split: 796,263 training events; 34,813 of them on or after 2026-06-22
test: 2,118 customers, with orders from any month
popular, recall@10: random split 0.113, time split 0.106

Write loyal_share() in section 4: the share of the test customers with 3 or more orders in the training data. Run it on both splits. Predict first: which split has more loyal customers?

Show the answer
def loyal_share(train, truth):
orders_before = train[train["event"] == "order"].groupby("customer_id")["session_id"].nunique()
n = orders_before.reindex(list(truth)).fillna(0)
return (n >= 3).mean()

It prints loyal: random split 45.2%, time split 28.8%, and both checks pass. An order is a session with an order, so we count different session_id values, not order lines. Section 5 then shows trending: 0.093 on the random split, 0.156 on the time split.

Check your understanding

Completion check: Tomás reports recall@10 0.336 for ALS on a random split. On the time split, ALS gets 0.142. Explain the inflated score in two sentences.

Show the answer

The random split trains on everything except the hidden orders. So the model learns from events after the orders it must guess: 34,813 training events are from the test period or later. It also tests heavier buyers, who are easier to guess (45.2% loyal against 28.8%), not the customers the shop faces.

Check your understanding

Predict and explain: trending scores 0.156 on the time split and 0.093 on the random split. Why does a random split make this method look worse, when it makes the others look better?

Show the answer

Trending is the top 10 of the last 28 days before 22 June: summer products. The random split’s hidden orders come from every month, winter included, and a summer list is a poor guess for a December order. The time split asks about the next two weeks, which is exactly what trending is built for. So a random split can change which method wins, not only the size of the scores.

Check your understanding

A paper evaluates a recommender with leave-last-out. Give two reasons why its score may not tell you how the method would do in Larkfield's shop.

Show the answer

First, the cut date differs per customer. For a customer whose last order is early, the training data has later orders of other customers: the future leaks in. Second, it tests one order per customer on many different days, while the shop trains once and suggests to everyone for the next days. A global time split, the same cut for everyone, matches the shop.

Recap

  • A time split trains on everything before one date and tests on the orders after it. Larkfield’s test period is 22 June to 5 July 2026, with 2,118 customers.
  • A random split leaks the future into the training data. On the same customers, item-cf goes from 0.088 to 0.310 (+251%) and ALS from 0.142 to 0.336 (+136%). Popular-category gains 59% and popular 7%. Trending loses (0.156 → 0.093, −40%).
  • There are two reasons. The training data has the future (34,813 events after the cut). And the random split tests heavier buyers (45.2% loyal against 28.8%).
  • A random split can change which method wins. Never choose a method on it.
  • Leave-last-out cuts at a different date for each customer, so other customers’ future leaks in. Use one global cut date.

Next, we set the scores that every clever method must beat: three popularity baselines on the time split.

Next: The baselines to beat