Train, validation and test sets

What you will be able to do

Judge a model on data that it did not learn from, and keep a separate set for one final score. A model that memorised its training data looks perfect on that data, and no error message tells you that it fails on new tickets.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. It runs the memorising model on 12 tickets and on the real data, the 100 rules of Tomás and the cross-validation. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Features and labels: a model learns from examples whose right answers are known, from Supervised learning: Features and labels.
• Larkfield’s decision, the target escalated_72h and the capacity of 20%, from Translate a need.
• The baselines and the rule “flag priority 1” (it catches 61 of the 183 May escalations), from Define a baseline.
• The time split of the data and the new warranty policy of 1 May 2026, from Choose a split in SQL and Data Preparation for AI.
• Leakage, and why every step that learns from data uses only the train set, from Fit transformations correctly in the SQL course.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, from Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv, data/valid.csv and data/test.csv, from Define a baseline: Download the data. If a file is missing or changed, run python get_data.py --reset.
• A notebook in the project folder, as in Define a baseline: Open a notebook in VS Code. The code of this lesson uses the helpers of the lesson notebook: copy its cell “Helpers for this lesson” into your notebook first.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The notebook was tested in Jupyter.

How a student learns mathematics

Think of a student who learns a new chapter of mathematics, for example fractions. First, she works through the practice exercises in her book. The answers are in the back of the book, so after each exercise she checks her answer and learns from her mistakes. She does many of them, and some of them more than once.

Then the teacher wants to know if she really understands fractions. So the teacher gives her the end-of-chapter exercises. These are new questions on the same topics, which she has not practised. Her marks show her weak points. Perhaps she adds fractions well, but makes mistakes when the bottom numbers are different. So she goes back, practises that kind of question again, and then tries a new set of chapter exercises. This loop can happen several times.

At the end of the term comes the final exam. Its questions are new again. The teacher keeps the paper sealed until the exam, the student takes it once, and its mark goes on her report. The school, her parents and her next teacher trust that mark, because nobody practised on those questions.

First the practice exercises, then end-of-chapter exercises that find a weak point and send the student back to practise, and last the final exam, sealed until the end.

Text description of the diagram

The picture has three parts. On the left is an open exercise book. Every exercise has an answer box and a green tick, because the student checks each answer in the back of the book. An arrow leads to the middle part, a sheet of end-of-chapter exercises. Four exercises have a green tick and one has a red cross, circled with a dashed red line: a weak point. A dashed red arrow goes from this sheet back to the book, because the student goes back to practise. A second arrow leads to the right part: a closed envelope with a padlock on its seal. It is the final exam, which nobody opens until the end.

Now imagine a lazy teacher who uses the practice exercises as the final exam. A student who memorised every answer in the back of the book scores 100%. She may understand nothing about fractions, and the exam cannot show it. The mark measures her memory of those exact questions, not her skill on new ones. That is why the three kinds of question are kept apart.

The same problem in machine learning

A supervised model learns in the same way as the student. It gets examples whose right answers, the labels, are known. It learns from them, and then it must give answers for new cases. For Larkfield, the examples are past tickets, and the new cases are the tickets that customers create tomorrow.

The labels also let us check a model. For each ticket where we know the answer, we compare the model’s prediction with the real answer and count the mistakes. But which tickets do we check on? If we check on the tickets that the model learned from, we repeat the lazy teacher’s mistake. A model that memorised those tickets gets a perfect score, and we learn nothing about tomorrow’s tickets.

This mistake has a name. A model overfits when it learns the details and the accidents of its training examples so well that it works much worse on new examples. It memorised instead of understanding. Overfitting is invisible on the training data: there, the overfitted model looks like the best one.

A tiny example: the memoriser and the rule

Here are 8 practice tickets with their answers. Three of them escalated: P-1, P-3 and P-6.

Ticket	Priority	Team	Words	Escalated
P-1	1	payment	120	1
P-2	3	delivery	40	0
P-3	2	warranty	210	1
P-4	1	returns	60	0
P-5	3	account	25	0
P-6	1	payment	95	1
P-7	2	delivery	70	0
P-8	3	returns	30	0

Two “models” learn from these tickets. The memoriser remembers the answer of every practice ticket. For any ticket, it looks for an exact copy: the same priority, team and number of words. If it finds one, it gives the remembered answer; if not, it says “no”. The rule learned one pattern instead: priority 1 tickets escalate more often, so it flags every priority 1 ticket.

On the practice tickets, the memoriser is right 8 times out of 8. The rule flags P-1, P-4 and P-6. It catches two escalations, gives one false alarm (P-4) and misses P-3, so it is right 6 times out of 8. Now four new tickets arrive. N-1 and N-3 escalated:

Ticket	Priority	Team	Words	Escalated	Memoriser	Rule
N-1	1	payment	110	1	no (no copy)	yes
N-2	3	delivery	45	0	no (no copy)	no
N-3	2	warranty	190	1	no (no copy)	no
N-4	2	returns	50	0	no (no copy)	no

No new ticket is an exact copy of a practice ticket, so the memoriser has nothing to remember. It says “no” four times and catches neither escalation. The rule catches N-1 and misses N-3. In the notebook, flag_report() counts this for you. It is the function of Define a baseline, which counts the escalations caught and missed, the false alarms and the tickets correctly left alone:

COLUMNS = ["priority", "team", "word_count"]
memory = {tuple(row[COLUMNS]): row["escalated"] for _, row in practice.iterrows()}

def memoriser(part):
    """The remembered answer for an exact copy of a practice ticket, else 0."""
    return [memory.get(tuple(row[COLUMNS]), 0) for _, row in part.iterrows()]

def rule(part):
    """Flag priority 1."""
    return (part["priority"] == 1).tolist()

tiny = pd.DataFrame.from_dict({
    "memoriser, practice": flag_report(practice["escalated"], memoriser(practice)),
    "rule, practice": flag_report(practice["escalated"], rule(practice)),
    "memoriser, new": flag_report(new["escalated"], memoriser(new)),
    "rule, new": flag_report(new["escalated"], rule(new)),
}, orient="index")
tiny[["flagged", "caught", "missed", "false alarms", "accuracy"]]

The output:

                     flagged  caught  missed  false alarms  accuracy
memoriser, practice        3       3       0             0      1.00
rule, practice             3       2       1             1      0.75
memoriser, new             0       0       2             0      0.50
rule, new                  1       1       1             0      0.75

On the practice tickets, the memoriser wins clearly. On the new tickets, it catches nothing, and the rule is better. The rule kept most of its skill because it learned a pattern. The memoriser learned tickets.

The real data: a model that memorises

Twelve tickets are easy to see through. Does the same happen with a real model and thousands of tickets? Priya tries it with a decision tree. A decision tree is a model that asks yes/no questions about a ticket’s features, one after another, until it reaches an answer. Is the priority 1? Is the team payment? Are there more than 150 words? Decision trees explains how a tree chooses its questions. Here, you need to know only one thing. With no limit, the tree keeps asking questions until every group of training tickets has a single answer. That is memorising, in a less obvious form.

In the notebook, the helper memorising_tree() makes this tree, with the preparation of the columns that it needs. Priya fits it on the 16,939 train tickets:

train = read("data/train.csv")
valid = read("data/valid.csv")
memoriser_tree = memorising_tree().fit(train[FEATURES], train[TARGET])
print("Groups of train tickets at the end of the questions:", memoriser_tree.named_steps["model"].get_n_leaves())

The output:

Groups of train tickets at the end of the questions: 2210

The tree split the train tickets into 2,210 small groups, each with one answer. Before you read on, predict: how many of the 1,983 train escalations will it catch? And how many of the 183 May escalations? Priya compares it with the rule “flag priority 1”, on the train tickets and on the validation set (May):

real = pd.DataFrame.from_dict({
    "memoriser on train": flag_report(train[TARGET], memoriser_tree.predict(train[FEATURES])),
    "memoriser on May": flag_report(valid[TARGET], memoriser_tree.predict(valid[FEATURES])),
    "rule on train": flag_report(train[TARGET], train["priority"] == 1),
    "rule on May": flag_report(valid[TARGET], valid["priority"] == 1),
}, orient="index")
real[["flagged", "caught", "missed", "precision", "recall"]]

The output:

                    flagged  caught  missed  precision  recall
memoriser on train     1983    1983       0      1.000   1.000
memoriser on May        177      46     137      0.260   0.251
rule on train          2385     693    1290      0.291   0.349
rule on May             163      61     122      0.374   0.333

Read the recall column first: the share of the escalations that were caught. On the tickets that it learned from, the tree is perfect. It flags exactly the 1,983 escalations and nothing else. On May, it catches 46 of 183, about one in four. That is fewer than the simple rule, which catches 61. The precision column, the share of flags that were right, tells the same story. About one tree flag in four is right, against more than one in three for the rule.

• Memorising tree
• Rule: flag priority 1

The share of escalations caught on the train tickets (the practice) and on May (new tickets). The memorising tree falls from 100% to 25.1%; the rule stays near one in three.

Text description of the chart

A column chart with four columns, on an axis from 0 to 100%. The memorising tree catches 100% of the escalations on the train tickets and 25.1% on the May tickets. The rule “flag priority 1” catches 34.9% on train and 33.3% on May. The tree has the highest column on train and the lowest on May. The rule’s two columns are almost the same height.

The rule changes little between train and May, because it learned a real pattern: urgent tickets escalate more. The tree learned the accidents of 16,939 old tickets, such as the exact number of words of one payment ticket at 14:00. Those accidents do not come back in May. This is overfitting, measured.

Omar, the product manager, asks: “The tree knows so much more than the rule. How can it be worse?” Priya answers: “It knows the old tickets, not what makes a ticket escalate. Most of what it remembered was chance.” Tomás has another idea: “Then let’s judge every model on the train tickets, and choose a model that does not memorise.” But you cannot see which model memorises if you look only at the tickets that it learned from. Every model looks better there than it really is. Only tickets that the model did not see show the difference.

Three sets, three jobs

So the data is divided, like the student’s exercises, into three parts with three different jobs:

• The training set is the practice exercises. The model learns from it, and so does every other step that learns from data. We never judge a model by its score on the training set.
• The validation set is the end-of-chapter exercises. The model did not learn from it. We use it to compare versions, to find weak points and to choose: which features, which model, which settings. We look at it many times.
• The test set is the final exam. Nobody looks at it while the work goes on. When every choice is made, we score the final model on it once, and that is the number we report to Grace.

Each set has one job. The loop between training and validation can run many times; the test set comes once, at the end.

Text description of the diagram

The diagram has three boxes from left to right (top to bottom on a phone). The first box says that the practice exercises are the training set: the model learns from it. Larkfield’s training set has 16,939 tickets from July 2024 to April 2026. An arrow labelled “check on new questions” leads to the second box: the end-of-chapter exercises are the validation set. You use it to compare versions, find weak points and choose, and you look at it many times. Larkfield’s validation set has 1,179 tickets from May 2026. A dashed arrow labelled “fix the weak points” goes back from the validation set to the training set. A last arrow, labelled “when every choice is made”, leads to the third box: the final exam is the test set. It gives one final score for everyone else, and you look at it once, at the end. Larkfield’s test set has 1,002 tickets from June 2026.

For Larkfield, the three sets are the split from the SQL course, which you already know. In code, they are the DataFrames train, valid and test, so the course often says “the train set” for short. The table puts the analogy and the numbers side by side:

Set	Like	Larkfield	Job	How often you look
Training set	Practice exercises	16,939 tickets, July 2024 to April 2026, 11.7% escalated	The model learns from it	Never to judge the model
Validation set	End-of-chapter exercises	1,179 tickets, May 2026, 15.5% escalated	Compare, find weak points, choose	Many times
Test set	The final exam	1,002 tickets, 1 to 27 June 2026, 15.7% escalated	One final, honest score	Once, at the end

In this course, you use the test set once, in Module 6, in Thresholds and calibration. A small script there refuses to run twice. Until then, you do not look at its score.

Why each look uses up the validation set a little

The validation set has a weakness too. Think of the student again. If she does the same chapter exercises again and again, she starts to remember them. After the fifth try, a good mark on them says less about fractions and more about those exact questions. The same happens with a model, in a quieter way. Each time you try a change and keep it because the validation score went up, part of that rise can be luck. Those 1,179 tickets happened to suit the change.

Tomás shows this without meaning to. He writes 100 rules, and each one flags about 1 ticket in 5, chosen only from the ticket’s ID. In the notebook, the helper id_rule() makes them. The rules know nothing about the tickets, so in the long run each one catches about 20% of the escalations. Tomás scores all 100 on May and keeps the best one:

may_recall = [flag_report(valid[TARGET], id_rule(number, valid))["recall"] for number in range(100)]
best = int(np.argmax(may_recall))
print("May, 100 rules: mean", round(np.mean(may_recall), 3), " lowest", min(may_recall), " highest", max(may_recall))
print("The best rule on May is rule", best)

The output:

May, 100 rules: mean 0.199  lowest 0.142  highest 0.251
The best rule on May is rule 7

“Rule 7 catches 25.1%,” says Tomás. “That is much better than chance.” Priya checks rule 7 on many more tickets that played no part in the choice: the 16,939 train tickets. That is fair here, because these rules learned nothing from train.

train_recall = [flag_report(train[TARGET], id_rule(number, train))["recall"] for number in range(100)]
print("Rule", best, "on train:", train_recall[best])
print("train, 100 rules: mean", round(np.mean(train_recall), 3), " lowest", min(train_recall), " highest", max(train_recall))

The output:

Rule 7 on train: 0.206
train, 100 rules: mean 0.2  lowest 0.177  highest 0.229

On train, rule 7 catches 20.6%, like every other rule. Its 25.1% on May was luck: more of the 183 May escalations happened to fall on its flags.

Each dot is one of the 100 rules, rounded to the nearest half percent. On May (183 escalations), luck spreads them from 14% to 25%. On train (1,983 escalations), the same rules stay close to 20%.

Text description of the chart

The chart has two rows of 100 dots each, on an axis from 10% to 30% of escalations caught. Dots with the same rounded value stack up. In the May row, the dots spread from 14.2% to 25.1%, with a mean of 19.9%. The dot at the right end, 25.1%, is rule 7. In the train row, the dots form a narrow, tall pile from 17.7% to 22.9%, with a mean of 20.0%. Rule 7 is near the middle of this pile, at 20.6%.

Two lessons follow from this chart. First, a choice made on the validation set picks some luck. The more versions you compare on it, the more the winner’s validation score is too good. So the validation score of your final model is a little optimistic, even when you did everything right. Second, a small set moves more by chance. With 183 escalations, rules that know nothing score anywhere from 14% to 25%. With 1,983 escalations, the same rules stay between 18% and 23%. Cross-validation, below, uses this second point.

Why the test set is used once

The first point is the reason for the test set. The validation set helped you choose, so its score is a little too good. You need one more set that played no part in any choice: the sealed exam paper. Its score is the honest estimate for the tickets that come tomorrow.

This works only if you look at the test set once, at the end. Suppose you look at the test score, do not like it, change something and look again. Now the test set helped you choose too. It has become a second validation set, with the same optimism, and you have no honest number left. The student who saw the exam paper the evening before is in the same position. Her mark is high, and it no longer says what she can do.

So when the test score is lower than the validation score, you report it as it is. A little lower is normal and expected. If it is much lower, you write that down and plan new work with new data.

How to split: at random or by time

Which rows go into which set? There are two common answers, and Choose a split in the SQL course compared them in detail.

A random split puts each row into a set by chance. It is like shuffling a pile of exercise cards and taking some of them for the exam. It is fair when the rows are independent and their order does not matter. Photos of flowers for a model that names the flower are a good example. A photo taken in May tells the model nothing special about a photo taken in June.

A time split puts the oldest rows in the training set and the newest rows in the validation and test sets. Use it when the model will predict the future from the past, which is how almost every model in a business is used. Then the test copies real use: the model learns from what happened before a date, and it is judged on what happened after that date.

Larkfield’s model will score tomorrow’s tickets with a model trained on the past, so Larkfield splits by time. The chart shows why this matters here:

• Training set
• Validation set (May 2026)
• Test set (June 2026)

The escalation rate of each month. In the 22 train months it stays between 9.5% and 13.9%. In May and June 2026, after the new warranty policy, it is above every train month.

Text description of the chart

The chart shows the share of tickets that escalated in each month from July 2024 to June 2026. The 22 months of the training set go up and down between 9.5% (September 2025) and 13.9% (August 2024). April 2026, the last train month, has 9.8%. Then the validation month, May 2026, has 15.5%, and the test month, June 2026, has 15.7%. Both are higher than any train month. The new warranty policy started on 1 May 2026.

On 1 May 2026, Larkfield started a new warranty policy, and more tickets escalate since then. A time split keeps this visible: the validation set is the model’s first meeting with the new conditions, exactly as in real use. A random split hides it, as the failure case below shows. A third kind, the group split, keeps all the tickets of one customer in one set. Use it when the model must work for customers that it never saw; the SQL course shows it.

Cross-validation

Back to the student. Suppose there is only one sheet of chapter exercises. Her mark on it depends partly on which questions happened to be on that sheet. A sheet full of her favourite kind of question gives a mark that is too good. A careful teacher can do better with the practice book itself. He divides it into five parts. The student practises on four parts and is tested on the fifth, which she has not seen. They repeat this five times, with a different part as the test each time, and average the five marks. Every question is used for testing once, and never by a student who practised on it.

Larkfield has the same problem with one sheet. May has only 183 escalations, so one score on it moves a lot by chance, as Tomás’s 100 rules showed. And each look at May uses it up a little. Cross-validation solves both problems with the teacher’s method, and it uses the large training set for the checks:

1. Cut the train tickets into 5 equal parts, called folds.
2. Fit the model on 4 folds, and score it on the fifth fold, which it did not see.
3. Repeat this 5 times, so that each fold plays the “chapter exercises” once.
4. Report the mean of the 5 scores and their standard deviation: how much the score moves from fold to fold.

This is 5-fold cross-validation, often written CV. The name describes the method. Each row helps the model learn in some rounds and checks it in one round, so learning and checking “cross over”.

5-fold cross-validation. In each round, the model learns from the four light folds and is scored on the dark fold. The five scores give a mean and a standard deviation.

Text description of the diagram

The picture has five rows, numbered 1 to 5. Each row is one round, and shows the train set as a bar of five equal blocks, the folds. In round 1, the first block is dark and the other four are light. In round 2, the second block is dark, and so on, until round 5, where the last block is dark. The model learns from the light blocks and is scored on the dark block. Each row ends with an arrow to a small circle: the score of that round. A bracket on the right joins the five circles into one large dot: the mean of the five scores.

Cut the folds with care

Each fold must be a fair small copy of the train set. Here are 20 rows, with the 5 escalations at the top of the file, as in a file sorted by the target. Plain KFold(5) cuts the rows in file order; StratifiedKFold(5) does not:

from sklearn.model_selection import KFold, StratifiedKFold

y_20 = np.array([1] * 5 + [0] * 15)
x_20 = np.arange(20).reshape(-1, 1)
for name, folds in [("KFold", KFold(5)), ("StratifiedKFold", StratifiedKFold(5))]:
    print(name, "escalated per fold:", [int(y_20[rows].sum()) for _, rows in folds.split(x_20, y_20)])

The output:

KFold escalated per fold: [4, 1, 0, 0, 0]
StratifiedKFold escalated per fold: [1, 1, 1, 1, 1]

With plain KFold, three folds have no escalation, so there is nothing to catch in them, and the first fold has four. A stratified split keeps the share of escalations the same in every fold. For a yes/no target, use StratifiedKFold. The course always uses StratifiedKFold(5, shuffle=True, random_state=0): stratified, with the rows shuffled first, and with a fixed random_state so that you get the same folds on every run.

Cross-validation on Larkfield’s train set

Priya runs 5-fold cross-validation on the memorising tree. cross_validate does the four steps above. With return_train_score=True, it also scores each round’s model on the four folds that it learned from:

from sklearn.model_selection import cross_validate

cv = StratifiedKFold(5, shuffle=True, random_state=0)
result = cross_validate(memorising_tree(), train[FEATURES], train[TARGET], cv=cv,
                        scoring="recall", return_train_score=True)
print("caught on the 4 folds it learned from:", result["train_score"].round(3))
print("caught on the fold it did not see:    ", result["test_score"].round(3))
print(f"mean {result['test_score'].mean():.3f}, standard deviation {result['test_score'].std():.3f}")

The output:

caught on the 4 folds it learned from: [1. 1. 1. 1. 1.]
caught on the fold it did not see:     [0.247 0.204 0.247 0.229 0.21 ]
mean 0.227, standard deviation 0.018

In every round, the tree catches all the escalations of the folds that it learned from. On the fold that it did not see, it catches about one in five. Cross-validation found the memorising problem without using May at all. Each fold has 3,388 tickets with 396 or 397 escalations, so these scores are steadier than one score on May. Later lessons often use another score, average precision, which The precision–recall trade-off explains. By that score, the same tree has 1.000 on train and 0.137 ± 0.004 in cross-validation: the same gap.

Note one confusing name. scikit-learn calls the score on the held-out fold test_score, but that fold is part of our training set. The real test set, June, is not used anywhere in cross-validation.

Now the standard deviation. Priya scores the rule “flag priority 1” on the same five folds. The rule learns nothing, so any change from fold to fold can only be chance:

rule_recall = [flag_report(train[TARGET].iloc[rows], train["priority"].iloc[rows] == 1)["recall"]
               for _, rows in cv.split(train, train[TARGET])]
print("rule, caught per fold:", rule_recall)
print(f"mean {np.mean(rule_recall):.3f}, standard deviation {np.std(rule_recall):.3f}")

The output:

rule, caught per fold: [0.371, 0.315, 0.343, 0.343, 0.376]
mean 0.350, standard deviation 0.022

The same rule catches from 31.5% to 37.6% of the escalations, only because each fold holds different tickets. So when two models differ by less than this kind of spread, the difference can be chance. Write a cross-validation result with its spread, for example “0.350 ± 0.022”. Tune hyperparameters turns this into a rule for choosing between models.

Cross-validation and the validation month

If cross-validation is so good, why keep May at all? Because all 22 train months come from before the new warranty policy. Cross-validation mixes them, so every fold has the old conditions, with monthly rates from 9.5% to 13.9%. It cannot show how a model works after the change. May can. So Larkfield uses both, each for its own job:

• Cross-validation on train compares models and settings. It has 1,983 escalations, so its averages are steady, and it does not use up May.
• The validation month checks the chosen model in the new conditions. Module 6 also sets the threshold on it.
• The test month gives the one final score.

Cross-validation has costs too. It fits the model 5 times instead of once, which matters for slow models. And on time-ordered data, it mixes older and newer rows inside the train set. That is acceptable here only because the validation month checks the newest conditions separately.

Keep the sets apart

Three sets help only if nothing passes between them. Leakage is information that reaches the model from where it should not: from the validation or test rows, or from the future. You met it in the SQL course. It is the student who saw the exam paper the evening before. Two forms matter most:

• A step that learns from all the rows. Suppose you compute the median for missing values, or the mean for scaling, on train, validation and test together. Then a little of the validation and test data goes into training. Every step that learns from data must use only the train rows, as in Fit transformations correctly in the SQL course. Module 4 builds a pipeline that does this for you.
• A column from the future. A column such as priority_now, which the escalation itself writes, contains the answer. Features and target timing in the SQL course and Translate a need showed it.

Leakage, like overfitting, raises no error. It makes the validation and test scores too good, and the model disappoints in real use.

Guided practice: overfitting and folds in the notebook

1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
2. Run the setup cells and the helper cell. The data cell ends with the Ready: line.
3. In section 2, predict which model is better on the practice tickets and which on the new tickets. Then run the cell. You see accuracy 1.00 for the memoriser on the practice tickets and 0.50 on the new ones.
4. In section 3, run the cell that fits the tree. You see 2210 groups.
5. Write your prediction for the tree’s recall on train and on May. Then run the next cell. You see 1983 caught on train and 46 on May.
6. In the Your turn cell, compute gap: the memoriser’s recall on train minus its recall on May.
7. Run the check cell. It prints Check passed.
8. In section 4, run the 100 rules. You see rule 7, with 0.251 on May and 0.206 on train.
9. Run the chart cell. The May row of dots is wider than the train row.
10. In section 6, run the 20-row cell, then the cross-validation of the tree. You see a mean of 0.227 and a standard deviation of 0.018.
11. Run the rule’s cross-validation. You see a mean of 0.350 and a standard deviation of 0.022.

Show the answer to step 6

gap = round(real.loc["memoriser on train", "recall"] - real.loc["memoriser on May", "recall"], 3)
print(gap)

0.749

The tree loses three quarters of the escalations (1.000 − 0.251 = 0.749) when it meets tickets that it did not learn from. For the rule, the same gap is 0.349 − 0.333 = 0.016.

Your turn: 10 folds

Change one thing: the number of folds. With 10 folds, the model learns from 9 parts in each round, and you score it on the tenth. So each fold has half as many tickets as before. Before you run it, predict: will the mean of the rule’s recall change? Will the standard deviation get smaller or larger? In the notebook, section 7, make cv_10 = StratifiedKFold(10, shuffle=True, random_state=0). Compute the rule’s recall on each fold, and put the mean and the standard deviation in mean_and_std.

Show the answer

cv_10 = StratifiedKFold(10, shuffle=True, random_state=0)
rule_recall_10 = [flag_report(train[TARGET].iloc[rows], train["priority"].iloc[rows] == 1)["recall"]
                  for _, rows in cv_10.split(train, train[TARGET])]
mean_and_std = [round(float(np.mean(rule_recall_10)), 3), round(float(np.std(rule_recall_10)), 3)]
print(mean_and_std)

[0.35, 0.037]

The mean stays at 0.350, because it is the same rule on the same tickets. The standard deviation grows from 0.022 to 0.037. Each fold now has only about 198 escalations, and a smaller set moves more by chance. With more folds, each model learns from more data (90% of train instead of 80%). But each fold’s score is noisier, and there are more fits to run. Five and ten folds are both common; this course uses five.

Common mistake

A random split hides the change

Symptom: Tomás says: “A random split is the standard. Let’s shuffle all 19,120 tickets and cut them into the same three sizes.” In the notebook, every is train, validation and test together. He gets this:

shuffled = every.sample(frac=1, random_state=0)
random_parts = {"test": shuffled.iloc[:1002], "valid": shuffled.iloc[1002:2181], "train": shuffled.iloc[2181:]}
for name, part in random_parts.items():
    recent = (part["created_at"] >= "2026-05-01").sum()
    print(f"random {name}: {len(part)} tickets, rate {part[TARGET].mean():.3f}, from May or June 2026: {recent}")

random test: 1002 tickets, rate 0.131, from May or June 2026: 112
random valid: 1179 tickets, rate 0.105, from May or June 2026: 122
random train: 16939 tickets, rate 0.122, from May or June 2026: 1947

The three sets look normal, and nothing raises an error.

Cause: Larkfield’s tickets have a time order that matters. The random test set mixes all 24 months, so its rate is 13.1%, while the real future, June, has 15.7%. Worse, the random training set contains 1,947 tickets from May and June. The model would learn from the future, and then you would test it partly on the past. A test like this answers the wrong question. It asks “how well does the model work on a mix of old tickets?”, not “how well will it work tomorrow?”

Fix: split by time, as Mei did in the SQL course: train before 1 May 2026, validation in May, test in June. Use a random split only when the rows are independent and their order does not matter.

Consequence in AI work: a random split gives a score that is too optimistic for a model that predicts the future. The model has seen later data, and the test never shows a change in the world, such as Larkfield’s new policy. The team trusts the number, and the model works worse in real use from the first day.

See it end to endForecast bike-share demandCase study

The bike-share case study shows how a random split flatters a forecast, and why a time split is honest.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study splits by patient, so that no patient is in two sets.

Knowledge check

Check your understanding

Map the student's three kinds of exercise to the three sets. For each set, say what it is used for and how often you look at its score.

Show the answer

• Practice exercises = the training set. The model, and every step that learns from data, learns from it. You never judge the model by its score on it.
• End-of-chapter exercises = the validation set. The model did not learn from it. You use it to compare versions, find weak points and choose. You look at it many times, and each look uses it up a little.
• The final exam = the test set. It plays no part in any choice. You score the final model on it once, at the end, and report that number.

Check your understanding

Predict and explain: a new model catches 96% of the escalations in the train set and 22% in May. What is happening? How could you have seen it without May?

Show the answer

The model overfits: it memorised the train tickets instead of learning a pattern that works on new tickets. The memorising tree of this lesson behaves in the same way (100% on train, 25.1% in May). The 96% on train says almost nothing about new tickets. Without May, cross-validation on the train set shows the problem. In each round, it scores the model on a fold that the model did not learn from. For the memorising tree, that gave about 23% on the held-out folds, against 100% on the folds that it learned from.

Check your understanding

Tomás tries 30 versions of a model and keeps the one with the best May score. Then he wants to try a few more versions and keep the best one on the June test set. What do you tell him about each step?

Show the answer

Choosing on May is the job of the validation set, but the winner’s May score is a little too good. Part of its lead is luck, as the 100 rules showed (25.1% on May, 20.6% on other tickets). That is why the test set exists. Choosing on June breaks it: June then becomes a second validation set, and its score is too good in the same way. Nobody would have an honest number for new tickets. Make every choice with cross-validation on train and with May. Then score the final model on June once, and report that score as it is.

Recap

• A model must be judged on data it did not learn from. A memorising decision tree catches all 1,983 train escalations, and only 46 of 183 in May: fewer than the rule “flag priority 1” (61). This is overfitting, and a score on the training data cannot show it.
• Three sets, three jobs. The training set is the practice exercises: the model learns from it. The validation set is the end-of-chapter exercises: compare and choose, with many looks. The test set is the final exam: one final score, with one look.
• Each choice on the validation set picks some luck. The best of 100 rules that know nothing caught 25.1% in May and 20.6% on other tickets. The test set gives the honest number, but only if nobody chooses with it.
• Larkfield splits by time, because the model predicts the future. A random split mixes the months and hides the change of 1 May 2026.
• Cross-validation scores a model 5 times on the train set, each time on a fold that it did not see. Report the mean with its standard deviation: the rule gives 0.350 ± 0.022.

Next, you measure how good a numeric prediction is, with the errors of a forecast of tickets per day.

Next: Regression metrics