Choose a split

What you will be able to do

Choose which tickets the model learns from and which tickets test it, so that the test looks like the model’s real work. A wrong split gives a score that you cannot trust, and it raises no error.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every script, task and check of this lesson, with the instructions. Its setup cells install pandas, download the practice data and rebuild the modelling table of Lesson 1, so it works without the local project. Read the explanations and use the explorer here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The modelling table, its target escalated_72h and why the last 72 hours are left out, from Features and target timing.
• groupby and agg in pandas, from Transform with Python.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ~/projects/ticket-data, from Set up a practice database: Build the database.
• The .venv environment with pandas 3.0.6, active in a terminal in ~/projects/ticket-data. You made it in Profile before changing: The same profile in pandas. If python -c "import pandas" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• The modelling table data/model/tickets_model.csv (1,369 rows, 188 escalated). You made it with python scripts/build_features.py in Features and target timing: Worked example: build_features.py, and checked it in Guided practice: build and check the table. It needs the clean tables data/clean/ from Handle messy values: Guided practice: finish clean.py.
• The table can be missing, or have other numbers, for example from the 48-hour version in Your turn: a 48-hour target. Then download the course’s script from C04/pipeline: curl -fsSL -o scripts/build_features.py https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C04/pipeline/scripts/build_features.py (curl.exe on Windows). Then run python scripts/build_features.py. If data/clean/ is missing too, follow the recovery steps in the “Before you start” box of Features and target timing first.
• This lesson adds scripts/split.py, train.csv, valid.csv, test.csv and split_manifest.json in data/model/.

Tested on macOS with Python 3.14.6 and pandas 3.0.6. The notebook was tested in Jupyter.

Why split at all

A model is useful only on tickets it has not seen. So you keep some tickets away from it, and test it on them later. A split divides the rows of the modelling table into parts with different jobs:

Part	Job	Rule
Train set	The model learns from these rows.	Everything that learns from data (the model, and in Lesson 3 the preprocessing) uses only these rows.
Validation set	You compare versions and choose settings with these rows.	You look at its score many times.
Test set	One final, honest score.	You look at its score once, at the end. If you change the model after you look, the test set becomes a second validation set.

The question of this lesson is which rows go into each part. There are three common answers.

Three ways to split, on eight tickets

Here are eight tickets from three customers, in order of time. Each split puts six tickets in train and two in test (there is no validation set in this small example):

Ticket	Customer	Created	Random	By time	By customer
T-1	A	10 Jan	train	train	train
T-2	B	2 Feb	test	train	test
T-3	A	20 Feb	train	train	train
T-4	C	15 Mar	train	train	train
T-5	B	5 Apr	train	train	test
T-6	A	30 Apr	test	train	train
T-7	C	20 May	train	test	train
T-8	A	12 Jun	train	test	train

• Random split: each row goes to train or test by chance. T-7 and T-8, the newest tickets, are in train. The model learns from May and June, then is tested on February and April.
• Split by time: train is everything before a date, test is everything after it. The model learns from the past and is tested on the future, as in real use.
• Split by customer (a group split): all tickets of one customer go to the same part. Customer B is only in test, so the test asks: how well does the model work for a customer it never saw?

None of these is always right. The correct split copies the way the model is used. Ask three questions:

1. Will the model predict the future from the past? Then split by time. Almost every model in a business does this.
2. Will the model meet new customers, patients, shops or machines? Then keep each group in one part.
3. Are the rows independent, with no time order and no repeated groups? Only then is a random split a fair test.

Try the three splits

The explorer has 60 made-up tickets from six customers, January to June. The escalation rate is 12.5% before 1 May and 20% after it, like the real data. Customer C-06 is new: their first ticket is in May.

Explore

Random, by time or by customer

Tickets in train / valid / test

42 / 9 / 9

Escalated in train

14.3%6 of 42 tickets

Escalated in test

11.1%1 of 9 tickets

Test tickets whose customer is also in train

9 of 9

Train tickets from June (the newest month)

6 of 42

Random split, draw 1: every part has tickets from many months. Train has 6 tickets from June, and test has 4 tickets from January to April. So the model learns from tickets that are newer than the tickets it is tested on. Press "Draw again" to see how much the rates change.

• Train
• Validation
• Test
• Ring: escalated ticket

Illustration: 60 made-up tickets, not the course data. The real split is in the text below the explorer.

Each dot is one ticket, in the lane of its customer. Colour and shape show the part: circle for train, square for validation, triangle for test. A ring marks an escalated ticket.

Text description of the explorer

The chart has six lanes, one for each customer from C-01 to C-06, and a time axis from January to June. Dashed lines mark 1 May and 1 June. There are 60 tickets. 40 are from before May, and 5 of them escalated (12.5%). May and June have 10 tickets each, with 2 escalated in each month (20%). C-06 has tickets only in May and June.

Random split, first draw: 42 train, 9 validation and 9 test tickets, spread over all months. Train has 6 escalated of 42 (14.3%), test 1 of 9 (11.1%). All 9 test tickets have a customer that is also in train. Train has 6 tickets from June. “Draw again” gives a new random split, and the test rate jumps, for example to 0% of 9 in the second draw.

Split by time: train is the 40 tickets before 1 May, validation the 10 May tickets, test the 10 June tickets. Train has 5 escalated (12.5%), test 2 (20%), so the rise after 1 May is visible. 7 of the 10 test tickets have a customer in train; the other 3 belong to the new customer C-06. Train has no ticket from June. “Draw again” is off, because a time split has no random part.

Split by customer, first draw: all 13 tickets of C-01 are in test. The 11 tickets of one other customer are in validation, and the other 36 tickets are in train. No test ticket has a customer in train. Train still has 6 tickets from June. Each new draw puts a different customer in test. The test rate changes a lot (38.5%, 20%, 0%), because one customer is a small sample.

What the explorer shows:

• A random split mixes the months. Every part has tickets from January to June. The model learns from June and is tested on February. In real use, the model never has next month’s tickets.
• Only the time split shows the change after 1 May. A random split puts tickets from before and after the change into both parts, so train and test have about the same rate.
• A random split is one draw. Press “Draw again” a few times. With 9 test tickets, the test rate moves from 0% to more than 40%. A real score moves too, only less.
• The group split tests new customers, but it mixes the months, like the random split.

The real data: something changed in May

Mei counts the tickets and escalations per month:

months.py

import pandas as pd

model = pd.read_csv("data/model/tickets_model.csv", parse_dates=["created_at"])
month = model["created_at"].dt.strftime("%Y-%m")
print(model.groupby(month)["escalated_72h"].agg(["size", "sum", "mean"]).round(3))

Output

            size  sum   mean
created_at
2026-01      210   23  0.110
2026-02      225   25  0.111
2026-03      249   32  0.129
2026-04      243   28  0.115
2026-05      227   45  0.198
2026-06      215   35  0.163

From January to April, about 11% to 13% of tickets escalate. In May, it is 19.8%. Omar, the product manager, asks what happened. Grace knows: “On 1 May, a new warranty policy started. Warranty questions now go to a senior agent more often.” This is the kind of change that a model meets in real work. The world after training is not the world of the training data.

Now look at a random 20% test set, made with pandas only:

random_split.py

import pandas as pd

model = pd.read_csv("data/model/tickets_model.csv", parse_dates=["created_at"])
test = model.sample(frac=0.2, random_state=0)
train = model.drop(test.index)
print(len(train), len(test))
print(test["created_at"].dt.strftime("%Y-%m").value_counts().sort_index())
print("train rate", round(train["escalated_72h"].mean(), 3), "test rate", round(test["escalated_72h"].mean(), 3))
print("train tickets from June:", (train["created_at"] >= "2026-06-01").sum())

Output

1095 274
created_at
2026-01    42
2026-02    44
2026-03    56
2026-04    40
2026-05    46
2026-06    46
Name: count, dtype: int64
train rate 0.14 test rate 0.128
train tickets from June: 169

The test set has tickets from every month, and train has 169 June tickets. The two rates, 14.0% and 12.8%, are close. Nothing in this split tells you that the escalation rate changed in May.

Worked example: split.py

Larkfield will train the model on past tickets and use it on new ones. So Mei splits by time:

• train: created before 2026-05-01 (January to April),
• valid: May,
• test: 1 to 27 June. June 28 to 30 is not in the table, because those targets are not final yet (Lesson 1).

The policy change falls between train and validation on purpose. The validation score then tells the truth: how well does a model from the old months work in the new ones?

scripts/split.py

"""Split the modelling table by time: train on the past, test on the future."""

import json
from pathlib import Path

import pandas as pd

IN = Path("data/model/tickets_model.csv")
OUT = Path("data/model")
VALID_FROM = pd.Timestamp("2026-05-01")
TEST_FROM = pd.Timestamp("2026-06-01")


def main():
    df = pd.read_csv(IN, parse_dates=["created_at"])
    df["split"] = "train"
    df.loc[df["created_at"] >= VALID_FROM, "split"] = "valid"
    df.loc[df["created_at"] >= TEST_FROM, "split"] = "test"
    manifest = {"rule": f"train < {VALID_FROM.date()} <= valid < {TEST_FROM.date()} <= test", "parts": {}}
    for name in ["train", "valid", "test"]:
        part = df[df["split"] == name].drop(columns="split")
        part.to_csv(OUT / f"{name}.csv", index=False, date_format="%Y-%m-%d %H:%M:%S")
        manifest["parts"][name] = {
            "rows": len(part), "escalated": int(part["escalated_72h"].sum()),
            "escalated_rate": round(part["escalated_72h"].mean(), 3),
            "first_created": str(part["created_at"].min()), "last_created": str(part["created_at"].max()),
        }
    train_customers = set(df.loc[df["split"] == "train", "customer_id"].dropna())
    test_customers = set(df.loc[df["split"] == "test", "customer_id"].dropna())
    manifest["test_customers_also_in_train"] = len(test_customers & train_customers)
    manifest["test_customers"] = len(test_customers)
    (OUT / "split_manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(manifest, indent=2))


if __name__ == "__main__":
    main()

Lines	Why
VALID_FROM, TEST_FROM	The two dates are the whole rule. They are constants at the top, so a reader finds them at once.
df.loc[... >= VALID_FROM, "split"] = "valid", then the same for test	Every row starts as train. Rows from May on become valid, and rows from June on become test. Each row is in exactly one part.
part.to_csv(OUT / f"{name}.csv", ...)	Three files: train.csv, valid.csv, test.csv. Nobody needs to repeat the split later, so nobody can repeat it wrongly.
The manifest dictionary	A split manifest: a small file that records the rule, and the rows, escalations and dates of each part. Priya can check the split without running code, and Module 6 checks it automatically.
test_customers_also_in_train	Counts how many test customers the model already met in training. It tells you what kind of test this is.

Mei runs it:

Terminal

python scripts/split.py

Output

{
  "rule": "train < 2026-05-01 <= valid < 2026-06-01 <= test",
  "parts": {
    "train": {
      "rows": 927,
      "escalated": 108,
      "escalated_rate": 0.117,
      "first_created": "2026-01-05 07:30:24",
      "last_created": "2026-04-30 23:54:47"
    },
    "valid": {
      "rows": 227,
      "escalated": 45,
      "escalated_rate": 0.198,
      "first_created": "2026-05-01 11:14:19",
      "last_created": "2026-05-31 20:57:12"
    },
    "test": {
      "rows": 215,
      "escalated": 35,
      "escalated_rate": 0.163,
      "first_created": "2026-06-01 08:42:16",
      "last_created": "2026-06-27 20:14:04"
    }
  },
  "test_customers_also_in_train": 77,
  "test_customers": 89
}

Count the rows: 927 + 227 + 215 = 1,369, the whole table. The escalations: 108 + 45 + 35 = 188. The rates show the change: 11.7% in train, 19.8% in validation, 16.3% in test. The dates do not overlap: train ends on 30 April, and validation starts on 1 May.

When to split by customer

77 of the 89 test customers also have tickets in train. 12 test customers are new, with 13 test tickets (one more test ticket is from a guest). So this test mostly measures how the model works for customers it already knows. For Larkfield, that is fine: most tickets come from customers who wrote before.

A group split is needed when the model’s real work is mostly about new groups. For example, Larkfield could sell the model to other shops, whose customers it never saw. A hospital model must work for new patients, and a fault model for new machines. Then keep every customer in one part. You can do both: split by time, and also leave some customers out of train completely.

Be honest about what the split changes on this data. Priya trained the same simple model with three kinds of split; you see this model in Lesson 3. Its score, ROC AUC, goes from 0.5 (guessing) to 1.0 (a perfect ranking of escalated tickets above the others):

Split	Score (ROC AUC)
Time: train on January to April, score on May	0.673
Random 80/20, 20 different random draws	mean 0.730, lowest 0.665, highest 0.827
By customer 80/20, 20 different random draws	mean 0.744, lowest 0.663, highest 0.823

Three things to see:

• The random and the customer splits are close on this data. The features do not identify a customer, so the model gains little from knowing a customer. That is a result for these features, not a rule. A feature such as a customer’s own escalation history can make a random split much too optimistic.
• The time split gives the lowest score. The model learns from the months before the policy change and is tested after it. That is the real situation, so 0.673 is the honest number.
• One random split is one draw. The same model and data give anything from 0.665 to 0.827. A report that says “0.827” from one lucky draw says more about the draw than about the model.

Guided practice: make and check the split

1. Make the file scripts/split.py with the code above, and save it.
2. Run python scripts/split.py. You should see the manifest above: 927, 227 and 215 rows.
3. Check that three files and the manifest exist. On macOS, run ls data/model. On Windows, run Get-ChildItem data/model. You should see build_summary.json, split_manifest.json, test.csv, tickets_model.csv, train.csv and valid.csv.
4. Make months.py with the code from “The real data”, and run it. You should see 0.198 for May.
5. Open split_manifest.json. Check that the rows add up to 1,369, and that the last train date is before the first validation date.

Your turn: move the dates back one month

Change one thing. In scripts/split.py, set VALID_FROM to 2026-04-01 and TEST_FROM to 2026-05-01. Before you run it, predict: where is the policy change now, and what does the validation rate tell you? Then run the script and read the manifest. At the end, set the dates back to 2026-05-01 and 2026-06-01, and run the script again, so the files are correct for Lesson 3.

Show the answer

Output (part)

"rule": "train < 2026-04-01 <= valid < 2026-05-01 <= test",
train: 684 rows, 80 escalated, 0.117
valid: 243 rows, 28 escalated, 0.115
test:  442 rows, 80 escalated, 0.181
"test_customers_also_in_train": 99,
"test_customers": 128

Now the policy change is between validation and test. The validation rate (11.5%) looks like the train rate (11.7%), so every check on validation says “all is stable”. The change appears only in the test set (18.1%), after all the decisions are made. Put the most recent, most realistic period in validation and test. If the world changed at a known date, make sure validation sees the change, so you can react to it before the final test.

Common mistake

The score that fell after launch

Symptom: Tomás splits the 1,369 rows at random, trains a model, and gets ROC AUC 0.827. Larkfield starts to use the model in July. A month later, Priya checks it on the July tickets, and it ranks escalations much worse than 0.827 promised.

Cause: two problems in one number. First, the random split let the model learn from May and June, and tested it on tickets from the same months. In July, no future tickets are in training, and the world may have changed again. Second, 0.827 was the highest of 20 random draws. The same model scored 0.665 on another draw.

Fix: split by time, as split.py does, so the validation score copies real use: 0.673 on May. If you also use random splits, report the spread across draws, not one number. Write the split rule and the counts in split_manifest.json, so a reader can see which test the score comes from.

Consequence in AI work: a score from the wrong split sets the wrong expectation for everyone. Omar plans staff with it, and the team stops looking for problems. The model does not get worse after launch; it was never as good as the number said.

See it end to endPlan overnight bike rebalancingCase study

The bike case study chooses its rule on weeks 2 and 3 of a month and tests it once on week 4.

Knowledge check

Check your understanding

Larkfield wants a model that predicts escalations for new tickets. Which split fits, and why is a random split not a fair test?

Show the answer

A split by time: train on older tickets, validate and test on newer ones. That is how the model is used: it learns from the past and predicts the future. A random split puts newer tickets in train and older tickets in test, so the model learns from information it will never have. It also hides changes over time, such as the rise in escalations after 1 May.

Check your understanding

Predict: you run the random 20% split with random_state=0, 1, 2, 3 and 4. Will the test escalation rate be the same each time? Will any of them show the change in May?

Show the answer

No, it changes with each draw, because different tickets land in test. On the real data, the test rates are 0.128, 0.142, 0.128, 0.113 and 0.139. None of them shows the change in May. Each random test set mixes all six months. Only a split by time separates the months before and after 1 May.

Check your understanding

Your split manifest says test_customers_also_in_train: 77 and test_customers: 89. What does this tell Priya? When would she need a different split?

Show the answer

Most test customers (77 of 89) already have tickets in train. So the test mostly measures how well the model works for customers it knows. That matches Larkfield’s normal work. She needs a split by customer if the model must work mainly for new customers. An example is a model for another shop’s customers.

Recap

• The train set is for learning, the validation set for choosing, and the test set for one final score.
• Choose the split that copies real use. Split by time when the model predicts the future. Split by group when it must work for new customers. Use a random split only for independent rows.
• On the real data, the escalation rate rises from 11.7% (train) to 19.8% (May) and 16.3% (June). A time split shows the change; a random split hides it.
• One random split is one draw: the same model scored from 0.665 to 0.827.
• split.py writes train.csv, valid.csv, test.csv and split_manifest.json.

Next, you prepare the features for a model, and learn where preprocessing can leak.

Next: Fit transformations correctly