Fit transformations correctly

What you will be able to do

Prepare the features for a model so that nothing from the validation or test set leaks into training. Then measure what leakage does to a score, so you can recognise a score that is too good to be true.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every script, task and check of this lesson, with the instructions. It also runs every leakage number on this page. Its setup cells install pandas and scikit-learn, download the practice data and rebuild the split of Lesson 2. So it works without the local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The train, validation and test sets and the time split, from Choose a split.
• Which columns are leaky and why, from Features and target timing.
• What a missing value means in each column, from Handle messy values. order_value is missing for account tickets (no order) and for the “unknown” rows; customer_tenure_days is missing for guests and deleted customers.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ~/projects/ticket-data, from Set up a practice database: Build the database.
• The .venv environment with pandas 3.0.6, active in a terminal in ~/projects/ticket-data. You made it in Profile before changing: The same profile in pandas. If python -c "import pandas" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/model/train.csv (927 rows), valid.csv (227) and test.csv (215). You made them with python scripts/split.py in Choose a split: Worked example: split.py. If you changed the dates in Your turn: move the dates back one month, set them back to 2026-05-01 and 2026-06-01 and run the script again.
• data/clean/tickets.csv, from Handle messy values: Guided practice: finish clean.py. This lesson takes the leaky columns from it for one experiment.
• If the split files are missing or have other numbers, download the course’s script from C04/pipeline: curl -fsSL -o scripts/split.py https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C04/pipeline/scripts/split.py (curl.exe on Windows). Then run python scripts/split.py. If tickets_model.csv is missing too, follow the recovery steps in the “Before you start” box of Choose a split first.
• scikit-learn 1.9.1, which you install in Install scikit-learn below. It needs an internet connection and no account.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1 (numpy 2.5.3). The Windows commands are not tested. The notebook was tested in Jupyter.

Install scikit-learn

scikit-learn is the most common Python library for classic machine learning. Here you use its preprocessing tools and one simple model. Install the version the course uses, and add it to requirements.txt:

Terminal

python -m pip install scikit-learn==1.9.1

Then open requirements.txt and add one line, so that it has both packages:

requirements.txt

pandas==3.0.6
scikit-learn==1.9.1

Check the installation:

Terminal

python -c "import sklearn; print(sklearn.__version__)"

Output

1.9.1

If you see ModuleNotFoundError: No module named 'sklearn', the environment is not active. Activate .venv and install again.

Fit, then transform

A model needs numbers with no gaps. So the features go through preprocessing first: steps that fill missing values, put numbers on a similar scale and turn categories into numbers. Each step has two parts:

• fit learns numbers from data. A median imputer learns the median of each column. (Imputation means filling a missing value with a substitute; you chose it in Module 4.) A scaler learns the mean and the standard deviation (SD).
• transform uses the learned numbers to change data. The imputer fills each gap with the learned median. The scaler computes (value − mean) ÷ SD.

Here is a tiny example: five train values of order_value, one of them missing, and two test values.

tiny_fit.py

import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

train = pd.DataFrame({"order_value": [20.0, 40.0, None, 60.0, 100.0]})
test = pd.DataFrame({"order_value": [None, 400.0]})

imputer = SimpleImputer(strategy="median")
imputer.fit(train)
print("median learned from train:", imputer.statistics_)

scaler = StandardScaler()
scaler.fit(imputer.transform(train))
print("mean and SD learned from train:", scaler.mean_.round(2), scaler.scale_.round(2))

print("test after transform:", scaler.transform(imputer.transform(test)).round(2).ravel())

Output

median learned from train: [50.]
mean and SD learned from train: [54.] [26.53]
test after transform: [-0.15 13.04]

Check it by hand. The known train values are 20, 40, 60 and 100, so the median is (40 + 60) ÷ 2 = 50. With the gap filled, the mean is (20 + 40 + 50 + 60 + 100) ÷ 5 = 54. The missing test value becomes 50, and (50 − 54) ÷ 26.53 = −0.15. The value 400 becomes (400 − 54) ÷ 26.53 = 13.04: far above anything in train, and the transformed value says so.

Now the mistake: fit on train and test together. Change both fit calls to fit on pd.concat([train, test]), then print the learned numbers and the transformed train values:

Output

median: [60.] mean and SD: [105.71] [122.22]
train after transform: [-0.7  -0.54 -0.37 -0.37 -0.05]

One test value moved every learned number. The median, the mean and the SD now contain information from the test set, before the test. That is leakage through preprocessing. The rule: fit on the train set only, then transform the train, validation and test sets with the same learned numbers.

Only the train set reaches fit. The validation and test sets are only transformed, with what fit learned from train.

Text description of the diagram

The train file, 927 tickets from January to April, is the only input to fit. Fit learns the medians, the means and SDs, and the list of categories: the fitted preprocessing. The fitted preprocessing then transforms three sets: train, valid (227 tickets from May) and test (215 tickets from June). The model is fit on the transformed train set. The model and the transformed validation set give the validation score, which you use to choose and improve. The model and the transformed test set give the test score, once, at the end. No arrow goes from valid or test to fit.

A pipeline keeps the rule for you

It is easy to forget the rule in a long script. A pipeline joins the preprocessing steps and the model into one object. pipeline.fit(train) fits every step on the train rows. pipeline.predict_proba(valid) only transforms the validation rows and gives scores. The code keeps the rule, not your memory.

Mei’s features are of two kinds, so the preprocessing has two branches. A ColumnTransformer sends each list of columns to its own steps:

Columns	Steps	What fit learns
Categories: channel, team, segment, region	OneHotEncoder(handle_unknown="ignore")	The list of categories in each column.
Numbers: priority, order_value, word_count, customer_tenure_days, prior_tickets_90d, created_hour	SimpleImputer(strategy="median", add_indicator=True), then StandardScaler()	The median and, after imputation, the mean and SD of each column.

One-hot encoding turns one category column into several columns of 0 and 1, one for each category that fit saw. A chat ticket gets 1 in channel_chat and 0 in the other channel columns.

add_indicator=True adds a column of 0 and 1 for each number column that had gaps in train: 1 where the value was missing. In Module 4, you saw that a missing order_value has a meaning (an account ticket, no order). The indicator keeps that meaning after the gap is filled with the median.

Worked example: model_check.py

Priya, the data scientist, asks Mei for a quick check before she starts on models. Is the split usable? Does the data have any signal? Mei writes a short check with a simple model, a logistic regression. It is a check of the data, not a product; the course Machine Learning: From Problem to Reliable Model trains and compares models.

scripts/model_check.py

"""Check the prepared data with a simple model: fit on train only, score on
valid. A check for the data, not a model for Larkfield (C05 trains models)."""

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

CATEGORIES = ["channel", "team", "segment", "region"]
NUMBERS = ["priority", "order_value", "word_count", "customer_tenure_days", "prior_tickets_90d", "created_hour"]
TARGET = "escalated_72h"


def build_pipeline():
    numbers = Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
    ])
    prepare = ColumnTransformer([
        ("categories", OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
        ("numbers", numbers, NUMBERS),
    ])
    return Pipeline([("prepare", prepare), ("model", LogisticRegression(max_iter=1000))])


def main():
    train = pd.read_csv("data/model/train.csv")
    valid = pd.read_csv("data/model/valid.csv")
    pipeline = build_pipeline()
    pipeline.fit(train[CATEGORIES + NUMBERS], train[TARGET])

    learned = pipeline.named_steps["prepare"].named_transformers_["numbers"]
    medians = learned["impute"].statistics_
    print("Medians learned from train:", {name: float(m) for name, m in zip(NUMBERS, medians)})
    channels = pipeline.named_steps["prepare"].named_transformers_["categories"].categories_[0]
    print("Channels learned from train:", [str(c) for c in channels])

    scores = pipeline.predict_proba(valid[CATEGORIES + NUMBERS])[:, 1]
    print("ROC AUC on valid:", round(roc_auc_score(valid[TARGET], scores), 3))


if __name__ == "__main__":
    main()

Lines	Why
CATEGORIES, NUMBERS, TARGET	Explicit lists. Identifiers and dates are not in them, so they cannot become features by accident.
build_pipeline()	Preprocessing and model in one object, made by one function. Every use gets the same steps.
pipeline.fit(train[...], train[TARGET])	The only fit in the script, and it gets the train rows only.
named_steps[...], named_transformers_[...]	Look inside the fitted pipeline, to see what fit learned.
pipeline.predict_proba(valid[...])[:, 1]	Transforms the validation rows with the numbers from train, then gives each ticket a score between 0 and 1.
roc_auc_score(...)	One number for how well the scores rank the tickets.

ROC AUC in one sentence: the chance that the model scores a random escalated ticket higher than a random other ticket (0.5 is guessing, 1.0 is a perfect ranking). The machine learning course explains it fully.

Terminal

python scripts/model_check.py

Output

Medians learned from train: {'priority': 2.0, 'order_value': 86.21, 'word_count': 88.0, 'customer_tenure_days': 1488.5, 'prior_tickets_90d': 3.0, 'created_hour': 14.0}
Channels learned from train: ['chat', 'email', 'phone', 'unknown', 'web_form']
ROC AUC on valid: 0.673

0.673 is a modest score: better than guessing, far from perfect. That is honest. The model learns from January to April and is tested on May, after the policy change. When Priya has finished choosing, she fits the same pipeline once on train and validation together. Then she scores June once: 0.761. Until then, nobody looks at the test set.

A category that train never saw

Grace says that Larkfield will soon answer messages on social media. The first ticket with channel = social will arrive, and no train ticket has that channel. What does the encoder do?

new_channel.py

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

train = pd.read_csv("data/model/train.csv")
new_ticket = pd.DataFrame({"channel": ["social"]})

strict = OneHotEncoder().fit(train[["channel"]])
try:
    strict.transform(new_ticket)
except ValueError as error:
    print("ValueError:", error)

tolerant = OneHotEncoder(handle_unknown="ignore").fit(train[["channel"]])
print(tolerant.get_feature_names_out())
print(tolerant.transform(new_ticket).toarray())
print(tolerant.transform(pd.DataFrame({"channel": ["chat"]})).toarray())

Output

ValueError: Found unknown categories ['social'] in column 0 during transform
['channel_chat' 'channel_email' 'channel_phone' 'channel_unknown'
 'channel_web_form']
[[0. 0. 0. 0. 0.]]
[[1. 0. 0. 0. 0.]]

With the default setting, one new category stops the transform with an error. In real use, the routing service would fail on the first social ticket. With handle_unknown="ignore", the social ticket gets 0 in every channel column: “none of the channels I know”. The model still scores it with the other features. Do not “fix” this by fitting the encoder on all rows: that covers the categories in your file today, not the next new one. In Module 6, you add a check that counts unknown categories in new data.

“Fit the preprocessing on all the data”

A common mistake is to prepare the whole table first, and split it after. Then the medians, means and SDs come from every row, including validation and test. How much does that change the score here? Mei measures it. She fits the preprocessing on all 1,369 rows and the model on train only, as the mistake does:

fit_on_all.py

import sys

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

sys.path.insert(0, "scripts")
from model_check import CATEGORIES, NUMBERS, TARGET, build_pipeline

train = pd.read_csv("data/model/train.csv")
valid = pd.read_csv("data/model/valid.csv")
test = pd.read_csv("data/model/test.csv")
everything = pd.concat([train, valid, test])
features = CATEGORIES + NUMBERS

# The mistake: fit the preprocessing on every row, then the model on train.
prepare = build_pipeline().named_steps["prepare"]
prepare.fit(everything[features])
model = LogisticRegression(max_iter=1000).fit(prepare.transform(train[features]), train[TARGET])
scores = model.predict_proba(prepare.transform(valid[features]))[:, 1]
print("Median prior_tickets_90d:", float(prepare.named_transformers_["numbers"]["impute"].statistics_[4]))
print("ROC AUC on valid:", round(roc_auc_score(valid[TARGET], scores), 4))

# The correct way, for comparison, with four decimals.
pipeline = build_pipeline().fit(train[features], train[TARGET])
print("Train only, ROC AUC on valid:", round(roc_auc_score(valid[TARGET], pipeline.predict_proba(valid[features])[:, 1]), 4))

Output

Median prior_tickets_90d: 5.0
ROC AUC on valid: 0.6726
Train only, ROC AUC on valid: 0.6728

The learned median of prior_tickets_90d changed from 3 to 5, but the score changed only in the fourth decimal. Be honest about this: on this data, fitting a median imputer and a scaler on all rows changes the score very little. A median or a mean of a feature carries almost no information about the target.

Fix it anyway. The mistake grows with small datasets, rare categories and steps that learn from the target. And a workflow that fits on all rows today will one day fit something stronger. The large inflations come from leaky features and leaky aggregates. The lesson notebook measures them on the same split and the same pipeline:

Features (fit on train, score on May)	ROC AUC on valid
The 10 honest features	0.673
+ priority_now (set to 1 when a ticket is escalated)	0.964
+ first_reply_minutes (known after the first reply)	0.856
+ each customer’s escalation rate, computed over all rows	0.767
Honest features, preprocessing fit on all rows	0.6726

The last leak is the sneakiest. The rate looks like a customer fact. But a May ticket’s value includes that ticket’s own target, and the targets of later tickets. The model reads part of the answer from it.

Guided practice: measure the leaks

The lesson notebook runs every number in the table above, so you see each leak happen. The box at the top of this page has the Colab and Kaggle links. It joins the two leaky columns back from the clean tickets for the experiments.

1. Open the notebook in Colab or Kaggle. To run it on your computer instead, see the box below the steps.
2. Run the setup cells. You should see pandas 3.0.6 and scikit-learn 1.9.1, then split.py: done.
3. In section 1, check the median 50, the mean 54 and the value −0.15 by hand.
4. In section 4, the preprocessing is fit on all rows. Predict the effect first. You should see 0.6726 and 0.6728.
5. In section 5, write your prediction for priority_now before you run the cell. You should see 0.964.
6. Do the tasks in section 8: Exercise 1 and Exercise 2. Each check cell prints Check passed. when your answer is right.
7. Restart and run all. Every cell must run, and every check cell must print Check passed.

Run the notebook on your computer

Download the notebook with View on GitHub, then the download button. Save it in a new, empty folder, not in your ticket-data project. The notebook makes its own ticket-data folder next to it, and its cells write scripts and data there. In VS Code, open it and select Select Kernel › Python Environments › .venv. If VS Code asks to install ipykernel, accept. It is a tool for you, not a dependency of the project, so do not add it to requirements.txt.

Your turn: mean instead of median

Change one thing in scripts/model_check.py: set strategy="mean" in the SimpleImputer. Before you run it, predict: will the value that fills a missing order_value go up or down? Will the score change much? Then run the script. (The first line still says “Medians”, but now it shows the means.) Set the strategy back to "median" at the end.

Show the answer

Output

Medians learned from train: {'priority': 2.348435814455232, 'order_value': 120.30397741530741, 'word_count': 107.33549083063646, 'customer_tenure_days': 1427.0087912087913, 'prior_tickets_90d': 5.858683926645091, 'created_hour': 13.627831715210355}
Channels learned from train: ['chat', 'email', 'phone', 'unknown', 'web_form']
ROC AUC on valid: 0.672

The fill value for order_value goes up, from 86.21 to 120.30. A few large orders pull the mean up, but not the median. The score hardly changes (0.672 instead of 0.673). The indicator column still marks every filled value, and the model can learn from it. The choice still matters for the meaning: a filled account ticket now “has” an order of $120.30. In Module 4, you decided that a missing order value is not an unknown number. So the median plus an indicator is the more honest choice here, and the score does not argue against it.

Common mistake

fit_transform on the validation set

Symptom: Tomás writes the preprocessing and the model as separate steps, without a pipeline. Training works. Then he prepares the validation set in the same way as the train set, and the prediction fails:

Tomás's code

prepare = build_pipeline().named_steps["prepare"]
train_x = prepare.fit_transform(train[features])
model = LogisticRegression(max_iter=1000).fit(train_x, train[TARGET])
valid_x = prepare.fit_transform(valid[features])  # fit again, on valid
model.predict_proba(valid_x)

Output

ValueError: X has 27 features, but LogisticRegression is expecting 28 features as input.

Cause: fit_transform on the validation set fits the preprocessing again, on the May tickets. The May tickets have no customer with the segment unknown (deleted accounts). So the encoder learned one category fewer, and the table has 27 columns instead of 28. The error is the lucky case. If May had the same categories, the code would run without an error. But the scaler would use the May mean and SD for every column. The same value would mean something different in train and validation, and the score would be quietly wrong.

Fix: fit or fit_transform only on the train set; only transform on everything else. Better, use one pipeline, as model_check.py does: then there is only one fit call, and it gets the train rows.

Consequence in AI work: the same mistake in production is worse. A service that refits its preprocessing on each new batch of tickets gives the same input a different meaning every day. Nobody sees an error.

Knowledge check

Check your understanding

Name what fit learns in each of the three preprocessing steps, and which rows it may learn from.

Show the answer

SimpleImputer(strategy="median") learns the median of each number column. StandardScaler learns the mean and the SD of each column after imputation. OneHotEncoder learns the list of categories in each category column. All three may learn only from the train rows. The validation and test rows are only transformed with what was learned.

Check your understanding

Tomás adds a column hours_open, the hours from created_at to closed_at. The validation ROC AUC goes up from 0.673 to 0.709. Explain why, and say what you would see in the real routing service.

Show the answer

closed_at is written when the ticket is closed, so hours_open is not known at prediction time. It is leakage. Escalated tickets stay open longer: in train, the median is about 53 hours for escalated tickets and 27 hours for the others. So the column carries information about the target, and the score goes up. In the routing service, a new ticket has no closed_at, so the column is empty for every ticket. The model’s real ranking is worse than its validation score said. A small rise is still a leak: the size of the jump does not decide, the time the value is written does.

Module practice

The module practice asks you to plan before you fit any model. A written plan makes each leakage decision visible, so a colleague can check it before it costs anything.

Worked example: Mei’s plan for the escalation model

Mei writes her plan on one page, with the numbers from this module:

Question	Decision	Evidence
Prediction time	When the ticket is created.	Larkfield routes tickets on arrival.
Target	escalated_72h: an escalated event within 72 hours after created_at.	188 of 1,369 rows (13.7%).
Target maturity	Only tickets created at or before 2026-06-28 06:00 (snapshot − 72 hours).	29 tickets removed and counted.
Features	channel, team, priority, segment, region, order_value, word_count, customer_tenure_days, prior_tickets_90d, created_hour.	All known at creation. History from earlier tickets only.
Never features	priority_now, first_reply_minutes, closed_at, outcome events, csat, any customer rate over all rows.	Written after creation, or contains targets.
Split	By time: train before May, valid May, test 1–27 June.	927 / 227 / 215 rows; rates 11.7% / 19.8% / 16.3%; 77 of 89 test customers in train.
Preprocessing	One pipeline: one-hot with handle_unknown="ignore"; median imputation with indicators; scaling. Fit on train only.	Medians from train, for example order_value 86.21.
Checks before any model	Row counts add up; dates do not overlap; no leaky column in the output; a single strong feature is a leakage alarm.	build_summary.json, split_manifest.json.

Your task: will a resolved ticket be reopened?

Grace has a new question. When an agent resolves a ticket, can a model say if the customer will reopen it within 14 days? Then a team lead could check those tickets before the customer comes back. Copy Mei’s table into a file, reports/reopen-plan.md, and fill in each row for this new model. Do not fit a model. For every decision, write one sentence of reason. Look carefully at the prediction time: it is no longer the creation of the ticket.

Show the model answer

Question	Decision	Reason
Prediction time	The moment of the resolved outcome event (recorded_at).	The team lead acts when the ticket is resolved.
Target	reopened_14d: a reopened event of the same ticket within 14 days after that resolved event.	Precise: which event, which window. A ticket that is resolved twice gives two rows, one per resolution.
Target maturity	Only resolutions at or before 2026-07-01 06:00 − 14 days = 2026-06-17 06:00. Count the rows removed.	A later resolution has not had its 14 days yet. Its “no reopen” is not final.
Features	The creation features, plus facts known at resolution: first_reply_minutes, whether the ticket was escalated before the resolution, resolution_code, the hours from creation to this resolution, earlier reopens of this ticket.	The prediction time moved. first_reply_minutes was a leak for the escalation model, but it is known at resolution, so it is a fair feature here. Leakage depends on the prediction time, not on the column.
Never features	csat (recorded after resolution), closed_at (for a reopened ticket, it is the final close, after the reopen), priority_now (the value at the snapshot, not at the resolution), any later outcome event, any customer reopen rate over all rows.	Each one is written after the prediction time or contains targets.
Split	By time of the resolution: for example train before May, valid May, test 1–16 June. Keep all resolutions of one ticket in the same part.	The model predicts future reopens from past ones. Two rows of one ticket in different parts would let the model see the ticket’s story in train.
Preprocessing	The same kind of pipeline, fit on train only. resolution_code is a new category column, with handle_unknown="ignore".	Same rule; a new resolution code must not stop the service.
Checks before any model	Rows per part and reopen rate per month; no overlap of dates or tickets between parts; for each feature, the time when it is written is before recorded_at of the resolution.	The rate may change after 1 May, too. The last check is the leakage check, done on paper.

Your answer can differ in the details, for example a different window or other dates. Check three things: the prediction time is the resolution; csat and closed_at are excluded; the target maturity uses the resolution time, not the creation time.

Module check

Check your understanding

Retrieval: give the three questions that decide between a random, a time-based and a group-based split.

Show the answer

1. Will the model predict the future from the past? Then split by time.
2. Will the model meet new customers or other new groups? Then keep each group in one part.
3. Are the rows independent, with no time order and no repeated groups? Only then is a random split a fair test.

For Larkfield, the answer to the first question is yes, so the split is by time.

Check your understanding

Predict and explain: Priya fits the imputer and the scaler on all 1,369 rows instead of on train. What happens to the validation score on this data? Why is it still worth fixing?

Show the answer

Almost nothing: 0.6726 instead of 0.6728. A median, a mean and an SD carry very little information about the target. It is still worth fixing. The learned numbers come from the future: the median of prior_tickets_90d moves from 3 to 5. And the effect grows with small data, rare categories and steps that learn from the target. A pipeline fixed on train costs nothing and prevents all of these.

Check your understanding

Completion check: in the notebook's Exercise 2, one of three new columns leaks. Name it, give the scores that show it, and explain how it inflated the evaluation. Then explain how you would find it without a model.

Show the answer

The leaky column is reply_speed. With the honest features, the validation ROC AUC is 0.673. Adding weekday gives 0.679 and order_size 0.687, small changes. Adding reply_speed gives 0.809.

How it inflated the evaluation: reply_speed is fast when first_reply_minutes is under 30. The first reply comes after the ticket is created, and escalated tickets get fast replies from senior agents. So the column carries a trace of the target into the validation rows. The model ranks escalated tickets higher because of it. A new ticket has no first reply yet, so in real use the column has no value. The real ranking is close to the honest 0.673, not 0.809.

Without a model: ask when each value is written. reply_speed comes from first_reply_minutes, which is in the after-creation list of build_features.py. A new name and a coarser form do not remove a leak.

A complete answer names the column and compares its score with the honest score. It links the leak to the time the value is written, and says what happens in real use.

Recap

In this module you prepared the data for a model without leaks:

• Timing: the prediction time is ticket creation. The target escalated_72h is exact, and the 29 tickets without a final target are removed. Features come from the past only; priority_now, first_reply_minutes, closed_at and outcome events are never features.
• Split: by time, as Larkfield will use the model. Train has 927 rows, validation 227 and test 215, with escalation rates of 11.7%, 19.8% and 16.3%. A random split hides the May change, and one random split is one draw (0.665 to 0.827).
• Fitting: preprocessing learns only from train, inside one pipeline. handle_unknown="ignore" keeps a new category from stopping the model.
• Evidence: honest features score 0.673. Leaks inflate it to 0.964 (priority_now), 0.856 (first_reply_minutes) and 0.767 (a customer rate over all rows). Fitting the preprocessing on all rows changed it only in the fourth decimal.

Next, in Module 6, you write the rules that every delivered file must pass. Then the next person can trust the dataset without repeating your checks.

Next: Validation rules