Preprocess with a pipeline

What you will be able to do

Use all ten features in one model, with exactly the same preparation for every ticket: in training, in validation and later for each new ticket. First see what goes wrong when you prepare the features by hand. Then solve it with a pipeline, and prove that it learned only from the train set.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every piece of code, task and check of this lesson, the module practice and the completion check. It shows each problem of preparing features by hand, then writes ticket_model.py with a %%writefile cell. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 80 minutes

What you need to know

• Logistic regression, scores, predict_proba and predict, and why 0.5 flags almost nothing, from Classification intuition: predict_proba and predict.
• The train, validation and test sets, and why the model must learn from the train set only, from Train, validation and test sets.
• Fit and transform, and leakage through preprocessing, from Fit transformations correctly: Fit, then transform in SQL and Data Preparation for AI.
• Precision and recall, from Classification metrics; average precision and ROC AUC, from The precision–recall trade-off.
• The baselines on the validation set, from Define a baseline.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model, with its .venv environment and the data in data/. You made them in Define a baseline: Set up your project. If the data is missing or changed, run python get_data.py --reset in the project folder.
• This lesson adds ticket_model.py to the project folder, in Write ticket_model.py. If you lose it or break it later, copy it again from that section. Do not copy the file from C05/project yet: that finished version already has the feature that Module 6 adds.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

Why the model stopped

In the last lesson, Priya’s logistic regression used only four of the ten features: the four numbers that have a value for every ticket. The other six features are useful too. The team (payment tickets escalate 20.7% of the time, account tickets 5.5%) and the customer’s history should help. So she tries to add one of each kind:

import pandas as pd
from sklearn.linear_model import LogisticRegression

TARGET = "escalated_72h"
CATEGORIES = ["channel", "team", "segment", "region"]
NUMBERS = ["priority", "order_value", "word_count", "customer_tenure_days", "prior_tickets_90d", "created_hour"]
FEATURES = CATEGORIES + NUMBERS
train = pd.read_csv("data/train.csv")
valid = pd.read_csv("data/valid.csv")

for columns in (["priority", "team"], ["priority", "order_value"]):
    try:
        LogisticRegression().fit(train[columns], train[TARGET])
    except ValueError as error:
        print(columns, "->", str(error).splitlines()[0])

The output:

Output

['priority', 'team'] -> could not convert string to float: 'delivery'
['priority', 'order_value'] -> Input X contains NaN.

Both attempts fail, for different reasons. To see why, look at six real tickets from the train set, with five of their columns:

six = train.head(6)[["ticket_id", "team", "order_value", "word_count", "customer_tenure_days", "priority"]]
print(six.to_string(index=False))

The output:

Output

ticket_id     team  order_value  word_count  customer_tenure_days  priority
 T-100001 delivery        53.76          48                 738.0         3
 T-100002 warranty        24.30          85                1496.0         1
 T-100003  returns       699.55         111                1243.0         3
 T-100004  returns       108.07         112                1625.0         3
 T-100005  account          NaN          45                1186.0         3
 T-100006  account          NaN          46                  46.0         2

Three problems are visible in these six rows. team is text, and a model can only multiply numbers by weights. order_value has gaps (NaN, “not a number”, is how pandas shows an empty cell). And the numbers live on very different scales: priority goes from 1 to 3, while tenure goes up to 1,625 days. Before you meet the tool that solves them, take one problem at a time.

Problem 1: text that a model cannot read

A logistic regression computes z = intercept + weight × feature for each feature. “Delivery” times a weight means nothing. So each text category must become numbers.

The usual way is one-hot encoding: one new column for each category. The column of the ticket’s category gets 1, and the other columns get 0. In pandas, get_dummies does it:

print(pd.get_dummies(six["team"], dtype=int).to_string())

The output:

Output

   account  delivery  returns  warranty
0        0         1        0         0
1        0         0        0         1
2        0         0        1         0
3        0         0        1         0
4        1         0        0         0
5        1         0        0         0

Each ticket has exactly one 1, in the column of its team. Now each team can get its own weight, so the model can learn that payment adds risk and account takes it away.

“But why not just number the teams?” asks Tomás. “Account is 1, delivery 2, payment 3, and so on. One column instead of five.” The problem is that numbers have an order and distances. With those numbers, the model would treat payment as “three times account”. It would also have to give delivery a risk exactly halfway between account and payment. The teams have no such order. One-hot columns make no such claim: each team is just itself.

Keep one detail in mind for later: get_dummies makes a column for each category in the rows that it gets. These six tickets have four teams, so it made four columns. The full train set has five.

Problem 2: gaps

The gaps in order_value are not errors. All account tickets are about an account, not an order, so they have no order value. About 1 in 100 orders were also not found. In the same way, customer_tenure_days is empty for the 202 guests, who have no account. A model cannot multiply an empty cell by a weight, so every gap needs a value.

“Why not just delete the rows with a gap?” asks Tomás. Two reasons. First, it throws away a lot of data: 2,629 of the 16,939 train tickets (15.5%) have at least one gap. Second, and worse, it does not work for new tickets. When a guest writes tomorrow, the routing service must give that ticket a score. It cannot skip it.

So you fill each gap with a value: this is imputation. A common choice is the median of the column, the middle value. Here is why the median is safer than the mean, on the six tickets:

known = six["order_value"].dropna()
print("median:", round(known.median(), 2), "| mean:", round(known.mean(), 2))
print("train tickets with a gap:", train[FEATURES].isna().any(axis=1).sum(), "of", len(train))
missing = train["order_value"].isna()
print("escalation rate, order_value missing:", round(train.loc[missing, TARGET].mean(), 3))
print("escalation rate, order_value known:", round(train.loc[~missing, TARGET].mean(), 3))

The output:

Output

median: 80.91 | mean: 221.42
train tickets with a gap: 2629 of 16939
escalation rate, order_value missing: 0.059
escalation rate, order_value known: 0.127

The four known order values are 24.30, 53.76, 108.07 and 699.55. One large order pulls the mean up to 221.42, a value that no typical ticket has. The median, 80.91, stays in the middle. Order values in the full train set are also uneven, with a few orders above $1,000. In the notebook, you compute the median and the mean of the whole train set and compare them.

The last two lines show something more. Tickets with no order value escalate less often (5.9%) than the others (12.7%), because they are mostly account questions. So the gap itself is information. A good imputer fills the gap and adds an indicator column: 1 where the value was missing, 0 where it was present. The model can then give “no order” its own weight.

Problem 3: very different scales

The six number features have very different ranges in the train set:

print(train[NUMBERS].agg(["min", "max"]).T.to_string())

The output:

Output

                      min     max
priority              1.0     3.0
order_value           4.5  1450.0
word_count            0.0   883.0
customer_tenure_days  0.0  2607.0
prior_tickets_90d     0.0    58.0
created_hour          0.0    23.0

You saw the effect in the last lesson. In the four-feature model, the weight of word_count was only 0.0052, yet 300 words added 1.56 to z, more than one step of priority. The size of a weight depended on the unit of its feature, so you could not compare weights. Very different scales also make the solver, the method that searches for the best weights, slow: it may stop before it finds them. The independent variation of this lesson shows that warning.

Scaling puts every number on a similar scale. The usual method, standardisation, subtracts the mean of the column and divides by its standard deviation (SD). In the train set, word_count has a mean of 106.13 and an SD of 70.81. So a 300-word message becomes (300 − 106.13) ÷ 70.81 = 2.74: “2.74 SDs above the usual length”. priority has a mean of 2.37 and an SD of 0.72, so priority 1 becomes −1.91. Both numbers now speak the same language.

Not every model needs scaling. The decision trees of Module 5 ask questions such as “is the word count above 200?”, and the answer does not change with the unit. Logistic regression needs it.

The real problem: every step learns something

The three fixes look simple. The difficulty is hidden in one fact: each fix learns numbers from data. The encoder learns the list of categories in each column. The imputer learns the median of each column. The scaler learns the mean and SD of each column. These learned numbers are part of the model, just like its weights.

Think of a café that makes lemonade. On the first day, the owner tastes and decides the recipe: two spoons of sugar for each glass. She writes it on a card. Every later glass, made by any cook on any day, follows the card. If each cook tasted their own glass and chose the sugar again, every glass would be different. If a cook forgot the sugar, the customer would get sour lemonade, and nobody would notice until a complaint.

The preparation of features works the same way, with two rules:

1. Learn the numbers from the train set only. The validation and test sets must stay unseen, as you learned in Train, validation and test sets. If the median comes from all the rows, information from the validation set leaks into the model. The SQL course measured this in Fit transformations correctly.
2. Apply exactly the same numbers, the same steps, in the same order, to every later row. That means the validation set, the test set and every single new ticket, months later.

By hand: four objects to keep in order

Priya first follows both rules by hand. She fits each step on the train set, then writes a function that applies the three fitted steps to any rows:

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

imputer = SimpleImputer(strategy="median", add_indicator=True).fit(train[NUMBERS])
scaler = StandardScaler().fit(imputer.transform(train[NUMBERS]))
encoder = OneHotEncoder().fit(train[CATEGORIES])


def prepare_by_hand(rows):
    """The three steps, with what they learned from train, for any rows."""
    numbers = scaler.transform(imputer.transform(rows[NUMBERS]))
    categories = encoder.transform(rows[CATEGORIES]).toarray()
    return np.hstack([categories, numbers])

SimpleImputer, StandardScaler and OneHotEncoder are scikit-learn’s versions of the three fixes. Each one has a fit method, which learns the numbers, and a transform method, which applies them. np.hstack joins the category columns and the number columns side by side. Then she trains the model and scores the validation set and one ticket:

from sklearn.metrics import average_precision_score

model = LogisticRegression(max_iter=1000).fit(prepare_by_hand(train), train[TARGET])
scores = model.predict_proba(prepare_by_hand(valid))[:, 1]
print("AP on valid:", round(average_precision_score(valid[TARGET], scores), 3))
one = valid.head(1)
print("score of", one["ticket_id"].iloc[0] + ":", round(model.predict_proba(prepare_by_hand(one))[0, 1], 3))

The output:

Output

AP on valid: 0.353
score of T-116940: 0.184

It works. The average precision (AP) is 0.353, much better than the rule “flag priority 1”, which gets 0.228 (the results table below shows both). But now there are four fitted objects: the imputer, the scaler, the encoder and the model. One function must call them in the right order. The routing service, the evaluation script and every later notebook must repeat all of it, exactly. Here is what goes wrong in real use.

What goes wrong: a forgotten step

Months later, Tomás writes the routing service. He copies the steps, but he forgets the scaler. Nothing stops him:

def forgot_scaler(rows):
    """The same steps, but the scaler is missing."""
    return np.hstack([encoder.transform(rows[CATEGORIES]).toarray(), imputer.transform(rows[NUMBERS])])


print("score of", one["ticket_id"].iloc[0] + ", scaler forgotten:", round(model.predict_proba(forgot_scaler(one))[0, 1], 3))
bad = model.predict_proba(forgot_scaler(valid))[:, 1]
print("valid tickets with a score of at least 0.5:", (bad >= 0.5).sum(), "of", len(valid))

The output:

Output

score of T-116940, scaler forgotten: 0.999
valid tickets with a score of at least 0.5: 875 of 1179

The same ticket now gets 0.999 instead of 0.184. The model learned its weights for scaled numbers, near 0. Without the scaler, they meet raw values such as 1,528 days of tenure, and z explodes. 875 of 1,179 tickets get a score of at least 0.5, against 18 with the scaler. And there is no error message: the service runs, and the flags flood the senior team.

What goes wrong: a category that train never saw

On 15 July 2026, Larkfield adds a new contact channel, social. The first social ticket reaches the by-hand code:

july = pd.read_csv("data/july_tickets.csv")
social = july[july["channel"] == "social"].head(1)
try:
    prepare_by_hand(social)
except ValueError as error:
    print("ValueError:", str(error).splitlines()[0])

The output:

Output

ValueError: Found unknown categories ['social'] in column 0 during transform

This time there is an error, and the service cannot score the ticket at all. The encoder knows only the five channels of the train set. You can tell it what to do with a new category (handle_unknown="ignore"), but you must remember to do it. If you use get_dummies instead, it is worse: as you saw in Problem 1, it makes columns from the rows it gets. One social ticket would get one channel column, channel_social, which the model has never seen, and the column count would not match.

What else goes wrong

Three more problems have the same root:

• The wrong column order. The model has no names for its 27 input columns, only positions. If the service sends the number columns in another order, each weight meets the wrong feature. scikit-learn stops with an error when it gets a table with named columns in another order. But a web service often sends plain values without names, and then scikit-learn can only print a warning.
• Fitting on the wrong rows. Tomás wants “more data for the medians” and fits the imputer on the train and validation sets together. Nothing complains. The validation set has now leaked into the model, and its score is no longer an honest check.
• Nothing to hand over. To use the model elsewhere, you must save four objects, the two column lists and the order of the steps. Module 8 packages the model into a file for another team. With loose objects, that file is a box of parts with no assembly plan.

None of these is a problem of the model. Each one is a problem of keeping many separate parts the same everywhere, for a long time, by people who did not write them.

The solution: a pipeline

A pipeline is one object that holds all the preparation steps and the model, in order. It is the recipe card of the café, made into code. You call fit once, on the train set, and the pipeline fits every step on those rows, one after the other. Each step learns from the output of the step before. You call predict_proba on any later rows, from a full month to one ticket. The pipeline then applies every fitted step in the same order, with the numbers it learned from train. Then it applies the model. The name comes from water pipes: what comes out of one section flows into the next.

The same steps, two ways to keep them. By hand, your code must call each fitted object, in the right order, for every ticket. A pipeline holds the steps itself, so one call does all of them.

Text description of the diagram

The left side shows the work by hand. One new ticket goes to two branches. The numbers go to the imputer, which fills gaps with the train medians, and then to the scaler, with the train means and SDs. The categories go to the encoder, with the train categories. A join step puts the columns side by side, in the same order as in training. The model, with its train weights, gives the score. Your code must call each of these five boxes. The right side shows a pipeline. The ticket goes into one object, pipeline.predict_proba(ticket), which contains the same five steps in the same order. One call gives the score.

Each problem from above has an answer in the pipeline:

Problem by hand	What the pipeline does
A step is forgotten at prediction time	The steps are part of the object; predict_proba always runs all of them.
A step is fitted on the wrong rows	One fit call fits every step on the same rows. If you pass train, all of them learn from train only.
The wrong column order	The pipeline selects the columns by name and joins them in the same order every time.
A new category	OneHotEncoder(handle_unknown="ignore") is written into the pipeline once, so every call handles it.
Nothing to hand over	The fitted pipeline is one object, so you can save it as one file (Module 8).

A pipeline does not choose the rows for you. If you call fit on the train and validation sets together, it learns from both. The failure case of this lesson shows it.

Two branches: the ColumnTransformer

The ten features need two kinds of preparation: the categories need one-hot encoding, and the numbers need imputation and then scaling. A ColumnTransformer sends each list of columns to its own steps and joins the results. It is the “join” box of the diagram. Inside the pipeline, it comes first, and the model comes last.

A pipeline on four tickets

Before the real data, try a pipeline on four made-up tickets, small enough to check by hand. They have one category and one number, and one order_value is missing. The ColumnTransformer sends channel to the encoder, and order_value to the imputer and then the scaler:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

tiny = pd.DataFrame({"channel": ["email", "chat", "email", "phone"],
                     "order_value": [20.0, None, 60.0, 100.0]})
numbers = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                    ("scale", StandardScaler())])
prepare = ColumnTransformer([("categories", OneHotEncoder(handle_unknown="ignore"), ["channel"]),
                             ("numbers", numbers, ["order_value"])], verbose_feature_names_out=False)
prepare.fit(tiny)
print(pd.DataFrame(prepare.transform(tiny), columns=prepare.get_feature_names_out()).round(2).to_string())

The output:

Output

   channel_chat  channel_email  channel_phone  order_value  missingindicator_order_value
0           0.0            1.0            0.0        -1.41                         -0.58
1           1.0            0.0            0.0         0.00                          1.73
2           0.0            1.0            0.0         0.00                         -0.58
3           0.0            0.0            1.0         1.41                         -0.58

Check it by hand:

• One-hot: fit saw three channels, so there are three channel columns. The chat ticket has 1 in channel_chat and 0 in the others.
• Imputation: the known values are 20, 60 and 100, so the median is 60, and the gap becomes 60.
• Scaling: the values are now 20, 60, 60 and 100. The mean is 60 and the SD is 28.28. So 20 becomes (20 − 60) ÷ 28.28 = −1.41, and 60 becomes 0.00.
• Indicator: the second ticket had a gap, so its indicator is 1. The scaler scales the indicator column too (0 becomes −0.58 and 1 becomes 1.73). That does no harm: it still says “missing or not”.

Now a new ticket arrives from a channel that fit never saw, social, with no order value:

new_ticket = pd.DataFrame({"channel": ["social"], "order_value": [None]})
print(pd.DataFrame(prepare.transform(new_ticket), columns=prepare.get_feature_names_out()).round(2).to_string())

The output:

Output

   channel_chat  channel_email  channel_phone  order_value  missingindicator_order_value
0           0.0            0.0            0.0          0.0                          1.73

This is the problem of the by-hand code, solved. The one ticket gets the same five columns as the four train tickets. handle_unknown="ignore" gives the unknown channel 0 in every channel column: “none of the channels I know”. The gap gets the median that fit learned, 60, which scales to 0.0. The pipeline learns nothing from the new ticket. Transform uses only what fit learned from the four train tickets.

Fit on train only

In the real pipeline, the rule from the SQL course stays the same: only the train set reaches fit. The validation and test sets only go through the fitted pipeline. The SQL course showed this in A pipeline keeps the rule for you.

Only the train set reaches fit. The validation and test sets only go through the fitted pipeline, which transforms them with what it learned from train and then scores them.

Text description of the diagram

The train file, 16,939 tickets from July 2024 to April 2026, is the only input to fit. Fit learns everything in the pipeline: the medians, the means and SDs, the lists of categories, and the model’s coefficients (its weights). The result is the fitted pipeline. The validation file, 1,179 tickets from May 2026, goes into predict_proba with the fitted pipeline. This gives the scores that you use to compare models and choose settings. The test file, 1,002 tickets from June 2026, goes into predict_proba once, at the end, in Module 6. No arrow goes from valid or test to fit.

Write ticket_model.py

Priya puts the column lists and the pipeline in one file, ticket_model.py. Every later lesson, script and notebook imports it, so every model uses the same columns and the same steps. This is the recipe card, written down once. In the notebook, a %%writefile ticket_model.py cell writes it. On your computer, create the file in ~/projects/ticket-model, next to data/, with exactly this content:

ticket_model.py

"""The shared parts of Larkfield's escalation model: the columns, how to load
a part of the data, and the preprocessing-plus-model pipeline.

Nextia Learning, C05, Module 4, lesson 3 (Preprocess with a pipeline).
"""

from pathlib import Path

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

DATA = Path("data")
CATEGORIES = ["channel", "team", "segment", "region"]
NUMBERS = ["priority", "order_value", "word_count", "customer_tenure_days", "prior_tickets_90d",
           "created_hour"]
FEATURES = CATEGORIES + NUMBERS
TARGET = "escalated_72h"


def read(path):
    """Read a ticket CSV file. Only an empty cell is missing: the text
    "unknown" stays a category."""
    return pd.read_csv(path, parse_dates=["created_at"], dtype={"customer_id": "string"},
                       keep_default_na=False, na_values=[""])


def load(part):
    """One part of the split: train, valid or test."""
    return read(DATA / f"{part}.csv")


def build_pipeline(model=None):
    """Preprocessing and model in one object. fit() learns the medians, the
    scaling and the categories from the rows it is given, and nothing else."""
    numbers = Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
    ])
    prepare = ColumnTransformer([
        ("categories", OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
        ("numbers", numbers, NUMBERS),
    ])
    return Pipeline([("prepare", prepare), ("model", model if model is not None else LogisticRegression(max_iter=1000))])

Each part has a reason:

Part	Why
CATEGORIES, NUMBERS, FEATURES, TARGET	Explicit lists. ticket_id, customer_id and created_at are not in them, so an identifier or a date cannot become a feature by accident.
read()	Reads a file the same way every time. keep_default_na=False, na_values=[""] makes only an empty cell missing, so the region unknown stays a category.
load(part)	One short call for "train", "valid" or "test".
build_pipeline(model=None)	The preparation and a model in one Pipeline, made by one function. Every call gets the same steps. You can pass another model, for example a baseline; the default is LogisticRegression(max_iter=1000).
max_iter=1000	Lets the solver run long enough to finish on this data.

build_pipeline() returns an unfitted pipeline. The pipeline learns nothing until you call fit. Check the file in a terminal, in the project folder, with .venv active:

Terminal

python -c "import ticket_model; print(len(ticket_model.FEATURES), 'features')"

The output:

Output

10 features

If you see ModuleNotFoundError: No module named 'ticket_model', the terminal is not in ~/projects/ticket-model. If you see a SyntaxError, compare your file with the code above, line by line.

Worked example: fit and look inside

Priya fits the pipeline on the train set only. Then she looks inside: did the imputer learn the train medians?

from ticket_model import CATEGORIES, FEATURES, NUMBERS, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
pipeline = build_pipeline()
pipeline.fit(train[FEATURES], train[TARGET])

prepare = pipeline.named_steps["prepare"]
learned = prepare.named_transformers_["numbers"]["impute"].statistics_
print(pd.DataFrame({"learned": learned, "train median": train[NUMBERS].median(),
                    "valid median": valid[NUMBERS].median()}).to_string())

The output:

Output

                      learned  train median  valid median
priority                 3.00          3.00          3.00
order_value             86.56         86.56         85.71
word_count              89.00         89.00         90.00
customer_tenure_days  1178.00       1178.00       1342.50
prior_tickets_90d        5.00          5.00          6.00
created_hour            14.00         14.00         14.00

named_steps["prepare"] is the ColumnTransformer; named_transformers_["numbers"]["impute"] is the fitted imputer inside it. Its statistics_ are the medians it learned. They are equal to the train medians in every row, and four of them differ from the validation medians. That is the proof that fit learned from train. If the validation rows had reached fit, customer_tenure_days would not be 1178.

Next, the categories that the encoder learned:

encoder = prepare.named_transformers_["categories"]
for column, categories in zip(CATEGORIES, encoder.categories_):
    print(column, list(categories))
print(len(prepare.get_feature_names_out()), "columns go into the model")

The output:

Output

channel ['chat', 'email', 'phone', 'unknown', 'web_form']
team ['account', 'delivery', 'payment', 'returns', 'warranty']
segment ['business', 'guest', 'home', 'trade']
region ['east', 'north', 'south', 'unknown', 'west']
27 columns go into the model

27 = 19 one-hot columns + 6 numbers + 2 indicators. Only order_value and customer_tenure_days had gaps in train, so only they get an indicator. A later ticket with a category outside these lists gets 0 in all the columns of that category.

Now the scores on the validation set:

from sklearn.metrics import average_precision_score, roc_auc_score

scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
print("ROC AUC on valid:", round(roc_auc_score(valid[TARGET], scores), 3))
print("AP on valid:", round(average_precision_score(valid[TARGET], scores), 3))

The output:

Output

ROC AUC on valid: 0.738
AP on valid: 0.353

These are the same scores as Priya’s by-hand code, because the steps are the same. predict_proba(valid) did not fit anything. It transformed the May tickets with the train medians, means, SDs and categories, then scored them with the train coefficients.

Results: baseline, rule and model

A number alone means nothing, so every result goes next to the baselines. Priya defines summary, one line of the results table. It prints the flags, the precision and the recall of a decision, and the ROC AUC and AP of a score:

def summary(name, scores, flags):
    """One line of the results table, on the validation set."""
    flagged = int(flags.sum())
    caught = int((flags & (valid[TARGET] == 1)).sum())
    precision = caught / flagged if flagged else float("nan")
    recall = caught / valid[TARGET].sum()
    auc = roc_auc_score(valid[TARGET], scores)
    ap = average_precision_score(valid[TARGET], scores)
    print(f"{name:9s} flagged {flagged:3d} ({flagged / len(valid):.1%}), precision {precision:.3f}, "
          f"recall {recall:.3f}, ROC AUC {auc:.3f}, AP {ap:.3f}")

The majority baseline goes through the same pipeline, with DummyClassifier as the model. It ignores the features and gives every ticket the train rate as its score. The rule’s 0/1 flags are also its score.

from sklearn.dummy import DummyClassifier

majority = build_pipeline(DummyClassifier(strategy="prior")).fit(train[FEATURES], train[TARGET])
summary("majority", majority.predict_proba(valid[FEATURES])[:, 1], majority.predict(valid[FEATURES]))

rule = (valid["priority"] == 1).astype(int)
summary("rule", rule, rule)

summary("logistic", scores, pipeline.predict(valid[FEATURES]))

The output:

Output

majority  flagged   0 (0.0%), precision nan, recall 0.000, ROC AUC 0.500, AP 0.155
rule      flagged 163 (13.8%), precision 0.374, recall 0.333, ROC AUC 0.615, AP 0.228
logistic  flagged  18 (1.5%), precision 0.556, recall 0.055, ROC AUC 0.738, AP 0.353

The same results in a table:

Validation set, 1,179 tickets	Flagged	Precision	Recall	ROC AUC	AP
Majority class: never escalate	0 (0%)	—	0.000	0.5	0.155
Rule: flag priority 1	163 (13.8%)	0.374	0.333	0.615	0.228
Logistic regression, 10 features (predict, 0.5)	18 (1.5%)	0.556	0.055	0.738	0.353

What the table says:

• The model ranks much better than the rule: AP 0.353 against 0.228, and ROC AUC 0.738 against 0.615. Ten features carry more information than one.
• Its decision at 0.5 is useless for Grace: 18 flags, 10 of them escalated. That is the default threshold of the last lesson again.
• A fair comparison of decisions uses the same workload. The module practice compares the model and the rule at 163 flags each.

When you need a pipeline, and what it costs

Use a pipeline whenever any preparation step learns from data: a median, a mean, a list of categories, a vocabulary of words. That covers almost every real model. The model that you evaluate is then exactly the model that you deliver, and one call fits it and one call uses it.

The cost is small but real. The code is less direct: to look inside, you need names such as named_steps["prepare"] and paths such as prepare__numbers__scale. An error message from deep inside a pipeline can be harder to read. And the pipeline protects the order and the consistency of the steps, not the choice of rows. You still decide what goes into fit.

Guided practice: what did fit learn?

1. Open the lesson notebook in Colab or Kaggle, and run the setup cells.
2. In sections 1 to 4, run the problem cells. You should see the two errors, the six tickets, the four team columns, median: 80.91 and the ranges of the numbers.
3. In section 5, run the by-hand cells. You should see score of T-116940: 0.184, then 0.999 with the scaler forgotten, then the ValueError for the social ticket.
4. In section 6, predict the median of 20, 60 and 100. Then run the cell. You should see 0.00 for the filled gap.
5. In section 7, run the %%writefile cell. You should see Writing ticket_model.py.
6. In section 8, run the fit cell. Compare the three columns of medians.
7. Do the Your turn task: list the number columns whose median would change if fit used the validation set.
8. Run the check cell. It prints Check passed when your list is right.

Show the answer to step 7

valid_medians = valid[NUMBERS].median()
would_change = sorted(name for name, value in zip(NUMBERS, learned) if value != valid_medians[name])
print(would_change)

The output:

Output

['customer_tenure_days', 'order_value', 'prior_tickets_90d', 'word_count']

Four of the six medians differ. Only priority (3) and created_hour (14) have the same median in both sets. So the medians are a fingerprint of the rows that fit used: they show that fit used train.

Run the notebook on your computer

Download the notebook with View on GitHub, then the download button. Save it in ~/projects/ticket-model/notebooks/. In VS Code, open it and select Select Kernel › Python Environments › .venv. The notebook makes its own folder ticket-model next to it and writes its own copy of ticket_model.py there. Your project’s file stays as you wrote it.

Independent variation: remove the scaler

Change one thing: skip the scaler in the pipeline, during training this time. set_params changes one step of a pipeline without editing the file, and "passthrough" means “skip this step”. The name prepare__numbers__scale is the path to the step: prepare, then its branch numbers, then its step scale. Before you run it, predict: will the score change a lot? Will anything else happen? Section 10 of the notebook has the cell.

Show the answer

import warnings

unscaled = build_pipeline().set_params(prepare__numbers__scale="passthrough")
with warnings.catch_warnings(record=True) as caught_warnings:
    warnings.simplefilter("always")
    unscaled.fit(train[FEATURES], train[TARGET])
for w in caught_warnings:
    print(type(w.message).__name__ + ":", str(w.message).splitlines()[0])
unscaled_scores = unscaled.predict_proba(valid[FEATURES])[:, 1]
print("ROC AUC", round(roc_auc_score(valid[TARGET], unscaled_scores), 3),
      "AP", round(average_precision_score(valid[TARGET], unscaled_scores), 3))

The output:

Output

ConvergenceWarning: lbfgs failed to converge after 1000 iteration(s) (status=1):
ROC AUC 0.739 AP 0.354

The scores hardly change (0.739 and 0.354, against 0.738 and 0.353). But scikit-learn warns: the solver, lbfgs, stopped after 1,000 steps without finding the best weights. This is Problem 3 in action. order_value goes up to $1,450 and customer_tenure_days up to 2,607 days, while the one-hot columns are 0 or 1. On such different scales, the solver needs many more steps.

Compare it with the forgotten scaler earlier. There, the model was trained with the scaler and used without it, and the scores went wrong. Here, the model is trained and used without it, consistently, so the scores are fine but the weights are not the best ones. Keep the scaler. A warning that you ignore today hides a real problem tomorrow.

Common mistake

A pipeline fitted on the wrong rows

Symptom: Tomás reads that a pipeline “keeps preprocessing consistent”. He wants more data for the medians, so he fits it on the train and validation sets together. The validation score goes up:

both = pd.concat([train, valid])
leaky = build_pipeline().fit(both[FEATURES], both[TARGET])
leaky_scores = leaky.predict_proba(valid[FEATURES])[:, 1]
print("ROC AUC on valid:", round(roc_auc_score(valid[TARGET], leaky_scores), 3))
print("AP on valid:", round(average_precision_score(valid[TARGET], leaky_scores), 3))
print("learned tenure median:", leaky.named_steps["prepare"].named_transformers_["numbers"]["impute"].statistics_[3])

The output:

Output

ROC AUC on valid: 0.742
AP on valid: 0.36
learned tenure median: 1191.0

The scores are 0.742 and 0.360, a little better than 0.738 and 0.353. The learned median of customer_tenure_days is 1191, not the train median 1178.

Cause: the pipeline did exactly what it was told. One fit call fits every step on the rows it gets, and here those rows include May. The medians, the scaling and the model’s weights all learned from the validation tickets, with their targets. The validation score now measures how well the model remembers tickets it has seen, not how it works on new ones.

Fix: call fit with the train set only, and use the validation set only to score. To check a fitted pipeline, compare its learned values with the train data, as in the worked example: the medians must equal train[NUMBERS].median().

Consequence in AI work: a pipeline makes consistent mistakes as easily as consistent work. Here the gain looks small, but every decision made with this validation set is now biased toward the model. With a step that learns more from the target, such as the leaky features of Module 6, the false gain can be large.

See it end to endEstimate house pricesCase study

The house-price case study puts one-hot encoding, scaling and a ridge model in one pipeline.

See it end to endSegment online-shop customersCase study

The customer-segments case study puts a log transform, a scaler and k-means in one pipeline.

Knowledge check

Check your understanding

What does fit learn in each of the three preparation steps of build_pipeline(), and how can you show that it learned from the train set only?

Show the answer

OneHotEncoder learns the list of categories of each category column (categories_). SimpleImputer learns the median of each number column (statistics_), and which columns had gaps. StandardScaler learns the mean and SD of each column after imputation (mean_, scale_). To show that fit used train only, compare the learned values with the data. The six learned medians equal train[NUMBERS].median(). Four of them differ from the validation medians, for example customer_tenure_days: 1178 in train, 1342.5 in valid.

Check your understanding

A ticket arrives with region = 'islands', a region that train never saw, and no order value. What does the fitted pipeline do with each of the two values?

Show the answer

With handle_unknown="ignore", the encoder gives islands 0 in all five region columns: “none of the regions I know”. The model scores the ticket with the other features. The missing order_value gets the median learned from train, 86.56, and the indicator missingindicator_order_value is 1. The pipeline learns nothing from the new ticket, and no error stops the service. Module 8 adds a check that counts unknown categories in new data.

Check your understanding

Tomás says: my by-hand code fits each step on train and gives the same validation AP as the pipeline, 0.353. So why do we need a pipeline? Give two reasons, with what you saw in this lesson.

Show the answer

The by-hand code is correct only while every user repeats every step, in the same order, with the same fitted objects. In this lesson, a forgotten scaler gave ticket T-116940 a score of 0.999 instead of 0.184. 875 of 1,179 tickets reached 0.5, with no error. A new channel, social, stopped the by-hand encoder with a ValueError. There are other reasons too. The column order can change, someone can fit a step on the wrong rows, and four loose objects are hard to save and hand over. A pipeline keeps the steps, their order and their learned numbers in one object: one fit on train, then one predict_proba for any rows.

Module practice

The module practice asks you to train a baseline and a linear classifier on prepared data, and to compare them honestly. You did the first part in the results table above. The same pipeline ran once with DummyClassifier and once with LogisticRegression. Both were fitted on train and compared on valid with the rule.

Your task: the model against the rule, at the same workload

The results table compares rankings (ROC AUC, AP), but Grace needs decisions. The rule flags 163 validation tickets. Flag the 163 validation tickets with the highest model scores instead. How many of them were escalated? What are the precision and the recall? Is the model better than the rule at the same workload? Section 12 of the notebook has the task and a check.

Show the model answer

top = scores.argsort()[::-1][:163]
caught_163 = int(valid[TARGET].iloc[top].sum())
precision_163 = round(caught_163 / 163, 3)
recall_163 = round(caught_163 / valid[TARGET].sum(), 3)
print(caught_163, precision_163, recall_163)

The output:

Output

66 0.405 0.361

The comparison at the same workload:

163 flags on the validation set (13.8%)	Escalated among the flags	Precision	Recall
Rule: flag priority 1	61	0.374	0.333
Logistic regression, 163 highest scores	66	0.405	0.361

At the same workload, the model catches 5 more escalations than the rule, with the same number of reviews. That is a real gain, but a modest one. It is still below the improvement target of Define a baseline: catch at least 40% of escalations while flagging at most 20%. With 163 flags, the model is not yet at 20%. Module 5 tries other model families, and Module 6 adds a feature and chooses the threshold for the 20% capacity.

scores.argsort()[::-1] sorts the positions from the highest score to the lowest; [:163] keeps the first 163. Your code can differ, for example valid.assign(score=scores).nlargest(163, "score"). Check the three numbers.

Module check

Check your understanding

Retrieval: name the three preparation steps of build_pipeline(), the columns that each one gets, and the model at the end.

Show the answer

• OneHotEncoder(handle_unknown="ignore") gets the four categories: channel, team, segment and region.
• SimpleImputer(strategy="median", add_indicator=True), then StandardScaler(), get the six numbers: priority, order_value, word_count, customer_tenure_days, prior_tickets_90d and created_hour.
• A ColumnTransformer joins the two branches, and the model at the end is LogisticRegression(max_iter=1000). The result has 27 columns.

Check your understanding

Predict and explain: a new ticket of 2,000 words arrives. What does the straight line of Classification intuition predict for its chance of escalation on word_count, what does the logistic regression predict, and which one can Grace use as a score?

Show the answer

The straight line predicts 0.0488 + 0.00064 × 2,000 = 1.335: a “chance” above 1, which is impossible. The logistic regression bends the same kind of weighted sum with the S-curve and gives 0.999, which stays between 0 and 1. Grace can use only the logistic regression’s number as a score. Even then, one feature that grows without a limit gives extreme tickets extreme scores. The model with all ten features, prepared by the pipeline, ranks the tickets much better (ROC AUC 0.738).

Check your understanding

Completion check: reproduce the results of this module and confirm that the preprocessing was fitted only on the training data. Write down the numbers you got and how you confirmed it.

Show the answer

Procedure (in the lesson notebook, or in your own notebook in ~/projects/ticket-model with the .venv kernel):

1. Restart the kernel and run all cells (in Colab: Runtime › Restart session and run all). A fresh start proves that no cell depends on an old value.
2. Section 8 must print ROC AUC on valid: 0.738 and AP on valid: 0.353. Section 9 must print the three lines of the results table. Majority: ROC AUC 0.500 and AP 0.155. Rule: 163 flagged, precision 0.374, recall 0.333. Logistic: 18 flagged at 0.5.
3. Run section 13. It compares the fitted pipeline with the data:

numbers_step = prepare.named_transformers_["numbers"]
filled = numbers_step["impute"].transform(train[NUMBERS])
print("medians equal train medians:", np.allclose(numbers_step["impute"].statistics_, train[NUMBERS].median()))
print("means equal train means:", np.allclose(numbers_step["scale"].mean_, filled.mean(axis=0)))
print("medians equal valid medians:", np.allclose(numbers_step["impute"].statistics_, valid[NUMBERS].median()))

The output:

Output

medians equal train medians: True
means equal train means: True
medians equal valid medians: False

What a complete answer says:

• The reproduced numbers: 0.738 and 0.353, and the three baseline lines.
• The learned medians equal the train medians (3, 86.56, 89, 1178, 5, 14). The scaler’s means equal the means of the filled train columns.
• The medians differ from the validation medians (for example 1342.5 for customer_tenure_days). So the validation rows did not reach fit.

The categories are a weaker check here, because the validation set has the same categories as train.

Reset to a known state

If something in your project is broken or changed, get back to the state at the end of this module:

1. Open a terminal in ~/projects/ticket-model and activate .venv (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
2. Run python get_data.py --reset. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
3. Replace ticket_model.py with the code in Write ticket_model.py. Check it with python -c "import ticket_model; print(len(ticket_model.FEATURES), 'features')". You should see 10 features.
4. If .venv is broken, delete the folder .venv and make it again, as in Define a baseline: Set up your project.
5. For a lesson notebook, download a fresh copy from its View on GitHub link, and run it from the top.

From Module 6, Select useful features, on, ticket_model.py has one more feature. Then the finished file in C05/project is the right one to copy.

Recap

In this module you built the first models:

• Regression: a linear regression, like a taxi fare, predicts the tickets per day. Least squares finds the line. The season model reaches a test MAE of 6.45 against 14.95 for the constant. Its residuals stay above zero in May and June (+4.98) because the shop grows, and a trend feature removes most of that bias.
• Classification: a straight line cannot give a chance (1.335 for 2,000 words). Logistic regression bends the weighted sum with the S-curve into a score from 0 to 1. predict uses 0.5 and flags almost nothing on this data; use predict_proba and a threshold that fits the decision.
• Preparation: text, gaps and scales each need a step. Each step learns numbers from train only, and you must apply them the same way to every ticket. By hand, a forgotten scaler gave a score of 0.999 instead of 0.184, and a new channel stopped the code. One pipeline in ticket_model.py holds the steps and the model, and fit used the train set only: the learned medians equal the train medians.
• Results on valid: the model ranks better than the rule (ROC AUC 0.738 and AP 0.353, against 0.615 and 0.228). At the rule’s workload of 163 flags, it catches 66 escalations instead of 61.

Next, Module 5 shows that machine learning is a loop. You try other model families on the same pipeline and the same validation set. You measure each one the same way, and keep what really helps.

Next: Machine learning is iterative