Select useful features

What you will be able to do

Find the columns that make the model better, and stop the columns that only make it look better. A column that leaks the answer gives the best score in every check before launch, and fails only with real tickets.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install the packages, download the practice data and write ticket_model.py as it was after Module 4. You change it to the final version in the notebook. It works without the local project.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Cross-validation on the train set, the mean ± standard deviation of the folds, and why a difference smaller than that spread is not a real difference, from Train, validation and test sets: Cross-validation.
• Average precision, the score that this course uses to compare models, from The precision–recall trade-off: Average precision.
• The experiment log and its helpers log_experiment() and show_log(), from Machine learning is iterative: The experiment log. This lesson adds the rows E15 to E18.
• Learning curves, from Control complexity.
• Leakage: a feature that is not known at prediction time, from Features and target timing: What each column knows in SQL and Data Preparation for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, active in a terminal, and the lesson notebooks open in VS Code or JupyterLab. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv, data/valid.csv, data/test.csv and data/extra_features.csv, from python get_data.py in Define a baseline: Set up your project. If a file is missing or changed, run python get_data.py --reset.
• experiment_log.csv with the rows E01 to E14, from Ensembles. If it is missing, the notebook’s setup cells write the rows again.
• ticket_model.py with 6 numbers in NUMBERS, from Preprocess with a pipeline: Write ticket_model.py. This lesson changes it in Change ticket_model.py. If your file is broken, write it again from that lesson, or download the final version as step 8 of Change ticket_model.py shows.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

The answer key on the desk

A student prepares for a test with a book of practice exercises. The answer key is printed at the back of the book. If she reads the answer before she solves each exercise, she gets every exercise right at home. Her practice score is perfect, and it tells her nothing. In the exam, there is no answer key, and her score falls.

A model can do the same thing. If one of its columns secretly contains the answer, the model learns to read that column. Every score that you compute from the same data agrees with it: train, cross-validation, even the validation month. Only real new tickets show the truth, because the column has no answer in it yet when a new ticket arrives.

Module 5 tried other model families. The trees lost to logistic regression, and the forests and boosting only tied it. The learning curves showed that more tickets will not help much either. So the next idea is better information: new columns. Mei, the data engineer, offers four. Each one is a candidate feature, a column that could become a feature if it passes a test. This lesson is that test.

You have met one of the four already. The points score of Module 3 gives a point for an earlier escalation of the customer, and that fact comes from prior_escalations_90d. The model has never used it. Now it must earn its place in the model in the same way as the other three.

Three questions for every candidate

A new column can go wrong in three ways. It can carry the answer, like the answer key. It can repeat a column that the model has already, like a second copy of the same notes. Or it can add nothing that you can measure. So Priya asks three questions, in this order:

1. Is it known when the ticket is created? The model predicts at that moment, the prediction time. A value that is written later is not available then. If a column holds information from after the prediction time, it leaks.
2. Does it add information that the model does not have yet? If it repeats another feature, it is redundant.
3. Does cross-validation on train say that it helps, by more than the noise? If not, leave it out. Every extra column must be collected, checked and explained, so it must earn its place.

So every candidate is one of three kinds. A candidate that passes all three questions is useful. One that fails question 2 or 3 is redundant: it adds nothing that you can measure. Usually it repeats a column that the model has. One that fails question 1 is leaking: it makes the model look better than it can be. Only a useful candidate is kept. Leakage is the name for this problem: information in the features that the model would not have at prediction time. You met it in the SQL course, in Features and target timing.

Question 1 comes first for a reason. Questions 2 and 3 use scores, and a leak wins every score. Only a question about time can catch it.

Eight tickets, by hand

Here are eight made-up tickets with three of the candidates. priority is the priority when the ticket was created. Before you read the notes under the table, try to answer the three questions for each candidate yourself.

Ticket	priority	order_value	order_value_band	prior_escalations_90d	priority_now	escalated
T-01	2	35.00	low	1	1	1
T-02	3	120.00	mid	0	3	0
T-03	1	80.00	mid	2	1	1
T-04	2	250.00	high	0	2	0
T-05	3	45.00	low	1	1	1
T-06	2	(empty)	none	0	2	0
T-07	1	60.00	mid	0	1	0
T-08	3	20.00	low	1	3	0

• prior_escalations_90d counts the customer’s escalations in the 90 days before this ticket. Four tickets have 1 or more (T-01, T-03, T-05, T-08), and 3 of them escalated. Four have 0, and none escalated. The count uses only earlier tickets, so it is known at creation. No other column has this information. Useful.
• order_value_band is low exactly when order_value is under 50 (T-01, T-05, T-08), none when it is empty, and so on. The model has order_value already, so the band adds nothing new. Redundant.
• priority_now is 1 for T-01, T-03, T-05 and T-07, and 3 of those 4 escalated. That is as good as the history column, so the rates alone do not show a problem. Question 1 does. T-01 was created with priority 2, and its priority became 1 when it escalated. The column records the answer. Leaking.

The fourth candidate, customer_escalation_rate, is the customer’s share of escalated tickets. Its problem is in which tickets it counts, and you find it on the real data below.

The picture puts each column on a time line. The model predicts when the ticket is created, at the blue line. It can use only what is written to the left of that line.

When each candidate column is written. Only the green columns exist when the model must predict. The amber columns are written later, so they leak.

Text description of the diagram

A horizontal time line runs from left to right (top to bottom on a phone). It has four marks: 90 days before the ticket, the moment the ticket is created, 72 hours later, and the data export. A dashed blue line at “ticket created” says that the model predicts there. The area to the left is labelled “Known when the ticket is created”, the area to the right “Written later”.

In the left area are two green boxes. prior_escalations_90d sits above a bracket that covers the 90 days before creation, because it counts the customer’s escalations in that window. priority sits at the moment of creation. In the right area are two amber boxes. priority_now sits at “72 hours later”, because the help desk changes the priority when it escalates a ticket. customer_escalation_rate sits at “data export”, because it is computed over all labelled tickets when the data is exported.

The four candidates on the real tickets

extra_features.csv has the four candidates, one row per labelled ticket. The code in this lesson starts with the 10 features of ticket_model.py, as you wrote it in Module 4. It also needs these imports (the notebook’s first cell has them all):

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

from ticket_model import CATEGORIES, NUMBERS, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

cv is the course’s cross-validation: 5 folds with the same escalation share in each fold. The fixed shuffle makes the folds the same on every run.

Join the candidates to train and valid by ticket_id. validate="one_to_one" stops the merge if a ticket appears twice:

extra = pd.read_csv("data/extra_features.csv")
candidates = train.merge(extra, on="ticket_id", how="left", validate="one_to_one")
candidates_valid = valid.merge(extra, on="ticket_id", how="left", validate="one_to_one")
print(len(extra), "rows in extra_features.csv; train after the merge:", len(candidates))
print(candidates[["ticket_id", "prior_escalations_90d", "order_value_band",
                  "customer_escalation_rate", "priority_now", TARGET]].head(4).to_string(index=False))

The output:

Output

19120 rows in extra_features.csv; train after the merge: 16939
ticket_id  prior_escalations_90d order_value_band  customer_escalation_rate  priority_now  escalated_72h
 T-100001                      0              mid                    0.0000             3              0
 T-100002                      0              low                    0.0541             1              0
 T-100003                      0             high                    0.0333             3              0
 T-100004                      0              mid                    0.2671             1              1

Now look at each candidate with one small question.

Does more history mean more risk? (5 means “5 or more”.)

history = candidates["prior_escalations_90d"].clip(upper=5)  # 5 means "5 or more"
print(candidates.groupby(history)[TARGET].agg(["size", "mean"]).round(3))

Output

                       size   mean
prior_escalations_90d
0                      9130  0.091
1                      3598  0.119
2                      1674  0.144
3                       902  0.144
4                       552  0.152
5                      1083  0.252

Yes. Tickets from customers with no recent escalation escalate 9.1% of the time. With 5 or more, they escalate 25.2% of the time. The column counts only earlier tickets, so it is known at creation. Guests always have 0, because they have no history.

Where do the bands come from?

print(candidates.groupby("order_value_band")["order_value"].agg(["size", "min", "max"]))
print("tickets with no order_value:", candidates["order_value"].isna().sum())

Output

                  size     min      max
order_value_band
high              2456  200.05  1450.00
low               3791    4.50    49.98
mid               8230   50.00   199.98
none              2462     NaN      NaN
tickets with no order_value: 2462

The bands do not overlap: under 50, 50 to 199.99, 200 and more, and none for the 2,462 tickets with no order value. The band is order_value cut into four pieces.

What does priority_now say about escalated tickets?

print(pd.crosstab(candidates["priority_now"], candidates[TARGET]))

Output

escalated_72h     0     1
priority_now
1              1831  1983
2              5140     0
3              7985     0

All 1,983 escalated train tickets have priority_now 1, and no ticket with 2 or 3 escalated. The help desk sets the priority to 1 when it escalates a ticket, as you saw in the SQL course.

Over which tickets is customer_escalation_rate computed? Check one customer, C-00078:

everything = pd.concat([load("train"), load("valid"), load("test")])
customer = "C-00078"
print("customer_escalation_rate in the file:",
      candidates.loc[candidates["customer_id"] == customer, "customer_escalation_rate"].iloc[0])
print("rate over all labelled tickets:      ", round(everything.loc[everything["customer_id"] == customer, TARGET].mean(), 4))
print("rate over the train tickets only:    ", round(train.loc[train["customer_id"] == customer, TARGET].mean(), 4))

Output

customer_escalation_rate in the file: 0.0959
rate over all labelled tickets:       0.0959
rate over the train tickets only:     0.0769

The value in the file is the rate over all labelled tickets: 7 escalations in 73 tickets, including May and June and each ticket’s own target. On 1 May, Larkfield knew only the train tickets: 5 in 65, which is 0.0769. So the column contains the future. It is the leaky aggregate from Fit transformations correctly in the SQL course, with a new name.

Worked example: decide with CV on train

Tomás has a fast idea: “Add all four columns and let the model decide which ones matter.” The problem is that the model cannot tell a leak from a good feature. It sees only that priority_now predicts the target almost perfectly, so it trusts that column most. The model will choose the leak every time. People must make this choice, with the three questions.

Priya tests each candidate in the same way: the 10 features plus that one column, logistic regression, and the same 5 folds of train. Two helpers do the work. pipeline_for() builds the same steps as build_pipeline(), for other lists of columns. cv_folds() returns the average precision (AP) on each fold:

def pipeline_for(categories, numbers):
    """The steps of build_pipeline(), for other lists of columns."""
    number_steps = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                             ("scale", StandardScaler())])
    prepare = ColumnTransformer([("categories", OneHotEncoder(handle_unknown="ignore"), categories),
                                 ("numbers", number_steps, numbers)])
    return Pipeline([("prepare", prepare), ("model", LogisticRegression(max_iter=1000))])


def cv_folds(data, categories, numbers):
    """The AP on each of the 5 folds of train, for logistic regression with these columns."""
    return cross_val_score(pipeline_for(categories, numbers), data[categories + numbers], data[TARGET],
                           cv=cv, scoring="average_precision")

order_value_band is text, so it goes with the categories. The other three are numbers:

tests = {
    "(nothing)": (CATEGORIES, NUMBERS),
    "prior_escalations_90d": (CATEGORIES, NUMBERS + ["prior_escalations_90d"]),
    "order_value_band": (CATEGORIES + ["order_value_band"], NUMBERS),
    "customer_escalation_rate": (CATEGORIES, NUMBERS + ["customer_escalation_rate"]),
    "priority_now": (CATEGORIES, NUMBERS + ["priority_now"]),
}
folds = {added: cv_folds(candidates, *columns) for added, columns in tests.items()}
for added, fold_scores in folds.items():
    gain = fold_scores - folds["(nothing)"]
    print(f"+ {added:25s} CV AP {fold_scores.mean():.3f} ± {fold_scores.std():.3f}   gain per fold {gain.round(3)}")

Before you read the output, predict: which candidate adds the most? Which result looks too good to be true?

Output

+ (nothing)                 CV AP 0.327 ± 0.012   gain per fold [0. 0. 0. 0. 0.]
+ prior_escalations_90d     CV AP 0.339 ± 0.016   gain per fold [0.01  0.01  0.005 0.025 0.013]
+ order_value_band          CV AP 0.328 ± 0.014   gain per fold [ 0.007 -0.005  0.004  0.001  0.001]
+ customer_escalation_rate  CV AP 0.445 ± 0.015   gain per fold [0.12  0.126 0.087 0.131 0.126]
+ priority_now              CV AP 0.851 ± 0.024   gain per fold [0.528 0.544 0.489 0.558 0.503]

Priya reads it in three steps.

First, the honest candidates. prior_escalations_90d adds 0.012 to the mean. That is about the size of the standard deviation (0.012 to 0.016), so the means alone cannot tell. But all five models use the same folds, and some folds are harder than others for every model. So most of the spread is shared, and a fairer comparison is fold by fold: the gain per fold. prior_escalations_90d helps on all 5 folds, by 0.005 to 0.025. A gain on every fold is a real difference. order_value_band helps on four folds and hurts on one, by 0.002 on average. That is no real difference, as you would expect from a column that repeats order_value.

Second, the jumps. One extra column takes the AP from 0.327 to 0.445, and another takes it to 0.851. An honest feature moved it by about 0.01. In the whole of Module 5, trees and ensembles did not raise it at all. “Too good to be true” is the signal for a leak. It is a signal, not a proof.

Third, question 1 decides. customer_escalation_rate uses later targets. priority_now is set by the escalation itself. Neither is known when a ticket is created.

The chart shows both steps. The upper panel puts the five results on one scale, so the two leaks stand far away from the rest. The lower panel zooms in on the two honest candidates and shows the gain on each fold.

The four candidates, tested with the same 5 folds of train. The two leaks are far above every honest result. Of the honest candidates, only prior_escalations_90d gains on every fold.

Text description of the chart

The upper panel has five rows on an axis from about 0.3 to 0.9, with a dashed line at 0.327 for the 10 features. Nothing added: 0.327 (0.315 to 0.339). Plus prior_escalations_90d: 0.339 (0.323 to 0.355). Plus order_value_band: 0.328 (0.314 to 0.342). Plus customer_escalation_rate: 0.445 (0.430 to 0.460). Plus priority_now: 0.851 (0.827 to 0.875). The first three rows overlap near the dashed line; the last two, in orange, stand far to the right.

The lower panel has two rows on an axis from about −0.01 to +0.03, with a dashed line at zero. Plus prior_escalations_90d: mean gain +0.013, from +0.005 to +0.025, so the whole bar is right of zero. Plus order_value_band: mean gain +0.002, from −0.005 to +0.007, so the bar crosses zero.

So Priya’s decision is to keep one candidate and reject three. The table sums it up, with the kind of each candidate and the reason for the decision:

Added to the 10 features	CV AP	Kind	Decision
(nothing)	0.327 ± 0.012
prior_escalations_90d	0.339 ± 0.016	Useful	keep: known at creation, and it helps on every fold
order_value_band	0.328 ± 0.014	Redundant	reject: it repeats order_value, and the gain is smaller than the noise
customer_escalation_rate	0.445 ± 0.015	Leaking	reject: it uses later targets
priority_now	0.851 ± 0.024	Leaking	reject: the escalation sets it

A rejected leak is different from a rejected redundant column. You may test a redundant column again one day, for example with another model. A leaking column stays out for good, because it can never be known at prediction time.

Confirm on the validation month

The decision is made, with train only. Now Priya checks it once on May, the validation set:

for added, (categories, numbers) in tests.items():
    pipeline = pipeline_for(categories, numbers).fit(candidates[categories + numbers], candidates[TARGET])
    scores = pipeline.predict_proba(candidates_valid[categories + numbers])[:, 1]
    print(f"+ {added:25s} valid AP {average_precision_score(valid[TARGET], scores):.3f}"
          f"  ROC AUC {roc_auc_score(valid[TARGET], scores):.3f}")

Output

+ (nothing)                 valid AP 0.353  ROC AUC 0.738
+ prior_escalations_90d     valid AP 0.391  ROC AUC 0.752
+ order_value_band          valid AP 0.358  ROC AUC 0.738
+ customer_escalation_rate  valid AP 0.485  ROC AUC 0.812
+ priority_now              valid AP 0.873  ROC AUC 0.981

Omar asks: “Why not just choose by May? It is the newest month.” There are two reasons for the order, CV first and May after:

• May must stay an independent check. If you choose by the May score, May becomes part of the choice. Try enough candidates, and one of them wins on May by chance. Then nothing is left to tell you that it was chance.
• May is small. It has 183 escalations, so its AP moves by about 0.03 by chance. CV uses all 16,939 train tickets and shows its spread on every fold.

May agrees: prior_escalations_90d raises the valid AP from 0.353 to 0.391, and ROC AUC from 0.738 to 0.752. But look at the leaks: they look great on May too, because the leaked information is in the May rows as well. A validation set cannot catch a leak that is in its own columns. Only the question “when is this value written?” catches it.

Write it in the experiment log

In Machine learning is iterative, you started an experiment log, experiment_log.csv. It has one row for each experiment, with its CV score, the decision and the reason. Module 5 ended at E14, with logistic regression on the 10 features (E03) as the model to keep. A log stops you from trying the same idea twice, and it lets Grace or a reviewer see why the model is what it is.

A rejected idea belongs in the log too. Next month, someone will suggest priority_now again, and the log already says why not. So Priya adds all four candidates, E15 to E18, with the fold scores from the worked example. She fills valid_ap because she checked each candidate once on May:

for row_id, added, valid_ap, decision, reason in [
    ("E15", "prior_escalations_90d", 0.391, "keep", "Known at creation, and higher than E03 on all 5 folds."),
    ("E16", "order_value_band", 0.358, "reject", "It repeats order_value; the gain is smaller than the noise."),
    ("E17", "customer_escalation_rate", 0.485, "reject", "Leak: computed over later targets, so not known at creation."),
    ("E18", "priority_now", 0.873, "reject", "Leak: the help desk sets it to 1 when a ticket escalates."),
]:
    change = f"E03 + {added}" + (" (11 features)" if row_id == "E15" else "")
    log_experiment(row_id, change, folds[added].mean(), folds[added].std(), valid_ap, decision, reason)
show_log()

The last rows of the output:

Output

E13                   Gradient boosting, default settings  0.316   0.010    0.369   reject
E14    Gradient boosting, max_depth=3, learning_rate=0.05  0.327   0.016    0.374   reject
E15             E03 + prior_escalations_90d (11 features)  0.339   0.016    0.391     keep
E16                                E03 + order_value_band  0.328   0.014    0.358   reject
E17                        E03 + customer_escalation_rate  0.445   0.015    0.485   reject
E18                                    E03 + priority_now  0.851   0.024    0.873   reject

E15 is the new model to keep, in place of E03. The log uses the same two decisions as in Module 5, keep and reject, and the reason says which kind of candidate it was. Note the two leaks: they have the best scores in the whole log, and the decision is reject. The reason column is what makes that clear to a reader. If your log is missing in the notebook, its setup cells write E01 to E14 again.

Change ticket_model.py

The decision is to add prior_escalations_90d. ticket_model.py needs two changes: NUMBERS gets a seventh entry, and load() merges the column from extra_features.csv. After that, every later lesson, train.py, evaluate.py and predict.py get the 11 features without any change of their own. Here is the change as a diff: lines with - go, and lines with + come.

ticket_model.py (the change)

 """The shared parts of Larkfield's escalation model: the columns, how to load
 a part of the data, and the preprocessing-plus-model pipeline.

Nextia Learning, C05, Module 4, lesson 3 (Preprocess with a pipeline).
Nextia Learning, C05. You write this file in Module 4, lesson 3 (Preprocess
with a pipeline), and add prior_escalations_90d in Module 6, lesson 1 (Select
useful features). train.py, evaluate.py and predict.py import it.
 """
@@
 NUMBERS = ["priority", "order_value", "word_count", "customer_tenure_days", "prior_tickets_90d",
           "created_hour"]
           "created_hour", "prior_escalations_90d"]
@@
 def load(part):
    """One part of the split: train, valid or test."""
    return read(DATA / f"{part}.csv")
    """One part of the split (train, valid or test), with the extra feature
    prior_escalations_90d from extra_features.csv."""
    df = read(DATA / f"{part}.csv")
    extra = pd.read_csv(DATA / "extra_features.csv", usecols=["ticket_id", "prior_escalations_90d"])
    return df.merge(extra, on="ticket_id", how="left", validate="one_to_one")

Each change has a reason:

Change	Why
"prior_escalations_90d" at the end of NUMBERS	It is a count, so it goes through the median imputer and the scaler, like the other numbers. FEATURES is CATEGORIES + NUMBERS, so it now has 11 entries.
usecols=["ticket_id", "prior_escalations_90d"]	Read only the column that you chose. The two leaks never enter the data that the model sees.
how="left"	Keep every ticket of the part, even one with no row in extra_features.csv.
validate="one_to_one"	Stop with an error if a ticket appears twice in either file, instead of quietly repeating rows.
The docstring	Says which lessons wrote the file. The file now equals the course’s finished ticket_model.py.

Here is the whole file after the change:

ticket_model.py

"""The shared parts of Larkfield's escalation model: the columns, how to load
a part of the data, and the preprocessing-plus-model pipeline.

Nextia Learning, C05. You write this file in Module 4, lesson 3 (Preprocess
with a pipeline), and add prior_escalations_90d in Module 6, lesson 1 (Select
useful features). train.py, evaluate.py and predict.py import it.
"""

from pathlib import Path

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

DATA = Path("data")
CATEGORIES = ["channel", "team", "segment", "region"]
NUMBERS = ["priority", "order_value", "word_count", "customer_tenure_days", "prior_tickets_90d",
           "created_hour", "prior_escalations_90d"]
FEATURES = CATEGORIES + NUMBERS
TARGET = "escalated_72h"


def read(path):
    """Read a ticket CSV file. Only an empty cell is missing: the text
    "unknown" stays a category."""
    return pd.read_csv(path, parse_dates=["created_at"], dtype={"customer_id": "string"},
                       keep_default_na=False, na_values=[""])


def load(part):
    """One part of the split (train, valid or test), with the extra feature
    prior_escalations_90d from extra_features.csv."""
    df = read(DATA / f"{part}.csv")
    extra = pd.read_csv(DATA / "extra_features.csv", usecols=["ticket_id", "prior_escalations_90d"])
    return df.merge(extra, on="ticket_id", how="left", validate="one_to_one")


def build_pipeline(model=None):
    """Preprocessing and model in one object. fit() learns the medians, the
    scaling and the categories from the rows it is given, and nothing else."""
    numbers = Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
    ])
    prepare = ColumnTransformer([
        ("categories", OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
        ("numbers", numbers, NUMBERS),
    ])
    return Pipeline([("prepare", prepare), ("model", model if model is not None else LogisticRegression(max_iter=1000))])

1. Open ticket_model.py in your project folder (in the notebook: the %%writefile ticket_model.py cell).

2. Add "prior_escalations_90d" at the end of NUMBERS, after "created_hour".

3. In load(), replace the return line with the three lines of the diff. Update the two docstrings.

4. Save the file.

5. Restart the notebook’s kernel, so that Python reads the new file. (The notebook does this with importlib.reload(ticket_model) instead.)

6. Run this check:

   from ticket_model import FEATURES, TARGET, build_pipeline, load
   
   train, valid = load("train"), load("valid")
   pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
   scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
   print(len(FEATURES), "features:", FEATURES)
   print("valid AP", round(average_precision_score(valid[TARGET], scores), 3),
         " ROC AUC", round(roc_auc_score(valid[TARGET], scores), 3))

   You should see:

   Output

   11 features: ['channel', 'team', 'segment', 'region', 'priority', 'order_value', 'word_count', 'customer_tenure_days', 'prior_tickets_90d', 'created_hour', 'prior_escalations_90d']
   valid AP 0.391  ROC AUC 0.752

7. If you see 10 features, the kernel still has the old file: restart it and run the check again. If you see MergeError, a ticket appears twice: run python get_data.py --reset.

8. If your file is still broken, replace it with the course’s finished file. This overwrites your version. Then run the check again.

   macOS and Linux

   Terminal

   curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py

   Windows

   PowerShell

   curl.exe -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py

This is the model that the rest of the course uses: logistic regression with its default settings, on 11 features, fitted on train only. Valid AP 0.391, ROC AUC 0.752. The next lesson asks if other settings would make it better.

Your turn: remove a feature instead

Change one condition: remove a feature instead of adding one. Selection works in both directions. Test region and channel, each removed from the 11 features, with the same folds. Before you run it, predict: will the model lose anything without region? Without channel?

all_folds = cv_folds(train, CATEGORIES, NUMBERS)
print(f"all {len(FEATURES)} features      CV AP {all_folds.mean():.3f} ± {all_folds.std():.3f}")
for removed in ["region", "channel"]:
    fold_scores = cv_folds(train, [c for c in CATEGORIES if c != removed], [n for n in NUMBERS if n != removed])
    change = fold_scores - all_folds
    print(f"without {removed:10s} CV AP {fold_scores.mean():.3f} ± {fold_scores.std():.3f}   change per fold {change.round(3)}")

Import CATEGORIES and NUMBERS again after you change ticket_model.py, so that they have the 11 features.

Show the answer

Output

all 11 features      CV AP 0.339 ± 0.016
without region     CV AP 0.340 ± 0.017   change per fold [-0.001 -0.    -0.002  0.005  0.002]
without channel    CV AP 0.332 ± 0.015   change per fold [-0.001 -0.007 -0.011 -0.006 -0.01 ]

• Without region: the change goes up and down, by 0.001 on average. The model does not need region. By question 3, you may remove it, and the model would be a little simpler.
• Without channel: the AP is lower on every fold, by 0.007 on average. channel carries information. Keep it.

Priya keeps region anyway, and writes down why in the log. It costs nothing to collect, Grace’s reports use it, and the slice analysis of Module 7 compares regions. Removing it changes nothing that she can measure. A selection decision is a number and a reason, both written down.

The leak that validation could not catch

Common mistake

The leak that validation could not catch

Symptom: Tomás adds priority_now and reports the best model of the project: CV AP 0.851, valid AP 0.873, ROC AUC 0.981. “It even works on May, the month it never saw.”

Cause: priority_now is the priority at the time of the export. The help desk sets it to 1 when it escalates a ticket, so the column records the target. It leaks in the May rows too, so May cannot catch it. When a new ticket is created, its priority_now equals its priority. Priya scores May with the values that a new ticket really has:

categories, numbers = tests["priority_now"]  # the 10 features + priority_now
columns = categories + numbers
leaky = pipeline_for(categories, numbers).fit(candidates[columns], candidates[TARGET])
reported = leaky.predict_proba(candidates_valid[columns])[:, 1]
at_creation = candidates_valid.assign(priority_now=candidates_valid["priority"])  # what a new ticket has
real = leaky.predict_proba(at_creation[columns])[:, 1]
print("valid AP as reported:               ", round(average_precision_score(valid[TARGET], reported), 3))
print("valid AP with the value at creation:", round(average_precision_score(valid[TARGET], real), 3))

Output

valid AP as reported:                0.873
valid AP with the value at creation: 0.353

The report says 0.873. In real use, the same model gets 0.353: no better than the model without the column. The model learned to trust priority_now, and at creation that column carries no information about the outcome. customer_escalation_rate does the same on a smaller scale. Its valid AP is 0.485 as reported, and 0.396 when the May tickets get the rate that Larkfield knew on 1 May.

Fix: for every candidate, ask when its value is written and how it was computed. Read the data card: it says that both columns leak. Treat any large jump from one column as a leak until you can prove the timing. Keep the leaks out of the code too: load() reads only prior_escalations_90d from extra_features.csv.

Consequence in AI work: a leak gives the best number in every report: on train, in CV and on the validation set. It fails only in production. Grace would plan her senior team around a model that seems to catch most escalations. In real use, it would catch a fraction of them. A score that is too good to be true is the most expensive mistake in this course, because every check before launch agrees with it.

See it end to endRank a call listCase study

The call-list case study finds the call-duration leak, a column that is known only after the call.

See it end to endForecast bike-share demandCase study

The bike-share case study finds two columns that add up to the target.

Knowledge check

Check your understanding

Priya decided with CV on train first, and looked at the validation month only after. Give two reasons for this order.

Show the answer

First, the validation month must stay an independent check. If you choose by it, it becomes part of the choice, and with enough candidates one of them wins on May by chance. Second, May is small: with 183 escalations, its AP moves by about 0.03 by chance. CV uses all 16,939 train tickets and shows its spread on every fold.

Check your understanding

A colleague offers minutes_to_first_reply. With it, the CV AP goes from 0.339 to 0.62, and the valid AP rises too. What do you conclude, and what do you check?

Show the answer

A jump of almost 0.3 from one column, when honest features add about 0.01, is too good to be true: suspect a leak. Check when the value is written. The first reply comes after the ticket is created. Escalated tickets get fast replies from senior agents, so the column carries a trace of the target. It is not known at prediction time, so never use it. The rise on May proves nothing, because the May rows contain the same leak.

Check your understanding

Tomás says: let the model choose among all four candidates, because it will give a useless column a weight near zero. Why does this not protect you from a leak?

Show the answer

The model judges a column only by how well it predicts the target in the data that it gets. A leak predicts the target very well, because it contains the answer, so the model gives it a large weight, not a small one. The model has no way to know when a value was written. Only a person who asks “is this known at prediction time?” can remove a leak, before the model ever sees it.

Recap

• Every candidate feature must pass three questions, in order: known at prediction time, new information, and a gain in CV larger than the noise.
• prior_escalations_90d is useful: CV AP 0.327 to 0.339, higher on every fold. order_value_band is redundant: it repeats order_value. customer_escalation_rate and priority_now are leaking. Keep the useful one; reject the others, with the reason in the log.
• A leak looks too good in every check, including the validation month. Ask when each value is written; the model cannot ask it for you.
• Decide with CV on train, then confirm once on the validation month, and write every decision and its reason in the experiment log.
• ticket_model.py now has 11 features: valid AP 0.391, ROC AUC 0.752.

Next, you keep the features fixed and ask about the model’s settings: are the default settings the best ones?

Next: Tune hyperparameters