Rank a call list

What you will be able to do

Rank a list of clients so that a call centre with room for only 10% of the calls reaches the most likely buyers first. You will find a column that looks perfect and cannot be used, and choose a tree that the team can read. Then you measure how much better than chance it is, and see why it needs watching after launch.

Case studyPortfolio piece

Dataset

Bank Marketing, Portugal 2008–2010 (UCI)

Licence

CC BY 4.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Leak detection
• Decision tree tuning
• Lift at capacity
• Readable rules
• Drift

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install pandas and scikit-learn and download the data (about 0.3 MB), so it needs no local setup. It runs in about one minute.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• Precision at a fixed number of flags, from The precision–recall trade-off, and average precision, from its section Average precision.
• How a tree splits and why a tree with no limit memorises, from Decision trees and Control complexity.
• Leaks, from Select useful features.
• Grid search and the one-standard-error rule, from Tune hyperparameters.
• Monitoring and retraining, from Plan production checks.

What you need on your computer (only for the local route; the notebook needs none of it)

• A folder with a virtual environment that has pandas 3.0.6, scikit-learn 1.9.1 and matplotlib, as in Define a baseline: Set up your project. The ticket-model project works; this case study does not change its files.
• The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell downloads the data into a new folder call-list. To start again, delete that folder.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1, in Jupyter from a fresh start. Windows and Linux are not tested.

The situation

Inês Duarte runs the call centre of a Portuguese bank. Her agents sell a term deposit: the client puts money in the bank for a fixed time and gets interest. This campaign’s list has thousands of clients, and her team has time to call only 1 in 10 of them. A call that ends with “no” costs about ten minutes of an agent’s time. A client who would have said yes but is never called is a deposit that the bank does not get.

So Inês does not need a yes/no answer for each client. She needs the list in order, the most likely clients first, so that the calls her team can make find as many yeses as possible. Her analyst, Tiago, builds the model. She has one more wish: “I want to understand why the list has this order. My team will not trust a black box.”

What you will build

You will build a decision tree that scores every client on the list. You will judge it the way Inês will use it: by the share of yeses among the top 10% of the list. Along the way you find a leak and compare trees of many sizes in an experiment log. You choose the simplest tree that is as good as the best at the team’s capacity, read it as rules, and test it once. Then you look at what happens when the economy changes, and plan the monitoring.

The skills come from the whole course: framing (Translate a need), baselines (Define a baseline), precision at a capacity, and trees. Leaks, tuning, slices (Slice analysis) and production checks come in too.

The data

The data is the Bank Marketing dataset from the UCI Machine Learning Repository, in its larger version, bank-additional-full. It has 41,188 real, anonymous phone calls of a Portuguese bank, from May 2008 to November 2010, and whether each client subscribed.

Source and licence

“Bank Marketing” by S. Moro, P. Rita and P. Cortez, UCI Machine Learning Repository, doi.org/10.24432/C5K306, licensed under CC BY 4.0. Nextia Learning saved the file bank-additional-full.csv with commas instead of semicolons and compressed it; nothing else changed. Paper: Moro, S., Cortez, P. and Rita, P. (2014), “A data-driven approach to predict the success of bank telemarketing”, Decision Support Systems 62, 22–31.

The notebook’s setup cell downloads the file and checks its checksum. Tiago loads it:

raw = pd.read_csv("bank_additional_full.csv.gz")
print(raw.shape)
print(raw.columns.tolist())

The output:

(41188, 21)
['age', 'job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays', 'previous', 'poutcome', 'emp.var.rate', 'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed', 'y']

What this tells us. One row is the last call to one client in one campaign. The columns describe the client, the call, earlier campaigns and the economy on the day of the call. The target y says whether the client subscribed.

The data dictionary

These are the columns that the case study uses, in plain words:

Column	What it means	Use
age, job, marital, education	Facts about the client	Inputs
default, housing, loan	Does the client have a credit in default, a housing loan, a personal loan? (yes, no, unknown)	Inputs
contact	Mobile (cellular) or landline (telephone)	Input
month, day_of_week	When the call was made	Inputs
duration	Length of the call, in seconds	Not an input (see below)
campaign	Calls to this client in this campaign, including this one	Input
pdays, previous, poutcome	Days since a call in an earlier campaign (999 = never), calls before this campaign, and its result	Inputs (pdays replaced, see below)
emp.var.rate, cons.price.idx, cons.conf.idx, euribor3m, nr.employed	Five figures for the whole Portuguese economy: employment change, consumer prices, consumer confidence, the 3-month Euribor interest rate, and the number of people employed	Inputs
y	Did the client subscribe?	Target

First checks

print("missing values:", int(raw.isna().sum().sum()), "| exact repeats:", int(raw.duplicated().sum()))
print("yes:", int((raw["y"] == "yes").sum()), f"({(raw['y'] == 'yes').mean():.1%})")
text = raw.select_dtypes(include=["object", "string"]).columns
print("unknown:", {c: int((raw[c] == "unknown").sum()) for c in text if (raw[c] == "unknown").any()})
print("pdays = 999:", f"{(raw['pdays'] == 999).mean():.1%}")

The output:

missing values: 0 | exact repeats: 12
yes: 4640 (11.3%)
unknown: {'job': 330, 'marital': 80, 'education': 1731, 'default': 8597, 'housing': 990, 'loan': 990}
pdays = 999: 96.3%

What this tells us. “No missing values” is not true. The missing answers are written as the word “unknown”: 8,597 clients for default alone, a fifth of the file. Tiago keeps “unknown” as its own category, because “the bank does not know” can itself say something about a client. pdays = 999 means “never called before” in 96.3% of rows. A model must not read it as 999 days, so Tiago replaces it with a yes/no column, contacted_before. The 12 exact repeats are dropped. And only 1 call in 9 ends with yes.

There is no client ID, so a client who was called in two campaigns appears twice, and nothing links the rows. The file is also in date order, which matters at the end of this case study.

Frame the problem

• The unit: one client on the campaign’s list.
• The target: subscribed, 1 if the client said yes.
• The prediction time: before the call, when the list is sorted.
• The decision: call the top 10% of the list. So the measure is the hit rate in the top 10%: the share of yes among the calls the team can make. This is precision at a fixed number of flags, from The precision–recall trade-off. Lift is the hit rate divided by the yes rate of the whole list: how many times better than calling at random.
• What is not known at prediction time: duration. The next section shows why that matters.

prepare() makes the fixes, and the column lists name every role:

def prepare(raw):
    """Drop exact repeats, add the 0/1 target, and replace the code 999 of pdays
    with a yes/no column. The rows stay in the file's order, which is time order."""
    calls = raw.drop_duplicates().reset_index(drop=True)
    calls["subscribed"] = (calls["y"] == "yes").astype(int)
    calls["contacted_before"] = (calls["pdays"] != 999).astype(int)
    return calls


calls = prepare(raw)
CATEGORIES = ["job", "marital", "education", "default", "housing", "loan", "contact", "month", "day_of_week", "poutcome"]
MACRO = ["emp.var.rate", "cons.price.idx", "cons.conf.idx", "euribor3m", "nr.employed"]  # the economy
NUMBERS = ["age", "campaign", "previous", "contacted_before"] + MACRO
FEATURES = CATEGORIES + NUMBERS
TARGET = "subscribed"
print(len(calls), "calls,", len(FEATURES), "features")

The output:

41176 calls, 19 features

What this tells us. 41,176 calls remain, with 19 inputs. duration is not one of them.

Explore what the decision needs

The leak

Tiago’s first quick model used every column and looked excellent. Before he shows it to Inês, he looks at the yes rate by the length of the call:

duration_band = pd.cut(calls["duration"], [-1, 60, 120, 180, 300, 600, 5000],
                       labels=["0-1 min", "1-2 min", "2-3 min", "3-5 min", "5-10 min", "10+ min"])
print(calls.groupby(duration_band, observed=True)[TARGET].agg(calls="size", yes_rate="mean").round(3))

The output:

          calls  yes_rate
duration
0-1 min    4284     0.000
1-2 min    8628     0.019
2-3 min    7784     0.051
3-5 min    9278     0.103
5-10 min   7738     0.186
10+ min    3464     0.486

Yes rate by the length of the call (minutes), all 41,176 calls. Almost no short call ends with yes, and half of the calls over 10 minutes do. The length is known only after the call, so it cannot rank a list before the calls.

Text description of the chart

A column chart with six columns for the length of the call in minutes. 0–1 (4,284 calls): 0%. 1–2 (8,628): 2%. 2–3 (7,784): 5%. 3–5 (9,278): 10%. 5–10 (7,738): 19%. 10 or more (3,464): 49%. The columns rise steadily from left to right.

What this tells us. A long call goes with a yes, of course: the agent explains the product and fills in the forms. A call under a minute almost never ends with yes. But the length of a call is known only after the call, when the answer is known too. A model that uses it cannot rank a list before anyone picks up the phone. This is a leak, as in Select useful features. The data’s own documentation warns about it, and many published models on this data still use it.

Who said yes before

One fact that is known before the call: what happened in an earlier campaign.

print(calls.groupby("poutcome")[TARGET].agg(calls="size", yes_rate="mean").round(3))

The output:

             calls  yes_rate
poutcome
failure       4252     0.142
nonexistent  35551     0.088
success       1373     0.651

What this tells us. Clients who said yes in an earlier campaign say yes again 65.1% of the time. “Call the earlier yeses first” is an obvious rule, and it is the baseline. But only 1,373 clients (3.3%) have an earlier yes, which is less than the team’s 10%. The rule says nothing about whom to call next.

Split and baseline

The file is in date order, but there is no date column. Inês’s question is “rank the rest of this campaign’s list”. The calls left on the list are like the ones already made. So the main split is random and stratified: train 60%, validation 20%, test 20%. A harder question is “will the ranking still work in the next period?”. Tiago plans a second check for it now, before any model: train on the earlier 80% of the file, and test on the later 20%. It comes at the end, as an epilogue.

hit_rate() sorts by score and takes the share of yes among the first 10%. When scores tie, it keeps the list order.

from sklearn.model_selection import train_test_split

rest, test = train_test_split(calls, test_size=0.2, stratify=calls[TARGET], random_state=0)
train, valid = train_test_split(rest, test_size=0.25, stratify=rest[TARGET], random_state=0)
for name, part in [("train", train), ("valid", valid), ("test", test)]:
    print(f"{name:5s} {len(part):6d} calls, {part[TARGET].sum():5d} yes ({part[TARGET].mean():.1%})")

CAPACITY = 0.10


def hit_rate(y, scores, capacity=CAPACITY):
    """Share of yes among the calls with the highest scores, for a capacity share of the list.
    Ties keep the list order (the sort is stable)."""
    calls_made = int(round(capacity * len(y)))
    order = np.argsort(-np.asarray(scores, dtype=float), kind="stable")
    return float(np.asarray(y)[order[:calls_made]].mean())

The output:

train  24705 calls,  2783 yes (11.3%)
valid   8235 calls,   928 yes (11.3%)
test    8236 calls,   928 yes (11.3%)

What this tells us. The three sets have the same yes rate. On the validation set, 10% means 824 calls.

Now the baselines on the validation set: “always no”, a random order, and the rule.

y_valid = valid[TARGET].to_numpy()
rule_valid = (valid["poutcome"] == "success").astype(int)
print("calls the team can make:", int(round(CAPACITY * len(valid))))
print("always no: accuracy", round(1 - y_valid.mean(), 3), "| catches", 0, "yes")
print("random order: hit rate", round(y_valid.mean(), 3))
print("rule, earlier success first: hit rate", round(hit_rate(y_valid, rule_valid), 3))

The output:

calls the team can make: 824
always no: accuracy 0.887 | catches 0 yes
random order: hit rate 0.113
rule, earlier success first: hit rate 0.273

What this tells us. A model that always says no is 88.7% accurate and wins zero deposits, so accuracy is useless here, as Classification metrics warned. A random order finds a yes in 11.3% of calls. The rule finds 27.3%: it calls the 266 earlier yeses first, then the rest in list order. The model must beat 0.273.

Models

Two pipelines

The tree gets one-hot categories, so that its rules read “contact is cellular”, not “contact ≤ 0.5”. It gets the numbers as they are, because a tree compares a number with a cut point and needs no scaling (Decision trees). Logistic regression, the check, gets one-hot categories and scaled numbers.

def build_tree(max_depth=None, min_samples_leaf=1, numbers=NUMBERS):
    prepare_step = ColumnTransformer(
        [("categories", OneHotEncoder(handle_unknown="ignore", sparse_output=False), CATEGORIES)],
        remainder="passthrough", verbose_feature_names_out=False)
    return Pipeline([("prepare", prepare_step), ("model", DecisionTreeClassifier(
        max_depth=max_depth, min_samples_leaf=min_samples_leaf, random_state=0))])


def build_logistic(numbers=NUMBERS):
    prepare_step = ColumnTransformer([("categories", OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
                                      ("numbers", StandardScaler(), numbers)])
    return Pipeline([("prepare", prepare_step), ("model", LogisticRegression(max_iter=2000))])

The experiment log

Tiago scores every experiment on the same 5 stratified folds of train, with two measures. Average precision (AP) looks at the whole ranking. top10 is the hit rate in the top 10%, which is Inês’s decision. The log has the columns from Machine learning is iterative, plus cv_top10 and valid_top10. The notebook defines cv_scores() and log_experiment().

The first rows are the baselines, then a tree with duration, to measure the leak:

base = np.array([train[TARGET].iloc[rows].mean() for _, rows in cv.split(train, train[TARGET])])
log_experiment("E01", "Baseline: random order", base, base, "baseline", "The floor: lift 1.")
rule_ap, rule_top = [], []
for _, rows in cv.split(train, train[TARGET]):
    part = train.iloc[rows]
    rule_ap.append(average_precision_score(part[TARGET], part["poutcome"] == "success"))
    rule_top.append(hit_rate(part[TARGET].to_numpy(), (part["poutcome"] == "success").astype(int)))
log_experiment("E02", "Baseline: earlier success first", np.array(rule_ap), np.array(rule_top), "baseline",
               "What the team would do without a model.")
leak = cv_scores(build_tree(None, 100, NUMBERS + ["duration"]), FEATURES + ["duration"])
log_experiment("E03", "Tree, min leaf 100, WITH duration", leak["test_ap"], leak["test_top10"], "reject",
               "Leak: the length of a call is known only after the call.")

The output:

E01  CV AP 0.113 ± 0.000  top 10% 0.113  baseline: Baseline: random order
E02  CV AP 0.215 ± 0.013  top 10% 0.284  baseline: Baseline: earlier success first
E03  CV AP 0.614 ± 0.014  top 10% 0.616  reject: Tree, min leaf 100, WITH duration

What this tells us. With the leak, the tree finds a yes in 61.6% of its top calls, more than twice the rule. On the validation set its ROC AUC is 0.944. Without duration, the same kind of tree reaches about 0.80. That number can never happen in real use. The row stays in the log with reject, so nobody tries it again.

Common mistake

The leak that looks like a great model

Symptom: a model has ROC AUC 0.94 on held-out data. On this problem, everyone expects about 0.75 to 0.80. Its most important feature is duration.

Cause: the feature is measured after the outcome. A long call is a consequence of a yes, not a sign of one. Held-out data does not catch this, because the test rows have the same leak.

Fix: for each input, ask “would I know this value at the moment I make the decision?” (the first question of Select useful features). Remove what fails, and keep the leaky result in the log as a warning.

Consequence in AI work: in production the leaky column is empty or meaningless, and the model’s real performance is far below the report. A team that promised 0.94 must explain why the campaign found half the yeses it expected.

Trees of every size

Next, a tree with no limits, and the depth sweep of Control complexity:

full = cv_scores(build_tree())
print("no limit: train AP", round(full["train_ap"].mean(), 3), "| CV AP", round(full["test_ap"].mean(), 3))
log_experiment("E04", "Tree, no limits", full["test_ap"], full["test_top10"], "reject", "It memorises train.")
sweep = []
for depth in [1, 2, 3, 4, 5, 6, 8, 10, 12, 15]:
    r = cv_scores(build_tree(depth))
    sweep.append({"depth": depth, "train_ap": r["train_ap"].mean(), "cv_ap": r["test_ap"].mean(),
                  "cv_top10": r["test_top10"].mean()})
sweep = pd.DataFrame(sweep).set_index("depth")
print(sweep.round(3))

The output:

no limit: train AP 0.999 | CV AP 0.178
E04  CV AP 0.178 ± 0.009  top 10% 0.308  reject: Tree, no limits
       train_ap  cv_ap  cv_top10
depth
1         0.270  0.271     0.447
2         0.348  0.348     0.468
3         0.376  0.371     0.487
4         0.413  0.395     0.491
5         0.441  0.403     0.484
6         0.472  0.405     0.485
8         0.540  0.377     0.474
10        0.610  0.337     0.451
12        0.681  0.308     0.441
15        0.784  0.261     0.401

• Train AP
• CV AP
• CV hit rate in the top 10% (dashed)

Depth sweep of the tree, 5-fold CV on train. Train AP keeps rising with depth; CV AP peaks at depth 6 and then falls. The hit rate in the top 10%, the team's measure, is nearly flat from depth 3 to 6.

Text description of the chart

A line chart with max_depth from 1 to 15 on the horizontal axis and scores from 0.2 to 0.8 on the vertical axis. Train AP rises steadily: 0.27 at depth 1, 0.38 at 3, 0.47 at 6, 0.78 at 15. CV AP rises with it to 0.405 at depth 6, then falls: 0.377 at 8, 0.337 at 10, 0.261 at 15. The gap between the two lines opens after depth 4. The dashed line for the CV hit rate in the top 10% is higher and flatter: 0.447 at depth 1, 0.468 at 2, 0.487 at 3 (marked), 0.491 at 4, 0.484 to 0.485 at 5 and 6, then down to 0.401 at 15.

What this tells us. The tree with no limit learns every training call by heart: train AP 0.999, CV AP 0.178. Its top 10% is barely better than the rule. A limit helps, but the two measures disagree about how much. AP, which judges the whole list, likes depth 5 or 6. The hit rate in the top 10%, which is the decision, is almost flat from depth 3 to 6. The extra depth improves the order at the bottom of the list, where nobody calls.

A grid search, and the simplest tree that is good enough

Tiago searches over depth and minimum leaf size together (Tune hyperparameters), with both measures:

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(build_tree(), {"model__max_depth": [2, 3, 4, 5, 6, 8, None],
                                     "model__min_samples_leaf": [1, 20, 50, 100, 200]},
                      scoring={"ap": "average_precision", "top10": top10}, refit="top10", cv=cv)
search.fit(train[FEATURES], train[TARGET])
grid = pd.DataFrame(search.cv_results_)[["param_model__max_depth", "param_model__min_samples_leaf", "mean_test_ap",
                                         "mean_test_top10", "std_test_top10"]]
grid.columns = ["max_depth", "min_leaf", "cv_ap", "cv_top10", "cv_top10_std"]
print("best by AP:\n", grid.sort_values("cv_ap", ascending=False).head(3).round(3).to_string(index=False))
print("best by top 10%:\n", grid.sort_values("cv_top10", ascending=False).head(3).round(3).to_string(index=False))

The output:

best by AP:
 max_depth  min_leaf  cv_ap  cv_top10  cv_top10_std
     None       100  0.433     0.483         0.017
        8       100  0.428     0.484         0.017
     None        50  0.423     0.487         0.014
best by top 10%:
 max_depth  min_leaf  cv_ap  cv_top10  cv_top10_std
        8        50  0.420     0.496         0.023
        5        50  0.412     0.494         0.023
        4        20  0.397     0.493         0.023

What this tells us. The winner depends on the measure, and the differences are small. Tiago uses the one-standard-error rule on the decision’s measure. The best top-10% hit rate is 0.496, with a standard error of 0.023 ÷ √5 = 0.010. So any tree at 0.486 or more is as good, within the noise, and he takes the simplest of them.

In the notebook, you compute the limit and the smallest depth that reaches it. The answer is depth 3, with 0.487: just inside the limit. Depth 2 (0.468) is not. A tree of depth 3 has at most 8 leaves, which means 8 rules that Inês’s team can read. Here are the candidates in the log, with logistic regression as a check:

E05  CV AP 0.433 ± 0.014  top 10% 0.483  reject: Tree, no depth limit, min leaf 100 (best AP)
E06  CV AP 0.420 ± 0.015  top 10% 0.496  reject: Tree, max_depth 8, min leaf 50 (best top 10%)
E07  CV AP 0.371 ± 0.015  top 10% 0.487  keep: Tree, max_depth 3
E08  CV AP 0.448 ± 0.013  top 10% 0.489  reject: Logistic regression, 19 features

What this tells us. The four candidates are close on the decision’s measure, from 0.483 to 0.496, and the depth-3 tree is the simplest of them. Logistic regression has the best AP, but at the team’s capacity it finds the same share of yeses as the tree (0.489 against 0.487), well inside the noise.

Rui, another analyst at the bank, asks: “Then why not use logistic regression?” Because Inês asked for rules her team can read. A tree of depth 3 can be drawn on one page and checked by people who never heard of AP. The epilogue will show one reason to keep logistic regression in mind.

Tiago confirms the choice once on the validation set, for the chosen tree and two candidates:

candidates = {"E05": build_tree(None, 100), "E07": build_tree(3), "E08": build_logistic()}
log = pd.read_csv(LOG)
for id, pipe in candidates.items():
    pipe.fit(train[FEATURES], train[TARGET])
    s = pipe.predict_proba(valid[FEATURES])[:, 1]
    log.loc[log["id"] == id, ["valid_ap", "valid_top10"]] = [round(average_precision_score(y_valid, s), 3),
                                                             round(hit_rate(y_valid, s), 3)]
    print(f"{id}  ROC AUC {roc_auc_score(y_valid, s):.3f}  AP {average_precision_score(y_valid, s):.3f}"
          f"  top 10% {hit_rate(y_valid, s):.3f}  lift {hit_rate(y_valid, s) / y_valid.mean():.2f}")
log.to_csv(LOG, index=False)
tree = candidates["E07"]
scores = tree.predict_proba(valid[FEATURES])[:, 1]

The output:

E05  ROC AUC 0.799  AP 0.469  top 10% 0.513  lift 4.56
E07  ROC AUC 0.757  AP 0.389  top 10% 0.510  lift 4.52
E08  ROC AUC 0.800  AP 0.474  top 10% 0.508  lift 4.51

What this tells us. At the team’s capacity, the 8-leaf tree finds as many yeses as the 183-leaf tree and logistic regression. That is about 51 in every 100 calls, 4.5 times the random rate. Its ROC AUC and AP are lower because it ranks the bottom of the list worse. That is the right trade for this decision. The log now holds eight experiments, with E07 as the one to keep.

Read the tree as rules

export_text() prints the tree. The weights of a leaf are its training calls, as [no, yes]:

from sklearn.tree import export_text

names = list(tree.named_steps["prepare"].get_feature_names_out())
print(export_text(tree.named_steps["model"], feature_names=names, show_weights=True, decimals=2))

The output:

|--- nr.employed <= 5087.65
|   |--- contacted_before <= 0.50
|   |   |--- contact_cellular <= 0.50
|   |   |   |--- weights: [276.00, 83.00] class: 0
|   |   |--- contact_cellular >  0.50
|   |   |   |--- weights: [1164.00, 713.00] class: 0
|   |--- contacted_before >  0.50
|   |   |--- day_of_week_mon <= 0.50
|   |   |   |--- weights: [161.00, 450.00] class: 1
|   |   |--- day_of_week_mon >  0.50
|   |   |   |--- weights: [68.00, 82.00] class: 1
|--- nr.employed >  5087.65
|   |--- cons.conf.idx <= -46.65
|   |   |--- euribor3m <= 1.54
|   |   |   |--- weights: [1251.00, 269.00] class: 0
|   |   |--- euribor3m >  1.54
|   |   |   |--- weights: [67.00, 62.00] class: 0
|   |--- cons.conf.idx >  -46.65
|   |   |--- month_oct <= 0.50
|   |   |   |--- weights: [18925.00, 1099.00] class: 0
|   |   |--- month_oct >  0.50
|   |   |   |--- weights: [10.00, 25.00] class: 1

The chosen tree of depth 3, fitted on the 24,705 training calls. Each box shows the calls that reach it and their yes rate (on a phone, only the yes rate); the leaves are the rules R1 to R8.

Text description of the diagram

The tree has seven questions and eight leaves. The first question splits the calls by the number of people employed in Portugal at the time of the call (at most 5,087.65 thousand, a period of low employment). In the low-employment period, the tree asks whether the client was contacted before, then about the phone type or the day. In the other period, it asks about consumer confidence, then the Euribor rate or the month. The leaves range from R7, with 20,024 calls and 5.5% yes, to R3, with 611 calls and 73.6% yes.

The same tree as rules for Inês’s team, in the order the list uses them (highest yes rate first):

Rule	When it applies	Train calls	Yes rate
R3	Low employment, contacted in an earlier campaign, not a Monday	611	73.6%
R8	Normal employment, consumer confidence above −46.65, October	35	71.4%
R4	Low employment, contacted in an earlier campaign, a Monday	150	54.7%
R6	Normal employment, low consumer confidence, Euribor above 1.54	129	48.1%
R2	Low employment, not contacted before, mobile phone	1,877	38.0%
R1	Low employment, not contacted before, landline	359	23.1%
R5	Normal employment, low consumer confidence, Euribor at most 1.54	1,520	17.7%
R7	Normal employment, consumer confidence above −46.65, not October	20,024	5.5%

What this tells us. The first question is not about the client. nr.employed is the number of people employed in the whole country, a figure published every three months. It splits the calls into periods of the economy: in the low-employment period, 44.3% of calls ended with yes; otherwise 6.7%. Only the next questions are about clients: an earlier contact, and a mobile phone. So the tree tells Inês two different things:

• When a campaign works: in this data, when employment was low and interest rates had fallen. People then preferred a safe deposit. The team cannot choose the economy, but it can plan the size of a campaign around it.
• Whom to call first: clients contacted in an earlier campaign, and then clients with a mobile phone.

Read the small leaves with care. R8 has only 35 training calls and R6 has 129. Their rates could change a lot with a few more calls. The Monday split in R3 and R4 is also based on only 150 calls. A tree reports every leaf with the same confidence; the count shows how much to trust it. Remember the first question, too: it matters in the epilogue.

Metric and operating point

Tiago chose the metric when he framed the problem: the hit rate in the top 10%, because the team calls only the top of the list. So the operating point is a capacity, not a threshold on the score: the team calls the first 10% of the list, whatever their scores are. What if Inês gets more agents? Tiago prints the hit rate, the lift and the share of all yeses found, at several capacities:

for capacity in [0.05, 0.10, 0.15, 0.20, 0.30]:
    rate = hit_rate(y_valid, scores, capacity)
    found = rate * round(capacity * len(y_valid)) / y_valid.sum()
    print(f"call {capacity:.0%}: hit rate {rate:.3f}, lift {rate / y_valid.mean():.2f}, finds {found:.1%} of all yes")

The output:

call 5%: hit rate 0.590, lift 5.23, finds 26.2% of all yes
call 10%: hit rate 0.510, lift 4.52, finds 45.3% of all yes
call 15%: hit rate 0.414, lift 3.67, finds 55.1% of all yes
call 20%: hit rate 0.347, lift 3.08, finds 61.6% of all yes
call 30%: hit rate 0.247, lift 2.20, finds 65.8% of all yes

• Tree, depth 3
• Logistic regression (dashed)
• Rule: earlier success first
• Random order (dashed)

Gains chart on the 8,235 validation calls: the share of all yeses found when the team calls the first part of the list. At 10% the tree and logistic regression find the same 45%; further down, logistic regression ranks better.

Text description of the chart

A square chart. The horizontal axis is the share of the list called, from 0% to 100%; the vertical axis is the share of all yeses found, from 0% to 100%. Random order is a dashed diagonal. The rule rises steeply for the first 3% of the list (the earlier yeses), then follows a line close to the diagonal: 24% at 10%, 33% at 20%, 59% at 50%. The tree rises steeply to 45% at 10% and 62% at 20%, then flattens: 66% at 30% and 76% at 50%. Logistic regression is almost the same as the tree up to 10% (45%), then higher: 64% at 20%, 72% at 30%, 82% at 50%.

What this tells us. At 10%, the tree finds 45.3% of all the yeses on the list with a tenth of the calls. The first calls are the best ones, and each extra 5% of the list adds fewer yeses. Doubling the team to 20% finds 16 percentage points more of the yeses, at a hit rate of 0.347. Beyond 20%, the tree’s line flattens. Most of the remaining calls sit in one big leaf, R7, where every call has the same score, so the tree cannot order them. Logistic regression keeps ordering them. If Inês’s capacity ever grows past 20%, the comparison should be made again.

In the notebook’s “change one thing” task, a busy week allows only 5% of the list. The hit rate rises to 0.590, because only the best leaves are called.

Errors and slices

The yeses the team never calls

Each validation call falls into one leaf. Tiago counts the missed yeses (not in the top 10%) by leaf:

order = np.argsort(-scores, kind="stable")
called = np.zeros(len(valid), dtype=bool)
called[order[: int(round(CAPACITY * len(valid)))]] = True
leaf = tree.named_steps["model"].apply(tree.named_steps["prepare"].transform(valid[FEATURES]))
errors = pd.DataFrame({"leaf": leaf, "score": scores.round(3), "called": called, "yes": y_valid,
                       "missed_yes": (y_valid == 1) & ~called})
print(errors.groupby(["leaf", "score"]).agg(calls=("yes", "size"), called=("called", "sum"), yes=("yes", "sum"),
                                            missed_yes=("missed_yes", "sum")))

The output:

            calls  called  yes  missed_yes
leaf score
3    0.231    120       0   27          27
4    0.380    612     542  252          27
6    0.736    182     182  140           0
7    0.547     47      47   28           0
10   0.177    518       0   94          94
11   0.481     36      36   17           0
13   0.055   6703       0  360         360
14   0.714     17      17   10           0

What this tells us. The team calls four leaves completely and most of a fifth (542 of 612 calls with the score 0.38). Of the 508 yeses it misses, 360 sit in the big leaf R7 (score 0.055), where only 1 call in 19 succeeds. The tree gives those 6,703 calls one score, so it cannot find the 360 among them. That is the price of 8 rules. A finer model could rank inside the leaf, but at 10% capacity the team would not reach them anyway.

Who ends up on the list

A ranking decides who gets a sales call, so Tiago checks it by age band, with the counts:

age_band = pd.cut(valid["age"], [16, 25, 40, 60, 100], labels=["17-25", "26-40", "41-60", "61+"])
frame = pd.DataFrame({"age band": age_band.to_numpy(), "called": called, "yes": y_valid})
slices = frame.groupby("age band", observed=True).agg(clients=("yes", "size"), yes_rate=("yes", "mean"),
                                                      share_called=("called", "mean"))
slices["hit rate in calls"] = frame[frame["called"]].groupby("age band", observed=True)["yes"].mean()
print(slices.round(3))

The output:

          clients  yes_rate  share_called  hit rate in calls
age band
17-25         308     0.214         0.224              0.551
26-40        4431     0.109         0.088              0.521
41-60        3317     0.090         0.073              0.490
61+           179     0.441         0.693              0.492

What this tells us. Clients aged 61 and over are only 179 of the 8,235. But 69.3% of them are on the call list, against 7% to 9% of the middle bands. The tree never asks about age. The reason is time. The bank made 82% of its calls to clients over 60 in the low-employment period, which the tree rates highly, and 75% of these clients are retired. The numbers script prints both shares. In that period older clients did say yes often (44.1%), so the list is efficient.

It still needs a human decision. A list that sends most older, retired clients to a sales call needs care. The product must suit them, the agent must not press for an answer, and refusing must be easy. A term deposit is a low-risk product, but the same model could rank a list for a riskier product. In many countries, rules on selling financial products ask firms to check that a product suits the client and to treat vulnerable clients with care. This is a point for Inês and the bank’s compliance team, not a reason to hide age from the analysis.

The final test, once

The tree (E07) and the capacity (10%) are fixed. Tiago opens the test set once and compares with the rule and a random order on the same list:

y_test = test[TARGET].to_numpy()
test_scores = tree.predict_proba(test[FEATURES])[:, 1]
calls_made = int(round(CAPACITY * len(test)))
rule_test = (test["poutcome"] == "success").astype(int)
print("calls:", calls_made, "| ROC AUC", round(roc_auc_score(y_test, test_scores), 3),
      "| AP", round(average_precision_score(y_test, test_scores), 3))
for name, rate in [("tree, depth 3", hit_rate(y_test, test_scores)), ("rule", hit_rate(y_test, rule_test)),
                   ("random (expected)", y_test.mean())]:
    print(f"{name:18s} hit rate {rate:.3f}  lift {rate / y_test.mean():.2f}  yes in {calls_made} calls: {rate * calls_made:.0f}")

The output:

calls: 824 | ROC AUC 0.752 | AP 0.365
tree, depth 3      hit rate 0.478  lift 4.24  yes in 824 calls: 394
rule               hit rate 0.263  lift 2.34  yes in 824 calls: 217
random (expected)  hit rate 0.113  lift 1.00  yes in 824 calls: 93

What this tells us. On calls that no step has touched, 824 calls find about 394 deposits with the tree. The rule finds 217, and a random order about 93. That is a lift of 4.24. The test is a little lower than validation (0.478 against 0.510), as expected, because validation helped to choose. This is the number to report, and it holds only for lists from the same period, as the next section shows.

Epilogue: the next period

The random split mixed calls from all periods, so every test call had neighbours from the same weeks in train. Inês’s next campaign will not. Tiago runs the second check that he planned at the start. First, how does the file change over time?

tenth = np.arange(len(calls)) * 10 // len(calls) + 1
by_tenth = calls.groupby(tenth).agg(yes_rate=(TARGET, "mean"), euribor3m=("euribor3m", "mean"),
                                    employed=("nr.employed", "mean"))
print(by_tenth.round(3))

The output:

    yes_rate  euribor3m  employed
1      0.028      4.857  5191.000
2      0.035      4.859  5195.270
3      0.042      4.941  5228.100
4      0.066      4.961  5228.100
5      0.062      4.965  5228.100
6      0.055      4.845  5222.484
7      0.102      3.370  5168.936
8      0.120      1.354  5099.100
9      0.157      1.260  5094.401
10     0.460      0.802  5014.834

Yes rate in each tenth of the file, which is in date order (May 2008 to November 2010). Fewer than 1 call in 15 succeeded in the first six tenths; in the last tenth, almost half did.

Text description of the chart

A line chart with the ten tenths of the file on the horizontal axis and the yes rate from 0% to 50% on the vertical axis. The rate stays low in the first six tenths: 2.8%, 3.5%, 4.2%, 6.6%, 6.2%, 5.5%. It then rises: 10.2% in the seventh, 12.0% in the eighth, 15.7% in the ninth, and jumps to 46.0% in the tenth.

The mean 3-month Euribor interest rate in each tenth of the file. It falls from about 4.9% to 0.8% in the last four tenths, as the yes rate rises.

Text description of the chart

A line chart with the ten tenths of the file on the horizontal axis and the mean Euribor rate from 0 to 5.5 on the vertical axis. The rate is flat near 4.9 for the first six tenths (4.86, 4.86, 4.94, 4.96, 4.97, 4.85), then falls: 3.37 in the seventh, 1.35 in the eighth, 1.26 in the ninth and 0.80 in the tenth.

What this tells us. The economy changed during the data, and so did the answers. In the first six tenths, fewer than 7 calls in 100 succeeded; in the last tenth, 46 did. Interest rates fell at the same time. The five economic columns are clocks: they tell the model which period a call comes from. That is why the tree’s first question was nr.employed.

Now the planned check: fit on the first 80% of the file (the earlier calls), and score the last 20% (the later calls).

cut = int(len(calls) * 0.8)
early, late = calls.iloc[:cut], calls.iloc[cut:]
y_late = late[TARGET].to_numpy()
print("yes rate: early", round(early[TARGET].mean(), 3), "| late", round(y_late.mean(), 3))
for name, pipe in [("tree, depth 3", build_tree(3)), ("logistic regression", build_logistic())]:
    s = pipe.fit(early[FEATURES], early[TARGET]).predict_proba(late[FEATURES])[:, 1]
    print(f"{name:20s} ROC AUC {roc_auc_score(y_late, s):.3f}  top 10% {hit_rate(y_late, s):.3f}"
          f"  lift {hit_rate(y_late, s) / y_late.mean():.2f}  mean score {s.mean():.3f}")
print("late calls with a Euribor rate below anything in early:",
      f"{(late['euribor3m'] < early['euribor3m'].min()).mean():.1%}")

The output:

yes rate: early 0.064 | late 0.308
tree, depth 3        ROC AUC 0.638  top 10% 0.416  lift 1.35  mean score 0.125
logistic regression  ROC AUC 0.748  top 10% 0.570  lift 1.85  mean score 0.256
late calls with a Euribor rate below anything in early: 98.5%

What this tells us. The tree trained on the past ranks the future much worse: ROC AUC 0.638, against 0.752 on the random test. Its lift falls to 1.35. Two things went wrong at once:

• The inputs moved outside what the tree has seen. 98.5% of the late calls have a Euribor rate lower than any early call. A tree can only answer with the leaves it learned. So all these calls land in a few leaves (the late calls fill only 6 of the 8), with the old yes rates. The tree’s mean score is 0.125 when 30.8% of the calls end with yes.
• The relationship moved. A rule such as “Euribor above 1.54” was learned when rates were near 5%. When every rate is below 1.3, the rule no longer separates anyone.

Logistic regression holds up better (ROC AUC 0.748). It follows the trend of a number beyond the range it saw, so it can say “even lower rates, even more yes”. That helped here, but it is not a guarantee: a trend can also reverse. Neither model was right about the level: both under-estimate the late yes rate.

This is drift, and no model choice made at the start would have prevented it. The answer is the one from Plan production checks: watch, and retrain.

• Watch the inputs, before the answers arrive. Each week, compare the economic columns with the range in the training data. An alarm when more than 5% of the list is outside that range would have fired at once here.
• Watch the results, when the answers arrive. Each week, compare the hit rate of the calls made with the tree’s expected rate for those leaves. A gap of 10 percentage points or more for two weeks means the ranking is out of date.
• Retrain on recent calls, and choose the next model with a time-based validation: train on older weeks, validate on the most recent ones. That is the split that matches the next campaign. With such a validation, the comparison between the tree and logistic regression could come out differently, so make it again.

Recommendations

Tiago writes to Inês in her words:

• Use the tree to order this campaign’s remaining list. With calls for 10% of the list, it finds about 48 yeses in every 100 calls. “Earlier yes first” finds 26, and a random order 11. That is about 394 deposits instead of 217 from the same 824 calls.
• Give the team the eight rules. They show whom to call first: clients contacted before, then mobile numbers. They also show that most of the success depends on the period, not on the client.
• If capacity grows past 20%, ask for a new comparison. Beyond that, the tree cannot order the large group of “normal period” calls.
• Take care with older clients. Seven in ten clients over 60 end up on the list. Check that the product suits them and that the call script lets them refuse easily.
• Before the next campaign, retrain and recheck. The tree learned the economy of 2008 to 2010. Watch the inputs and the weekly hit rate, as above.

And what not to use it for:

• Not for a new campaign in a changed economy without retraining and a time-based check. The epilogue shows the lift falling from 4.2 to 1.4.
• Not for another product. The tree learned who buys a term deposit. A credit card or an insurance product has other buyers and other risks for the client.
• Not as a statement about a client. “Low score” means “few clients like this said yes in this period”, not “this client will never buy”.

Limits and next steps

• One bank, 2008 to 2010. The years of a financial crisis, with unusual interest rates. The patterns may not hold today or elsewhere.
• The main split mixes periods. The random test answers “rank the rest of this list”; only the time check answers “the next campaign”, and it shows a large drop.
• campaign is a little suspicious. It counts calls in this campaign including the last one. Before the first call it is not known; the case study treats it as “calls so far”. A stricter version would use only calls before this one, which this data does not record.
• No client ID. A client may appear in several campaigns, possibly in both train and test.
• The day of the call. month and day_of_week describe when the call was made. The team knows the month of a campaign before it starts, but the day of a call is set when the agent calls. So read the Monday split of R3 and R4 as a hint for planning the calls, not as a fact about the client.
• Small leaves. R8 (35 calls) and R6 (129 calls) rest on little evidence.
• Unknown answers. A fifth of default is “unknown”. If the bank can fill it, the rules might change.

Make it a portfolio piece

This case study shows three things that employers look for. You found a leak that others miss. You chose a model for the decision, not for a leaderboard. And you tested what happens over time. Make it yours with one change, for example a capacity of 5%, the time-based split as the main one, or the same study on another open marketing dataset. Then publish it.

What to put in the repository:

• README.md (template below).
• The notebook, run from top to bottom, with its outputs.
• experiment_log.csv, including the leaky E03, with its reason.
• RULES.md: the eight rules as the table above, with the counts. This is the page that a call-centre team would read.
• MODEL_CARD.md, as in Document use boundaries: intended use (ordering one campaign’s list at 10% capacity), not for (new periods without retraining, other products), evaluation, the drift result, monitoring signals.
• The data source and licence, with the attribution text from the box above.

A README template:

# Ranking a bank's call list under a 10% capacity

## Problem
A call centre can call 10% of a campaign's list. Order the list so those calls find
as many term-deposit subscribers as possible.

## Data and licence
UCI "Bank Marketing" (Moro, Rita and Cortez, 2014), bank-additional-full,
41,188 calls, Portugal 2008-2010, CC BY 4.0, https://doi.org/10.24432/C5K306.
Changes: 12 exact repeats dropped; pdays=999 replaced by contacted_before.

## Approach
Removed the duration leak (known only after the call). Random stratified split
60/20/20, plus a planned time check (first 80% -> last 20%). Decision trees in a
grid; chose the simplest within one standard error of the best top-10% hit rate:
depth 3, 8 rules. Logistic regression as a check.

## Results (test set, used once)
Top-10% hit rate 0.478 (lift 4.24): 394 yes in 824 calls vs 217 for the rule
"earlier yes first" and about 93 at random.

## Drift
Trained on the earlier 80%, the tree's ROC AUC on the later 20% falls to 0.64
(logistic regression 0.75): interest rates left the training range.

## Limits
One bank, crisis years; random split mixes periods; small leaves; no client ID.

How to describe it in an interview, in two sentences: “I ranked a bank’s marketing call list with an 8-rule decision tree chosen for a 10% call capacity; it found 4.2 times as many subscribers per call as random calling, and I removed a call-length leak that makes models on this data look far better than they can be. A time-based check showed the ranking breaking when interest rates moved out of the training range, so I wrote input-range and weekly hit-rate alarms and a retraining plan.”

Go further:

• Make the time-based validation the main one: train on the first 60%, choose on the next 20%, test on the last 20%. Does depth 3 still win?
• Remove the five economic columns and see what client-only rules the tree finds, and what lift they give.
• Compare a random forest or boosting at the same capacity. Decide whether its gain is worth losing the rules (When a simple model is better).
• Write the weekly monitoring check as a small script that reads a new list and prints the share of calls outside the training range.

Knowledge check

Check your understanding

A colleague's model on this data has ROC AUC 0.94 and uses the length of the call. Why is that model useless for ordering a call list, even though its test score is honest arithmetic?

Show the answer

The length of a call is known only after the call, when the answer is known too. To order a list, the model must score clients before anyone calls them, so the column is empty at that moment. The test score is computed on rows that also contain the length, so the test has the same leak and cannot catch it. Without the column, trees reach about 0.76 to 0.80. The question for every input is: would I know this value at the moment I decide?

Check your understanding

The grid search found a tree with a higher average precision (0.433) than the depth-3 tree (0.371). Why did Tiago choose depth 3?

Show the answer

The decision is to call the top 10% of the list, so the measure that matters is the hit rate in the top 10%. On that measure, the best tree reached 0.496 with a standard error of about 0.010. Depth 3 reached 0.487: within one standard error, so as good within the noise. The higher AP comes from a better order at the bottom of the list, which nobody calls. Depth 3 has 8 leaves, so the team can read it as 8 rules, which Inês asked for.

Check your understanding

Trained on the earlier 80% of the file, the tree's ROC AUC on the later 20% is 0.638, while logistic regression keeps 0.748. Explain the drop, and name two monitoring signals that would warn the team.

Show the answer

The economy changed. 98.5% of the late calls have a Euribor rate below anything the tree saw, and the yes rate rose from 6.4% to 30.8%. A tree can only answer with its learned leaves. So the late calls fall into a few leaves with old rates (mean score 0.125), and rules such as “Euribor above 1.54” no longer separate anyone. Logistic regression follows the trend of a number beyond its range, which helped here. Two signals help. Before any call: the share of calls whose economic inputs are outside the training range. When the answers arrive: the weekly hit rate of the calls made, against the rate the model expects.

Recap

• Accuracy is useless when the team calls 10% of a list: “always no” scores 88.7%. The hit rate in the top 10% and the lift match the decision.
• The call length looked like the best feature, but it is a leak: it is known only after the call. ROC AUC is 0.944 with it, and about 0.76 to 0.80 without.
• A full tree memorised train (AP 0.999 against 0.178 in CV). With the one-standard-error rule on the decision’s measure, a depth-3 tree with 8 readable rules was as good as the best.
• On the test, used once: hit rate 0.478, lift 4.24, 394 yes in 824 calls against 217 for the rule.
• The tree’s first rule was about the economy, not the client. When the economy moved, the tree’s ROC AUC fell to 0.638. Monitor the input ranges and the weekly hit rate, and retrain with a time-based validation.

Next, a comparison of four model families under one protocol, where the honest answer is how little the data can predict.

Next: Choose a model for hospital readmissions