Predict credit-card default

What you will be able to do

Build a credit-risk model from real data to a tested result that you can show to others. You will choose the threshold from what each error costs, and check that the scores mean what they say. You will also find out whether the model treats young clients differently, although age is not one of its inputs.

Case studyPortfolio piece

Dataset

Default of credit card clients, Taiwan 2005 (UCI)

Licence

CC BY 4.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Cost matrix
• Logistic regression pipeline
• Cost-based threshold
• Calibration
• Fairness slices
• Challenger model

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. Its setup cells install pandas and scikit-learn and download the data (about 1 MB), so it needs no local setup. It runs in about one minute.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• Precision, recall and the confusion matrix, from Classification metrics, and average precision, from The precision–recall trade-off.
• The pipeline with one-hot encoding and scaling, from Preprocess with a pipeline.
• The experiment log, from Machine learning is iterative.
• Thresholds, calibration and the test set used once, from Thresholds and calibration.
• Slices with counts, from Slice analysis, and one ticket’s contributions, from Explain predictions.
• Why lenders use readable models, from When a simple model is better.

What you need on your computer (only for the local route; the notebook needs none of it)

• A folder with a virtual environment that has pandas 3.0.6, scikit-learn 1.9.1 and matplotlib, as in Define a baseline: Set up your project. The ticket-model project works; this case study does not change its files.
• The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell downloads the data into a new folder credit-default. To start again, delete that folder.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1, in Jupyter from a fresh start. Windows and Linux are not tested.

The situation

Mei-Ling Chen leads the card-risk team of a bank in Taiwan. Each month, a few days before the next statement, her team chooses clients for an early-help call. An agent calls the client, offers a payment plan, and pauses any increase of the credit limit. The team does not refuse or close anything. The call is help and a reminder, made before the client falls behind.

Each choice can go wrong in two ways, and Mei-Ling has put a rough price on each:

• A false alarm: the team calls a client who would have paid. That costs agent time and annoys a good client. She counts it as 1 unit.
• A missed default: a client who will not pay gets no call. Part of that debt never comes back. She counts it as 5 units.

So a miss is five times as bad as a false alarm. Today the team calls every client who was late in the last month. Mei-Ling asks her analyst, Arjun, for something better: “Who is most likely to miss the next payment? And how many should we call?”

In this case study, default means “did not make the next payment”. It is a fact about repayment, not a judgement of the person’s honesty or intentions.

What you will build

You will build a logistic regression pipeline that gives each client a score for the risk of default next month. Then you choose a threshold that turns the score into “call” or “do not call” at the lowest total cost. Along the way you keep an experiment log and check the scores against reality. You also explain one client’s score and check the model for groups of clients. At the end you test it once and write it up as a portfolio piece.

The skills come from the whole course: framing (Translate a need), baselines (Define a baseline), honest splits (Train, validation and test sets), pipelines, the experiment log, thresholds and calibration, slices and explanations, and the model card (Document use boundaries).

The data

The data is the Default of Credit Card Clients dataset from the UCI Machine Learning Repository. It has 30,000 real, anonymous clients of one bank in Taiwan. For each client, it has the history from April to September 2005, and whether the client paid in October.

Source and licence

“Default of Credit Card Clients” by I-Cheng Yeh, UCI Machine Learning Repository, doi.org/10.24432/C55S3H, licensed under CC BY 4.0. Nextia Learning converted the Excel file to a compressed CSV, removed its extra first header row, and renamed the target column to default; nothing else changed. Paper: Yeh, I.-C. and Lien, C.-H. (2009), Expert Systems with Applications 36(2), 2473–2480.

The notebook’s setup cell downloads the file and checks its checksum. Arjun loads it and looks at its size and columns:

raw = pd.read_csv("credit_default_taiwan.csv.gz")
print(raw.shape)
print(raw.columns.tolist())

The output:

(30000, 25)
['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'default']

What this tells us. One row is one client. Apart from an ID, there are the credit limit, four personal facts, and three groups of six monthly columns. In each group the months run backwards, from September to April. The last column is the target.

The data dictionary

These are only the columns that the case study uses, in plain words. Amounts are in New Taiwan dollars (NT$).

Column	What it means	Use
LIMIT_BAL	The card’s credit limit	Input
EDUCATION	1 graduate school, 2 university, 3 high school, 4 others	Input
SEX, MARRIAGE, AGE	Sex (1 male, 2 female), marital status (1 married, 2 single, 3 others), age in years	Not inputs: only for the fairness checks
PAY_0, PAY_2 … PAY_6	Repayment status in September, August … April: −1 paid on time, 1 to 9 months late	Inputs
BILL_AMT1 … BILL_AMT6	The bill in September … April	Inputs
PAY_AMT1 … PAY_AMT6	The amount paid in September … April	Inputs
default	1 if the client did not pay in October	Target

First checks

Before any model, Arjun checks for gaps, repeats and the share of defaults:

print("missing values:", int(raw.isna().sum().sum()))
print("repeated IDs:", int(raw["ID"].duplicated().sum()))
print("same row, other ID:", int(raw.drop(columns="ID").duplicated().sum()))
print("defaults:", int(raw["default"].sum()), "of", len(raw), f"({raw['default'].mean():.1%})")

The output:

missing values: 0
repeated IDs: 0
same row, other ID: 35
defaults: 6636 of 30000 (22.1%)

What this tells us. There are no gaps. 35 rows repeat another row in every column but the ID, and most of them are clients with no bills and no payments at all. They have different IDs, so they are probably different people with the same empty history, and Arjun keeps them. About one client in five defaults. That is rare enough for accuracy to mislead: a model that always says “pays” is right 77.9% of the time and helps nobody. It is also common enough for a model to learn from.

Odd codes

The source documents education codes 1 to 4 and marital status 1 to 3. For the repayment status, it documents −1 (paid on time) and 1 to 9 (months late). Arjun checks whether the file keeps to that:

for column in ["EDUCATION", "MARRIAGE", "PAY_0", "PAY_2"]:
    print(column, raw[column].value_counts().sort_index().to_dict())
print("negative September bills:", int((raw["BILL_AMT1"] < 0).sum()))

The output:

EDUCATION {0: 14, 1: 10585, 2: 14030, 3: 4917, 4: 123, 5: 280, 6: 51}
MARRIAGE {0: 54, 1: 13659, 2: 15964, 3: 323}
PAY_0 {-2: 2759, -1: 5686, 0: 14737, 1: 3688, 2: 2667, 3: 322, 4: 76, 5: 26, 6: 11, 7: 9, 8: 19}
PAY_2 {-2: 3782, -1: 6050, 0: 15730, 1: 28, 2: 3927, 3: 326, 4: 99, 5: 25, 6: 12, 7: 20, 8: 1}
negative September bills: 590

What this tells us. There are four problems, and each one changes a later step:

• Codes that nobody documented. Education has 0, 5 and 6 (345 clients), and marital status has 0 (54 clients). Arjun puts the three education codes into “others” (4). Marital status is not an input, so he leaves it.
• The two most common status codes are not documented. −2 and 0 cover 17,496 clients in September. A common reading is “no card use” and “paid the minimum”, but the source does not confirm it. They are too common to drop, and they are clearly not “months late”. So Arjun treats every status code as a category, not as a number.
• September is coded differently. Code 1 (one month late) appears 3,688 times in September and only 28 times in August. The same number may not mean the same thing in every month. That is one more reason to give each month’s codes their own categories.
• The names skip a number. The status columns are PAY_0, PAY_2, …, and there is no PAY_1. Arjun renames PAY_0 to PAY_1, so that 1 means September in every group.

The 590 negative bills belong to clients who paid more than they owed. They are real, so they stay.

Frame the problem

Translate a need asks a few questions before any model. Here are Arjun’s answers:

• The unit: one client, on the statement date in September 2005.
• The target: default, 1 if the client did not make the October payment.
• The prediction time: before the October payment is due, when the September statement is known.
• The inputs: the credit limit, education, and the six months of status, bills and payments. All of them are known at the prediction time, so nothing leaks from October.
• Sensitive columns: sex, marital status and age. They are personal facts that should not decide who gets a call, and in many countries laws limit their use in credit decisions. Arjun keeps them out of the inputs. He keeps them in the data, because without them he cannot check whether the model treats these groups differently.

prepare() makes the two fixes, and the column lists name every role:

def prepare(raw):
    """Rename PAY_0 to PAY_1 (September), and put the undocumented
    education codes 0, 5 and 6 into 4 (others)."""
    clients = raw.rename(columns={"PAY_0": "PAY_1"})
    clients["EDUCATION"] = clients["EDUCATION"].replace({0: 4, 5: 4, 6: 4})
    return clients


clients = prepare(raw)
STATUS = [f"PAY_{m}" for m in range(1, 7)]  # repayment status, 1 = September ... 6 = April
AMOUNTS = ["LIMIT_BAL"] + [f"BILL_AMT{m}" for m in range(1, 7)] + [f"PAY_AMT{m}" for m in range(1, 7)]
CATEGORIES = ["EDUCATION"] + STATUS
FEATURES = CATEGORIES + AMOUNTS
SENSITIVE = ["SEX", "MARRIAGE", "AGE"]  # never inputs: only for slice checks
TARGET = "default"
print(len(FEATURES), "features | kept out of the inputs:", SENSITIVE)

The output:

20 features | kept out of the inputs: ['SEX', 'MARRIAGE', 'AGE']

What this tells us. The model will see 20 columns. The three sensitive ones are named in one place, so no later step adds them by accident.

Explore what the decision needs

Arjun does not look at every column. He asks two questions that the decision needs. First: how does the risk change with the most recent status?

by_status = clients.groupby(clients["PAY_1"].clip(upper=3))[TARGET].agg(clients="size", default_rate="mean")
print(by_status.round(3))

The output:

       clients  default_rate
PAY_1
-2        2759         0.132
-1        5686         0.168
 0       14737         0.128
 1        3688         0.339
 2        2667         0.691
 3         463         0.719

Default rate in October by the repayment status in September, all 30,000 clients. The undocumented codes −2 and 0 behave like 'paid on time' (−1); a delay of one month or more raises the rate sharply.

Text description of the chart

A column chart with six columns, one per September status code. −2 (2,759 clients): 13%. −1, paid on time (5,686): 17%. 0 (14,737): 13%. 1, one month late (3,688): 34%. 2, two months late (2,667): 69%. 3 or more months late (463): 72%. The first three columns are low and similar; the last three rise steeply.

What this tells us. A delay in September is the strongest single sign: of the clients two months late, about 7 in 10 default. The three codes −2, −1 and 0 have similar, low rates, and 0 is the lowest. If the model read the codes as numbers, it would assume that the risk rises from −2 to 0. The data says that it does not, which confirms the choice of categories.

Second: what do the amounts look like, and does the credit limit matter?

print(clients[["LIMIT_BAL", "BILL_AMT1", "PAY_AMT1"]].agg(["median", "mean", "max"]).round(0))
limit_band = pd.cut(clients["LIMIT_BAL"], [0, 50_000, 140_000, 240_000, 1_000_000],
                    labels=["up to 50k", "50k-140k", "140k-240k", "over 240k"])
print(clients.groupby(limit_band, observed=True)[TARGET].agg(clients="size", default_rate="mean").round(3))

The output:

        LIMIT_BAL  BILL_AMT1  PAY_AMT1
median   140000.0    22382.0    2100.0
mean     167484.0    51223.0    5664.0
max     1000000.0   964511.0  873552.0
           clients  default_rate
LIMIT_BAL
up to 50k     7676         0.318
50k-140k      7614         0.247
140k-240k     7643         0.173
over 240k     7067         0.140

What this tells us. The amounts are very skewed. The median September payment is NT$2,100, and the largest is NT$873,552, more than 400 times as much. In a linear model, a few huge values would pull the weights, so the pipeline takes a logarithm of the amounts first. The credit limit matters too. Clients with a limit up to NT$50,000 default 31.8% of the time, more than twice the rate over NT$240,000. Keep this in mind. The limit is not a fact about the client alone, because the bank set it. It comes back in the fairness checks.

Split and baseline

Arjun splits the clients into three sets, as in Train, validation and test sets. Train (60%) is for fitting, validation (20%) for choosing, and test (20%) for one final check.

Kevin, an analyst on another team, asks: “Why not split by time, as Larkfield did?” Because there is no time to split by. All 30,000 clients come from the same months, and nothing says which statement came first. The model will score next month’s clients of the same bank, who are like these. So a random split, stratified by the target, matches the use: each set gets the same share of defaults. The cost is that this data cannot show what happens when the economy changes. That goes into the limits.

from sklearn.model_selection import train_test_split

rest, test = train_test_split(clients, test_size=0.2, stratify=clients[TARGET], random_state=0)
train, valid = train_test_split(rest, test_size=0.25, stratify=rest[TARGET], random_state=0)
for name, part in [("train", train), ("valid", valid), ("test", test)]:
    print(f"{name:5s} {len(part):6d} clients, {part[TARGET].sum():5d} defaults ({part[TARGET].mean():.1%})")

The output:

train  18000 clients,  3982 defaults (22.1%)
valid   6000 clients,  1327 defaults (22.1%)
test    6000 clients,  1327 defaults (22.1%)

What this tells us. The three sets have the same default rate. The test set is now sealed until the end.

The cost of a decision

The team’s decision has a price, so the evaluation counts the price. report() compares yes/no flags with what happened. It gives the counts of Classification metrics, precision, recall, and the total cost: 5 for each missed default plus 1 for each false alarm. Then Arjun scores three baselines on the validation set: call nobody, call everybody, and today’s rule, “call every client who was late in September”.

MISS_COST, ALARM_COST = 5, 1


def report(y, flags):
    """Flags against the real answers: counts, precision, recall and the cost."""
    y, flags = np.asarray(y, dtype=bool), np.asarray(flags, dtype=bool)
    caught, false_alarms, missed = int((flags & y).sum()), int((flags & ~y).sum()), int((~flags & y).sum())
    cost = MISS_COST * missed + ALARM_COST * false_alarms
    return {"flagged": round(flags.mean(), 3), "caught": caught, "false alarms": false_alarms, "missed": missed,
            "precision": round(caught / max(caught + false_alarms, 1), 3), "recall": round(caught / y.sum(), 3),
            "cost": cost, "cost per 1,000": round(1000 * cost / len(y))}


y_valid = valid[TARGET]
baselines = pd.DataFrame([report(y_valid, np.zeros(len(valid))), report(y_valid, np.ones(len(valid))),
                          report(y_valid, valid["PAY_1"] >= 1)],
                         index=["call nobody", "call everybody", "rule: late in September"])
print(baselines.to_string())

The output:

                         flagged  caught  false alarms  missed  precision  recall  cost  cost per 1,000
call nobody                0.000       0             0    1327      0.000   0.000  6635            1106
call everybody             1.000    1327          4673       0      0.221   1.000  4673             779
rule: late in September    0.228     678           689     649      0.496   0.511  3934             656

What this tells us. Calling nobody costs 6,635 (1,327 misses × 5). Calling everybody is cheaper, 4,673, because misses are so expensive. Today’s rule is the best of the three. It calls 22.8% of clients, catches about half of the defaults (recall 0.511), and costs 3,934. That is the number to beat. If a model cannot beat a one-line rule, the team should keep the rule.

Models

The pipeline

The pipeline follows Preprocess with a pipeline, with two choices from the checks above. The status codes and education go through one-hot encoding, so each code gets its own column. A code that fewer than 50 train clients have goes into one shared “infrequent” column, so a rare code cannot get a wild weight. The amounts go through a signed log: log(1 + |x|) with the sign of x. It shrinks large amounts and keeps the negative bills negative. Then the amounts are scaled.

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler


def signed_log(values):
    """log(1 + |x|) with the sign of x: it shrinks large amounts and keeps negative bills negative."""
    return np.sign(values) * np.log1p(np.abs(values))


def build_pipeline(categories=CATEGORIES, amounts=AMOUNTS, log=True, C=1.0):
    """Status codes one-hot (rare codes grouped), amounts log-scaled and standardised, then logistic regression."""
    amount_steps = [("log", FunctionTransformer(signed_log, feature_names_out="one-to-one"))] if log else []
    amount_steps.append(("scale", StandardScaler()))
    prepare_step = ColumnTransformer([
        ("status", OneHotEncoder(handle_unknown="infrequent_if_exist", min_frequency=50), categories),
        ("amounts", Pipeline(amount_steps), amounts),
    ])
    return Pipeline([("prepare", prepare_step), ("model", LogisticRegression(max_iter=2000, C=C))])

Why logistic regression first? It gives a score from 0 to 1 that is often close to a real chance, and each client’s score splits into reasons. A bank that must tell a client why it acted needs both (see When a simple model is better).

The experiment log

Every experiment is scored the same way: average precision on the same 5 stratified folds of train, as in Machine learning is iterative. Why average precision? While the models change, the threshold is not chosen yet. Average precision judges how well the scores put the defaults at the top of the list, over all thresholds. The 78% of clients who pay cannot make it look good, as they can with accuracy. The threshold comes later, from the costs. The notebook defines cv_ap() for a pipeline, rule_ap() for a fixed rule, and log_experiment(). The last one adds a row to experiment_log.csv with this course’s columns: id, date, change, cv_ap, cv_std, valid_ap, decision and reason.

The first three rows are the floor (one score for every client), today’s rule as a 0/1 score, and a model with one column:

log_experiment("E01", "Baseline: one score for every client (the train rate)",
               rule_ap(lambda part: np.full(len(part), train[TARGET].mean())), "baseline", "The floor.")
log_experiment("E02", "Baseline: the rule 'late in September' as a 0/1 score",
               rule_ap(lambda part: (part["PAY_1"] >= 1).astype(int)), "baseline", "What the team does today.")
log_experiment("E03", "Logistic regression, PAY_1 only (one-hot)", cv_ap(build_pipeline(["PAY_1"], []), ["PAY_1"]),
               "reject", "Above the rule, but it ignores five months of history.")

The output:

E01  CV AP 0.221 ± 0.000  baseline: Baseline: one score for every client (the train rate)
E02  CV AP 0.367 ± 0.010  baseline: Baseline: the rule 'late in September' as a 0/1 score
E03  CV AP 0.453 ± 0.012  reject: Logistic regression, PAY_1 only (one-hot)

What this tells us. The floor is 0.221, the share of defaults. The rule reaches 0.367. One column used well, with a separate risk for each code, already reaches 0.453. The model knows that “2 months late” is much worse than “1 month late”, while the rule treats them the same.

Next come all 20 features, in three versions. One of them is a mistake on purpose: the status codes read as numbers.

log_experiment("E04", "All 20 features, status codes as numbers",
               cv_ap(build_pipeline(["EDUCATION"], AMOUNTS + STATUS), FEATURES),
               "reject", "The codes -2, -1, 0 are not amounts; as numbers they lose information.")
log_experiment("E05", "All 20 features, amounts without the log", cv_ap(build_pipeline(log=False), FEATURES),
               "reject", "Lower than E06 on 4 of 5 folds.")
log_experiment("E06", "All 20 features, status one-hot, signed log of amounts", cv_ap(build_pipeline(), FEATURES),
               "keep", "Best so far; train AP close to CV, so it does not overfit.")

The output:

E04  CV AP 0.495 ± 0.015  reject: All 20 features, status codes as numbers
E05  CV AP 0.536 ± 0.007  reject: All 20 features, amounts without the log
E06  CV AP 0.542 ± 0.007  keep: All 20 features, status one-hot, signed log of amounts

What this tells us. Treating the codes as categories is worth 0.047 AP (0.495 against 0.542). That is much more than the noise of 0.007 to 0.015. It is the price of an undocumented code read the wrong way. The log adds only 0.006, but it is higher on 4 of the 5 folds and costs nothing. So E06 is the model to beat.

Common mistake

Codes read as numbers

Symptom: Kevin gives the status columns to the model as numbers, because “they are already numbers”. There is no error. The CV AP is 0.495 instead of 0.542.

Cause: a linear model gives a numeric column one weight. So it assumes that each step from −2 to −1 to 0 to 1 adds the same risk. The data says otherwise: −2 has a default rate of 0.132, −1 has 0.168 and 0 has 0.128. These codes are labels that happen to be written as digits.

Fix: read the documentation, and print the rate for each code before you choose an encoding. Codes go through one-hot encoding, or into a tree, which can split anywhere.

Consequence in AI work: this mistake hides in plain sight. Postcodes, product codes, error codes and diagnosis codes all look like numbers. A model that reads them as numbers still trains and still gives scores, only worse ones, and nothing warns you.

Arjun tries his own idea next: two ratios, the utilisation (September bill ÷ limit) and the paid share (September payment ÷ August bill). In the notebook, this is your task. The result:

E07  CV AP 0.541 ± 0.008  reject: E06 + utilisation and paid share

What this tells us. There is no gain. The model already sees the bills, the payments and the limit, so the ratios add nothing new. A feature that sounds clever is not always useful, and two more columns would need explaining for nothing.

Then the regularization setting C (see Tune hyperparameters), with a small grid on the same folds:

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(build_pipeline(), {"model__C": [0.01, 0.1, 1, 10]}, cv=cv, scoring="average_precision")
search.fit(train[FEATURES], train[TARGET])
grid = pd.DataFrame(search.cv_results_)[["param_model__C", "mean_test_score", "std_test_score"]]
print(grid.round(4).to_string(index=False))
log_experiment("E08", "E06 with C = 10, the best of the grid", cv_ap(build_pipeline(C=10), FEATURES), "reject",
               "C = 0.1, 1 and 10 tie (0.541 to 0.542); keep the default C = 1 (E06).")

The output:

 param_model__C  mean_test_score  std_test_score
           0.01           0.5349          0.0084
           0.10           0.5413          0.0070
           1.00           0.5416          0.0069
          10.00           0.5417          0.0068
E08  CV AP 0.542 ± 0.007  reject: E06 with C = 10, the best of the grid

What this tells us. C = 0.1, 1 and 10 are within 0.0004 of each other, far inside the noise. With 18,000 clients and 55 prepared columns, the model has little room to overfit, so the default stays.

Two experiments that test a policy

Two more rows test questions that are not only about the score. First, what do the personal facts add? Second, does a more complex model earn its place?

from sklearn.ensemble import HistGradientBoostingClassifier

log_experiment("E09", "E06 + SEX, MARRIAGE and AGE as inputs",
               cv_ap(build_pipeline(CATEGORIES + ["SEX", "MARRIAGE"], AMOUNTS + ["AGE"]), FEATURES + SENSITIVE),
               "reject", "Policy: personal facts are not inputs; the gain (0.001) is smaller than the noise.")
e10_folds = cv_ap(HistGradientBoostingClassifier(random_state=0), FEATURES)
print("E10 - E06 by fold:", (e10_folds - cv_ap(build_pipeline(), FEATURES)).round(3))

The output:

E09  CV AP 0.543 ± 0.007  reject: E06 + SEX, MARRIAGE and AGE as inputs
E10 - E06 by fold: [ 0.005  0.014  0.013  0.013 -0.004]

What this tells us. The personal facts add 0.001, which is nothing: six months of history already carry the signal. Arjun would reject them even with a gain, because the bank decided that they are not inputs, and the log records that reason. Gradient boosting is better on 4 of 5 folds, by about 0.008. That is small but steady, so Arjun checks both on the validation set, once:

from sklearn.metrics import brier_score_loss, roc_auc_score

pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
booster = HistGradientBoostingClassifier(random_state=0).fit(train[FEATURES], train[TARGET])
booster_scores = booster.predict_proba(valid[FEATURES])[:, 1]
for name, s in [("logistic regression (E06)", scores), ("boosting (E10)", booster_scores)]:
    print(f"{name:26s} ROC AUC {roc_auc_score(y_valid, s):.3f}  AP {average_precision_score(y_valid, s):.3f}")

The output:

logistic regression (E06)  ROC AUC 0.775  AP 0.555
boosting (E10)             ROC AUC 0.783  AP 0.559

What this tells us. Boosting ranks a little better: +0.008 ROC AUC. In money, at each model’s cheapest threshold, it would save 56 cost units on 6,000 clients. That is about 9 per 1,000 clients, or 1.7% of the cost. Against that, boosting cannot split one client’s score into exact reasons, and the team must tell clients why they got the call. When a simple model is better describes this trade, and a 1.7% gain does not pay for that loss here. Boosting stays in the log as a challenger: a model to try again if the data grows or the gain becomes larger.

Here is the log after ten experiments:

 id                                                          change  cv_ap  cv_std  valid_ap decision
E01           Baseline: one score for every client (the train rate)  0.221   0.000       NaN baseline
E02           Baseline: the rule 'late in September' as a 0/1 score  0.367   0.010       NaN baseline
E03                       Logistic regression, PAY_1 only (one-hot)  0.453   0.012       NaN   reject
E04                        All 20 features, status codes as numbers  0.495   0.015       NaN   reject
E05                        All 20 features, amounts without the log  0.536   0.007       NaN   reject
E06          All 20 features, status one-hot, signed log of amounts  0.542   0.007     0.555     keep
E07                                E06 + utilisation and paid share  0.541   0.008       NaN   reject
E08                           E06 with C = 10, the best of the grid  0.542   0.007       NaN   reject
E09                           E06 + SEX, MARRIAGE and AGE as inputs  0.543   0.007       NaN   reject
E10 Gradient boosting (HistGradientBoosting, defaults), 20 features  0.550   0.006     0.559   reject

What this tells us. Ten tries gave one keeper, with a reason for every rejection. Most of the gain came from understanding the data (E03 to E06), not from tuning (E08) or a stronger algorithm (E10). That pattern is common.

Metric and operating point

A threshold from the costs

The model gives scores, but the team needs “call” or “do not call”. At Larkfield, the team’s capacity chose the threshold. Here the team can call as many clients as it needs, so the costs choose it. Arjun tries every threshold from 0.01 to 0.99 on the validation set and keeps the cheapest:

THRESHOLDS = np.round(np.arange(0.01, 1.0, 0.01), 2)
costs = pd.Series([report(y_valid, scores >= t)["cost"] for t in THRESHOLDS], index=THRESHOLDS)
threshold = float(costs.idxmin())
print("cheapest threshold:", threshold, "| cost:", int(costs.min()))
table = pd.DataFrame([report(y_valid, scores >= t) for t in [0.5, 0.3, 0.2, threshold, 0.1]],
                     index=[0.5, 0.3, 0.2, threshold, 0.1])
print(table.to_string())

The output:

cheapest threshold: 0.17 | cost: 3355
      flagged  caught  false alarms  missed  precision  recall  cost  cost per 1,000
0.50    0.115     478           213     849      0.692   0.360  4458             743
0.30    0.206     687           547     640      0.557   0.518  3747             624
0.20    0.325     851          1100     476      0.436   0.641  3480             580
0.17    0.399     946          1450     381      0.395   0.713  3355             559
0.10    0.778    1240          3425      87      0.266   0.934  3860             643

Total cost of the 6,000 validation clients at each threshold, for a missed default at 5 and a false alarm at 1. The curve is lowest at 0.17 and nearly flat from 0.14 to 0.17.

Text description of the chart

A line chart. The horizontal axis is the threshold from 0.05 to 0.60; the vertical axis is the total cost from 3,000 to 5,200. The line starts high on the left, because a very low threshold calls almost everybody, and falls to its lowest point at 0.17 (3,355). From 0.14 (3,375) to 0.17 it is almost flat. To the right it rises: 3,480 at 0.20, 3,747 at 0.30, and 4,458 at the default 0.50, because more and more defaults are missed.

What this tells us. The cheapest threshold is 0.17. There the team calls 39.9% of clients and catches 946 of the 1,327 defaults (recall 0.713), for a cost of 3,355. That is 14.7% less than today’s rule (3,934), and 24.7% less than the same model at the default threshold of 0.5. At 0.5 the model looks precise (0.692), but it misses almost two defaults in three: precise and expensive.

Why is the threshold so low? A miss costs five times a false alarm. So a call is worth it even when the chance of default is well below one half. Suppose a score is a true chance p. Then the expected cost of a call is (1 − p) × 1, and of no call p × 5. The call is cheaper when p is above 1 ÷ (1 + 5) = 0.167. The search on the data found 0.17, so theory and data agree. That agreement has a condition, which the next section checks.

The threshold belongs to the costs, not to the model. In the notebook’s “change one thing” task, a miss costs 3 instead of 5. Then the cheapest threshold moves up to 0.25, as the theory predicts (1 ÷ 4). If Mei-Ling’s costs change, the threshold changes, and the model stays the same.

Calibration

The rule “call above 0.167” assumes that a score of 0.17 means about a 17% chance of default. That property is calibration (Thresholds and calibration). Arjun groups the validation scores into bins:

bins = [0, 0.1, 0.2, 0.3, 0.5, 0.7, 1]
calibration = pd.DataFrame({"score": scores, "default": y_valid.to_numpy()}).groupby(
    pd.cut(scores, bins), observed=True).agg(clients=("default", "size"), mean_score=("score", "mean"),
                                             default_rate=("default", "mean"))
print(calibration.round(3))
print("Brier:", round(brier_score_loss(y_valid, scores), 4), "| mean score", round(scores.mean(), 3),
      "| default rate", round(y_valid.mean(), 3))

The output:

            clients  mean_score  default_rate
(0.0, 0.1]     1335       0.081         0.065
(0.1, 0.2]     2714       0.138         0.143
(0.2, 0.3]      717       0.239         0.229
(0.3, 0.5]      543       0.375         0.385
(0.5, 0.7]      366       0.605         0.628
(0.7, 1.0]      325       0.789         0.763
Brier: 0.1346 | mean score 0.223 | default rate 0.221

• Validation clients
• Perfect calibration (dashed)

Reliability chart of the model on the 6,000 validation clients, in six score bins. Every dot is close to the diagonal, so the scores can be read as chances.

Text description of the chart

A square chart. The horizontal axis is the mean score in a bin and the vertical axis the observed default rate, both from 0 to 0.85, with a dashed diagonal for perfect calibration. Six dots lie close to the diagonal: (0.081, 0.065) with 1,335 clients, (0.138, 0.143) with 2,714, (0.239, 0.229) with 717, (0.375, 0.385) with 543, (0.605, 0.628) with 366, and (0.789, 0.763) with 325. The largest gap is in the top bin, 2.6 percentage points.

What this tells us. In every bin, the mean score is within 2.6 percentage points of the real default rate. On average the model says 22.3%, and 22.1% default. So the scores can be read as chances on this data, and the cost rule is sound. Compare Larkfield, whose scores were too low after a policy change: there, train and validation came from different months. Here they come from the same months, so this check cannot see a future change. The monitoring plan must watch for one.

Reasons for one client

A client who gets the call may ask “why me?”, and the agent needs an honest answer. For logistic regression, a score splits into contributions: coefficient times prepared value (Explain predictions). Twenty columns with one-hot codes give 55 small numbers, too many for an agent. So contributions() adds them up into six reason groups. Arjun then compares one client with the average train client:

REASON_OF = {"PAY_1": "late status in September", "LIMIT_BAL": "credit limit", "EDUCATION": "education",
             **{f"PAY_{m}": "late status, April to August" for m in range(2, 7)},
             **{f"BILL_AMT{m}": "bill amounts" for m in range(1, 7)},
             **{f"PAY_AMT{m}": "payments made" for m in range(1, 7)}}


def contributions(pipeline, part):
    """Coefficient times prepared value, summed into the six reason groups, one row per client."""
    prepare_step, model = pipeline.named_steps["prepare"], pipeline.named_steps["model"]
    values = np.asarray(prepare_step.transform(part[FEATURES])) * model.coef_[0]
    names = [n.split("__", 1)[1] for n in prepare_step.get_feature_names_out()]
    columns = [next(c for c in FEATURES if n == c or n.startswith(c + "_")) for n in names]
    table = pd.DataFrame(values, columns=[REASON_OF[c] for c in columns], index=part.index)
    return table.T.groupby(level=0).sum().T


average = contributions(pipeline, train).mean()
client = valid.assign(score=scores)[valid["ID"] == 3008]
print(client[["ID", "LIMIT_BAL"] + STATUS + ["PAY_AMT1", "PAY_AMT2", "score"]].to_string(index=False))
reasons = (contributions(pipeline, client).iloc[0] - average).sort_values(ascending=False)
print(reasons.round(2).to_string())

The output:

  ID  LIMIT_BAL  PAY_1  PAY_2  PAY_3  PAY_4  PAY_5  PAY_6  PAY_AMT1  PAY_AMT2  score
3008      10000      1      2      2      2      2      2         0      1600 0.7283
late status, April to August    0.87
payments made                   0.72
credit limit                    0.46
late status in September        0.37
education                       0.04
bill amounts                    0.01

What this tells us. Client 3008 has a score of 0.73, far above the threshold. Compared with an average client, the three largest reasons are the status from April to August, the payments, and the limit. The client was two months late in each of those months, paid nothing in September, and has a very low limit (NT$10,000). The agent can say: “You were two months behind in each of the last five months, and no payment arrived in September. We would like to offer you a payment plan.” That is specific, and true to the model. One more fact: this client did pay in October. The reasons explain the score, not the future. This call was a false alarm, and a polite one.

Errors and slices

Where the model fails

At the threshold of 0.17, the model misses 381 defaults and makes 1,450 false alarms on the validation set. Arjun looks at who they are:

errors = valid.assign(score=scores, flag=scores >= threshold)
missed = errors[~errors["flag"] & (errors[TARGET] == 1)]
false_alarms = errors[errors["flag"] & (errors[TARGET] == 0)]
print("missed defaults:", len(missed), "| no delay in any of the 6 months:", int((missed[STATUS] <= 0).all(axis=1).sum()),
      "| median score", round(missed["score"].median(), 3))
print("false alarms:", len(false_alarms), "| late in September:", int((false_alarms["PAY_1"] >= 1).sum()),
      "| limit up to 50,000:", int((false_alarms["LIMIT_BAL"] <= 50_000).sum()))

The output:

missed defaults: 381 | no delay in any of the 6 months: 350 | median score 0.126
false alarms: 1450 | late in September: 639 | limit up to 50,000: 568

What this tells us. 350 of the 381 missed defaults had no delay in any of the six months. They looked like good payers until the day they did not pay. A lost job or an illness could cause that, and no column records it. No model on these columns can catch them, so the honest promise to Mei-Ling is “about 7 in 10 defaults”, not “all of them”. The false alarms are of two kinds. Some clients were late in September but caught up (639). Others have low limits (568 have NT$50,000 or less). The second kind leads to the fairness question.

Fairness slices

The model never sees sex, marital status or age. That does not prove that it treats these groups alike, because other columns can carry the same information. So Arjun checks each group, as in Slice analysis, always with the count. For fairness, the most important column is the false-alarm rate: of the clients who would have paid, the share who still got the call. These are the people who carry the cost of the model’s mistakes.

AGE_BANDS, AGE_LABELS = [20, 25, 30, 40, 50, 80], ["21-25", "26-30", "31-40", "41-50", "51+"]


def slice_table(part, part_scores, column, threshold=threshold):
    """Clients, default rate, flag rate, recall, false-alarm rate and ROC AUC per group."""
    frame = part.assign(score=part_scores, flag=part_scores >= threshold)
    rows = {}
    for group, g in frame.groupby(column, observed=True):
        good, bad = g[g[TARGET] == 0], g[g[TARGET] == 1]
        rows[group] = {"clients": len(g), "default rate": round(g[TARGET].mean(), 3),
                       "flagged": round(g["flag"].mean(), 3), "recall": round(bad["flag"].mean(), 3),
                       "false-alarm rate": round(good["flag"].mean(), 3),
                       "ROC AUC": round(roc_auc_score(g[TARGET], g["score"]), 3) if g[TARGET].nunique() == 2 else None}
    return pd.DataFrame(rows).T


valid_bands = valid.assign(age_band=pd.cut(valid["AGE"], AGE_BANDS, labels=AGE_LABELS))
for column in ["SEX", "MARRIAGE", "age_band"]:
    print(slice_table(valid_bands, scores, column).to_string(), end="\n\n")

The output:

   clients  default rate  flagged  recall  false-alarm rate  ROC AUC
1   2438.0         0.246    0.420   0.718             0.323    0.770
2   3562.0         0.204    0.385   0.708             0.302    0.777

   clients  default rate  flagged  recall  false-alarm rate  ROC AUC
0      8.0         0.125    0.625   1.000             0.571    0.571
1   2709.0         0.237    0.401   0.697             0.309    0.766
2   3208.0         0.209    0.396   0.726             0.308    0.785
3     75.0         0.213    0.480   0.750             0.407    0.739

       clients  default rate  flagged  recall  false-alarm rate  ROC AUC
21-25    822.0         0.260    0.523   0.794             0.428    0.787
26-30   1394.0         0.196    0.392   0.758             0.302    0.800
31-40   2102.0         0.212    0.363   0.649             0.286    0.754
41-50   1220.0         0.234    0.382   0.719             0.279    0.788
51+      462.0         0.238    0.413   0.682             0.330    0.734

What this tells us. By sex (1 male, 2 female), the false-alarm rates are close: 0.323 and 0.302. So is the ranking quality (ROC AUC 0.770 and 0.777). By marital status, the two small groups are too small to judge: 8 clients with code 0, and 75 with “others”. Their rates would move a lot with a few clients either way.

Age is different. Clients aged 21 to 25 are flagged 52.3% of the time, against 36% to 41% for the other bands. Part of that is real, because they default more often: 26.0%, against 21.2% for ages 31 to 40. But look at the good payers. A client aged 21 to 25 who would have paid gets the call 42.8% of the time. For a good payer aged 31 to 40, it is 28.6%. That is 1.5 times as often, and age is not an input. The model must get the information from somewhere else.

The proxy

The credit-limit table above gives a clue: low limits go with high default rates. Arjun checks the limits by age, and then the false-alarm rates within the same limit:

print(valid_bands.groupby("age_band", observed=True)["LIMIT_BAL"].median())
low = (valid_bands["LIMIT_BAL"] <= 50_000).to_numpy()
print("\nlimit up to 50,000:\n", slice_table(valid_bands[low], scores[low], "age_band")[["clients", "default rate", "false-alarm rate"]])
print("\nlimit over 50,000:\n", slice_table(valid_bands[~low], scores[~low], "age_band")[["clients", "default rate", "false-alarm rate"]])

The output:

age_band
21-25     50000.0
26-30    140000.0
31-40    180000.0
41-50    160000.0
51+      100000.0
Name: LIMIT_BAL, dtype: float64

limit up to 50,000:
        clients  default rate  false-alarm rate
21-25    481.0         0.304             0.573
26-30    301.0         0.302             0.514
31-40    339.0         0.336             0.520
41-50    301.0         0.336             0.485
51+      155.0         0.271             0.478

limit over 50,000:
        clients  default rate  false-alarm rate
21-25    341.0         0.199             0.249
26-30   1093.0         0.167             0.254
31-40   1763.0         0.188             0.249
41-50    919.0         0.200             0.223
51+      307.0         0.221             0.259

• All clients
• Limit up to NT$50,000
• Limit over NT$50,000

False-alarm rate by age band on the validation set, at the threshold 0.17. Over all clients, good payers aged 21–25 are called far more often. Within one limit band, the age bands are close: the credit limit carries the age effect.

Text description of the chart

A line chart with five age bands on the horizontal axis (21–25, 26–30, 31–40, 41–50, 51+) and the false-alarm rate from 0% to 60% on the vertical axis. The solid line for all clients starts high at 43% for 21–25, drops to 30%, 29% and 28%, and rises to 33% for 51+. The dashed line for limits up to NT$50,000 is high and fairly flat: 57%, 51%, 52%, 49%, 48%. The dashed line for limits over NT$50,000 is low and flat: 25%, 25%, 25%, 22%, 26%. The high value of the solid line at 21–25 comes from the mix: most young clients have low limits.

What this tells us. This is a proxy: a column that stands in for one that was left out. The median limit is NT$50,000 for clients aged 21–25, against NT$100,000 to NT$180,000 for the others. 59% of the youngest band have NT$50,000 or less. The model has learned that a low limit means risk. Within one limit band, the age bands are treated much more alike: the dashed lines are close to flat. So most of the age gap comes through the limit.

And the limit was set by the bank. If the bank gave young clients low limits because they were young, the model now repeats that old decision and calls it a prediction. Removing a column from the inputs does not remove its effect when another column carries it.

What the team can do

There is no single right answer, and some options raise legal questions that an analyst cannot settle. So Arjun lays out the options with evidence, for Mei-Ling and the compliance team to decide.

1. Measure the cost of removing the proxy. In the notebook task, Arjun refits the model without LIMIT_BAL. The false-alarm rate of good payers aged 21–25 falls from 0.428 to 0.373. The total cost rises from 3,355 to 3,425, about 2%. The gap shrinks but does not close, because the payment history also differs by age. The numbers script records it as E11 (CV AP 0.540, against 0.542).
2. Make a false alarm cheap and kind. The call offers help and pauses limit increases; it refuses nothing. If the script is respectful and the pause ends after a few good months, a false alarm costs the client little. A harsher action, such as a lower limit, would make the same gap far more serious.
3. Review the limits themselves. The deeper question is why young clients get low limits. That is a question about the bank’s limit policy, not about this model.
4. Do not “fix” it with age. Adding age as an input, or a different threshold per age band, uses age directly in a decision about a person. In the numbers script, age as an input moves the false alarms from the young to the clients over 50: 0.385 for 21–25, 0.406 for 51+. It does not remove the gap; it moves it. Laws on credit and on equal treatment limit the use of age in many countries, sometimes with exceptions.
5. Monitor it. Whatever the bank decides, report the false-alarm rate by age band every month, with the counts, next to the cost.

This is not legal advice

Rules on the use of age, sex, marital status and their proxies in credit differ by country and change over time. This case study shows how to find and measure the effect. Whether a model or a response is allowed is a question for a qualified lawyer or compliance team. When a simple model is better names some of the rules and their sources.

Arjun recommends options 2 and 5 now, and gives option 1 to Mei-Ling as a decision, with the numbers in front of her. The test below uses the model as it is, because no change was agreed before the test.

The final test, once

Every choice is made: the model E06, the threshold 0.17, the costs 5 and 1. Arjun opens the test set once, as in Thresholds and calibration, and compares with the rule on the same clients:

y_test = test[TARGET]
test_scores = pipeline.predict_proba(test[FEATURES])[:, 1]
print("ROC AUC", round(roc_auc_score(y_test, test_scores), 3), "| AP", round(average_precision_score(y_test, test_scores), 3),
      "| Brier", round(brier_score_loss(y_test, test_scores), 4))
final = pd.DataFrame([report(y_test, test_scores >= threshold), report(y_test, test["PAY_1"] >= 1),
                      report(y_test, np.zeros(len(test)))],
                     index=[f"model, threshold {threshold}", "rule: late in September", "call nobody"])
print(final.to_string())

The output:

ROC AUC 0.77 | AP 0.549 | Brier 0.1349
                         flagged  caught  false alarms  missed  precision  recall  cost  cost per 1,000
model, threshold 0.17      0.382     930          1361     397      0.406   0.701  3346             558
rule: late in September    0.224     688           657     639      0.512   0.518  3852             642
call nobody                0.000       0             0    1327      0.000   0.000  6635            1106

What this tells us. On 6,000 clients that no step has touched, the model costs 3,346, against 3,852 for the rule: 13.1% less. It catches 930 of 1,327 defaults (recall 0.701), against 688 for the rule, and it calls 38.2% of clients. ROC AUC (0.770) and AP (0.549) are a little below validation (0.775 and 0.555), as expected, because validation helped to choose. These are the numbers to report, and nothing changes after reading them.

The age slice on test tells the same story. Good payers aged 21–25 get the call 40.7% of the time, against 26.9% to 28.4% for every older band. The numbers script cs03_credit.py prints the full table. It is reported, not used to change the model.

Recommendations

Arjun writes to Mei-Ling in her words, not in model words:

• Use the model to choose the early-help calls. Call every client whose score is 0.17 or more. On new clients, this costs about 13% less than “call everyone who was late last month”. It reaches about 7 in 10 clients who will miss the next payment, against 5 in 10 today.
• Plan for the volume. It means calling about 38% of clients each month, against 22% today. If the team cannot make that many calls, call the highest scores first; the cost rises as the list gets shorter.
• Keep the call helpful. About 6 in 10 called clients would have paid anyway. The script should offer help, not warn of consequences.
• Use the reasons. Each call can come with the client’s top three reasons, such as “two months behind for five months”.
• Decide about the credit limit. Young clients who pay are called 1.5 times as often as clients aged 31 to 40, because their limits are lower. Dropping the limit from the model narrows the gap for about 2% more cost. This is a business and compliance decision.

And what not to use it for:

• Not for refusing credit, lowering limits or pricing. The costs, the threshold and the fairness check were made for a supportive call. A harsher action needs a new analysis and, probably, a new review.
• Not as a judgement of a person. A score of 0.7 means “about 7 in 10 clients like this did not pay in October 2005”. It does not mean “this client is unreliable”.
• Not for another bank, country or year without checking it on that data first.

Limits and next steps

• One bank, one place, 2005. The data comes from Taiwan during a period of high card debt. Default rates and patterns may be very different today or elsewhere.
• No time check. All clients come from the same months, so the test cannot show what happens when the economy changes. The first months of use are the real test. Monitor the flag rate, the mean score against the default rate, and the false-alarm rate by age band, as in Plan production checks.
• Undocumented codes. The status codes −2 and 0 and three education codes have no official meaning. If the bank can document them, check whether the model’s use of them makes sense.
• The costs are rough. 5 and 1 are Mei-Ling’s estimates, and the threshold moves with them (0.25 if a miss costs 3). Revisit them with real collection and call costs.
• Sudden defaults are invisible. 350 of 381 missed defaults had a clean history. New data, such as changes in income or other debts, might help; these columns cannot.
• The challenger. Boosting was better by about 1.7% of the cost. Try it again when there is more data, and only if the team can still give each client honest reasons.

Make it a portfolio piece

A case study like this shows an employer that you can take a real problem from data to a decision. Make it yours with one change, for example other costs, a capacity limit for the calls, or the same study on another open credit dataset. Then make it public, and make it easy to read in five minutes.

What to put in the repository:

• README.md, the page that people read first (template below).
• The notebook, run from top to bottom, with its outputs.
• experiment_log.csv, with every experiment and its reason.
• MODEL_CARD.md, as in Document use boundaries: intended use (the early-help call), not for (refusals, lower limits, pricing), data, evaluation, the age finding, limits.
• model_info.json from the notebook’s last cell: the features, the threshold, the costs and the data checksum. Leave out default_model.joblib unless you also say which versions load it; a reader can rebuild it from the notebook.
• The data source and licence, with the attribution text from the box above. CC BY 4.0 lets you share the data with credit, but a link and a checksum are enough.

A README template:

# Predicting credit-card default for early-help calls

## Problem
A card-risk team calls clients who may miss the next payment. A missed default costs
about 5 units, an unneeded call about 1. Today's rule: call everyone late last month.

## Data and licence
UCI "Default of Credit Card Clients" (Yeh, 2009), 30,000 clients, Taiwan 2005,
CC BY 4.0, https://doi.org/10.24432/C55S3H. Changes: renamed PAY_0 to PAY_1,
grouped undocumented education codes.

## Approach
Random stratified split 60/20/20. Logistic regression pipeline (status codes one-hot,
signed log of amounts). 10 experiments in experiment_log.csv. Threshold chosen on
validation by expected cost. Sex, marital status and age kept out of the inputs and
used for slice checks.

## Results (test set, used once)
Cost 3,346 vs 3,852 for the current rule (-13%); recall 0.70 vs 0.52; ROC AUC 0.77.

## Fairness finding
Good payers aged 21-25 are called 1.5x as often, through the credit limit (a proxy).
Removing the limit narrows the gap for +2% cost.

## Limits
One bank, one period, no time check; rough costs; undocumented codes.

How to describe it in an interview, in two sentences: “I built a logistic-regression model that chooses which credit-card clients get an early-help call, with the threshold set by the cost of each error; on a held-out test it cost 13% less than the bank’s existing rule. I also found that young clients were called more often through their lower credit limits, measured what removing that proxy would cost, and documented it for a business decision.”

Go further:

• Fit CalibratedClassifierCV and compare the reliability chart. Predict first whether it changes much here, and say why.
• Add a bootstrap interval for the cost difference between the model and the rule on the test set.
• Try boosting with monotonic constraints (the risk may only rise with the months of delay), and compare its cost and its reasons.
• Write the client-facing text for each of the six reason groups, and test it with a reader who is not technical.

Knowledge check

Check your understanding

The cheapest threshold on the validation set is 0.17, not 0.5. Explain why, and say what would change if a missed default cost the same as a false alarm.

Show the answer

A missed default costs 5 and a false alarm 1. So a call pays off even when the chance of default is well below one half. With calibrated scores, a call is cheaper when the chance p is above 1 ÷ (1 + 5) = 0.167. The search on the data found 0.17. If both errors cost the same, the break-even chance is 1 ÷ (1 + 1) = 0.5. The threshold would move up to about 0.5, and the team would make far fewer calls. The threshold comes from the costs, not from the model or the library’s default.

Check your understanding

Age is not an input, yet good payers aged 21 to 25 get the call 1.5 times as often as good payers aged 31 to 40. How can that happen, and how did the case study show it?

Show the answer

Another input carries the information: the credit limit. Young clients have much lower limits (median NT$50,000, against NT$180,000 for ages 31–40). The model learned that a low limit means risk. The case study showed it by comparing the false-alarm rates within one limit band. There the age bands were close: 0.478 to 0.573 up to NT$50,000, and 0.223 to 0.259 above it. A column that stands in for a removed one is a proxy. Leaving a sensitive column out of the inputs does not prove that the model ignores it; only a slice check can show that.

Check your understanding

Boosting has a higher ROC AUC (0.783 against 0.775) and would cut the validation cost by 56 units. Mei-Ling asks why the team does not use it. What do you answer?

Show the answer

The gain is small: 56 units on 6,000 clients, about 1.7% of the cost, and CV showed only about 0.008 AP. Against it, the team must tell each called client why, and logistic regression gives exact reasons per client while boosting does not. The bank also needs a model that a reviewer can check. So the simple model wins this trade. Boosting stays in the log as a challenger. If more data makes the gain larger, the team can weigh it again.

Recap

• You framed a credit-risk decision with costs: a miss 5, a false alarm 1. You kept sex, marital status and age out of the inputs, and treated undocumented codes as categories.
• The experiment log tried ten things. Most of the gain came from understanding the data; tuning and boosting added little.
• The threshold came from the costs: 0.17, which matches the theory, 1 ÷ (1 + 5), because the scores are calibrated.
• Slices found a proxy: young good payers are called 1.5 times as often, through the credit limit. You measured one response (without the limit: 0.428 to 0.373, for 2% more cost) and left the decision to the business and compliance.
• The test, used once: 13.1% less cost than the current rule, recall 0.701.

Next, a different decision: a call centre that can call only 10% of a list, with a decision tree that the team can read.

Next: Rank a call list