Deep network or gradient boosting?

What you will be able to do

Find out, on a real table, whether a neural network is worth its cost. You will train a network and gradient boosting under the same split, the same metric and the same tuning budget. Then you compare the score, the spread across seeds, the preparation, the training time and the explanations, and write an honest verdict.

Case studyPortfolio piece

Dataset

Default of credit card clients, Taiwan 2005 (UCI)

Licence

CC BY 4.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Fair model comparison
• Network for a table
• Early stopping
• Seeds and spread
• Bootstrap interval
• Permutation importance

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page with the same code, plus three tasks with checks. Its setup cells download the data (about 1 MB) and check its checksum, so it needs no local setup. It trains about 70 small models and runs in about 2 minutes on a CPU.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

• The five-step training loop, from The training loop, and model.eval() with torch.no_grad(), from Batches, epochs and validation.
• Early stopping and dropout, from Overfitting controls.
• Gradient boosting, from Ensembles in the machine learning course, and average precision, from The precision–recall trade-off.
• The pipeline with one-hot encoding and scaling, from Preprocess with a pipeline.
• Helpful, not required: the machine learning course’s case study Predict credit-card default. It explains this data in full. This page repeats only what the comparison needs.

What you need on your computer (only for the local route; the notebook needs none of it)

• The virtual environment of Set up PyTorch, with pandas and scikit-learn added (pip install pandas scikit-learn).
• The notebook, downloaded from the link above. Its setup cell downloads the data into a new folder credit-network. To start again, delete that folder.

Tested on macOS (Apple M4, CPU only), in Jupyter from a fresh start. Versions: Python 3.12, PyTorch 2.14.1, scikit-learn 1.9.1 and pandas 3.0.6, with two CPU threads for each model. Windows and Linux are not tested. Training times depend on your computer. The scores were the same in every run on this computer. On another computer, the last digit can differ.

The situation

Mei-Ling Chen leads the card-risk team of a bank in Taiwan. Each month her team chooses clients for an early-help call: an agent offers a payment plan before the client falls behind. A call to a client who would have paid costs about 1 unit of agent time and goodwill. A missed default costs about 5 units, because part of that debt never comes back.

In the machine learning course, her analyst Arjun built a logistic regression pipeline for this decision. It cut the cost of today’s rule by 13% on the test set. Gradient boosting was a little better, but Arjun kept it only as a challenger. Logistic regression gives each client exact reasons.

Now Daniel joins the team, fresh from a deep learning course: “Networks recognise faces and translate languages. Shouldn’t a network beat both of your models here?” Arjun does not argue. He proposes a fair contest, a network against gradient boosting under the same rules, and Mei-Ling will decide from the result.

What you will build

You will build two challengers for the same decision. The first is scikit-learn’s HistGradientBoostingClassifier, which reads the table as it is. The second is a small network for a table in PyTorch, with the course’s loop, early stopping and dropout. Both get the same tuning budget and 10 seeds each, then meet the sealed test set once. At the end you write a verdict: the score, its uncertainty, the work, the training time and the explanations.

Why this question matters

Module 1 gave a rule of thumb in When deep learning is worth it. Networks win on raw input such as images, sound and free text. On a table of facts that people chose, gradient boosting is very hard to beat. Published comparisons on many tables agree: boosted trees usually win or tie. But a rule of thumb is not evidence for your data, so Arjun and Daniel measure instead of guessing.

The data

The data is the Default of Credit Card Clients dataset from the UCI Machine Learning Repository. It has 30,000 real, anonymous clients of one bank in Taiwan. Each client has six months of history, from April to September 2005, and the outcome: did the client pay in October?

Source and licence

“Default of Credit Card Clients” by I-Cheng Yeh, UCI Machine Learning Repository, doi.org/10.24432/C55S3H, licensed under CC BY 4.0. Nextia Learning converted the Excel file to a compressed CSV, removed its extra first header row, and renamed the target column to default; nothing else changed. Paper: Yeh, I.-C. and Lien, C.-H. (2009), Expert Systems with Applications 36(2), 2473–2480.

The machine learning course’s case study checked this file. Three findings matter for a network:

• Codes that look like numbers. The repayment status uses −2, −1, 0 and 1 to 9; −2 and 0 are not documented. These are categories, not amounts.
• Skewed amounts. The median September payment is NT$2,100; the largest is NT$873,552.
• Sensitive columns. Sex, marital status and age are kept out of the inputs, by the bank’s decision.

So the inputs are 20 columns: education, six months of repayment status, the credit limit, six bills and six payments.

One split for both

A fair contest needs the same split for both models. Arjun reuses the split from the machine learning course: 60% train, 20% validation, 20% test, stratified by the target, with random_state=0. The data has no dates, so a random split matches the use. The cost is that it cannot show what happens when the economy changes.

compare.py

from sklearn.model_selection import train_test_split

raw = pd.read_csv("credit_default_taiwan.csv.gz")
clients = raw.rename(columns={"PAY_0": "PAY_1"})                      # 1 = September in every group
clients["EDUCATION"] = clients["EDUCATION"].replace({0: 4, 5: 4, 6: 4})  # undocumented codes -> "others"
STATUS = [f"PAY_{m}" for m in range(1, 7)]
AMOUNTS = ["LIMIT_BAL"] + [f"BILL_AMT{m}" for m in range(1, 7)] + [f"PAY_AMT{m}" for m in range(1, 7)]
CATEGORIES = ["EDUCATION"] + STATUS
FEATURES = CATEGORIES + AMOUNTS        # 20 inputs; SEX, MARRIAGE and AGE are never inputs
TARGET = "default"

rest, test = train_test_split(clients, test_size=0.2, stratify=clients[TARGET], random_state=0)
train, valid = train_test_split(rest, test_size=0.25, stratify=rest[TARGET], random_state=0)
y_train, y_valid, y_test = (part[TARGET].to_numpy() for part in (train, valid, test))
for name, part in [("train", train), ("valid", valid), ("test", test)]:
    print(f"{name:5s} {len(part):6d} clients, {part[TARGET].sum():5d} defaults ({part[TARGET].mean():.1%})")

The output:

train  18000 clients,  3982 defaults (22.1%)
valid   6000 clients,  1327 defaults (22.1%)
test    6000 clients,  1327 defaults (22.1%)

What this tells us. These are the same sets as in the machine learning course, so the numbers compare directly with its published results. Nobody looks at the test set until the end.

The rules of the contest

A comparison is fair only when the two models differ in one thing: the model. Arjun writes the rules down before any training:

1. The same split and the same inputs. Both models see the same 20 columns of the same 18,000 training clients.
2. The same metric. Average precision (AP) on the validation set chooses between settings. It says how well the scores put the defaults at the top of the list. The cost at the cheapest threshold (miss 5, false alarm 1) says what the decision costs. The threshold is always chosen on validation.
3. The same held-out share for stopping. Boosting holds out 10% of train inside fit() and stops adding trees when that part stops improving. The network gets a stopping set of the same size for early stopping.
4. The same tuning budget. Each model gets 8 settings, and each setting runs with 3 seeds.
5. Several seeds. The final recipes run with 10 seeds each, and the page reports the spread, not only the best run.
6. The test set once, at the end, for every model.

The contest. Only the model changes; the split, the inputs, the metric, the held-out share and the tuning budget stay the same.

Text description of the diagram

A left-to-right flow. A box “30,000 clients, 20 input columns” leads to “One split: train 18,000, valid 6,000, test 6,000 (sealed)”. From there two lanes run side by side. The gradient boosting lane: “Columns as they are, no preparation”, then “fit() holds out 10% and stops adding trees”. The network lane: “One-hot, log, scale, 20 to 55 columns”, then “Training loop, stops on a 10% stopping set”. Both lanes join in “Same budget: 8 settings × 3 seeds, median AP on valid”, then “10 seeds each, spread on valid”, then “Test, once: AP and cost”.

The notebook defines three small scoring functions, the same as in the machine learning course. cost() adds 5 for each missed default and 1 for each false alarm. cheapest() tries every threshold from 0.01 to 0.99 and returns the cheapest one with its cost. metrics() returns AP and ROC AUC.

The models to beat

Before any challenger, Arjun scores the two models that exist. One is today’s rule: “call every client who was late in September”. The other is the machine learning course’s logistic regression. preprocessor() is that course’s preparation. It one-hot encodes the codes. It takes a signed log of the amounts, log(1 + |x|) with the sign of x, and then scales them. The network will need the same function, so it has switches that leave out each step:

compare.py

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler


def signed_log(values):
    return np.sign(values) * np.log1p(np.abs(values))


def preprocessor(log=True, scale=True, codes_as_categories=True):
    """The machine learning course's preparation: codes one-hot, amounts signed log, then scaled."""
    steps = [("log", FunctionTransformer(signed_log))] if log else []
    steps += [("scale", StandardScaler())] if scale else []
    steps = steps or [("keep", "passthrough")]
    if not codes_as_categories:
        return ColumnTransformer([("all", Pipeline(steps), FEATURES)])
    return ColumnTransformer([
        ("codes", OneHotEncoder(handle_unknown="infrequent_if_exist", min_frequency=50, sparse_output=False),
         CATEGORIES),
        ("amounts", Pipeline(steps), AMOUNTS)])


logistic = Pipeline([("prepare", preprocessor()), ("model", LogisticRegression(max_iter=2000))])
logistic.fit(train[FEATURES], y_train)
logistic_valid = logistic.predict_proba(valid[FEATURES])[:, 1]
rule_valid = (valid["PAY_1"] >= 1).to_numpy()
print("rule: late in September  ", metrics(y_valid, rule_valid.astype(float)), "cost", cost(y_valid, rule_valid))
print("logistic regression      ", metrics(y_valid, logistic_valid), "cheapest threshold and cost",
      cheapest(y_valid, logistic_valid))

The output:

rule: late in September   {'ap': 0.3616, 'auc': 0.6817} cost 3934
logistic regression       {'ap': 0.5546, 'auc': 0.7748} cheapest threshold and cost (0.17, 3355)

What this tells us. These are the machine learning course’s numbers: AP 0.555, and a cost of 3,355 against 3,934 for the rule. A challenger must beat 0.555 clearly enough to pay for what it costs.

Gradient boosting, as it is

Gradient boosting builds many small trees, one after another. Each tree corrects the mistakes of the trees before it. A tree asks questions such as “is the September status above 0.5?” or “is the bill above NT$12,000?”. So the scale of a column does not matter, and a code can be split at any value. Boosting reads the 20 columns as they are:

compare.py

import time
from sklearn.ensemble import HistGradientBoostingClassifier


def fit_boosting(params, seed):
    """Gradient boosting on the 20 columns as they are: no one-hot, no log, no scaling."""
    model = HistGradientBoostingClassifier(random_state=seed, **params)
    start = time.perf_counter()
    model.fit(train[FEATURES], y_train)
    seconds = time.perf_counter() - start
    return model, model.predict_proba(valid[FEATURES])[:, 1], seconds


booster, booster_valid, seconds = fit_boosting({}, seed=0)
print("boosting, defaults", metrics(y_valid, booster_valid), f"| {booster.n_iter_} trees in {seconds:.2f} s")

The output:

boosting, defaults {'ap': 0.5589, 'auc': 0.7827} | 45 trees in 0.73 s

What this tells us. With its defaults and no preparation, boosting reaches AP 0.559, a little above logistic regression, as in the machine learning course. fit() held out 10% of train and stopped after 45 trees. The time depends on the computer and its load: in another run on the same Mac it was 0.19 s.

A network for a table

The inputs: why a network needs preparation

Think of a recipe that lists “2 eggs” and “300 grams of flour”. If you add the numbers as they are, the flour wins every comparison, only because grams are small units. A network has the same problem. Each neuron computes a weighted sum, as in Neurons and layers. A credit limit of 800,000 next to a status code of 2 swamps the sum, and the gradients of the large columns swamp the gradients of the small ones.

So the network needs the preparation that logistic regression needed. The codes become one-hot columns, and the amounts go on a log scale. Then every column is scaled to a mean of 0 and a standard deviation of 1. The preparation learns its scaling from the rows that the network trains on. Arjun also keeps 10% of train apart as the stopping set. It is the network’s own small validation set for early stopping:

compare.py

fit_part, stop_part = train_test_split(train, test_size=0.1, stratify=train[TARGET], random_state=0)


def tensors(prepare, frames):
    """Prepared columns as float32 tensors, one per frame."""
    return [torch.tensor(prepare.transform(f[FEATURES]), dtype=torch.float32) for f in frames]


prepare = preprocessor().fit(fit_part[FEATURES])   # learn the scaling from the fitting rows only
x_fit, x_stop, x_valid, x_test = tensors(prepare, (fit_part, stop_part, valid, test))
y_fit = torch.tensor(fit_part[TARGET].to_numpy())
y_stop = torch.tensor(stop_part[TARGET].to_numpy())
for name, x in [("x_fit", x_fit), ("x_stop", x_stop), ("x_valid", x_valid), ("x_test", x_test)]:
    show_shape(name, x)
print("first client, first 8 prepared numbers:", [round(v, 2) for v in x_fit[0, :8].tolist()])

The output:

x_fit        shape (16200, 55)
x_stop       shape (1800, 55)
x_valid      shape (6000, 55)
x_test       shape (6000, 55)
first client, first 8 prepared numbers: [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0]

What this tells us. The 20 columns became 55: the 7 code columns became 42 one-hot columns, and the 13 amounts stay 13. The first numbers of a client are the 0s and 1s of the one-hot codes. Why not stop on the 6,000 validation clients? Validation chooses the settings of both models. If the network also stopped on it, the network would use it twice, and boosting would not.

The network

TableNet is the network of this course with a table in place of a bag of words. It has linear layers, ReLU and dropout, and two scores at the end: “pays” and “defaults”. Softmax turns the two scores into probabilities, as in Loss, and the probability of “defaults” is the client’s score.

compare.py

class TableNet(nn.Module):
    """A network for a table: (Linear -> ReLU -> Dropout) x depth, then 2 scores."""

    def __init__(self, n_inputs, hidden=64, depth=2, dropout=0.2):
        super().__init__()
        layers, width = [], n_inputs
        for _ in range(depth):
            layers += [nn.Linear(width, hidden), nn.ReLU(), nn.Dropout(dropout)]
            width = hidden
        layers.append(nn.Linear(width, 2))           # 2 scores: pays, defaults
        self.layers = nn.Sequential(*layers)

    def forward(self, x):                            # x: (batch, n_inputs) -> (batch, 2)
        return self.layers(x)


torch.manual_seed(0)
model = TableNet(x_fit.shape[1])
print(model)
print("parameters:", sum(p.numel() for p in model.parameters()))
show_shape("scores", model(x_fit[:8]))

The output:

TableNet(
  (layers): Sequential(
    (0): Linear(in_features=55, out_features=64, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.2, inplace=False)
    (3): Linear(in_features=64, out_features=64, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.2, inplace=False)
    (6): Linear(in_features=64, out_features=2, bias=True)
  )
)
parameters: 7874
scores       shape (8, 2)

What this tells us. Count the parameters by hand, as in Module 1: 55 × 64 + 64 = 3,584, then 64 × 64 + 64 = 4,160, then 64 × 2 + 2 = 130. The total is 7,874. A batch of 8 clients gives 8 rows of 2 scores. This is the starting recipe; tuning will try others.

Train with early stopping

train_network() is the course’s training loop. The inner loop is the five steps of The training loop, on shuffled mini-batches of 256 clients. After each epoch, the function measures the loss on the stopping set in evaluation mode. A new best keeps a copy of the weights. Ten epochs without a new best stop the training, and the best weights come back. This is early stopping from Overfitting controls, with a patience of 10. AdamW adds a small weight decay of 0.0001.

compare.py

import copy


def predict(model, x):
    """Probability of default, with dropout off and no gradients."""
    model.eval()
    with torch.no_grad():
        return torch.softmax(model(x), dim=1)[:, 1].numpy()


def train_network(model, x_fit, y_fit, x_stop, y_stop, lr=1e-3, weight_decay=1e-4, batch_size=256,
                  max_epochs=200, patience=10, seed=0):
    """The course's five steps per mini-batch; early stopping on the stopping set."""
    torch.manual_seed(seed)
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=weight_decay)
    loss_fn = nn.CrossEntropyLoss()
    best_loss, best_state, best_epoch, waited, history = float("inf"), None, 0, 0, []
    for epoch in range(1, max_epochs + 1):
        model.train()
        order = torch.randperm(len(x_fit))
        total = 0.0
        for i in range(0, len(order), batch_size):
            idx = order[i:i + batch_size]
            optimizer.zero_grad()                          # 1. forget the last batch's gradients
            loss = loss_fn(model(x_fit[idx]), y_fit[idx])  # 2-3. forward pass, how wrong?
            loss.backward()                                # 4. backward pass: gradients
            optimizer.step()                               # 5. update the weights
            total += loss.item() * len(idx)
        model.eval()
        with torch.no_grad():
            stop_loss = loss_fn(model(x_stop), y_stop).item()
        history.append({"epoch": epoch, "train_loss": round(total / len(x_fit), 4), "stop_loss": round(stop_loss, 4)})
        if stop_loss < best_loss - 1e-4:                   # a new best: keep a copy of the weights
            best_loss, best_state, best_epoch, waited = stop_loss, copy.deepcopy(model.state_dict()), epoch, 0
        else:
            waited += 1
            if waited >= patience:
                break
    model.load_state_dict(best_state)                      # early stopping: back to the best epoch
    return history, best_epoch


history, best_epoch = train_network(model, x_fit, y_fit, x_stop, y_stop, seed=0)
print(f"best epoch {best_epoch}, stopped after {len(history)} | valid", metrics(y_valid, predict(model, x_valid)))

The output:

best epoch 7, stopped after 17 | valid {'ap': 0.5503, 'auc': 0.78}

One call is new. model.state_dict() returns the model’s weights by name, and copy.deepcopy() keeps a copy that later updates cannot change. Save and inspect results uses the same state_dict() to save weights to a file.

• Training loss
• Stopping-set loss

The starting network (seed 0). The training loss keeps falling slowly; the stopping-set loss is lowest at epoch 7 and then stays flat, so training stops after epoch 17 and the weights of epoch 7 come back.

Text description of the chart

A line chart from epoch 1 to 17, loss from 0.40 to 0.54. The dashed training loss starts at 0.529 and falls to 0.423 at epoch 17. The solid stopping-set loss starts at 0.462 and falls quickly to 0.434 at epoch 5. It reaches its lowest point, 0.433, at epoch 7. Then it stays between 0.433 and 0.436 without a new best. Training stops after epoch 17.

What this tells us. Most of the drop happens in the first three epochs. Then the stopping-set loss stays flat while the training loss keeps falling: the start of the overfitting picture of Module 3. The starting network has an AP of 0.550, a little below logistic regression. It needs tuning before the contest is fair.

The failure: raw columns into the network

Boosting needed no preparation. What if Daniel gives the network the raw columns too? run_network() puts the steps above in one function, with the preparation switches as an argument; the notebook shows the whole function. Here it trains the starting network three times on the raw columns:

compare.py

START = {"hidden": 64, "depth": 2, "dropout": 0.2}
for seed in range(3):
    result, _ = run_network(START, seed, {"log": False, "scale": False, "codes_as_categories": False})
    print("raw columns:", result)

The output:

raw columns: {'ap': 0.2214, 'auc': 0.5007, 'seed': 0, 'seconds': 1.19, 'best_epoch': 44}
raw columns: {'ap': 0.2219, 'auc': 0.5007, 'seed': 1, 'seconds': 0.79, 'best_epoch': 27}
raw columns: {'ap': 0.2213, 'auc': 0.5004, 'seed': 2, 'seconds': 0.89, 'best_epoch': 31}

Common mistake

Raw amounts into a network

Symptom: the network trains without an error, and the loss falls a little. But its AP is 0.221, the share of defaults, and its ROC AUC is 0.50. It ranks clients no better than chance. Boosting on the same raw columns reached 0.559.

Cause: the inputs are on very different scales. Credit limits in train go from 10,000 to 800,000, and status codes from −2 to 8. The first weighted sums are huge, ReLU passes them on, and the optimizer, with one learning rate for all inputs, cannot find a useful direction. The network settles on the safe answer: almost the same score, about 22%, for every client.

Fix: scale the inputs. With scaling alone (codes still as numbers, no log), the same network reaches an AP of 0.547 to 0.556 over three seeds. One-hot codes and the log add a little more. Learn the scaling on the training rows only, as in Debug the experiment.

Consequence in AI work: a network does not warn you about unscaled inputs. A model at chance level still gives scores, and its accuracy of 77.9% (it always says “pays”) can look fine. Always compare with a trivial baseline, here the share of defaults.

So the first cost of the network is work before training, and without it the network fails silently. The other side: boosting on the network’s 55 prepared columns gets 0.564 (seed 0), against 0.559 on the raw columns. The trees do not need the preparation, and it does not hurt them.

The same tuning budget for both

A network has many settings (width, depth, dropout, learning rate, batch size, patience); boosting has a few that matter (learning rate, tree size, smallest leaf). To be fair, each model gets 8 settings, and each setting runs with 3 seeds. With another seed, a network starts from other random weights and sees the batches in another order. One run can be lucky, so the median of three is a steadier judge.

compare.py

NET_GRID = [{"hidden": h, "depth": d, "dropout": p} for h in (64, 256) for d in (1, 2) for p in (0.0, 0.3)]
BOOST_GRID = [{"learning_rate": r, "max_leaf_nodes": n, "min_samples_leaf": m}
              for r in (0.05, 0.1) for n in (15, 31) for m in (20, 100)]
rows = []
for cfg in NET_GRID:                     # 8 settings x 3 seeds = 24 networks
    aps = [run_network(cfg, seed)[0]["ap"] for seed in (0, 1, 2)]
    rows.append({"model": "network", "setting": str(cfg), "median AP": np.median(aps), "lowest": min(aps)})
for cfg in BOOST_GRID:                   # 8 settings x 3 seeds = 24 boosted models
    aps = [metrics(y_valid, fit_boosting(cfg, seed)[1])["ap"] for seed in (0, 1, 2)]
    rows.append({"model": "boosting", "setting": str(cfg), "median AP": np.median(aps), "lowest": min(aps)})
tuning = pd.DataFrame(rows)
print(tuning.sort_values(["model", "median AP"], ascending=False).groupby("model").head(3).to_string(index=False))
print(tuning.groupby("model")["median AP"].agg(["min", "max"]))

The output:

   model                                                                setting  median AP  lowest
 network                            {'hidden': 256, 'depth': 1, 'dropout': 0.0}     0.5602  0.5542
 network                            {'hidden': 256, 'depth': 1, 'dropout': 0.3}     0.5593  0.5592
 network                             {'hidden': 64, 'depth': 1, 'dropout': 0.0}     0.5584  0.5560
boosting {'learning_rate': 0.05, 'max_leaf_nodes': 15, 'min_samples_leaf': 100}     0.5664  0.5580
boosting  {'learning_rate': 0.1, 'max_leaf_nodes': 31, 'min_samples_leaf': 100}     0.5656  0.5591
boosting  {'learning_rate': 0.05, 'max_leaf_nodes': 15, 'min_samples_leaf': 20}     0.5634  0.5603
             min     max
model
boosting  0.5589  0.5664
network   0.5491  0.5602

What this tells us. Three things.

• The best network is shallow. It has one hidden layer of 256 neurons and no dropout (median 0.560). Depth did not help on this table: each two-layer setting is below its one-layer partner.
• The settings matter more for the network. Its eight medians spread from 0.549 to 0.560, against 0.559 to 0.566 for boosting. The worst boosting setting is about as good as the best network setting.
• Boosting’s defaults were already good. They gave 0.559, close to its best. The network’s starting recipe gave 0.550, below logistic regression.

The tuning took about 12 seconds for the 24 networks and 5 seconds for the 24 boosted models. Both are cheap at this size, but the network needed the tuning more.

Ten seeds each

The chosen recipes now run with 10 seeds each. For the network, the seed sets the starting weights and the batch order. For boosting, it sets which 10% of train fit() holds out. So both models vary from run to run:

compare.py

best_net = {"hidden": 256, "depth": 1, "dropout": 0.0}
best_boost = {"learning_rate": 0.05, "max_leaf_nodes": 15, "min_samples_leaf": 100}
final, parts = {"network": [], "boosting": []}, {"network": {}, "boosting": {}}
for seed in range(10):
    result, (model, prepare, x_test) = run_network(best_net, seed)
    final["network"].append(result)
    parts["network"][seed] = (model, prepare, predict(model, x_test))
    booster, scores, seconds = fit_boosting(best_boost, seed)
    final["boosting"].append(metrics(y_valid, scores) | {"seed": seed, "seconds": round(seconds, 2)})
    parts["boosting"][seed] = (booster, None, booster.predict_proba(test[FEATURES])[:, 1])
for name, results in final.items():
    table = pd.DataFrame(results)
    print(f"{name:9s} valid AP median {table['ap'].median():.3f} (lowest {table['ap'].min():.3f}, "
          f"highest {table['ap'].max():.3f}) | fit time median {table['seconds'].median():.2f} s")

The output:

network   valid AP median 0.561 (lowest 0.551, highest 0.564) | fit time median 0.29 s
boosting  valid AP median 0.567 (lowest 0.558, highest 0.571) | fit time median 0.51 s

• Network
• Boosting

Ten seeds of each tuned recipe on the 6,000 validation clients. The two clouds overlap: 7 of the 10 network runs beat the weakest boosting run, but the boosting median is higher.

Text description of the chart

A dot plot with two rows, average precision from 0.535 to 0.58; each dot is one seed. Network row, ten dots: 0.551, 0.554, 0.557, 0.560, 0.561, 0.562, 0.563, 0.563, 0.563 and 0.564; median 0.561. Boosting row, ten dots: 0.558, 0.562, 0.564, 0.565, 0.566, 0.567, 0.567, 0.569, 0.570 and 0.571; median 0.567. The rows overlap between 0.558 and 0.564. Logistic regression, for reference, has 0.555.

What this tells us. Boosting’s median is higher by about 0.006, and both models beat logistic regression’s 0.555 in most runs. But the clouds overlap. If Daniel reported only his best seed (0.564) and Arjun only his worst (0.558), the network would “win”. That is why the page reports all seeds.

Many people expect a network to vary much more than boosting across seeds. Here it did not: the range was 0.013 for both. The tuned network is small and stops after about 5 epochs, so chance has little room to grow. Report what you measure, not what you expect.

The test set, once

Arjun has chosen everything. Each seed of each model now scores the 6,000 test clients once, with the threshold that the same seed chose on validation:

compare.py

logistic_test = logistic.predict_proba(test[FEATURES])[:, 1]
print(f"{'rule: late in September':28s} AP   -    cost {cost(y_test, test['PAY_1'].to_numpy() >= 1)}")
print(f"{'logistic regression':28s} AP {metrics(y_test, logistic_test)['ap']:.3f} "
      f"cost {cost(y_test, logistic_test >= cheapest(y_valid, logistic_valid)[0])}")
for name in ("boosting", "network"):
    costs, aps = [], []
    for seed, (model, prepare, test_scores) in parts[name].items():
        valid_scores = (model.predict_proba(valid[FEATURES])[:, 1] if prepare is None
                        else predict(model, tensors(prepare, (valid,))[0]))
        threshold = cheapest(y_valid, valid_scores)[0]          # chosen on valid, as before
        costs.append(cost(y_test, test_scores >= threshold))
        aps.append(metrics(y_test, test_scores)["ap"])
    print(f"{name + ', median of 10 seeds':28s} AP {np.median(aps):.3f} cost {np.median(costs):.0f} "
          f"(seeds: {min(costs)} to {max(costs)})")

The output:

rule: late in September      AP   -    cost 3852
logistic regression          AP 0.549 cost 3346
boosting, median of 10 seeds AP 0.562 cost 3362 (seeds: 3323 to 3381)
network, median of 10 seeds  AP 0.556 cost 3342 (seeds: 3319 to 3358)

What this tells us. On the ranking, boosting is ahead again: a test AP of 0.562, against 0.556 for the network and 0.549 for logistic regression. On the cost, which is what the bank pays, the order changes and the gaps are tiny. The network costs 3,342 (median), logistic regression 3,346, and boosting 3,362. The seeds of one model spread over 40 to 60 units, more than the gaps between the models. All three cost about 13% less than the rule’s 3,852. For Mei-Ling, the three models cost the same.

Is the AP difference real?

Arjun asks one more question: is boosting’s lead in AP real, or luck in which clients landed in the test set? The machine learning course answered the same question with a bootstrap, in Choose a model for readmissions. Draw 6,000 test clients with replacement, measure the difference, and repeat this 1,000 times. The middle 95% of the differences is the interval.

compare.py

rng = np.random.default_rng(0)
boost_scores, net_scores = parts["boosting"][0][2], parts["network"][0][2]   # seed 0 of each
differences = []
for _ in range(1000):
    idx = rng.integers(0, len(y_test), len(y_test))          # 6,000 test clients, drawn with replacement
    differences.append(average_precision_score(y_test[idx], boost_scores[idx])
                       - average_precision_score(y_test[idx], net_scores[idx]))
low, high = np.percentile(differences, [2.5, 97.5])
point = average_precision_score(y_test, boost_scores) - average_precision_score(y_test, net_scores)
print(f"boosting minus network, test AP: {point:+.4f}, 95% interval {low:+.4f} to {high:+.4f}")
print(f"share of resamples where boosting is ahead: {np.mean(np.array(differences) > 0):.0%}")

The output:

boosting minus network, test AP: +0.0083, 95% interval -0.0038 to +0.0200
share of resamples where boosting is ahead: 92%

Top: the test AP of each model; the bars show the lowest and highest of 10 seeds. Bottom: the 95% bootstrap interval of the AP difference between boosting and the network includes zero.

Text description of the chart

Two panels. First panel, test average precision from 0.53 to 0.58: logistic regression 0.549 (one model, no range); network median 0.556, seeds from 0.554 to 0.559; boosting median 0.562, seeds from 0.557 to 0.566. Second panel, the difference in test AP between boosting and the network (seed 0 of each): +0.008, with a 95% interval from −0.004 to +0.020. The interval crosses the zero line.

What this tells us. Boosting is ahead in 92% of the resamples, and the point difference is +0.008. But the 95% interval runs from −0.004 to +0.020, so it includes zero. With the average scores of all 10 seeds of each model, the difference is smaller: +0.005 (interval −0.006 to +0.016). So the honest summary is: boosting is probably a little better at ranking, and the network is not better. The test set is too small to say more.

Explanations

The team must tell each called client why. In Explain predictions, the machine learning course used permutation importance. Shuffle one column, so that its link to the target breaks, and measure how much the score drops. It needs only the model’s scores, so it is a fair tool for both models:

compare.py

def permutation_importance(score, frame, y, repeats=5, seed=0):
    """How much AP drops when one column is shuffled (its link to the target is broken)."""
    rng = np.random.default_rng(seed)
    base = average_precision_score(y, score(frame))
    drops = {}
    for column in FEATURES:
        losses = []
        for _ in range(repeats):
            shuffled = frame.copy()
            shuffled[column] = rng.permutation(shuffled[column].to_numpy())
            losses.append(base - average_precision_score(y, score(shuffled)))
        drops[column] = round(float(np.mean(losses)), 4)
    return pd.Series(drops).sort_values(ascending=False)


boost0 = parts["boosting"][0][0]
net0, prepare0 = parts["network"][0][0], parts["network"][0][1]
importance = pd.DataFrame({
    "boosting": permutation_importance(lambda f: boost0.predict_proba(f[FEATURES])[:, 1], valid, y_valid),
    "network": permutation_importance(lambda f: predict(net0, tensors(prepare0, (f,))[0]), valid, y_valid)})
print(importance.sort_values("boosting", ascending=False).head(6))

The output:

           boosting  network
PAY_1        0.2071   0.1549
PAY_2        0.0137   0.0027
PAY_AMT3     0.0105   0.0102
BILL_AMT1    0.0098   0.0034
PAY_AMT6     0.0093   0.0008
LIMIT_BAL    0.0089   0.0130

Permutation importance of the six most important columns of each model. Both depend mostly on the September status. After that, the order differs, and every other column matters little.

Text description of the chart

Two panels, the drop in validation AP from 0 to 0.22. Boosting: status in September 0.207, status in August 0.014, paid in July 0.011, bill in September 0.010, paid in April 0.009, credit limit 0.009. Network: status in September 0.155, credit limit 0.013, paid in July 0.010, paid in September 0.009, status in April 0.007, education 0.006.

What this tells us. Both models lean on the status in September. Shuffling it costs boosting 0.21 AP and the network 0.15. Every other column adds about 0.01, in a different order per model. That is normal when columns carry the same information, as the six months of bills do.

Now the limit of this tool. Permutation importance answers “which columns matter overall?”, not why one client got a high score. Logistic regression splits each client’s score into exact contributions; neither boosting nor the network can. Tools that estimate per-client reasons exist for both, but they are estimates and add work. On explanations, the network is no worse than boosting, and both are worse than logistic regression.

One new client, end to end

A made-up client arrives. The client is two months late in September and August. The bill is NT$48,200 on a limit of NT$50,000, and the client paid nothing in September. The notebook scores this client with every seed of each model:

compare.py

new_client = pd.DataFrame([{
    "LIMIT_BAL": 50_000, "EDUCATION": 2,
    "PAY_1": 2, "PAY_2": 2, "PAY_3": 0, "PAY_4": 0, "PAY_5": 0, "PAY_6": 0,
    "BILL_AMT1": 48_200, "BILL_AMT2": 47_100, "BILL_AMT3": 46_000,
    "BILL_AMT4": 30_500, "BILL_AMT5": 29_800, "BILL_AMT6": 29_000,
    "PAY_AMT1": 0, "PAY_AMT2": 1_500, "PAY_AMT3": 1_600, "PAY_AMT4": 1_200, "PAY_AMT5": 1_100, "PAY_AMT6": 1_000}])
for name in ("boosting", "network"):
    scores = [float(model.predict_proba(new_client[FEATURES])[0, 1]) if prepare is None
              else float(predict(model, tensors(prepare, (new_client,))[0])[0])
              for model, prepare, _ in parts[name].values()]
    print(f"{name:9s} seed 0: {scores[0]:.3f} | 10 seeds: {min(scores):.3f} to {max(scores):.3f}")
print(f"logistic           {logistic.predict_proba(new_client[FEATURES])[0, 1]:.3f}")

The output:

boosting  seed 0: 0.596 | 10 seeds: 0.533 to 0.611
network   seed 0: 0.742 | 10 seeds: 0.658 to 0.742
logistic           0.682

What this tells us. Every model and every seed puts this client far above the cheapest thresholds (0.13 to 0.20), so every model calls the client. The decision is the same; the scores are not: 0.596, 0.742 and 0.682. Even one recipe moves by about 0.08 between seeds. So use these scores to rank and to decide, not as exact chances, unless you check the calibration first, as the machine learning course did.

The work each model needs

Training time first. Both models train in under a second with two CPU threads. In three timing rounds of 10 seeds each, the median was 0.29 s for the network and 0.45 s for boosting. In an earlier run, it was 0.42 s and 0.22 s. Other work shared the Mac, so the times change by a factor of two between runs. At this size, training time decides nothing. Scoring 6,000 clients took about 6 ms (network) and 12 ms (boosting). The table puts all the measurements side by side:

	Gradient boosting	Network
Preparation	None: the 20 columns as they are	One-hot codes, signed log, scaling (55 columns); without them, chance level
Held-out data for stopping	Built into fit() (10%)	A stopping set (10%) that you split and pass yourself
Code you write	One call to fit()	A model class, a training loop, early stopping, predict()
Defaults (valid AP)	0.559, close to the tuned 0.567	0.550, below logistic regression until tuned
8 settings (median valid AP)	0.559 to 0.566	0.549 to 0.560
10 seeds (valid AP)	0.558 to 0.571	0.551 to 0.564
Test AP, median	0.562	0.556
Test cost, median	3,362	3,342
Explanations	Permutation importance; no exact per-client reasons	The same

What this tells us. On the score, it is a tie, with a small edge for boosting in ranking. On the work, boosting wins clearly: no preparation, no loop to write, good defaults, less sensitivity to its settings.

Does it hold on another table?

One dataset can mislead. So the numbers script ran the same protocol on a second open table, UCI Adult (census income, CC BY 4.0). It has 48,842 people, and the target is an income above $50,000. Sex and race were not inputs. The network used its credit recipe without new tuning, so this check is rough. Over 5 seeds, boosting reached a test AP of 0.833 and the network 0.774. The test ROC AUC was 0.929 against 0.910. Here the gap is large, and every seed agrees. It fits the published pattern: on tables, boosted trees win or tie.

The verdict

Arjun writes the result for Mei-Ling in five points:

1. Score: the network does not beat boosting. Boosting ranks a little better (test AP 0.562 against 0.556); the cost is equal.
2. Uncertainty: the 95% interval of the AP difference includes zero. The median costs of all three models are within 20 units. The seed alone moves a model’s cost by 40 to 60 units.
3. Work: the network needs preparation, fails silently without it, and needs tuning to reach even logistic regression.
4. Time: both train in under a second, so time does not decide.
5. Explanations: neither gives exact per-client reasons; logistic regression does.

Recommendation: keep logistic regression for the calls. It costs the same as both challengers (3,346), and it gives each client exact reasons. Keep boosting as the challenger, as the machine learning course decided. Drop the network for this decision.

Daniel reads it and agrees. “So the network isn’t wrong. It just doesn’t earn its cost here.” That is the right reading.

When would a network be the better choice here?

The verdict is about this table, this size and this decision. A network would earn a new contest if one of these changed:

• Much more data. With millions of clients and many more columns, a network has room to find patterns that trees miss. Even then, measure: on many large tables, boosting still wins or ties.
• Raw data next to the table. Suppose the bank added the text of call notes, scanned documents, or the sequence of every card payment. A network can read those directly, as this course’s ticket router reads text, and one network can combine them with the table. Boosting needs a person to turn the raw data into columns first.
• One model for several outputs. Suppose the team wanted one model to predict default, the best payment plan and the chance of a complaint. A network can share its hidden layers across the three outputs. With boosting, that means three separate models.
• Learned features to reuse. A network’s hidden layer is a learned summary of a client, and other models can reuse it. Pretrained models are reused in the same way, as Training versus inference shows.

None of these is true for Mei-Ling’s team today.

Responsible use

• The decision is help, not refusal. The models choose who gets an early-help call. Do not reuse them for credit limits, prices or refusals without a new evaluation, new costs and a legal review.
• Sensitive columns stay out, but proxies stay in. The machine learning course showed that the credit limit acts as a proxy for age. The credit limit is the network’s second most important column, so a network does not remove the proxy. Run the same slice checks on any model that you use.
• Report every seed. A model reported only by its best seed looks better than it is. Report the median and the range, and keep the test set for the end.
• Scores are not chances. The same client got 0.60, 0.68 or 0.74 from three models. Do not show a score to a client as “your risk” without a calibration check.
• One bank, one period. All rows come from one bank in Taiwan in 2005. Nothing here shows how the models behave in another economy or another year.

Insights and limits

• The data decided more than the model. All three models land between a test AP of 0.549 and 0.562. Six months of payment history carry most of the signal, and every model finds it.
• A tie is a result. The network did not lose badly; it gave nothing extra for more work. A fair contest exists to find exactly that.
• Limits of this study. The tuning grids were small (8 settings each); a much larger search might move either model a little. The network used plain inputs; methods made for tables, such as embeddings for the codes, were not tried. The timings come from one shared computer. The check on the Adult table used an untuned network.

Make it a portfolio piece

A fair comparison shows an employer something rare: you can test an exciting idea and report an honest “no”. Make it yours with one change: a third model family, other costs, or another open table.

What to put in the repository:

• README.md, from the template below.
• The notebook, run from top to bottom, with its outputs.
• results.csv, with every run: model, setting, seed, validation AP, test AP, cost and fit time.
• The rules of the contest, written before the results, so that a reader sees that you did not change them afterwards.
• The data source and licence, with the attribution text from the box above.

A README template:

# Deep network or gradient boosting? A fair test on credit-card default

## Question
On a table of 20 columns about 30,000 credit-card clients, is a neural network worth
it compared with gradient boosting and logistic regression?

## Data and licence
UCI "Default of Credit Card Clients" (Yeh, 2009), Taiwan 2005, CC BY 4.0,
https://doi.org/10.24432/C55S3H. Sex, marital status and age are not inputs.

## Rules of the contest
Same split (60/20/20, stratified), same inputs, same metric (average precision; cost
with miss 5, false alarm 1), 10% held out for stopping in both, 8 settings x 3 seeds
each, 10 final seeds each, test once.

## Results (test, median of 10 seeds)
Boosting AP 0.562, cost 3,362. Network AP 0.556, cost 3,342. Logistic AP 0.549,
cost 3,346. Rule cost 3,852. Bootstrap 95% interval of the AP difference
(boosting - network): -0.004 to +0.020.

## Verdict
A tie in cost; boosting a little ahead in ranking. The network needs preparation and
tuning and gives no explanation advantage. Keep logistic regression; boosting stays
the challenger.

## Limits
One bank, one period; small tuning grids; timings from one shared CPU.

How to describe it in an interview, in two sentences: “I set up a fair contest between a PyTorch network and gradient boosting on a real credit table, with one split, one metric, one tuning budget and ten seeds each. The network tied on cost but needed more work, so I recommended the simpler model and wrote down when to test a network again.”

Go further:

• Give the network an embedding for each status code instead of one-hot columns, as Module 5 did for words, and measure whether it helps.
• Add one more setting to each grid (the network’s learning rate, boosting’s number of trees) and check whether the verdict changes.
• Repeat the whole protocol with five different splits (random_state 0 to 4) and report how often each model wins.

Knowledge check

Check your understanding

Daniel reports his best network seed (validation AP 0.564) and Arjun's worst boosting seed (0.558), and concludes that the network wins. What is wrong with this, and what should they report instead?

Show the answer

He compares the luckiest run of one model with the unluckiest run of the other. Both models vary with the seed: the network from 0.551 to 0.564, boosting from 0.558 to 0.571. A choice of seeds can turn a small loss into a win. They should run the same number of seeds for both and report the median and the range (network 0.561, boosting 0.567). Then they check the difference on the test set once, with an interval such as the bootstrap.

Check your understanding

The network trained on the raw columns reaches an AP of 0.221 and a ROC AUC of 0.50, while boosting on the same raw columns reaches 0.559. Explain both results.

Show the answer

A tree asks yes-or-no questions such as “is the limit above 140,000?”, so the scale of a column does not matter. A network computes weighted sums. A limit of 800,000 next to a status code of 2 swamps the sums and the gradients, so the optimizer cannot learn. The network gives almost the same score to everyone. So its AP equals the share of defaults (0.221), and its ROC AUC is 0.50, which is chance. Scaling the inputs fixes it.

Check your understanding

Mei-Ling's bank plans to add the free-text notes of every call agent and the full list of card payments to the data. Should the team run this contest again? What would make a network more promising?

Show the answer

Yes. The verdict was for a table of 20 chosen columns. Free text and payment sequences are raw data. A network can read them directly and learn its own features, as this course’s ticket router reads text. One network can also combine them with the table. Boosting would need a person to turn the notes and payments into columns first. More data also helps a network. The team should repeat the contest under the same rules. It should also check the explanations and the slices again, because the reasons for a call matter to clients.

Recap

• A fair contest changes only the model. The split, the inputs, the metric, the held-out share, the tuning budget and the seeds stay the same. The test set comes once, at the end.
• On this credit table, boosting ranked a little better (test AP 0.562 against 0.556), but the bootstrap interval of the difference included zero. In cost, the network, boosting and logistic regression were the same.
• A network for a table needs preparation. Without scaling, it fell to chance, without any warning.
• Seeds matter for both models. Report the median and the range, never only the best run.
• Permutation importance works for any model, but it gives no exact per-client reasons. The machine learning course’s logistic regression stays the right choice for this bank.
• A network earns a new contest with much more data, raw inputs next to the table, or several outputs from one model.

Next, a case study where deep learning is the right tool: text, and a model that has already read a lot of it.

Next: Fine-tune a small transformer