Ensembles

What you will be able to do

A single tree is unstable, and the next step is to combine many models into one. Learn why a group of models can beat one model, and how the two main kinds of ensemble work. Then measure what the extra complexity costs, so that you can decide when it is worth it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson and of the module practice, with the instructions. Its setup cells install the packages, download the practice data, write ticket_model.py and write your experiment log so far, so it works without the local project. It runs in about two minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Decision trees, tree_pipeline() and evaluate(), from Decision trees.
• Bias, variance and the five random halves of train, from Control complexity: Bias and variance. Hyperparameters, from Control complexity: Hyperparameters.
• The rules of the loop and the experiment log, from Machine learning is iterative: The experiment log.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, active in a terminal, and the lesson notebooks open in VS Code or JupyterLab. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv and data/valid.csv, from Define a baseline: Download the data. If a file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 6 numbers in NUMBERS, from Preprocess with a pipeline: Write ticket_model.py. If it is missing or broken, write it again from that section. Do not copy the finished file from C05/project yet: it has an extra feature that Module 6 adds.
• Your experiment log with the rows E01 to E10, from Control complexity: Update the experiment log. If it is missing, the notebook’s setup cell writes the ten rows again.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

The problem: one tree is unstable

In Control complexity, Priya fitted a tree with no limit on five random halves of train. For the same May ticket, T-117529, the five trees gave the scores 0, 1, 0, 1 and 0. Over all May tickets, the five scores of a ticket differed by 0.194 on average. A deep tree has high variance: its answer depends a lot on which train tickets it happened to see.

Limiting the tree lowers the variance, but it raises the bias: the depth-3 tree was stable and too simple. So a single tree forces a choice between two problems. Is there a way to keep a flexible model and still remove most of its randomness? There is, and the idea is older than computers.

The wisdom of the crowd

Put a big jar of sweets on a table, and ask 100 people to guess how many sweets it holds. Most single guesses are far off: some people guess too many, some too few. But if you take the average of all the guesses, it is often very close to the true number. In 1906, the scientist Francis Galton saw this at a country fair in England. About 800 visitors guessed the weight of an ox, and the middle guess was within 1% of the true weight.

Why does it work? Each person makes their own error, for their own reasons. Some errors are too high and some too low, so in the average they cancel. Nobody in the crowd is an expert, but the average of the crowd is better than almost any one person.

The same idea is behind a second medical opinion. If one doctor is unsure, a second doctor can catch a mistake. But only if the second doctor examines the patient herself. If she just reads the first doctor’s notes and agrees, she adds nothing: she repeats the same mistake.

This simulation shows both cases with 100 guesses of a jar that holds 437 sweets:

rng = np.random.default_rng(0)
true_count = 437
own_errors = np.round(true_count + rng.normal(0, 120, 100))      # each person makes their own error
same_hint = np.round(true_count + 150 + rng.normal(0, 40, 100))  # everyone read the same wrong hint
for name, guesses in [("own errors", own_errors), ("same hint", same_hint)]:
    print(f"{name:10s} one guess misses by {np.abs(guesses - true_count).mean():.0f} on average; "
          f"the average guess is {guesses.mean():.0f}, a miss of {abs(guesses.mean() - true_count):.0f}")

The output:

Output

own errors one guess misses by 95 on average; the average guess is 447, a miss of 10
same hint  one guess misses by 148 on average; the average guess is 585, a miss of 148

Crowd 1: 100 people who each make their own error. Single guesses spread widely, but their average is close to the true count.

Text description of the chart

The guesses spread from about 150 to about 750 sweets, with most between 350 and 550. A dashed line marks the true count, 437. Another dashed line marks the average guess, 447, only 10 away from the true count. One guess misses by 95 on average.

Crowd 2: 100 people who all read the same wrong hint. Their guesses agree with each other, and their average is as wrong as one guess.

Text description of the chart

The guesses are close together, from about 450 to about 700 sweets, almost all to the right of the true count, 437. The average guess is 585, a miss of 148: the same as the average miss of one guess. Averaging did not help, because everyone made the same error.

In the first crowd, one guess misses by 95 on average, but the average guess misses by only 10. In the second crowd, the guesses agree more with each other, and that is the problem. They share one error, and an average cannot remove an error that everyone makes.

The idea: an ensemble

A model made of many models is an ensemble. (The word comes from French and means “together”, as in a group of musicians.) Its members are often trees. The ensemble combines their scores into one score.

The jar shows when this helps. Two conditions must both be true:

1. Each member must be useful on its own: better than guessing. An average of bad guesses is still bad.
2. The members must make different errors. If they all make the same mistake, the average makes it too.

The second condition is the hard one. If you train 200 trees on the same tickets, in the same way, you get 200 copies of the same tree, with the same errors. The two methods below create different members on purpose, each in its own way.

Bagging and the random forest

The first method trains many trees at the same time, independently, and makes them different with randomness.

• Each tree learns from its own bootstrap sample of the train set. A bootstrap sample has as many rows as train, drawn at random with replacement: after each draw, the row goes back, so it can be drawn again. Some tickets appear twice or more, and about a third do not appear at all. (You met the bootstrap in Estimate uncertainty.)
• The ensemble’s score is the average of the trees’ scores.

This is called bagging, short for bootstrap aggregating (aggregating means combining). A random forest adds a second source of randomness: at each split, a tree may choose only from a random part of the features. So even two trees with similar samples often ask different questions.

A random forest: 200 trees learn in parallel, each from its own random sample of the rows and with random features at each split. Their scores are averaged.

Text description of the diagram

The train set, 16,939 tickets, feeds 200 random samples in parallel. Each sample has random rows, some of them twice. Each sample trains its own tree, and each tree chooses from random features at each split. The 200 trees do not see each other. For a new ticket, each tree gives a score, and the forest’s score is the average of the 200 scores.

A forest on the ten tickets

Here is a forest of 5 small trees on the 10 tickets of Decision trees, with a new ticket of priority 1 and 100 words:

tiny_forest = RandomForestClassifier(n_estimators=5, max_depth=2, random_state=0)
tiny_forest.fit(tiny[["priority", "word_count"]], tiny["escalated"])
print("tree 1 learned from:", sorted(tiny["ticket"].iloc[tiny_forest.estimators_samples_[0]]))

new_ticket = pd.DataFrame({"priority": [1], "word_count": [100]})
each_tree = [round(float(tree.predict_proba(new_ticket.to_numpy())[0, 1]), 2) for tree in tiny_forest.estimators_]
print("score of each tree:", each_tree)
print("forest score (the average):", round(float(tiny_forest.predict_proba(new_ticket)[0, 1]), 2))

Output

tree 1 learned from: ['T-01', 'T-01', 'T-03', 'T-03', 'T-04', 'T-04', 'T-07', 'T-08', 'T-10', 'T-10']
score of each tree: [1.0, 0.4, 1.0, 1.0, 1.0]
forest score (the average): 0.88

Tree 1 saw T-01, T-03, T-04 and T-10 twice, and never saw T-02, T-05, T-06 or T-09. So each tree learns a slightly different version of the data. Four trees give the new ticket 1.0 and one gives 0.4. The forest’s score is the average: (1.0 + 0.4 + 1.0 + 1.0 + 1.0) ÷ 5 = 0.88. One tree’s mistake moves the average only a little.

Why it helps: averaging lowers the variance

Does the randomness matter? Priya trains 200 deep trees (no depth limit) in three ways, and scores the average of the first 1, 2, 5, … 200 trees with the course’s cross-validation:

• Similar trees: every tree sees all the rows and all the features. Only small random choices differ.
• Bagging: every tree sees its own bootstrap sample.
• Random forest: a bootstrap sample, and random features at each split.

Before you look at the chart, predict: which line rises most as trees are added?

• Random forest (random rows and features)
• Bagging (random rows)
• Similar trees (same rows, all features)
• Logistic regression (E03), CV AP 0.327

The CV AP of an average of deep trees, by the number of trees. Different trees improve a lot when averaged; similar trees hardly improve.

Text description of the chart

The horizontal axis is the number of deep trees averaged: 1, 2, 5, 10, 20, 50, 100 and 200. The vertical axis is the CV average precision, from 0.10 to 0.35. All three lines start near 0.14 with one tree. Similar trees: 0.139, 0.142, 0.146, 0.148, then 0.149 from 20 trees on. Bagging: 0.141, 0.157, 0.189, 0.217, 0.243, 0.263, 0.269 and 0.273. Random forest: 0.137, 0.153, 0.187, 0.220, 0.249, 0.272, 0.287 and 0.290. A dashed line marks logistic regression’s CV AP, 0.327, above all three lines.

One deep tree scores about 0.14 in each case. Averaging 200 similar trees brings it only to 0.149. Their scores have a correlation of 0.80: they share most of their errors, like the crowd with the same hint. Averaging 200 bagged trees brings it to 0.273, and the random forest to 0.290. Their trees are much less alike (correlations of 0.15 and 0.12), so their errors cancel. The extra randomness of the forest makes each tree a little worse and the average better.

This is the reason for a forest: averaging lowers the variance. Priya repeats the five-halves test of Control complexity with a forest of 200 deep trees:

ticket = valid[valid["ticket_id"] == "T-117529"]
for name, make in [("one tree, no limit", lambda: tree_pipeline(DecisionTreeClassifier(random_state=0))),
                   ("forest, 200 trees", lambda: tree_pipeline(
                       RandomForestClassifier(n_estimators=200, n_jobs=-1, random_state=0)))]:
    ticket_scores, all_scores = [], []
    for seed in range(5):
        half = train.sample(frac=0.5, random_state=seed)
        pipeline = make().fit(half[FEATURES], half[TARGET])
        ticket_scores.append(round(float(pipeline.predict_proba(ticket[FEATURES])[0, 1]), 2))
        all_scores.append(pipeline.predict_proba(valid[FEATURES])[:, 1])
    spread = pd.DataFrame(all_scores).std(ddof=0).mean()
    print(f"{name:20s} T-117529 gets {ticket_scores}; mean spread over May {spread:.3f}")

Output

one tree, no limit   T-117529 gets [0.0, 1.0, 0.0, 1.0, 0.0]; mean spread over May 0.194
forest, 200 trees    T-117529 gets [0.1, 0.04, 0.09, 0.05, 0.1]; mean spread over May 0.034

The same deep trees, averaged, give answers that move about six times less when the train tickets change: 0.034 instead of 0.194.

Boosting: learn from the mistakes

The second method builds the members one after another, and each new member works on the mistakes of the ones before it.

Think of a student who prepares for an exam with a set of practice questions. After the first round, she does not study everything again. She looks at the questions that she got wrong, and studies those. After the second round, she studies what is still wrong, and so on. Each round fixes a part of the remaining errors.

Gradient boosting does this with small trees:

1. Start with the same score for every ticket: the escalation rate.
2. Compute the error of each ticket: escalated (1 or 0) minus its score.
3. Fit a small tree that predicts these errors.
4. Add a part of this tree’s correction to the scores. The learning rate sets the part, for example 0.1: each tree corrects only a tenth, so no single tree can push the scores too far.
5. Repeat from step 2, with the new errors.

The final score is the start plus all the small corrections, added up. (“Gradient” refers to how each tree measures the errors, with the gradient of the loss from Loss and gradients; the idea above is enough here.)

Gradient boosting: small trees learn one after another, each one from the errors that the ones before it left. Their corrections are added up, each scaled by the learning rate.

Text description of the diagram

The train set gives a start: every ticket gets the escalation rate. Small tree 1 learns the errors of the start. Small tree 2 learns the errors that are left after tree 1. This continues, one tree at a time, to tree n. Each tree depends on the trees before it, so they cannot learn in parallel. The final score adds the start and 0.1 times each tree’s correction.

Boosting on the ten tickets

Here are two rounds by hand, on the 10 tickets, with a learning rate of 0.5 so that the steps are easy to see. Each small tree has one question. (Real boosting works on another scale, so that a score always stays between 0 and 1. The idea is the same.)

x_tiny = tiny[["priority", "word_count"]]
boost = tiny[["ticket", "escalated"]].copy()
boost["start"] = 0.4                                # the escalation rate, for every ticket
score = boost["start"]
for step in (1, 2):
    error = tiny["escalated"] - score               # what the scores still miss
    small_tree = DecisionTreeRegressor(max_depth=1).fit(x_tiny, error)
    print(f"tree {step} asks:", export_text(small_tree, feature_names=["priority", "word_count"]).split()[1:4])
    score = score + 0.5 * small_tree.predict(x_tiny)  # learning rate 0.5: half of the correction
    boost[f"after tree {step}"] = score.round(3)
print(boost.to_string(index=False))

Output

tree 1 asks: ['priority', '<=', '1.50']
tree 2 asks: ['word_count', '<=', '42.50']
ticket  escalated  start  after tree 1  after tree 2
  T-01          1    0.4         0.575         0.657
  T-02          0    0.4         0.575         0.385
  T-03          1    0.4         0.575         0.657
  T-04          1    0.4         0.575         0.657
  T-05          1    0.4         0.283         0.365
  T-06          0    0.4         0.283         0.365
  T-07          0    0.4         0.283         0.093
  T-08          0    0.4         0.283         0.093
  T-09          0    0.4         0.283         0.365
  T-10          0    0.4         0.283         0.365

Follow T-01. Its error at the start is 1 − 0.4 = 0.6. Tree 1 asks about priority. The mean error of the priority 1 group is +0.35, so T-01 goes up by half of it, 0.175, to 0.575. Then tree 2 works on what tree 1 got wrong. T-02 has priority 1 but only 40 words, and it did not escalate. Tree 2 asks about the word count and moves T-02 down, from 0.575 to 0.385. After two trees, the three escalated priority 1 tickets have the highest scores.

Why it helps: adding corrections lowers the bias

Boosting attacks the other problem. Its members are small and simple, so each one alone has high bias. To see this, Priya boosts the simplest possible trees, with one question each (max_depth=1), and scores the model after 1, 20 and 200 trees:

for n_trees in (1, 20, 200):
    stumps = HistGradientBoostingClassifier(max_depth=1, learning_rate=0.1, max_iter=n_trees,
                                            early_stopping=False, random_state=0)
    print(f"{n_trees:3d} trees:", evaluate(tree_pipeline(stumps)))

Output

  1 trees: {'train AP': 0.178, 'CV AP': 0.178, 'std': 0.008}
 20 trees: {'train AP': 0.285, 'CV AP': 0.279, 'std': 0.017}
200 trees: {'train AP': 0.341, 'CV AP': 0.333, 'std': 0.018}

• Train AP
• CV AP
• Logistic regression (E03), CV AP 0.327

Boosting trees of one question each, learning rate 0.1. Each round lowers the bias: both scores rise from the level of a one-question rule to the level of logistic regression.

Text description of the chart

The horizontal axis is the number of trees: 1, 2, 5, 10, 20, 50, 100, 200 and 500. The vertical axis is the average precision, from 0.10 to 0.40. With 1 and 2 trees, both scores are 0.178, the same as the rule “flag priority 1”. They rise together: with 20 trees, train 0.285 and CV 0.279; with 100 trees, 0.331 and 0.325; with 200 trees, 0.341 and 0.333; with 500 trees, 0.350 and 0.336. The two lines stay close together. A dashed line marks logistic regression’s CV AP, 0.327: the CV line reaches it at about 100 trees and stays close to it.

One tree of one question is the rule “flag priority 1” again: 0.178. Each added tree corrects some of the remaining error, and both scores rise together, up to 0.333 with 200 trees. The members were too simple; the sum is not. So: a forest averages to lower the variance of deep trees; boosting adds up corrections to lower the bias of small trees.

Boosting has its own danger. With many deep trees or a large learning rate, it keeps correcting errors that are only noise, and it overfits. So boosting has more hyperparameters to choose: the number of trees, the learning rate and the depth. scikit-learn’s HistGradientBoostingClassifier stops adding trees by default when a part of the train set (10%) stops improving. With its default settings, it stopped at 36 trees on Larkfield’s data.

Worked example: the ensembles on Larkfield’s tickets

Now the real test, with the rules of the loop: the same folds and the same metric as every row of the log. Priya scores three ensembles next to logistic regression. min_samples_leaf=20 stops any split that leaves fewer than 20 tickets in a leaf, as in Control complexity. n_jobs=-1 uses all the processor cores. It takes about 20 seconds.

results = {
    "logistic regression": evaluate(build_pipeline()),
    "forest, default": evaluate(tree_pipeline(
        RandomForestClassifier(n_estimators=200, n_jobs=-1, random_state=0))),
    "forest, min leaf 20": evaluate(tree_pipeline(
        RandomForestClassifier(n_estimators=200, min_samples_leaf=20, n_jobs=-1, random_state=0))),
    "boosting, default": evaluate(tree_pipeline(HistGradientBoostingClassifier(random_state=0))),
}
print(table(results))

Before you read the output, predict: will any ensemble beat logistic regression’s 0.327 ± 0.012 by more than one standard deviation?

Output

                     train AP  CV AP    std
logistic regression     0.328  0.327  0.012
forest, default         1.000  0.290  0.014
forest, min leaf 20     0.483  0.325  0.018
boosting, default       0.477  0.316  0.010

The ensembles fix most of what a single tree does wrong. The best single tree had 0.278 (E09), and the forest with min_samples_leaf=20 gets 0.325. But compare them with logistic regression:

• The forest with min leaf 20 is 0.002 below logistic regression. That is far smaller than one standard deviation (0.012 to 0.018): a tie.
• Boosting with its default settings is 0.011 below, also within the noise.
• The forest with default settings is lower, 0.290, with a train AP of 1.000: its deep trees memorise. The failure case below comes back to it.

The model families in the experiment log: CV AP with a bar of one standard deviation on each side. The best ensembles overlap logistic regression (E03).

Text description of the chart

The chart shows eight rows from the log, on an axis from 0.15 to 0.35. E02, the rule “flag priority 1”: 0.178. E03, logistic regression: 0.327, from 0.315 to 0.339, highlighted. E07, the tree of depth 6: 0.258. E09, the tree with min_samples_leaf=100: 0.278. E11, the random forest with default settings: 0.290. E12, the random forest with min_samples_leaf=20: 0.325, from 0.307 to 0.343. E13, gradient boosting with default settings: 0.316, from 0.306 to 0.326. E14, gradient boosting with max_depth=3 and learning_rate=0.05: 0.327, from 0.311 to 0.343. The bars of E03, E12, E13 and E14 overlap.

Why only a tie? The ensembles can find interactions and complex shapes. But the risk at Larkfield comes mostly from facts that each add or remove some risk on their own. The pattern is mostly additive, which is exactly the shape of logistic regression. Even the boosted one-question trees above, an additive model made of trees, end at the same level. There is little left for the extra flexibility to find. On another dataset, with strong interactions, the ensembles could win clearly. That is why you measure.

Priya confirms on the validation month once, for the candidates. It confirms; it does not choose:

for name, pipeline in [("logistic regression", build_pipeline()),
                       ("forest, min leaf 20", tree_pipeline(RandomForestClassifier(
                           n_estimators=200, min_samples_leaf=20, n_jobs=-1, random_state=0))),
                       ("boosting, default", tree_pipeline(HistGradientBoostingClassifier(random_state=0)))]:
    pipeline.fit(train[FEATURES], train[TARGET])
    print(f"{name:20s} valid AP {average_precision_score(valid[TARGET], pipeline.predict_proba(valid[FEATURES])[:, 1]):.3f}")

Output

logistic regression  valid AP 0.353
forest, min leaf 20  valid AP 0.382
boosting, default    valid AP 0.369

On May, the forest is 0.029 above logistic regression. Is that a reason to switch? May has only 183 escalations, so its AP moves by about 0.03 by chance alone. Priya measured this with the bootstrap: she resampled the May tickets 1,000 times, and the forest’s lead had a standard deviation of 0.018. The middle 95% of the resampled leads went from −0.006 to +0.063, which includes zero. Cross-validation, on 16,939 tickets, said “tie”, and May does not contradict it.

Guided practice: log the ensembles

Add the three ensembles to the log. Do this in the notebook. Each step has a check.

1. Run the cell with the four models. You should see the table above.
2. Write the row E11 for the forest with default settings. Use None for the valid AP: it is not a candidate.
3. Write the row E12 for the forest with min_samples_leaf=20, with its valid AP, 0.382, because you checked it once.
4. Write the row E13 for boosting with default settings, with its valid AP, 0.369.
5. Use the decision "reject" for all three, and write a reason for each. The reason for a tie should say so.
6. Run show_log() and the check cell. You should see Check passed: the log from E11 to E13 is right.

Show an example answer

for row_id, name, change, valid_ap, reason in [
    ("E11", "forest, default", "Random forest, 200 trees, default settings", None,
     "Its trees memorise (train AP 1.000); below E03."),
    ("E12", "forest, min leaf 20", "Random forest, 200 trees, min_samples_leaf=20", 0.382,
     "A tie with E03 (higher on 3 folds of 5), and 2,000 times bigger as a file."),
    ("E13", "boosting, default", "Gradient boosting, default settings", 0.369,
     "Lower than E03 on all 5 folds."),
]:
    log_experiment(row_id, change, results[name]["CV AP"], results[name]["std"], valid_ap, "reject", reason)
show_log(columns=("id", "change", "cv_ap", "cv_std", "valid_ap", "decision"))

The forest is rejected although it ties: with no difference that the data can show, the simpler model stays. The next section explains what “2,000 times bigger” means.

Your turn: a slower, smaller boosting

Change one condition: give boosting smaller trees and smaller steps, with max_depth=3 and learning_rate=0.05. Before you run it, predict: will the train AP go up or down from 0.477? Will the CV AP change much? Then log it as E14.

slow_boosting = evaluate(tree_pipeline(
    HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, random_state=0)))
print(slow_boosting)

Show the answer

Output

{'train AP': 0.366, 'CV AP': 0.327, 'std': 0.016}

The train AP falls from 0.477 to 0.366: smaller trees and smaller steps fit the train tickets less closely. The CV AP goes from 0.316 to 0.327, exactly the score of logistic regression. The gap between train AP and CV AP shrinks from 0.161 to 0.039. It is the best boosting so far, and still only a tie. On May, it gets 0.374. Decision: reject. A possible reason: “A tie with E03; more complex and harder to explain.”

log_experiment("E14", "Gradient boosting, max_depth=3, learning_rate=0.05", slow_boosting["CV AP"],
               slow_boosting["std"], 0.374, "reject", "A tie with E03; more complex and harder to explain.")

The cost of complexity

Tomás still prefers the forest: “It is the more powerful model, and it won on May.” Omar asks a better question: “What would the forest change for our customers, and what would it cost us?” A tie in the score is not a tie in the cost. Priya measures the cost:

def costs(name, pipeline):
    """Time to train, time to score the 1,179 May tickets, and the size of the saved file."""
    start = time.perf_counter()
    pipeline.fit(train[FEATURES], train[TARGET])
    fit_seconds = time.perf_counter() - start
    start = time.perf_counter()
    pipeline.predict_proba(valid[FEATURES])
    score_ms = (time.perf_counter() - start) * 1000
    file = io.BytesIO()          # a file in memory
    joblib.dump(pipeline, file)  # save the pipeline as a file, as Module 8 does
    print(f"{name:20s} train {fit_seconds:5.2f} s   score May {score_ms:5.1f} ms   file {file.tell() / 1000:8,.0f} KB")
    return pipeline


simple = costs("logistic regression", build_pipeline())
forest = costs("forest, min leaf 20", tree_pipeline(
    RandomForestClassifier(n_estimators=200, min_samples_leaf=20, n_jobs=-1, random_state=0)))
boosted = costs("boosting, default", tree_pipeline(HistGradientBoostingClassifier(random_state=0)))
print("logistic regression:", simple.named_steps["model"].coef_.size, "weights")
print("forest:", sum(tree.get_n_leaves() for tree in forest.named_steps["model"].estimators_), "leaves in 200 trees")

The output on the test laptop (your times will differ; the sizes will not):

Output

logistic regression  train  0.02 s   score May   2.0 ms   file        5 KB
forest, min leaf 20  train  0.27 s   score May  14.1 ms   file   10,459 KB
boosting, default    train  0.14 s   score May   3.5 ms   file      145 KB
logistic regression: 27 weights
forest: 64934 leaves in 200 trees

The forest takes about ten times longer to train and to score, and its file is about 2,000 times bigger: 10.5 MB against 5 KB. That is not a problem for 1,179 tickets. But every cost grows with the data, and it comes back each time you retrain or use the model. The less visible costs are larger:

• It is harder to explain. Logistic regression has 27 weights, one for each column after preprocessing. Priya can show Grace which facts raise a ticket’s score, and by how much. The forest has 64,934 leaves in 200 trees. No person can read them. To explain one decision, you need extra tools, and they give only an approximation. Explain predictions shows what you can do.
• It is harder to check and to trust. A bank, a hospital or a regulator may require that a person can understand and check the model. When a simple model is better looks at such rules.
• It is harder to debug. When the scores go wrong after a change in the data, a few weights are easy to inspect. Thousands of leaves are not.
• It has more hyperparameters. The number of trees, the depth, the leaf size, the learning rate. More settings mean more experiments, and every look at the validation month uses it up a little. Try enough settings, and one of them wins on May by luck.
• It can overfit in more ways. As the forest with default settings shows below.

So a complex model must earn its place: it must beat the simple model by more than the noise, and the gain must be worth these costs. At Larkfield, the data shows no gain at all. Priya keeps logistic regression, E03. Omar’s answer: the forest would change nothing for customers that the data can show, and it would cost more to run, to explain and to maintain.

This is not a rule against ensembles. A forest or boosting often wins clearly on data with strong interactions, such as many kinds of sensor data. They are also worth it when a small gain is worth a lot of money. Gradient boosting is one of the strongest model families for tables of data. Measure, then decide.

The perfect score

Common mistake

The perfect score

Symptom: Tomás reports the forest with default settings by its score on train:

print("random forest, default settings:", results["forest, default"])

Output

random forest, default settings: {'train AP': 1.0, 'CV AP': 0.29, 'std': 0.014}

“Train AP 1.000: it finds every escalation.” But in cross-validation, the same forest gets 0.290, below the forest with min_samples_leaf=20 (0.325) and below logistic regression (0.327).

Cause: with the default settings, every tree grows until each leaf is pure. Each tree memorises its bootstrap sample, so the forest scores the tickets it learned from almost perfectly. Averaging 200 memorising trees helps a lot (one such tree gets only 0.137), but not enough.

Fix: never judge a model by its score on the tickets it learned from. Compare models by cross-validation, as mean ± std, and confirm on the validation month. Limit the trees, for example with min_samples_leaf.

Consequence in AI work: a score on training data is the most common “too good to be true” number in reports. A team that ships the forest because of “1.000” plans the senior team’s work with a model that is weaker than the simple one, and more expensive.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study compares a forest and boosting with logistic regression, and asks if the gain is worth it.

Knowledge check

Check your understanding

Explain the difference between a random forest and gradient boosting in three sentences: how the members are trained, how they are combined, and which problem (bias or variance) each one mainly reduces.

Show the answer

A random forest trains many deep trees independently, at the same time, each on its own bootstrap sample and with random features at each split. It averages their scores, and the averaging mainly reduces the variance of the deep trees. Gradient boosting trains small trees one after another, each on the errors that the earlier trees left. It adds up their corrections, scaled by the learning rate, and so mainly reduces the bias of the small trees.

Check your understanding

Name three costs of replacing Larkfield's logistic regression with the forest (min leaf 20), even though their CV scores tie.

Show the answer

Any three of these: it is slower to train and to score (about ten times); its file is about 2,000 times bigger; it cannot be read by a person (64,934 leaves against 27 weights), so it is harder to explain to Grace or a regulator; it is harder to debug; it has more hyperparameters, so more experiments and more risk of choosing by luck on the validation month.

Module practice

The module task: compare two model families under the same validation procedure, and explain the difference without assuming that the more complex model is better. The same procedure means the same features, the same 5 folds, the same score and the same order of steps. Decide with CV on train, then confirm once on May.

Worked example: logistic regression against the forest, fold by fold

A mean ± std summarises five numbers. With the same folds, you can also compare the two models on each fold. compare() runs two pipelines on the same 5 folds. It prints the AP per fold, the mean ± std, the train AP, and the difference on each fold:

def compare(name_a, pipeline_a, name_b, pipeline_b):
    """Two pipelines on the same 5 folds of train: AP per fold, mean ± std, the
    difference per fold, and the train AP (fit and scored on all train tickets)."""
    a = cross_val_score(pipeline_a, train[FEATURES], train[TARGET], cv=cv, scoring="average_precision")
    b = cross_val_score(pipeline_b, train[FEATURES], train[TARGET], cv=cv, scoring="average_precision")
    for name, pipeline, folds in ((name_a, pipeline_a, a), (name_b, pipeline_b, b)):
        pipeline.fit(train[FEATURES], train[TARGET])
        train_ap = average_precision_score(train[TARGET], pipeline.predict_proba(train[FEATURES])[:, 1])
        print(f"{name:20s} folds {folds.round(3)}  CV AP {folds.mean():.3f} ± {folds.std():.3f}  train AP {train_ap:.3f}")
    print(f"{'b minus a':20s} folds {(b - a).round(3)}  mean {(b - a).mean():+.3f}")
    return a, b


folds_a, folds_b = compare("logistic regression", build_pipeline(), "forest, min leaf 20", tree_pipeline(
    RandomForestClassifier(n_estimators=200, min_samples_leaf=20, n_jobs=-1, random_state=0)))

Output

logistic regression  folds [0.316 0.31  0.342 0.339 0.327]  CV AP 0.327 ± 0.012  train AP 0.328
forest, min leaf 20  folds [0.318 0.294 0.326 0.344 0.342]  CV AP 0.325 ± 0.018  train AP 0.483
b minus a            folds [ 0.002 -0.016 -0.016  0.005  0.015]  mean -0.002

The forest wins on three folds and loses on two, and the sign of the difference changes from fold to fold. It is not consistently better or worse: a tie. Its train AP, 0.483, is far above its CV AP, so it learns details that do not repeat. On May, it was 0.029 higher, inside May’s chance range of about 0.03. Decision: keep logistic regression, because the more complex model shows no real advantage and costs more.

Your task

1. Choose two models from two different families: logistic regression, a limited tree, a random forest or gradient boosting. Do not use the pair of the worked example.
2. Predict which one wins, and by how much.
3. Run compare(). Read the difference per fold, and the gap between the train AP and the CV AP of each model.
4. Decide from the CV results only. Write one paragraph: the decision, the numbers that support it, and the reason.
5. Then confirm on May, once, and say whether May changes anything.

Show the model answer: logistic regression against gradient boosting

folds_a, folds_b = compare("logistic regression", build_pipeline(),
                           "boosting, default", tree_pipeline(HistGradientBoostingClassifier(random_state=0)))

Output

logistic regression  folds [0.316 0.31  0.342 0.339 0.327]  CV AP 0.327 ± 0.012  train AP 0.328
boosting, default    folds [0.313 0.298 0.323 0.322 0.324]  CV AP 0.316 ± 0.010  train AP 0.477
b minus a            folds [-0.003 -0.012 -0.019 -0.017 -0.003]  mean -0.011

Decision: keep logistic regression.

• Boosting is lower on all five folds, by 0.003 to 0.019 (mean 0.011). On the same folds, a difference with the same sign every time is consistent, although it is small.
• Boosting’s train AP is 0.477, against a CV AP of 0.316: a gap of 0.161. It learns details of the train tickets that do not repeat. Logistic regression has almost no gap (0.328 and 0.327).
• Why the simpler model wins: the pattern in this data is mostly additive, which is exactly the shape of logistic regression. Boosting’s extra flexibility finds mostly noise here.

Then the confirmation on May, once: logistic regression 0.353, boosting 0.369. Boosting is 0.016 higher on May, far inside May’s chance range of about 0.03. It does not change the decision. Switching models because of it would mean choosing by May.

If you chose a limited tree (for example min_samples_leaf=100): it is lower on all five folds, by 0.036 to 0.061 (mean 0.049), a clear loss. If you chose the slow boosting (E14): the differences are −0.003, −0.007, −0.003, −0.002 and +0.015, a mean of 0.000: a tie.

Module check

Check your understanding

Retrieval: an ensemble averages its members. What two conditions must the members meet for the average to be better than one member?

Show the answer

Each member must be useful on its own (better than guessing), and the members must make different errors. Errors that differ cancel in the average, like the guesses of the sweets in a jar. Errors that all members share stay in the average. This is why a random forest uses bootstrap samples and random features: they make the trees different.

Check your understanding

Explain or predict: Tomás builds a 'forest' of 200 deep trees in which every tree sees all the train rows and all the features. Predict its CV AP compared with one deep tree (0.137), and explain.

Show the answer

It will be almost the same as one tree, a little above 0.137: on this data, about 0.149. Without bootstrap samples and random features, the trees are almost the same, so they make the same errors. Averaging copies of one mistake does not remove the mistake. A real random forest of deep trees reaches 0.290, because its trees are different.

Check your understanding

Completion: the forest (min leaf 20) has CV AP 0.325 ± 0.018 and valid AP 0.382; logistic regression has 0.327 ± 0.012 and 0.353. Explain the difference to Grace, and say which model you would hand over and what result would change your mind.

Show the answer

On 16,939 train tickets, scored on the same 5 folds, the two models tie: the forest is 0.002 lower, and it wins on three folds and loses on two. The May lead of 0.029 is inside May’s chance range of about 0.03, because May has only 183 escalations. So the data shows no real difference. I hand over logistic regression: it is as good, smaller, faster, and Grace’s team can see why a ticket was flagged. I would change my mind if the forest were consistently higher on the same folds, by more than the noise, and the gain were worth the extra cost. That could happen, for example, after new features that create strong interactions.

Module 5 recap and reset

In this module you went around the loop of model development several times. Your experiment log now has 14 rows: two baselines, logistic regression, and eleven experiments with other settings and other model families. Each was measured on the same 5 folds with the same metric, and each has a decision and a reason. The result: logistic regression, E03, is still the model to keep. The trees lose, and the ensembles tie.

Reset: to get back to a known state before Module 6:

1. Run python get_data.py --reset in the project folder. It downloads the data again and checks it.
2. If ticket_model.py is broken, write it again from Preprocess with a pipeline: Write ticket_model.py. Module 6 changes it in Select useful features.
3. If your experiment log is lost, run the solution notebook of this lesson from the top. It writes the rows E01 to E14 into its own experiment_log.csv. Download that file, and put it in your project folder.

Recap

• Machine learning is iterative: change one thing, measure it on the same folds with the same metric, log it, keep what helps.
• A decision tree asks questions in order. A deep tree memorises (train AP 1.000, CV AP 0.137); limits such as max_depth and min_samples_leaf are hyperparameters, chosen with CV.
• An ensemble combines many models. It helps only if the members are useful and make different errors, like a crowd whose guesses are independent.
• A random forest averages deep trees trained on bootstrap samples with random features: it lowers the variance (spread 0.194 → 0.034). Gradient boosting adds small trees that each correct the errors before them: it lowers the bias (0.178 → 0.333 with 200 one-question trees).
• On Larkfield’s tickets, the best ensembles tie logistic regression (0.325 and 0.327 against 0.327 ± 0.012), because the pattern is mostly additive. A complex model costs more to run, explain and maintain, so it must earn its place with a gain larger than the noise.

Next, in Module 6, you improve the model by changing its features instead of its family.

Next: Select useful features