Decision trees

What you will be able to do

Logistic regression is one family of models. A decision tree is another: it scores a ticket by asking questions in order. Learn how a tree chooses its questions, and try it on Larkfield’s tickets with the same fair test as before. Then add the result to your experiment log.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install the packages, download the practice data, write ticket_model.py and write the experiment log of the previous lesson, so it works without the local project.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The loop and the experiment log, from Machine learning is iterative: The experiment log.
• Overfitting: a model that memorises its practice data, from Train, validation and test sets.
• Cross-validation, from Train, validation and test sets: Cross-validation, and average precision, from The precision–recall trade-off: Average precision.
• Logistic regression, scores and predict_proba, from Classification intuition.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, active in a terminal, and the lesson notebooks open in VS Code or JupyterLab. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv (16,939 rows), from Define a baseline: Download the data. If a file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 6 numbers in NUMBERS, from Preprocess with a pipeline: Write ticket_model.py. If it is missing or broken, write it again from that section. Do not copy the finished file from C05/project yet: it has an extra feature that Module 6 adds, so your numbers would differ.
• Your experiment log with the rows E01 to E05, from Machine learning is iterative: The experiment log. If it is missing, the notebook’s setup cell writes the five rows again.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

Twenty questions

In the game of twenty questions, one player thinks of a thing, and the other player finds it with yes-or-no questions. A good player does not start with “Is it my cat?”. She starts with a question that splits the possibilities well, such as “Is it alive?”. Each answer sends her down a different path, and the next question depends on the answers before it.

A nurse at a hospital entrance works in a similar way. She follows a short checklist: “Is the patient breathing normally? Is there heavy bleeding? Is the pain severe?”. A “no” to the first question leads straight to the most urgent group. The order of the questions matters, and each answer narrows the group.

A model can work like this too. Instead of adding up weighted facts, it asks questions about the ticket, one at a time, in a fixed order.

The problem: facts that work together

Logistic regression gives each feature one weight, and adds the weighted features up. A long message adds the same amount of risk to every ticket, whatever its priority. That is a strong assumption. Maybe a long message matters a lot for a priority 1 ticket and only a little for a priority 3 ticket. When the effect of one fact depends on another fact, the two facts interact. Logistic regression cannot learn an interaction unless someone builds a special column for it.

A question-based model does not have this limit. If the first question is “Is the priority 1?”, the next question is asked only inside each group. So it can use one cut point for the word count of priority 1 tickets and another cut point for the other tickets.

This model is a decision tree. It has this name because its diagram looks like a tree drawn upside down. The first question is the root at the top, and the answers are branches. The groups at the ends are the leaves. Three more words:

• A split is one question and the two groups that it makes.
• The depth is the number of questions from the root to a leaf.
• The tree’s score for a ticket is the share of escalated train tickets in the leaf where the ticket ends.

A tree on ten tickets

Here are 10 made-up tickets, small enough to check by hand. Four of them escalated.

Ticket	T-01	T-02	T-03	T-04	T-05	T-06	T-07	T-08	T-09	T-10
priority	1	1	1	1	2	2	2	3	3	3
word_count	150	40	120	45	130	160	20	30	145	60
escalated	1	0	1	1	1	0	0	0	0	0

import pandas as pd
from sklearn.metrics import average_precision_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier, export_text

from ticket_model import FEATURES, TARGET, build_pipeline, load

train = load("train")

tiny = pd.DataFrame({
    "ticket": [f"T-{i:02d}" for i in range(1, 11)],
    "priority": [1, 1, 1, 1, 2, 2, 2, 3, 3, 3],
    "word_count": [150, 40, 120, 45, 130, 160, 20, 30, 145, 60],
    "escalated": [1, 0, 1, 1, 1, 0, 0, 0, 0, 0],
})

Which question first?

Like the good player of twenty questions, the tree wants a first question that separates the tickets well. The best possible groups are pure: all escalated, or all not escalated. Impurity measures how mixed a group is.

The tree uses the Gini impurity. Take p, the share of escalated tickets in a group. The impurity is 2 × p × (1 − p). A pure group has p = 0 or p = 1, so its impurity is 0. A half-and-half group has 2 × 0.5 × 0.5 = 0.5, the most mixed.

Work it out for the 10 tickets. Four of ten escalated, so p = 0.4, and the impurity is 2 × 0.4 × 0.6 = 0.48. Now compare two candidate questions. After a split, the tree takes the impurity of each group and weights it by the group’s size:

Question	Yes group	No group	Impurity after the split
priority = 1?	4 tickets, 3 escalated: 2 × 0.75 × 0.25 = 0.375	6 tickets, 1 escalated: 2 × 1/6 × 5/6 = 0.278	(4 × 0.375 + 6 × 0.278) ÷ 10 = 0.317
more than 100 words?	5 tickets, 3 escalated: 0.48	5 tickets, 1 escalated: 0.32	(5 × 0.48 + 5 × 0.32) ÷ 10 = 0.40

“priority = 1?” lowers the impurity more, from 0.48 to 0.317. The tree tries every possible question on every feature, and takes the one with the lowest impurity after the split. Then it does the same inside each group, until it reaches its maximum depth or the groups are pure.

Check this with scikit-learn. DecisionTreeClassifier(max_depth=2) grows a tree with at most two questions on each path. export_text prints its questions. The last two lines score a new ticket with priority 2 and 140 words:

tiny_tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tiny_tree.fit(tiny[["priority", "word_count"]], tiny["escalated"])
print(export_text(tiny_tree, feature_names=["priority", "word_count"], show_weights=True))

new_ticket = pd.DataFrame({"priority": [2], "word_count": [140]})
print("score of the new ticket:", tiny_tree.predict_proba(new_ticket)[0, 1].round(3))

The output:

Output

|--- priority <= 1.50
|   |--- word_count <= 42.50
|   |   |--- weights: [1.00, 0.00] class: 0
|   |--- word_count >  42.50
|   |   |--- weights: [0.00, 3.00] class: 1
|--- priority >  1.50
|   |--- word_count <= 95.00
|   |   |--- weights: [3.00, 0.00] class: 0
|   |--- word_count >  95.00
|   |   |--- weights: [2.00, 1.00] class: 0

score of the new ticket: 0.333

The first question is “priority = 1?”, written as priority <= 1.50: the tree puts its cut point halfway between two values. weights: [2.00, 1.00] means 2 tickets not escalated and 1 escalated in that leaf. Notice the interaction: the second question uses 42.5 words on the priority 1 side and 95 words on the other side. Logistic regression could not do that with one weight for word_count.

The tree on the 10 tickets. Each ticket follows the answers down to one leaf. Three leaves are pure; the last one mixes T-05 (escalated) with T-06 and T-09.

Text description of the diagram

The root asks “priority = 1?” of all 10 tickets, of which 4 escalated, with an impurity of 0.48.

• Yes: T-01 to T-04, 3 of 4 escalated. The next question is “more than 42 words?”. No: a leaf with T-02 only, score 0.0. Yes: a leaf with T-01, T-03 and T-04, score 1.0.
• No: T-05 to T-10, 1 of 6 escalated. The next question is “more than 95 words?”. No: a leaf with T-07, T-08 and T-10, score 0.0. Yes: a leaf with T-05, T-06 and T-09, score 0.333.

A new ticket with priority 2 and 140 words follows “no”, then “yes”, and lands in the last leaf: its score is 1 ÷ 3 = 0.333.

The real tree

Now the same on the 16,939 train tickets, with the 10 features. One change comes first: a tree does not need the scaler of the pipeline. A tree compares a feature with a cut point. Scaling changes the numbers but not their order, so every ticket goes to the same side of every cut. tree_pipeline() takes the scaler out of build_pipeline() ("passthrough" means “no step here”):

def tree_pipeline(model):
    """build_pipeline() without the scaler: a tree does not need it."""
    return build_pipeline(model).set_params(prepare__numbers__scale="passthrough")


pipeline = tree_pipeline(DecisionTreeClassifier(max_depth=2, random_state=0))
pipeline.fit(train[FEATURES], train[TARGET])
names = list(pipeline.named_steps["prepare"].get_feature_names_out())
print(export_text(pipeline.named_steps["model"], feature_names=names, show_weights=True))

The output (the prefix numbers__ is the name of the pipeline’s branch for numbers):

Output

|--- numbers__priority <= 1.50
|   |--- numbers__word_count <= 66.50
|   |   |--- weights: [635.00, 128.00] class: 0
|   |--- numbers__word_count >  66.50
|   |   |--- weights: [1057.00, 565.00] class: 0
|--- numbers__priority >  1.50
|   |--- numbers__word_count <= 100.50
|   |   |--- weights: [7954.00, 507.00] class: 0
|   |--- numbers__word_count >  100.50
|   |   |--- weights: [5310.00, 783.00] class: 0

The real depth-2 tree, fitted on train. Each leaf's score is the share of escalated train tickets in it.

Text description of the diagram

The tree starts with all 16,939 train tickets, 11.7% escalated, and asks “priority = 1?”.

• Yes: 2,385 tickets, 29.1% escalated. The next question is “more than 66 words?”. No: leaf 1, 763 tickets, 128 escalated, score 0.168. Yes: leaf 2, 1,622 tickets, 565 escalated, score 0.348.
• No (priority 2 or 3): 14,554 tickets, 8.9% escalated. The next question is “more than 100 words?”. No: leaf 3, 8,461 tickets, 507 escalated, score 0.060. Yes: leaf 4, 6,093 tickets, 783 escalated, score 0.129.

The tree found two facts on its own: the priority, and the length of the first message. A priority 1 ticket with more than 66 words is the riskiest: 565 of 1,622 escalated, a score of 0.348. A short ticket with priority 2 or 3 is the safest, at 0.060. The weights read as [not escalated, escalated]. For leaf 2: 1,057 + 565 = 1,622 tickets, and 565 ÷ 1,622 = 0.348. The four leaves add up to 16,939, every train ticket.

You can explain this tree to Grace in one minute, and she can check it against her own experience. That is a real strength. But it gives only four different scores, so it ranks the tickets coarsely. A deeper tree has more leaves and more different scores. Is a deeper tree better? Only a fair test can say.

Compare the tree with logistic regression

The comparison follows the rules of the loop: the same 5 folds of train, the same metric, and the result in the log. Two short reminders:

• Cross-validation cuts the train set into 5 folds. It fits the model 5 times, each time on 4 folds, and scores it on the fold that it did not see. The mean of the 5 scores is the CV score, and their standard deviation shows how much it moves by chance. Train, validation and test sets explains it fully.
• Average precision (AP) sorts the tickets by score and measures how early the escalated ones come. One score for every ticket gives the escalation rate, 0.117. The precision–recall trade-off explains it fully.

evaluate() returns two numbers for a pipeline. The train AP fits the model on all train tickets and scores the same tickets. The CV AP is the fair score, with its standard deviation. table() shows the results, one row per model.

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)


def evaluate(pipeline):
    """Train AP (fit on all train tickets) and CV AP (mean ± std of the 5 folds)."""
    fold_scores = cross_val_score(pipeline, train[FEATURES], train[TARGET], cv=cv, scoring="average_precision")
    pipeline.fit(train[FEATURES], train[TARGET])
    train_ap = average_precision_score(train[TARGET], pipeline.predict_proba(train[FEATURES])[:, 1])
    return {"train AP": round(float(train_ap), 3), "CV AP": round(float(fold_scores.mean()), 3),
            "std": round(float(fold_scores.std()), 3)}


def table(results):
    """The results as a table, one row per model (rows still set to ... are left out)."""
    return pd.DataFrame({name: r for name, r in results.items() if isinstance(r, dict)}).T


results = {
    "logistic regression": evaluate(build_pipeline()),
    "tree, depth 3": evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=3, random_state=0))),
    "tree, depth 6": evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=6, random_state=0))),
    "tree, no limit": evaluate(tree_pipeline(DecisionTreeClassifier(random_state=0))),
}
print(table(results))

Before you read the output, predict: which tree has the highest train AP? Which has the highest CV AP?

Output

                     train AP  CV AP    std
logistic regression     0.328  0.327  0.012
tree, depth 3           0.244  0.234  0.011
tree, depth 6           0.346  0.258  0.010
tree, no limit          1.000  0.137  0.004

Read the table one row at a time:

• Logistic regression has almost the same train AP and CV AP: 0.328 and 0.327. What it learned on some tickets works on others.
• The depth-3 tree is far below it: 0.234. Three questions cannot separate the tickets well.
• The depth-6 tree is the best tree, 0.258, but still 0.069 below logistic regression. That is more than five standard deviations: a real difference.
• The tree with no limit scores a perfect 1.000 on train and the worst CV AP, 0.137, close to the floor of 0.117. It kept asking questions until every leaf was pure, so it learned each train ticket by heart, chance details included. This is the memorising model of Train, validation and test sets: perfect on the practice exercises, close to useless on new ones.

Why does the tree lose? The trees can find interactions, but this data has few strong ones. Most of the risk comes from facts that each add or remove some risk on their own, which is exactly the shape of logistic regression. A tree must describe such a smooth pattern with steps, one leaf at a time, and it needs many leaves and many tickets to do it.

Guided practice: log the trees

Add the three trees to your experiment log. Do this in the notebook. Each step has a check.

1. Run the cell with evaluate(). You should see the table above.
2. Write the row E06 for the depth-3 tree. Take the numbers from the table: results["tree, depth 3"]["CV AP"] and results["tree, depth 3"]["std"]. Leave the valid AP empty (None). Write "reject" and a reason.
3. Write the rows E07 (depth 6) and E08 (no limit) in the same way.
4. Run show_log(). You should see eight rows.
5. Run the check cell. You should see Check passed: the log from E06 to E08 is right.

Show an example answer

for row_id, name, change, reason in [
    ("E06", "tree, depth 3", "Decision tree, max_depth=3", "Far below E03."),
    ("E07", "tree, depth 6", "Decision tree, max_depth=6", "The best of the three trees, still far below E03."),
    ("E08", "tree, no limit", "Decision tree, no depth limit",
     "It memorises train (train AP 1.000) and fails on new tickets."),
]:
    log_experiment(row_id, change, results[name]["CV AP"], results[name]["std"], None, "reject", reason)
show_log()

Output

 id                                                change  cv_ap  cv_std valid_ap decision
E01 Baseline: one score for every ticket (the train rate)  0.117   0.000    0.155 baseline
E02  Baseline: the rule 'flag priority 1', as a 0/1 score  0.178   0.008    0.228 baseline
E03              Logistic regression, the 10 C04 features  0.327   0.012    0.353     keep
E04                      E03 with class_weight='balanced'  0.323   0.012            reject
E05  Baseline: the rule 'flag priority 1 or team payment'  0.172   0.006            reject
E06                            Decision tree, max_depth=3  0.234   0.011            reject
E07                            Decision tree, max_depth=6  0.258   0.010            reject
E08                         Decision tree, no depth limit  0.137   0.004            reject

None of the trees is a candidate, so the valid AP stays empty. E03 is still the model to keep.

Your turn: depth 4

Change one condition: grow a tree with max_depth=4. Before you run it, predict: will its CV AP be between the depth-3 tree (0.234) and the depth-6 tree (0.258)? Will its train AP be closer to its CV AP than for depth 6?

print(evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=4, random_state=0))))

Show the answer

Output

{'train AP': 0.275, 'CV AP': 0.247, 'std': 0.008}

Yes to both. The CV AP, 0.247, is between the two other trees. The gap between train AP and CV AP is 0.028, against 0.010 at depth 3 and 0.088 at depth 6. Each extra level of questions fits the train tickets more closely, and the gap grows. The next lesson follows this gap for every depth from 1 to 20.

When a tree is a good choice

A single tree has real strengths:

• You can read it. A small tree is a checklist that a person can follow and check, like the nurse’s questions.
• It needs little preparation. No scaling, and it handles numbers and categories in the same way.
• It finds interactions. It can ask a different question in each branch.

It also has real weaknesses:

• It is unstable. A small change in the train tickets can change the first question, and then the whole tree changes.
• Its scores are coarse. All tickets in a leaf get the same score.
• A deep tree memorises. With no limit, it learns noise, as the 1.000 and 0.137 show.

So a single small tree is useful when people must follow or check the rules. As a predictor on its own, it is usually weaker than other models. The next lesson shows how to limit a tree’s complexity. The lesson after it, Ensembles, shows how many trees together fix the instability.

A rule from two tickets

Common mistake

A rule from two tickets

Symptom: Tomás reads the depth-6 tree and finds a leaf with a score of 1.0. He tells Grace: “Priority 3 tickets with an order value between $190.53 and $190.80 always escalate.” The tree really has that leaf:

deep = tree_pipeline(DecisionTreeClassifier(max_depth=6, random_state=0)).fit(train[FEATURES], train[TARGET])
leaf_of_ticket = deep.named_steps["model"].apply(deep.named_steps["prepare"].transform(train[FEATURES]))
leaf_sizes = pd.Series(leaf_of_ticket).value_counts()
small = leaf_sizes[leaf_sizes < 10].index
pure = [leaf for leaf in small if train[TARGET][leaf_of_ticket == leaf].nunique() == 1]
print(len(leaf_sizes), "leaves;", len(small), "with fewer than 10 train tickets;", len(pure), "of these are pure")
print(train.loc[leaf_of_ticket == 113, ["ticket_id", "priority", "order_value", TARGET]].to_string(index=False))

apply() gives the number of the leaf where each ticket ends. The output:

Output

59 leaves; 18 with fewer than 10 train tickets; 13 of these are pure
ticket_id  priority  order_value  escalated_72h
 T-113665         3       190.56              1
 T-116778         3       190.64              1

Cause: the “rule” comes from two tickets. A cut point of $190.53 and another of $190.80 only separate these two tickets from their neighbours. Of the 59 leaves of the depth-6 tree, 18 have fewer than 10 train tickets, and 13 of these are pure, only by chance. A leaf of two tickets has a score of 0 or 1 very easily, the same way two coin flips are both heads one time in four.

Fix: never read a leaf’s score without its number of tickets. Limit the tree so that each leaf has enough tickets: min_samples_leaf, in the next lesson. Judge the tree by its CV AP, not by its most striking leaves.

Consequence in AI work: rules read from small leaves sound precise and convincing. A team that acts on them changes a process because of two tickets. The same trap appears with any small group of data: always show the count next to the rate.

See it end to endRank a call listCase study

The call-list case study turns a depth-3 tree into eight rules that a call team can read.

Knowledge check

Check your understanding

For the 10 tickets, a third question is 'priority = 3?'. Its groups are 3 tickets with 0 escalated and 7 tickets with 4 escalated. Compute the impurity after this split. Would the tree choose it over 'priority = 1?' (0.317)?

Show the answer

The yes group is pure: 2 × 0 × 1 = 0. The no group has p = 4/7, so 2 × 4/7 × 3/7 = 0.490. Weighted by size: (3 × 0 + 7 × 0.490) ÷ 10 = 0.343. That is higher than 0.317, so the tree still chooses “priority = 1?”. A pure group alone does not win: the other group must also become purer.

Check your understanding

The tree with no limit has train AP 1.000 and CV AP 0.137. Logistic regression has 0.328 and 0.327. Omar asks which model learned more. What do you answer?

Show the answer

The tree learned more about the train tickets: it fits each one exactly. But most of what it learned is noise that does not repeat in new tickets, so its CV AP is close to the floor of 0.117. Logistic regression learned less detail, and what it learned works on tickets it has not seen. For Larkfield, only the score on new tickets matters, so logistic regression is the better model.

Check your understanding

A colleague says: 'A tree can find interactions and logistic regression cannot, so a tree is always better.' Use the CV results to answer.

Show the answer

Being able to find interactions helps only when the data has strong ones. Here the best tree has CV AP 0.258 and logistic regression 0.327, a real difference of more than five standard deviations. The risk at Larkfield comes mostly from facts that each add some risk on their own, the shape of logistic regression. No model family is always better: the data decides, and you measure it.

Recap

• A decision tree asks yes-or-no questions in order. At each split it chooses the question that lowers the Gini impurity most. A leaf’s score is its share of escalated train tickets.
• A tree can use a different question in each branch, so it can find interactions. It needs no scaling.
• The real depth-2 tree splits on priority 1, then on the word count. Its riskiest leaf scores 0.348.
• On the same folds, the trees lose to logistic regression: depth 6 gets 0.258, logistic regression 0.327 ± 0.012. A tree with no limit memorises: train AP 1.000, CV AP 0.137.
• Your log has eight rows. Never read a leaf’s score without its number of tickets.

Next, you control how complex a model is, and see when it learns too little or too much.

Next: Control complexity