Control complexity

What you will be able to do

A model can be too simple to learn the pattern, or so complex that it learns the noise of its train tickets. Learn to see both from two numbers, and to choose the settings that make a model work best on tickets it has not seen.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions and the charts. Its setup cells install the packages, download the practice data, write ticket_model.py and write your experiment log so far, so it works without the local project. Use the explorer here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Decision trees, tree_pipeline(), evaluate() and the table of logistic regression against three trees, from Decision trees: Compare the tree with logistic regression.
• Overfitting, from Train, validation and test sets, and cross-validation with its standard deviation, from Cross-validation there.
• The rules of the loop and the experiment log, from Machine learning is iterative: The experiment log.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, active in a terminal, and the lesson notebooks open in VS Code or JupyterLab. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv and data/valid.csv, from python get_data.py in Define a baseline: Download the data. If a file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 6 numbers in NUMBERS, from Preprocess with a pipeline: Write ticket_model.py. If it is missing or broken, write it again from that section. Do not copy the finished file from C05/project yet: it has an extra feature that Module 6 adds.
• matplotlib for the charts, from the requirements.txt of Define a baseline: Install the packages.
• Your experiment log with the rows E01 to E08, from Decision trees: Guided practice. If it is missing, the notebook’s setup cell writes the eight rows again.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

Three students

Three students prepare for a mathematics exam with the same book of practice exercises.

• The first student learns one rule: “the answer is usually the biggest number”. It works on a few exercises. On the exam, he gets most questions wrong. He learned too little: his rule is too simple for the subject.
• The second student learns every practice exercise by heart, with its answer. On the practice exercises, she is perfect. On the exam, the questions are new, and she is lost. She learned too much of the wrong thing: the details of the exercises, not the method.
• The third student learns the methods. She makes a few mistakes on the practice exercises, and about the same number on the exam.

A model can be any of these students. The first two are the two ways a model fails. The third is the model you want. The problem is that you cannot look inside a model’s head. You see only its scores. This lesson shows how two scores tell the three students apart. One is the score on the train tickets (the practice exercises). The other is the cross-validated score (new questions).

Too simple and too complex

The code in this lesson uses the helpers of Decision trees: cv (the 5 folds), tree_pipeline() and evaluate(), and the table tiny of the 10 made-up tickets. The notebook defines them again in its first cell, and also imports learning_curve, LogisticRegression and matplotlib.

Start with the 10 tickets of Decision trees. Grow three trees on them: 1 question deep, 2 deep, and with no limit (max_depth=None):

x_tiny = tiny[["priority", "word_count"]]
for depth in (1, 2, None):
    tiny_tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(x_tiny, tiny["escalated"])
    print(f"max_depth={depth}: {tiny_tree.get_n_leaves()} leaves, scores", tiny_tree.predict_proba(x_tiny)[:, 1].round(2))
print(export_text(tiny_tree, feature_names=["priority", "word_count"]))

Output

max_depth=1: 2 leaves, scores [0.75 0.75 0.75 0.75 0.17 0.17 0.17 0.17 0.17 0.17]
max_depth=2: 4 leaves, scores [1.   0.   1.   1.   0.33 0.33 0.   0.   0.33 0.  ]
max_depth=None: 5 leaves, scores [1. 0. 1. 1. 1. 0. 0. 0. 0. 0.]
|--- priority <= 1.50
|   |--- word_count <= 42.50
|   |   |--- class: 0
|   |--- word_count >  42.50
|   |   |--- class: 1
|--- priority >  1.50
|   |--- word_count <= 95.00
|   |   |--- class: 0
|   |--- word_count >  95.00
|   |   |--- word_count <= 137.50
|   |   |   |--- class: 1
|   |   |--- word_count >  137.50
|   |   |   |--- class: 0

With no limit, the tree needs 5 leaves, and its scores are exactly the targets. Look at its third question, word_count <= 137.50. It separates T-05 (130 words, escalated) from T-06 (160 words) and T-09 (145 words). The rule it makes: “a priority 2 or 3 ticket with 96 to 137 words escalates, and one with 138 words or more does not”. Would you bet on that rule for the next ticket? It comes from a single ticket.

These are the two failures of the students:

• Overfitting is the second student. The model learns details of the train tickets that do not repeat in new tickets. The score on train is high, and the score on new tickets is lower. The tree with no limit overfits: its “96 to 137 words” rule comes from one ticket.
• Underfitting is the first student. The model is too simple to learn the pattern that is there, so both scores are low. The tree with one question underfits: it gives the same score (0.17) to T-05, which escalated, and to T-08, which did not.

The tree with no limit, on the real data

In Decision trees, the tree with no limit scored 1.000 on train and 0.137 in CV. Here is how it gets there:

full_tree = tree_pipeline(DecisionTreeClassifier(random_state=0))
print("tree, no limit:", evaluate(full_tree))
model = full_tree.named_steps["model"]
print("leaves:", model.get_n_leaves(), " depth:", model.get_depth())

Output

tree, no limit: {'train AP': 1.0, 'CV AP': 0.137, 'std': 0.004}
leaves: 2210  depth: 31

The tree asks up to 31 questions in a row and makes 2,210 leaves, each of them pure. Every train ticket gets exactly its own target, so the train AP is 1.000. On tickets it did not see, the AP is 0.137, only a little above the floor of 0.117 (one score for all tickets). The tree learned 2,210 small rules like “96 to 137 words”, and most of them are noise.

Bias and variance

Think of two bad bathroom scales. The first one always shows 2 kg too much. Weigh yourself five times, and it shows the same wrong number five times. The second one is right on average, but it jumps: 68 kg, then 73 kg, then 66 kg. Both scales are wrong, in different ways. The first has a bias: a fixed error in one direction. The second has a large variance: its answers move a lot when nothing important changed.

Models fail in the same two ways, and these are the technical names:

• Bias is the error of a model that is too simple. It misses the same part of the pattern, whatever train tickets it gets. An underfitting model has high bias.
• Variance is how much the model changes when its train tickets change. An overfitting model has high variance: it follows the chance details of its tickets, and other tickets have other chance details.

To see them, Priya fits trees on five different random halves of train. Then she scores one May ticket, T-117529, which escalated. She also measures the spread of the five scores for every May ticket, and takes the mean:

valid = load("valid")
ticket = valid[valid["ticket_id"] == "T-117529"]
for depth in (None, 3):
    ticket_scores, all_scores = [], []
    for seed in range(5):
        half = train.sample(frac=0.5, random_state=seed)
        pipeline = tree_pipeline(DecisionTreeClassifier(max_depth=depth, random_state=0))
        pipeline.fit(half[FEATURES], half[TARGET])
        ticket_scores.append(round(float(pipeline.predict_proba(ticket[FEATURES])[0, 1]), 3))
        all_scores.append(pipeline.predict_proba(valid[FEATURES])[:, 1])
    spread = pd.DataFrame(all_scores).std(ddof=0).mean()
    print(f"max_depth={depth}: T-117529 gets {ticket_scores}; mean spread over May {spread:.3f}")

Output

max_depth=None: T-117529 gets [0.0, 1.0, 0.0, 1.0, 0.0]; mean spread over May 0.194
max_depth=3: T-117529 gets [0.117, 0.099, 0.099, 0.12, 0.113]; mean spread over May 0.024

• The tree with no limit has high variance. For the same ticket, its answer jumps between 0 and 1 with the train tickets it got. Over all May tickets, its five scores differ by 0.194 on average.
• The depth-3 tree is stable (0.024), but it gives this escalated ticket only about 0.1. Its few leaves cannot follow the pattern closely: it has higher bias.

A good model sits between the two. That is the bias–variance trade-off: more complexity lowers the bias and raises the variance. You find the best point with cross-validation, as the next sections show. Ensembles, the next lesson, attacks the variance from another side: it averages many high-variance trees.

Hyperparameters: settings you choose

How deep should a tree be? The tree cannot answer this itself. When you call fit(), the tree learns its questions and cut points from the data. These learned values are the model’s parameters: the weights of logistic regression are parameters too. But max_depth is not learned. You choose it before training, and it controls how the training works.

A setting that you choose before training is a hyperparameter. “Hyper” means “above”: a hyperparameter sits above the parameters and controls how they are learned. Think of a camera. The photo records what is in front of the lens. But you set the zoom before you take it, and the zoom decides how much detail the photo can show. max_depth is the zoom of a tree. Every algorithm has hyperparameters: max_depth and min_samples_leaf for a tree, C for logistic regression, as you see below.

You choose a hyperparameter like everything else in the loop. Try several values, score each one with cross-validation on train, and keep the best. Tune hyperparameters in Module 6 does this for several hyperparameters at once. Here you do it for one.

Worked example: the depth sweep

Priya fits one tree for every max_depth from 1 to 20 and adds the gap: train AP − CV AP. The gap shows how much better the tree is on the tickets it learned from than on new ones. It takes about 10 seconds.

rows = []
for depth in range(1, 21):
    result = evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=depth, random_state=0)))
    rows.append({"depth": depth, **result})
sweep = pd.DataFrame(rows).set_index("depth")
sweep["gap"] = (sweep["train AP"] - sweep["CV AP"]).round(3)
print(sweep.loc[[1, 2, 3, 4, 5, 6, 7, 8, 10, 15, 20]])

Before you read the output, predict: which depth gives the best CV AP, 2, 6 or 20?

Output

       train AP  CV AP    std    gap
depth
1         0.178  0.178  0.008  0.000
2         0.217  0.215  0.010  0.002
3         0.244  0.234  0.011  0.010
4         0.275  0.247  0.008  0.028
5         0.308  0.255  0.010  0.053
6         0.346  0.258  0.010  0.088
7         0.391  0.250  0.012  0.141
8         0.441  0.235  0.012  0.206
10        0.579  0.211  0.014  0.368
15        0.917  0.159  0.009  0.758
20        0.996  0.139  0.006  0.857

• Train AP (the tickets it learned from)
• CV AP (tickets it did not see)

Train AP and CV AP of a decision tree for each max_depth from 1 to 20. The train line keeps rising; the CV line peaks at depth 6 and then falls.

Text description of the chart

The horizontal axis is the max depth, from 1 to 20. The vertical axis is the average precision, from 0 to 1. At depth 1, both lines start at 0.178. The train line (orange) rises at every depth: 0.346 at depth 6, 0.579 at depth 10, 0.917 at depth 15 and 0.996 at depth 20. The CV line (blue) rises to its highest point, 0.258, at depth 6. Then it falls: 0.211 at depth 10, 0.159 at depth 15 and 0.139 at depth 20, close to the floor of 0.117.

Read the chart from left to right:

• Depth 1 to 3, underfitting. Both lines are low and close together. The tree is too simple. Each extra question still helps on new tickets.
• Depth 6, the best CV AP: 0.258 ± 0.010. Depth 5 (0.255) and depth 7 (0.250) are within one std of it: they are as good.
• Depth 8 and deeper, overfitting. The train line keeps rising, but the CV line falls. The gap grows from 0.088 at depth 6 to 0.857 at depth 20. Each new level of questions learns more noise than pattern.

The best depth comes from CV on train only. The validation month stays free, to confirm the final choice later. And the best tree, 0.258, is still far below logistic regression (0.327 ± 0.012).

Explore the depth

The explorer has the same numbers, plus the number of leaves at each depth. Move the slider, and watch the gap tile and the status line.

Explore

Train AP, CV AP and the gap, by depth

Max depth

3

Train AP

0.244

CV AP (± std)

0.234 ± 0.011

Gap: train − CV

0.010

Leaves

8

Too simple (underfitting): train AP and CV AP are both low, and the gap is only 0.010. The tree cannot follow the pattern yet.

• Train AP (the tickets it learned from)
• CV AP (tickets it did not see)

Precomputed from the real train set: a decision tree on the 10 features from the SQL course for each max_depth from 1 to 20, scored on train and by 5-fold stratified CV.

Text description of the explorer

The explorer has one slider, the max depth, from 1 to 20. It starts at depth 3. The readouts show the max depth, the train AP, the CV AP with its standard deviation, the gap (train AP − CV AP, highlighted) and the number of leaves. A chart shows the train line and the CV line for all depths. At the chosen depth, it adds a dashed line, two dots and a shaded bar for the gap.

The values: depth 1: train 0.178, CV 0.178 ± 0.008, gap 0.000, 2 leaves. Depth 2: 0.217, 0.215, gap 0.002, 4 leaves. Depth 3: 0.244, 0.234, gap 0.010, 8 leaves. Depth 4: 0.275, 0.247, gap 0.028, 16 leaves. Depth 5: 0.308, 0.255, gap 0.053, 31 leaves. Depth 6: 0.346, 0.258 ± 0.010, gap 0.088, 59 leaves. Depth 7: 0.391, 0.250, gap 0.141, 104 leaves. Depth 8: 0.441, 0.235, gap 0.206, 176 leaves. Depth 10: 0.579, 0.211, gap 0.368, 416 leaves. Depth 12: 0.735, 0.185, gap 0.550, 817 leaves. Depth 15: 0.917, 0.159, gap 0.758, 1,507 leaves. Depth 20: 0.996, 0.139, gap 0.857, 2,093 leaves.

The status line says “too simple” for depths 1 to 4 and “the best CV AP” at depth 6. It says “close to the best” at depths 5 and 7 (within one standard deviation), and “too complex” from depth 8.

From depth 6 to depth 7, the CV AP falls by 0.008 while the gap grows by 0.053. From then on, every extra level adds more gap than it adds fit. At depth 10, the tree has 416 leaves for 16,939 tickets; at depth 20, it has 2,093.

Guided practice: choose the depth with CV

Do this in the notebook. Each step has a check.

1. Run the sweep cell. You should see the table above: 0.346 0.258 0.010 0.088 at depth 6.
2. Run the chart cell. The CV line must peak at depth 6.
3. Set best_depth to the depth with the highest CV AP. Use int(sweep["CV AP"].idxmax()): idxmax() gives the index of the largest value.
4. Set gap_at_best to the gap at that depth: float(sweep.loc[best_depth, "gap"]).
5. Run the check cell. You should see Check passed: best_depth is 6. and Check passed: gap_at_best is 0.088.

Other limits: leaf size and regularization

max_depth is one way to limit a tree. min_samples_leaf is another: the smallest number of train tickets that a leaf may have. A rule from one ticket, like “96 to 137 words”, is then not allowed.

Logistic regression has a limit too. Regularization is a penalty on large weights, which keeps a model simpler. The model must “pay” for each large weight, so it uses large weights only where the data clearly asks for them. In scikit-learn, the hyperparameter C sets it: a smaller C means a stronger penalty, and the default is C = 1.0. With a very strong penalty, C = 0.0001, logistic regression underfits:

strong_penalty = evaluate(build_pipeline(LogisticRegression(C=0.0001, max_iter=1000)))
print("logistic regression, C = 1 (default):", evaluate(build_pipeline()))
print("logistic regression, C = 0.0001:     ", strong_penalty)

Output

logistic regression, C = 1 (default): {'train AP': 0.328, 'CV AP': 0.327, 'std': 0.012}
logistic regression, C = 0.0001:      {'train AP': 0.301, 'CV AP': 0.302, 'std': 0.012}

Both scores fall by about 0.025, and the gap stays near zero. At the default C = 1, logistic regression has almost no gap: it does not overfit, so a stronger penalty can only take something away. Tune hyperparameters tries several values of C in Module 6.

Learning curves

Would more data help? A learning curve answers that question. It shows the train score and the CV score as the model gets more train tickets. scikit-learn’s learning_curve fits the model on a part of each fold’s train tickets, from 500 to 13,551. That is four fifths of train, the most that a fold allows:

sizes = [500, 1000, 2000, 4000, 8000, 13551]
curves = {}
for name, pipeline in {"logistic regression": build_pipeline(),
                       "tree, depth 6": tree_pipeline(DecisionTreeClassifier(max_depth=6, random_state=0)),
                       "tree, no limit": tree_pipeline(DecisionTreeClassifier(random_state=0))}.items():
    n, train_scores, cv_scores = learning_curve(pipeline, train[FEATURES], train[TARGET], cv=cv, train_sizes=sizes,
                                                scoring="average_precision", shuffle=True, random_state=0)
    curves[name] = (train_scores.mean(axis=1), cv_scores.mean(axis=1))
    print(f"{name:20s} train {train_scores.mean(axis=1).round(3)}")
    print(f"{'':20s} CV    {cv_scores.mean(axis=1).round(3)}")

Output

logistic regression  train [0.408 0.386 0.371 0.349 0.337 0.329]
                     CV    [0.279 0.304 0.317 0.323 0.325 0.327]
tree, depth 6        train [0.681 0.669 0.545 0.459 0.396 0.355]
                     CV    [0.157 0.171 0.21  0.221 0.242 0.258]
tree, no limit       train [1. 1. 1. 1. 1. 1.]
                     CV    [0.139 0.135 0.138 0.136 0.141 0.137]

• Logistic regression
• Tree, depth 6
• Tree, no limit
• Dashed: train AP. Solid with dots: CV AP.

Learning curves of three models, from 500 to 13,551 train tickets. Dashed lines are the train AP, solid lines with dots the CV AP.

Text description of the chart

The horizontal axis is the number of train tickets used to fit, from 500 to 13,551. The vertical axis is the average precision, from 0 to 1. Logistic regression (blue): the train line falls from 0.408 to 0.329, and the CV line rises from 0.279 to 0.327. At 4,000 tickets they are 0.349 and 0.323, and at 13,551 they almost meet. The depth-6 tree (orange): the train line falls from 0.681 to 0.355, and the CV line rises from 0.157 to 0.258, still rising at the end. The tree with no limit (green): the train line stays at 1.0 at every size, and the CV line stays between 0.135 and 0.141.

Each model tells a different story:

• Logistic regression has learned what it can. The two lines almost meet, and the CV line is flat: from 4,000 to 13,551 tickets, it rises only from 0.323 to 0.327. Twice as many tickets of the same kind would help little. To do better, the model needs better features: Select useful features tests them in Module 6.
• The depth-6 tree still improves. Its CV line rises at every step, and the gap closes. More tickets would help this tree, but it has a long way to go to reach 0.327.
• The tree with no limit does not improve at all. It memorises any number of tickets: train 1.000 at every size, CV about 0.137. More data does not cure a model that learns noise.

A learning curve with a large gap says “overfitting: limit the model, or get more data”. Two lines that meet low say “underfitting: a more flexible model or better features”.

Your turn: limit the leaves instead of the depth

Change one condition. Keep no depth limit, and limit the leaf size instead. Try min_samples_leaf = 1, 5, 20, 50, 100, 200 and 500. Before you run it, predict: which value gives the best CV AP? Will it beat the best depth (0.258)?

leaf_rows = []
for leaf in [1, 5, 20, 50, 100, 200, 500]:
    result = evaluate(tree_pipeline(DecisionTreeClassifier(min_samples_leaf=leaf, random_state=0)))
    leaf_rows.append({"min_samples_leaf": leaf, **result})
leaf_sweep = pd.DataFrame(leaf_rows).set_index("min_samples_leaf")
print(leaf_sweep)

Show the answer

Output

                  train AP  CV AP    std
min_samples_leaf
1                    1.000  0.137  0.004
5                    0.742  0.180  0.006
20                   0.466  0.228  0.016
50                   0.373  0.265  0.017
100                  0.328  0.278  0.017
200                  0.302  0.266  0.015
500                  0.269  0.255  0.012

The best value is min_samples_leaf=100: CV AP 0.278 ± 0.017, with a gap of only 0.050. The tree then has 124 leaves. The same pattern as the depth: too small a leaf overfits (1 and 5), too large a leaf underfits (500). The best leaf size beats the best depth by 0.020, about one std, so it is a little better at most. Both trees stay below logistic regression (0.327 ± 0.012). Limits make a tree honest; they do not make it the right shape for an additive pattern.

Update the experiment log

Two experiments of this lesson go in the log: the best tree so far (min_samples_leaf=100, from your turn) and the strong penalty for logistic regression. The depth sweep found depth 6, which is already in the log as E07. Neither new experiment beats E03, so both are rejected:

log_experiment("E09", "Decision tree, no depth limit, min_samples_leaf=100", leaf_sweep.loc[100, "CV AP"],
               leaf_sweep.loc[100, "std"], None, "reject", "The best tree so far, still far below E03.")
log_experiment("E10", "E03 with C=0.0001 (a very strong penalty)", strong_penalty["CV AP"], strong_penalty["std"],
               None, "reject", "It underfits: both train and CV AP fall.")
show_log(columns=("id", "change", "cv_ap", "cv_std", "decision"))

The last rows of the output:

Output

 id                                                change  cv_ap  cv_std decision
...
E08                         Decision tree, no depth limit  0.137   0.004   reject
E09   Decision tree, no depth limit, min_samples_leaf=100  0.278   0.017   reject
E10             E03 with C=0.0001 (a very strong penalty)  0.302   0.012   reject

Ten experiments, and logistic regression with its default settings is still the best. That is a normal result of the loop: most experiments are rejected. Each rejected row still taught something. The trees showed that this data has few interactions, and the penalty showed that logistic regression does not overfit.

No gap, so no problem?

Common mistake

No gap, so no problem?

Symptom: Tomás has learned that a gap means overfitting. He wants a model with no gap at all, and finds one:

print("tree, depth 1:", evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=1, random_state=0))))
print("tree, depth 6:", evaluate(tree_pipeline(DecisionTreeClassifier(max_depth=6, random_state=0))))

Output

tree, depth 1: {'train AP': 0.178, 'CV AP': 0.178, 'std': 0.008}
tree, depth 6: {'train AP': 0.346, 'CV AP': 0.258, 'std': 0.01}

“Depth 1 has a gap of zero. It does not overfit, so it is the safest model.”

Cause: a zero gap means only that the model does not learn noise. The depth-1 tree asks one question, “priority = 1?”, and gives every ticket one of two scores. It underfits: its CV AP, 0.178, is only a little above the floor of 0.117, and far below the depth-6 tree (0.258) and logistic regression (0.327). It has high bias. Logistic regression with C = 0.0001 shows the same: no gap, and a CV AP of 0.302 instead of 0.327.

Fix: read two things, always together. The gap tells you about overfitting. The level of the CV score, next to the baseline and the best model, tells you about underfitting. Choose the complexity with the best CV score, and among models within one std of it, the simplest.

Consequence in AI work: a model chosen only to “not overfit” can be too weak to be useful. Grace’s senior team would get flags that are barely better than “flag every priority 1 ticket”, the rule that the model was supposed to beat.

See it end to endRank a call listCase study

The call-list case study shows a full tree that memorises the training data, and the depth that fixes it.

Knowledge check

Check your understanding

A model has train AP 0.40 and CV AP 0.39 at every train size from 2,000 to 13,551 tickets. Is it overfitting or underfitting? Will twice as many tickets of the same kind help? What would you try instead?

Show the answer

The lines have met and stay flat, so the model does not overfit: it has learned what it can from these features. More tickets of the same kind will help very little. To do better, give it more information (better features) or a more flexible model, and check the result with CV. If its level is also low compared with other models, it underfits.

Check your understanding

Predict: a tree with max_depth 25 on the same data. Will its train AP be higher or lower than at depth 20 (0.996)? Its CV AP? Explain with the words bias and variance.

Show the answer

The train AP goes up to 1.000, because more questions let the tree fit every train ticket. The CV AP stays low: on this data it is 0.138, close to the tree with no limit (0.137). A deeper tree has lower bias but much higher variance. Its leaves describe single tickets, so it changes a lot with the train tickets. It works worse on new ones.

Check your understanding

Priya says: 'Depth 6 has CV AP 0.258 and depth 5 has 0.255. Depth 6 is the better tree.' Is she right? Which would you choose, and why?

Show the answer

The difference, 0.003, is much smaller than the std (about 0.010), so it is not a real difference. Both depths are equally good on new tickets. Choose depth 5: it is simpler (31 leaves instead of 59), easier to read and a little more stable. The rule: among models within one std of the best, choose the simplest.

Recap

• Underfitting: both scores low and close (high bias). Overfitting: a large gap between the train score and the CV score (high variance).
• The tree with no limit has 2,210 pure leaves: train AP 1.000, CV AP 0.137. Its score for one ticket jumps between 0 and 1 with the train tickets.
• A hyperparameter is a setting that you choose before training: max_depth, min_samples_leaf, C. The model learns its parameters; you choose its hyperparameters, by CV on train. The best depth is 6 (0.258 ± 0.010), the best leaf size 100 (0.278 ± 0.017).
• Regularization keeps a model simpler. Logistic regression at C = 1 has almost no gap; C = 0.0001 underfits (0.302).
• A learning curve shows whether more data would help. For logistic regression, it would not: better features are the next step, in Module 6.
• Your log has ten rows. E03, logistic regression, is still the best.

Next, you combine many trees into one model, and see what that costs.

Next: Ensembles