Tune hyperparameters

What you will be able to do

Choose the settings of the model with evidence, not with a guess, and without spending the test set. Then keep the simplest setting that is as good as the best, because a “better” score smaller than the noise is not better.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. It also runs the random search and the simulation of tuning on the test set. Its setup cells install pandas and scikit-learn, and download the practice data and ticket_model.py, so it works without the local project. Each search takes about 15 seconds.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 65 minutes

What you need to know

• The three sets and their jobs, from Train, validation and test sets: Three sets, three jobs, and cross-validation, from the same lesson.
• Average precision, from The precision–recall trade-off: Average precision.
• Underfitting, overfitting and the setting C of logistic regression, from Control complexity.
• Why a more complex model must earn its place, from Ensembles: The cost of complexity.
• The standard error as the typical size of a chance error, from Estimate uncertainty in Essential Mathematics and Statistics for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its active .venv and the data in data/, from Define a baseline: Set up your project. If a data file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 11 features, from Select useful features: Change ticket_model.py. To recover it: curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py (curl.exe on Windows).
• experiment_log.csv with the rows E01 to E18, from Select useful features: Write it in the experiment log. This lesson adds E19 to E21. If it is missing, the notebook’s setup cells write the rows again.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows commands are not tested. The notebook was tested in Jupyter.

The oven does not learn

You bake a cake from a new recipe. The recipe says which ingredients to mix, and the mix becomes the cake. But two numbers are not in the bowl: the oven temperature and the baking time. The cake does not choose them, and it does not learn them while it bakes. You set them before you start.

The packet says “180 °C for 40 minutes”. That is a sensible start for a typical oven, not a promise for yours. If your oven runs hot, the cake burns at the edges. If you take it out too early, the middle is raw. So a careful baker tries a few settings, tastes each cake, and keeps the setting that works. She also writes it down, so she does not have to guess next time.

A machine-learning model has both kinds of number too. You met them in Control complexity, where you chose one setting, the depth of a tree, with cross-validation. This lesson explains the idea in full, and then chooses several settings at once, honestly.

Parameters and hyperparameters

When you call fit(), the model learns numbers from the train tickets. These learned numbers are the parameters of the model. For logistic regression, they are the coefficients (one weight for each prepared column) and the intercept. For a decision tree, they are the questions at each split and the values in the leaves. Nobody types them in. The data decides them.

Other numbers must be fixed before fit() starts. These are the hyperparameters. The prefix “hyper” means “above”: they sit above the parameters and control how the parameters are learned. C of logistic regression is one: it sets how strongly the fit keeps the coefficients small, as you saw in Control complexity. The maximum depth of a tree is another, and so are the number of trees in a forest and the learning rate of gradient boosting.

You can see both kinds in the final model. As in every notebook of this course, pd is pandas and np is NumPy; the notebook’s setup cells import them.

from ticket_model import FEATURES, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
model = pipeline.named_steps["model"]
print("parameters learned by fit():", model.coef_.size, "coefficients and 1 intercept")
print("hyperparameters chosen before:", {name: model.get_params()[name] for name in ("C", "max_iter", "class_weight")})

The output:

Output

parameters learned by fit(): 28 coefficients and 1 intercept
hyperparameters chosen before: {'C': 1.0, 'max_iter': 1000, 'class_weight': None}

The 11 features become 28 columns after preparation. Each category gets its own column, and the imputer adds a column for each number with missing values. So the fit learned 28 coefficients. For example, team_account got −1.02 and prior_escalations_90d got +0.43. The hyperparameters did not come from the data at all. C = 1.0 is scikit-learn’s default, and max_iter = 1000 is the value that ticket_model.py sets.

You choose the hyperparameters before the fit. The fit learns the parameters from the train tickets. Tuning means choosing the hyperparameters with data too.

Text description of the diagram

A flow from left to right (top to bottom on a phone). Two boxes feed into a box named fit. The first is Hyperparameters: “you choose them before the fit: C = 1, max_iter = 1000”. The second is the data file train.csv, 16,939 tickets. From fit, an arrow goes to Parameters: “the fit learns them from the data: 28 coefficients and 1 intercept”. From there, an arrow goes to Scores for new tickets.

Model	Parameters (learned by fit())	Some hyperparameters (chosen before)
Logistic regression	One coefficient per prepared column, and the intercept	C, class_weight
Decision tree	The question at each split, the value in each leaf	max_depth, min_samples_leaf
Random forest	All the splits of all the trees	n_estimators (number of trees), max_depth, min_samples_leaf
Gradient boosting	All the splits of all the trees, and their weights	learning_rate, max_depth, max_iter (number of trees)

Why the defaults are only a guess

Every hyperparameter in scikit-learn has a default value. The library’s authors chose each one to work reasonably on many datasets. But your data is one dataset, and the default knows nothing about it.

You have already seen what a default can do. DecisionTreeClassifier has no depth limit by default. In Control complexity, that tree memorised the train tickets: train AP 1.000, CV AP 0.137. A tree limited to depth 6 reached CV AP 0.258. So for trees, the default was the worst setting in the sweep. For gradient boosting, the default scored CV AP 0.316, and depth 3 with a learning rate of 0.05 scored 0.327. A default can be fine, or it can be far from the best. You cannot know until you measure.

The reason is the trade-off from Module 5. A setting that makes the model too simple underfits: it misses real patterns, like a cake taken out too early. A setting that makes it too complex overfits: it learns the noise of the train tickets, like a cake left in too long. The best setting is somewhere between, and where it lies depends on the data. Tuning means finding it with evidence: try several values, measure each one fairly, and choose.

Omar asks: “Can an expert not just tell us the right value?” An expert can tell you a sensible range to try. The right value inside that range depends on these tickets, so only a measurement can find it.

How to compare settings honestly

To choose between settings, you need a score for each one. Which data gives that score? Think of the three sets from Train, validation and test sets.

• Not the train score. A model that memorises wins on the tickets it learned from. If you choose by train score, you choose the tree with no limit (train AP 1.000), the worst one on new tickets.
• Not the test set. The test set is the sealed final exam. If it helps to choose, its score stops being an honest estimate. The failure case at the end of this lesson shows by how much.
• Cross-validation on train. CV cuts the train tickets into 5 folds, fits on 4, scores the fifth, and repeats 5 times. Every score comes from tickets that the fitted model did not see, and you get 5 scores, so you also see the spread. You learned it in Cross-validation.

So tuning has five steps:

1. Choose a few values for each hyperparameter that you want to tune.
2. Score each setting with the same CV on train, with the same metric.
3. Choose a setting, with a rule that respects the noise (the one-standard-error rule, below).
4. Check the chosen setting once on the validation set.
5. Write the result in the experiment log.

The course always uses StratifiedKFold(5, shuffle=True, random_state=0). “Stratified” means that each fold gets the same share of escalations, about 12%, so every fold score measures the same task. Here is the final pipeline under that CV:

from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(5, shuffle=True, random_state=0)
fold_scores = cross_val_score(build_pipeline(), train[FEATURES], train[TARGET], cv=cv, scoring="average_precision")
print("AP per fold:", fold_scores.round(3))
print(f"mean {fold_scores.mean():.4f}, std {fold_scores.std():.4f}, standard error {fold_scores.std() / 5 ** 0.5:.4f}")

Output

AP per fold: [0.325 0.321 0.347 0.364 0.34 ]
mean 0.3393, std 0.0156, standard error 0.0070

The same pipeline scores from 0.321 to 0.364, only because the folds hold different tickets. The mean of the 5 scores is more stable than any one of them. How stable? The standard error of the mean is std ÷ √5 = 0.0156 ÷ 2.24 = 0.0070. It is the typical size of the chance error in the mean, as in the mathematics course. Keep this number in mind: two settings whose means differ by less than about 0.007 cannot be told apart.

Why the validation month still matters

Step 4 checks the chosen setting on May, the validation set. Why, if CV already scored it? Because all 22 train months come from before the new warranty policy of 2026-05-01, and May comes after it. Priya compares the model’s mean score, its average guess of the escalation rate, with the real rate:

from sklearn.model_selection import cross_val_predict

oof = cross_val_predict(build_pipeline(), train[FEATURES], train[TARGET], cv=cv, method="predict_proba")[:, 1]
scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
print(f"train, out of fold: mean score {oof.mean():.3f}, escalation rate {train[TARGET].mean():.3f}")
print(f"valid (May 2026):   mean score {scores.mean():.3f}, escalation rate {valid[TARGET].mean():.3f}")
monthly = train.groupby(train["created_at"].dt.strftime("%Y-%m"))[TARGET].mean()
print(f"train months: escalation rate from {monthly.min():.3f} to {monthly.max():.3f}")

Output

train, out of fold: mean score 0.117, escalation rate 0.117
valid (May 2026):   mean score 0.120, escalation rate 0.155
train months: escalation rate from 0.095 to 0.139

cross_val_predict gives each train ticket an out-of-fold score: the score from the fold model that did not see it. On train, the mean score equals the rate. In May, 15.5% of tickets escalate, more than in any train month, and the model still guesses 12.0%. CV cannot show this, because every fold is from before the change. So the two sets have two jobs. CV on train chooses: it has 1,983 escalations, so its averages are stable. The validation month checks the choice in the new conditions: with only 183 escalations, it is too small to rank ten settings. The next lesson uses May for one more job, the threshold.

The diagram shows the whole workflow. This lesson does the first two steps. The next lesson does the threshold and the single test.

Each set has one job. CV on train chooses the settings (this lesson), the validation month sets the threshold, and the test set gives one final report (the next lesson). Nothing goes back from the report.

Text description of the diagram

The diagram is a flow from left to right (top to bottom on a phone). train.csv (16,939 tickets, July 2024 to April 2026) goes to 5-fold CV on train, with GridSearchCV and 10 settings. CV leads to Choose settings, where the one-standard-error rule picks logistic regression with C = 1. That leads to Threshold on valid, which also takes valid.csv (1,179 tickets, May 2026): for a capacity of 20%, the threshold is 0.19. The threshold and test.csv (1,002 tickets, June 2026) go to Test once, with evaluate.py. The last box is the Report: average precision 0.345, recall 0.446, and no changes after this. On the wide diagram, a dashed arrow from the report back to Choose settings ends in a cross, with the label “never back”.

Grid search and random search

Back in the kitchen. You want to try three temperatures (160, 180 and 200 °C) and three baking times (30, 40 and 50 minutes). If you bake one cake for every combination, you bake 3 × 3 = 9 cakes. That plan is a grid: every value of one setting with every value of the other. It is easy to plan and easy to read.

A grid grows fast. With a third setting, such as the shelf height, at three values, you bake 27 cakes. With five values for each of three settings, you bake 125. In ML, each “cake” is 5 fits of CV, so the cost of a grid multiplies with every hyperparameter that you add.

There is a second problem. Often only one or two settings matter much, and the others hardly change the result. Suppose that the temperature matters and the shelf height does not. A grid of 9 cakes tries only 3 different temperatures, each three times. A random search chooses 9 combinations at random from the ranges that you give. Then all 9 cakes have different temperatures, so you learn much more about the setting that matters, for the same cost. Bergstra and Bengio showed this in 2012 (“Random Search for Hyper-Parameter Optimization”, Journal of Machine Learning Research 13).

Nine settings each. The grid tries only three values of the setting that matters; the random search tries nine, so it gets closer to the best value for the same cost.

Text description of the diagram

There are two panels: “Grid: 9 settings” and “Random: 9 settings” (stacked on a phone). In each panel, a square shows the settings tried: the horizontal direction is the setting that matters, and the vertical direction is a setting that matters little. Under each square, short marks show the values of the setting that matters. Above each square, a curve shows the score against the setting that matters. It is flat and low, with one peak a little right of the middle.

In the grid panel, the nine dots form a 3 by 3 pattern, so they cover only three values of the setting that matters. The best of them, marked in amber, sits on the slope below the peak. In the random panel, the nine dots are scattered, so they cover nine different values. The best of them, in amber, sits near the top of the peak.

So which one should you use? Use a grid when you tune one or two hyperparameters with a few values each. Then you see every combination, for example all four values of C. Use a random search when there are three or more hyperparameters, or when a value can be anywhere in a range. A learning rate between 0.01 and 0.3 is an example. You fix the budget (the number of settings) in advance, so the cost stays under control.

Worked example: tune with GridSearchCV

GridSearchCV runs the same CV for every setting in a grid, and ranks them. Priya’s grid has 4 logistic regressions and 6 gradient-boosting models: 10 settings × 5 folds = 50 fits. Before you read on, predict: which setting will rank first?

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

grid = [
    {"model": [LogisticRegression(max_iter=1000)], "model__C": [0.01, 0.1, 1, 10]},
    {"model": [HistGradientBoostingClassifier(random_state=0)],
     "model__max_depth": [2, 3, None], "model__learning_rate": [0.05, 0.1]},
]
search = GridSearchCV(build_pipeline(), grid, cv=cv, scoring="average_precision", n_jobs=-1)
search.fit(train[FEATURES], train[TARGET])
print("settings:", len(search.cv_results_["params"]))

Output

settings: 10

Each part of the call has a job:

Lines	Why
"model": [...]	The pipeline’s last step is named model in ticket_model.py. The grid replaces it with each model in the list.
"model__C": [...]	Two underscores reach inside a step: the C of the step model.
A list of two dictionaries	Two grids: C only for logistic regression, depth and learning rate only for boosting.
cv=cv, scoring="average_precision"	The same folds and the same metric for every setting, so the comparison is fair.
n_jobs=-1	Use every processor core. It takes about 15 seconds.

The pipeline keeps the scaler for the boosting models. Trees split on the order of the values, so scaling changes nothing for them.

A short helper names each setting, and the results become a table, best first:

def describe(params):
    """A short name for one setting of the grid."""
    if isinstance(params["model"], LogisticRegression):
        return f"logistic, C={params['model__C']}"
    return f"boosting, rate {params['model__learning_rate']}, depth {params['model__max_depth']}"

results = pd.DataFrame(search.cv_results_)
results["setting"] = results["params"].map(describe)
results["se"] = results["std_test_score"] / 5 ** 0.5
table = results[["setting", "mean_test_score", "std_test_score", "se", "rank_test_score"]].sort_values("rank_test_score")
print(table.round(4).to_string(index=False))

Output

                        setting  mean_test_score  std_test_score     se  rank_test_score
    boosting, rate 0.1, depth 2           0.3424          0.0178 0.0080                1
                 logistic, C=10           0.3394          0.0157 0.0070                2
                  logistic, C=1           0.3393          0.0156 0.0070                3
                logistic, C=0.1           0.3391          0.0154 0.0069                4
   boosting, rate 0.05, depth 2           0.3379          0.0179 0.0080                5
               logistic, C=0.01           0.3375          0.0168 0.0075                6
    boosting, rate 0.1, depth 3           0.3367          0.0153 0.0069                7
   boosting, rate 0.05, depth 3           0.3362          0.0147 0.0066                8
boosting, rate 0.05, depth None           0.3212          0.0138 0.0062                9
 boosting, rate 0.1, depth None           0.3188          0.0101 0.0045               10

Tomás points at the first row: “Boosting wins. Ship it.” Priya asks a different question: is 0.3424 really better than 0.3393?

The one-standard-error rule

Back in the kitchen once more. Five friends taste two cakes and give marks out of 10. Cake A gets 8.4 on average and cake B gets 8.3. But the same cake gets marks that vary by half a point from one friend to the next. A difference of 0.1 is much smaller than that variation, so the friends cannot really tell the cakes apart. If cake B is much easier to make, a sensible baker makes cake B.

The one-standard-error rule turns that thought into arithmetic:

1. Take the best mean: 0.3424.
2. Subtract its standard error: 0.3424 − 0.0080 = 0.3344.
3. Every setting at or above 0.3344 ties with the best.
4. From the tied settings, choose the simplest one.

The chart shows it. Each bar is a setting’s mean ± one standard error, and the dashed line is the limit.

The ten settings of the grid, best first. Eight means are at or above the limit (the best mean minus its standard error), so they tie with the best. Only the two boosting settings without a depth limit are clearly worse.

Text description of the chart

Ten rows on an axis from 0.31 to 0.355, with a dashed vertical line at the limit, 0.3344. Each row is a dot at the mean with a bar of ± one standard error. From the top: boosting with rate 0.1 and depth 2, the best, 0.3424 (0.3344 to 0.3504), in orange. Logistic regression with C = 10: 0.3394. Logistic regression with C = 1, the chosen one, 0.3393 (0.3323 to 0.3463), in blue. Logistic with C = 0.1: 0.3391. Boosting with rate 0.05 and depth 2: 0.3379. Logistic with C = 0.01: 0.3375. Boosting with rate 0.1 and depth 3: 0.3367. Boosting with rate 0.05 and depth 3: 0.3362. These eight dots are all right of the dashed line. The last two rows, boosting with no depth limit at rates 0.05 and 0.1, are at 0.3212 and 0.3188, well left of the line.

Eight of the ten settings are at or above 0.3344. Only the two boosting models without a depth limit are clearly worse. Logistic regression with C = 1 is 0.0031 below the best: less than half a standard error.

Priya’s decision: logistic regression with the default C = 1.0. It ties with the best, and it is the simplest model in the tie. Its folds vary less (std 0.0156 against 0.0178). Its coefficients can be explained to Grace (Module 7). And it is smaller, faster and easier to check, which is the cost of complexity from Module 5 in reverse. The four values of C are all within 0.002 of each other, so on this data tuning C does not matter. Say that honestly in a report, instead of reporting a “tuned C”.

Then step 4: check the choice once on the validation month, next to the setting that ranked first.

from sklearn.metrics import average_precision_score

best_boosting = build_pipeline(HistGradientBoostingClassifier(learning_rate=0.1, max_depth=2, random_state=0))
best_boosting.fit(train[FEATURES], train[TARGET])
for name, fitted in [("boosting, rate 0.1, depth 2", best_boosting), ("logistic, C=1", pipeline)]:
    print(f"{name:28s} valid AP {average_precision_score(valid[TARGET], fitted.predict_proba(valid[FEATURES])[:, 1]):.3f}")

Output

boosting, rate 0.1, depth 2  valid AP 0.395
logistic, C=1                valid AP 0.391

The difference on May is 0.004. May’s AP moves by about 0.03 by chance, so May agrees: the two settings are a tie. This is the same pipeline as before the search. The search did not find a better model. It showed, with evidence, that no setting is clearly better, and that is a useful result.

Tomás is not convinced: “Then why not keep boosting? It was first.” Because “first” by less than the noise is not “better”. The simpler model gives the same results, and it is cheaper in every other way. A later lesson, When a simple model is better, comes back to this choice.

Random search in code

RandomizedSearchCV works like GridSearchCV, but you give it a range or a list for each hyperparameter and a budget, n_iter. Priya gives boosting three hyperparameters. A grid with 5 values of each would need 125 settings. She tries 10 random ones:

from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV

space = {"model__learning_rate": loguniform(0.01, 0.3), "model__max_depth": [2, 3, 4, 6, None],
         "model__min_samples_leaf": [20, 50, 100, 200]}
random_search = RandomizedSearchCV(build_pipeline(HistGradientBoostingClassifier(random_state=0)), space,
                                   n_iter=10, cv=cv, scoring="average_precision", random_state=0, n_jobs=-1)
random_search.fit(train[FEATURES], train[TARGET])
found = random_search.cv_results_
for i in found["rank_test_score"].argsort():
    p = found["params"][i]
    print(f"rate {p['model__learning_rate']:.3f}  depth {p['model__max_depth']!s:4}  min leaf {p['model__min_samples_leaf']:3}"
          f"   CV AP {found['mean_test_score'][i]:.4f}  se {found['std_test_score'][i] / 5 ** 0.5:.4f}")

loguniform(0.01, 0.3) draws learning rates between 0.01 and 0.3, as often near 0.01 as near 0.1. random_state=0 makes the 10 draws the same on every run. Predict first: will the best random setting beat logistic regression by more than one standard error?

Output

rate 0.208  depth 2     min leaf  20   CV AP 0.3441  se 0.0077
rate 0.233  depth 2     min leaf  50   CV AP 0.3437  se 0.0086
rate 0.259  depth 2     min leaf 100   CV AP 0.3418  se 0.0083
rate 0.091  depth 2     min leaf 200   CV AP 0.3393  se 0.0087
rate 0.065  depth 2     min leaf 200   CV AP 0.3385  se 0.0080
rate 0.185  depth 3     min leaf 200   CV AP 0.3366  se 0.0082
rate 0.050  depth 3     min leaf 200   CV AP 0.3343  se 0.0076
rate 0.037  depth 2     min leaf  50   CV AP 0.3330  se 0.0079
rate 0.090  depth None  min leaf 200   CV AP 0.3307  se 0.0062
rate 0.059  depth 6     min leaf 100   CV AP 0.3302  se 0.0061

With the same budget of 10 settings, the random search found a slightly better boosting setting than the grid: 0.3441 against 0.3424. It tried 10 different learning rates, and the best ones are higher than the grid’s two values. The top settings all have depth 2, so the depth matters and a small depth is best. But apply the rule: 0.3441 − 0.0077 = 0.3364, and logistic regression with C = 1 (0.3393) is above that limit. It is still a tie, and the decision does not change.

Write it in the experiment log

Priya adds three rows: the best setting of the grid (E19), the best value of C (E20) and the best random setting (E21). All three are rejects, because each one ties with E15. A search that changes nothing still earns its rows. It is the evidence that the simple model is a measured choice, not a lazy one.

best = table.iloc[0]
c_10 = table.set_index("setting").loc["logistic, C=10"]
top = found["rank_test_score"].argmin()
log_experiment("E19", "Grid search best: gradient boosting, learning_rate=0.1, max_depth=2",
               best["mean_test_score"], best["std_test_score"], 0.395, "reject",
               "Within one standard error (0.008) of E15, and more complex.")
log_experiment("E20", "E15 with C=10 (the best C; C=0.01 to 10 are within 0.002)",
               c_10["mean_test_score"], c_10["std_test_score"], None, "reject",
               "No real difference from C=1, so keep the default.")
log_experiment("E21", "Random search best: gradient boosting, learning_rate=0.208, max_depth=2, min_samples_leaf=20",
               found["mean_test_score"][top], found["std_test_score"][top], None, "reject",
               "Less than one standard error (0.0077) above E15.")
show_log()

The last rows of the output:

Output

E19                          Grid search best: gradient boosting, learning_rate=0.1, max_depth=2  0.342   0.018    0.395   reject
E20                                    E15 with C=10 (the best C; C=0.01 to 10 are within 0.002)  0.339   0.016            reject
E21 Random search best: gradient boosting, learning_rate=0.208, max_depth=2, min_samples_leaf=20  0.344   0.017            reject

E19 has valid_ap because Priya checked it once on May to confirm the tie. E15 stays the model to keep.

Guided practice: run the search and apply the rule

Use the notebook, sections 4 and 5.

1. Run the setup cells and sections 1 to 3. You should see mean 0.3393, std 0.0156, standard error 0.0070 in section 2.
2. Predict which setting will rank first, and write it down.
3. Run the grid search in section 4. You should see settings: 10 after about 15 seconds.
4. Run the table cell. The first row should be boosting, rate 0.1, depth 2 with 0.3424.
5. In section 5, set limit to the best mean minus its standard error.
6. Set tied to the list of settings in table with a mean at or above limit.
7. Run the check cell. It prints Check passed when the list is right.
8. Pick the simplest setting in tied, and write one sentence on why.

Show the answer

best = table.iloc[0]
limit = best["mean_test_score"] - best["se"]
tied = table.loc[table["mean_test_score"] >= limit, "setting"].tolist()
print(round(limit, 4), tied)

Output

0.3344 ['boosting, rate 0.1, depth 2', 'logistic, C=10', 'logistic, C=1', 'logistic, C=0.1', 'boosting, rate 0.05, depth 2', 'logistic, C=0.01', 'boosting, rate 0.1, depth 3', 'boosting, rate 0.05, depth 3']

Eight settings tie. The simplest is logistic regression. Among its four values of C, keep the default, C = 1: the four are within 0.002, and the default needs no explanation. A good sentence: “Logistic regression with C = 1 is 0.0031 below the best setting, less than one standard error (0.0080). The difference is noise, and the simpler model is easier to explain and to check.”

Your turn: rank by ROC AUC

Change one thing: rank the same grid by ROC AUC instead of average precision. In section 9 of the notebook, set scoring_name = "roc_auc" and run the cell. Before you run it, predict: does the decision change? Is logistic regression with C = 1 still within one standard error of the best?

Show the answer

Output

                    setting  mean_test_score     se  rank_test_score
boosting, rate 0.1, depth 2           0.7719 0.0026                1
boosting, rate 0.1, depth 3           0.7699 0.0018                2
              logistic, C=1           0.7696 0.0013                3
            logistic, C=0.1           0.7696 0.0013                4
True

The same boosting setting is first (0.7719, standard error 0.0026). Logistic regression with C = 1 is 0.0023 lower: less than one standard error. The decision does not change. ROC AUC has smaller standard errors here, but the gaps between the settings are just as small. A different metric can change the order of a table. So choose the metric before the search, as Module 3 taught, never after you see which metric makes your favourite win.

Tuning on the test set

Common mistake

Tuning on the test set

Symptom: Tomás says: “We have a test set. Let us try all 10 settings on it and keep the best. It is only one more number.” The best test score is higher than any honest estimate, and he reports it as the model’s result.

Cause: when you choose a setting because it scored best on a set, part of its score is luck on that set. So the chosen score is too high. Priya shows it without touching the real test set. She cuts the 1,179 May tickets at random into two halves, 200 times. Half A plays the test set that Tomás tunes on; half B plays new tickets:

truth = valid[TARGET].to_numpy()
rng = np.random.default_rng(0)
reported, on_new, fixed = [], [], []
for draw in range(200):
    order = rng.permutation(len(truth))
    half_a, half_b = order[:589], order[589:]
    ap_a = {name: average_precision_score(truth[half_a], s[half_a]) for name, s in may_scores.items()}
    chosen = max(ap_a, key=ap_a.get)
    reported.append(ap_a[chosen])
    on_new.append(average_precision_score(truth[half_b], may_scores[chosen][half_b]))
    fixed.append(average_precision_score(truth[half_b], may_scores["logistic, C=1"][half_b]))
print(f"reported (best of 10 on half A): {np.mean(reported):.3f}")
print(f"the same setting on half B:      {np.mean(on_new):.3f}")
print(f"logistic, C=1 on half B:         {np.mean(fixed):.3f}")

Output

reported (best of 10 on half A): 0.411
the same setting on half B:      0.389
logistic, C=1 on half B:         0.399

may_scores holds the May scores of the 10 settings, each fitted on train (section 8 of the notebook). The best of 10 on half A looks 0.022 better than the same setting does on new tickets: about three CV standard errors. And the setting chosen this way does no better on new tickets than logistic regression with C = 1, chosen in advance (0.399).

Fix: make every choice before the test: settings with CV on train, the threshold on valid. Then look at the test set once, in the next lesson, and report what it says. If the result disappoints you, write that down, and plan new work with new data, not with the same test set.

Consequence in AI work: a test set that helped to choose becomes a second validation set. Its score is optimistic, and nobody has an honest estimate any more. Grace would plan the senior team’s work with that number, and the model would disappoint in production without ever getting worse.

See it end to endRank a call listCase study

The call-list case study uses the one-standard-error rule to choose the tree depth.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study tunes four model families and chooses between them with the same rule.

Knowledge check

Check your understanding

Sort these into parameters and hyperparameters: the coefficient of priority, max_depth of a tree, the learning rate of boosting, the question at the first split of a tree, C of logistic regression. Say how you decided.

Show the answer

Parameters: the coefficient of priority and the question at the first split of a tree. Hyperparameters: max_depth, the learning rate and C. The test is when the value is set. fit() learns a parameter from the train tickets. You choose a hyperparameter before fit() starts, and the fit does not change it. So you tune hyperparameters with CV, and the fit finds the parameters.

Check your understanding

Predict: Mei adds two more boosting settings to the grid. One of them ranks first with CV AP 0.3430, standard error 0.0081. What does the one-standard-error rule decide now, and why?

Show the answer

The limit is 0.3430 − 0.0081 = 0.3349. Logistic regression with C = 1 has 0.3393, above the limit, so it still ties with the best. The rule still chooses the simplest tied setting: logistic regression with C = 1. A new first place that wins by less than one standard error does not change the decision.

Check your understanding

Tomás wants to choose the depth of a tree by its score on the train tickets, because 'it uses the most data'. What goes wrong, and what do you use instead?

Show the answer

On the tickets that it learned from, a deeper tree always scores at least as well, so the train score always picks the deepest tree. The tree with no limit scores 1.000 on train and 0.137 in CV: it memorised the train tickets. The train score measures memory, not skill on new tickets. Use cross-validation on train instead: every fold score comes from tickets that the fitted tree did not see. Then check the chosen depth once on the validation set.

Recap

• Parameters are learned by fit() (28 coefficients and an intercept). Hyperparameters are chosen before it (C, depth, learning rate). Defaults are a guess that works on many datasets, not on yours.
• Score each setting with the same CV on train and the same metric; never with the train score, never with the test set. Then check the choice once on the validation month.
• A grid search tries every combination. A random search tries a fixed number of random ones, and often finds good values of the setting that matters with fewer fits.
• The one-standard-error rule: every setting within one standard error of the best ties with it; choose the simplest. Boosting led by 0.0031 (standard error 0.008), so the decision is logistic regression with C = 1.
• Choosing on a set inflates its score: 0.411 reported against 0.389 on new tickets in the simulation.

Next, you turn the scores into flags for the senior team and check if a score is a chance. Then you open the test set, once.

Next: Thresholds and calibration