Machine learning is iterative

What you will be able to do

Nobody can say in advance which model will work best on Larkfield’s tickets. Learn the loop that practitioners use to find out: change one thing, measure it in the same way as before, and keep what helps. Write every result in an experiment log.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install the packages, download the practice data, write ticket_model.py and define the helpers for the experiment log, so it works without the local project.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The three sets and why you choose with the train set only, from Train, validation and test sets: Three sets, three jobs.
• Cross-validation, its 5 folds and its standard deviation, from Train, validation and test sets: Cross-validation.
• Average precision (AP), from The precision–recall trade-off: Average precision.
• The baselines and the improvement target, from Define a baseline: The improvement target.
• Logistic regression and the pipeline in ticket_model.py, from Preprocess with a pipeline: Write ticket_model.py.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment, active in a terminal, and the lesson notebooks open in VS Code or JupyterLab. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• data/train.csv and data/valid.csv, from Define a baseline: Download the data. If a file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 6 numbers in NUMBERS, from Preprocess with a pipeline: Write ticket_model.py. If it is missing or broken, write it again from that section. Do not copy the finished file from C05/project yet: it has an extra feature that Module 6 adds, so your numbers would differ.
• This lesson starts one new file, experiment_log.csv. The notebook writes it in its own ticket-model folder.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

A cook with a new recipe

Think of a cook who wants to make the best bread in her town. She starts from a simple recipe and bakes one loaf. It is good, but a little dry. The next day she adds a little more water and changes nothing else. She tastes the bread, and she writes in her notebook: “more water: softer, better, keep”. The day after, she tries a hotter oven. The crust burns, so she writes “hotter oven: worse, go back”.

After two weeks, her notebook shows what helped, what did not, and why. Three habits make her method work:

• She changes one thing at a time. If she changed the water and the oven together, she could not say which change helped.
• She tastes in the same way every time: the same kind of bread, at the same time after baking.
• She writes everything down, also the changes that failed. Without the notes, she would try the hot oven again next month.

Machine learning works in the same way. This lesson shows why, and how to do it well.

Why you cannot know the best model in advance

Tomás, the AI engineer, asks Priya a direct question: “Which algorithm is the best? Let us use that one and finish.” It sounds like a fair question, but nobody can answer it for Larkfield’s tickets before trying.

Each algorithm assumes a certain shape of pattern. Logistic regression assumes that each fact adds or removes some risk, on its own. A decision tree, which you meet in the next lesson, assumes that a few questions in order give the answer, such as “priority 1, and a long message”. Which assumption fits better depends on the data. On one dataset, the trees win. On another one, logistic regression wins. Researchers proved a form of this: no algorithm is the best for every problem. They call it the “no free lunch” theorem: you cannot get a good result for free, without looking at your data.

So practitioners do not choose the model by reputation. They try several, measure each one in the same way, and keep what helps. This is why machine learning is called iterative: the work goes in rounds (iterations), and each round learns from the one before. What machine learning is showed the ML workflow as a loop, not a line. This module is the part of the loop where you try other models.

The loop

Here are the steps of one round. The first time, you start at the top. After that, you go around the loop again and again.

1. Baseline. Know the result that a simple rule gets without ML. Larkfield’s rule is “flag priority 1”.
2. A first model. Start simple and fast: logistic regression.
3. Evaluate. Score the model with the same cross-validation and the same metric every time.
4. Look and log. Look at the result next to the baseline and the best model so far. Write it in the log, with your decision.
5. Stop, or change one thing. If you go on, change one thing, and evaluate again.

The loop of model development. Steps 3, 4 and 5 repeat until a stop condition is true. The test set is used only after the loop.

Text description of the diagram

The diagram has five steps and one question. Step 1 is the baseline, the rule to beat. Step 2 is a first model, simple and fast. Step 3 evaluates it with the same cross-validation and the same metric. Step 4 looks at the result and writes it in the experiment log. Then comes the question “stop?”. The answer is yes if the target is met, if the gain is smaller than the noise, or if there is no more time. If the answer is no, step 5 changes one thing (the data, the features, the algorithm, the hyperparameters or the threshold), and the loop goes back to step 3. If the answer is yes, the work goes to the handover: the test set is used once, and then the model is documented.

What you can change

In step 5, there are five kinds of change. Each one has a place in this course:

• The data. Get more rows, fix wrong labels, or use more recent rows. At Larkfield, Module 8 plans to retrain the model with the May and June tickets, after the warranty policy changed.
• The features. Add a column that carries new information, or remove one that adds nothing. Module 6 tests four new columns. It finds one useful column and two that leak the answer.
• The algorithm. Try another family of models. The next lessons try decision trees, random forests and gradient boosting.
• The hyperparameters. These are the settings of an algorithm that you choose before training, such as the depth of a tree. Control complexity explains them, and Module 6 tunes them.
• The threshold. The score at which a ticket is flagged. Module 6 chooses it from Grace’s capacity of 20%.

The same five kinds, as a summary:

What you change	Larkfield example	Where in the course
Data	Retrain with the tickets after the policy change	Module 8
Features	Add prior_escalations_90d	Select useful features
Algorithm	A tree, a random forest, gradient boosting	This module
Hyperparameters	max_depth of a tree, C of logistic regression	Control complexity, Tune hyperparameters
Threshold	Flag at a score of 0.19	Thresholds and calibration

When to stop

The loop has no natural end, so you decide in advance when to stop. There are three good reasons:

• The target is met. Larkfield’s target is to flag at most 20% of new tickets and catch at least 40% of the escalations. When a model meets it on the validation month, the model is useful. You can still improve it, but each extra round must be worth its cost.
• The gains are smaller than the noise. If the last few changes moved the score by less than its standard deviation, the data cannot show a difference. More rounds will only find luck.
• The time or the budget is used up. A model that helps Grace’s team next month is often worth more than a slightly better model next year.

Rules that keep the loop honest

Each round compares a new result with the best result so far. The comparison is fair only if the rounds follow the same rules. Each rule below prevents a mistake that looks like progress.

• The same data, the same folds, the same metric. Every model gets the same 5 folds of the train set and the same score, average precision. If two models were scored on different tickets, a difference can come from the tickets, not from the models.
• The baseline every time. Show every new result next to the rule and the best model so far. A number alone means nothing.
• Choose with the train set; confirm with the validation set; keep the test set sealed. Cross-validation on train chooses. The validation month, the data that you keep to check the model while you work, only confirms a choice, once. The test set is used once, after the loop. Train, validation and test sets explained why: each look at a set uses it up a little.
• A difference smaller than the noise is not a difference. The standard deviation of the 5 fold scores shows how much a score moves by chance. If a change moves the mean by less than that, write “no real difference”.
• One change at a time. If you change two things and the score goes up, you do not know which change helped. One of them could even make the model worse.

Omar, the product manager, asks: “Why so strict? If a model scores higher, it is better.” Priya answers with a number. Logistic regression scores 0.310 on one fold and 0.342 on another, with no change at all. A careless comparison can easily find a gain of 0.03 that does not exist. Then Larkfield would pay for a more complex model and get nothing for it.

The experiment log

The cook’s notebook has a name in machine learning: the experiment log. It is a table with one row for each experiment, in the order you ran them. An experiment is one try: one model, with one set of features and settings, scored in the agreed way.

Here is the cook’s log as a table, so you can see the shape:

id	change	result	decision	reason
1	The simple recipe	good, a little dry	baseline	the start
2	More water	softer	keep	better, and no extra cost
3	Hotter oven	the crust burns	reject	worse

Priya’s log for Larkfield has the same idea, with numbers. It has these columns:

• id: E01, E02 and so on, in order.
• date: the day you ran the experiment.
• change: the one thing that changed, in words.
• cv_ap and cv_std: the mean average precision of the 5 folds, and their standard deviation.
• valid_ap: the average precision on the validation month. Fill it only for a candidate that you check once to confirm a decision. Leave it empty for the others.
• decision: baseline, keep or reject.
• reason: one sentence that says why.

A log helps in three ways. It is your memory: you do not repeat a failed experiment. It keeps you honest: the rejected experiments stay visible, so nobody can show only the best result. And it tells the story to others: Grace or a reviewer can see why the final model was chosen.

Write the log as you work, not at the end. You will keep this log for the rest of Module 5 and through Module 6.

Worked example: the first rows of the log

Priya starts the log with what Larkfield already knows: the two baselines and logistic regression. To compare them fairly, she scores all three on the same 5 folds of train.

The baselines do not learn, but they still give a score to each ticket. “One score for every ticket” gives each ticket the train rate, 0.117. Such a score cannot rank the tickets, so its AP equals the share of escalations. This is the floor from Average precision: 0.117 on train. The rule gives 1 to a priority 1 ticket and 0 to the others. rule_scores() computes the AP of such a fixed rule on each of the 5 folds. cv_scores() does the same for a pipeline, with cross-validation.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score
from sklearn.model_selection import StratifiedKFold, cross_val_score

from ticket_model import FEATURES, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)


def cv_scores(pipeline):
    """The AP of a pipeline on each of the 5 folds of train (cross-validation)."""
    return cross_val_score(pipeline, train[FEATURES], train[TARGET], cv=cv, scoring="average_precision")


def rule_scores(score_of):
    """The AP of a fixed rule (it learns nothing) on each of the same 5 folds."""
    return np.array([average_precision_score(train[TARGET].iloc[rows], score_of(train.iloc[rows]))
                     for _, rows in cv.split(train, train[TARGET])])

Before you run the next cell, predict: how far above the floor of 0.117 is the rule? And logistic regression?

one_score = rule_scores(lambda tickets: [0.117] * len(tickets))
priority_rule = rule_scores(lambda tickets: (tickets["priority"] == 1).astype(int))
logistic = cv_scores(build_pipeline())
for name, folds in [("one score for all", one_score), ("rule: priority 1", priority_rule),
                    ("logistic regression", logistic)]:
    print(f"{name:20s} folds {folds.round(3)}  CV AP {folds.mean():.3f} ± {folds.std():.3f}")

The output:

Output

one score for all    folds [0.117 0.117 0.117 0.117 0.117]  CV AP 0.117 ± 0.000
rule: priority 1     folds [0.185 0.164 0.182 0.173 0.186]  CV AP 0.178 ± 0.008
logistic regression  folds [0.316 0.31  0.342 0.339 0.327]  CV AP 0.327 ± 0.012

The rule is 0.061 above the floor. Logistic regression is 0.149 above the rule, more than ten times its standard deviation. That difference is real. The validation month confirms the order once:

pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
print("one score for all  ", round(average_precision_score(valid[TARGET], [0.117] * len(valid)), 3))
print("rule: priority 1   ", round(average_precision_score(valid[TARGET], (valid["priority"] == 1).astype(int)), 3))
print("logistic regression", round(average_precision_score(valid[TARGET], pipeline.predict_proba(valid[FEATURES])[:, 1]), 3))

Output

one score for all   0.155
rule: priority 1    0.228
logistic regression 0.353

The numbers are higher in May because 15.5% of the May tickets escalated, against 11.7% in train. The order is the same. Now Priya writes the three rows. log_experiment() adds one row to experiment_log.csv with today’s date, and show_log() prints the log. The notebook defines both helpers in its setup cells.

log_experiment("E01", "Baseline: one score for every ticket (the train rate)", one_score.mean(), one_score.std(),
               0.155, "baseline", "The floor: any useful model must be above it.")
log_experiment("E02", "Baseline: the rule 'flag priority 1', as a 0/1 score", priority_rule.mean(), priority_rule.std(),
               0.228, "baseline", "What Larkfield does today: the model must beat it.")
log_experiment("E03", "Logistic regression, the 10 C04 features", logistic.mean(), logistic.std(),
               0.353, "keep", "Far above both baselines; train AP 0.328, so it does not overfit.")
show_log()

Output

 id                                                change  cv_ap  cv_std  valid_ap decision
E01 Baseline: one score for every ticket (the train rate)  0.117   0.000     0.155 baseline
E02  Baseline: the rule 'flag priority 1', as a 0/1 score  0.178   0.008     0.228 baseline
E03              Logistic regression, the 10 C04 features  0.327   0.012     0.353     keep

One round of the loop

Tomás has an idea. “Only 11.7% of the train tickets escalated. The model sees few escalations, so it does not learn them well. Tell it to count each escalated ticket more.” scikit-learn has a setting for this: class_weight="balanced". In training, it gives each escalated ticket about 7.5 times the weight of another ticket, as if the two groups were the same size.

This is a good experiment: it is one change to the best model so far. Before you read the output, predict: will the CV AP go up, go down, or stay about the same?

balanced = cv_scores(build_pipeline(LogisticRegression(max_iter=1000, class_weight="balanced")))
print("logistic regression", logistic.round(3), f"CV AP {logistic.mean():.3f} ± {logistic.std():.3f}")
print("balanced           ", balanced.round(3), f"CV AP {balanced.mean():.3f} ± {balanced.std():.3f}")
print("balanced − logistic", (balanced - logistic).round(3))

Output

logistic regression [0.316 0.31  0.342 0.339 0.327] CV AP 0.327 ± 0.012
balanced            [0.314 0.309 0.339 0.335 0.32 ] CV AP 0.323 ± 0.012
balanced − logistic [-0.002 -0.002 -0.003 -0.004 -0.007]

The balanced model is lower on all 5 folds. Because the folds are the same, Priya can compare fold by fold. The change does not help. Why not? Average precision depends only on the order of the tickets by score. The weights move all the scores up, but they change the order very little. The change also costs something: the scores no longer match the real escalation rate, which matters in Thresholds and calibration.

Decision: reject. It still goes in the log, so that nobody tries it again next month without a new reason.

The experiment log after this lesson: the CV AP of each experiment, with a bar of one standard deviation on each side. E03, logistic regression, is the best so far.

Text description of the chart

The chart shows five rows, one for each experiment, on an axis from 0.10 to 0.35. A dashed line marks the floor, 0.117. E01, one score for every ticket: 0.117, with no spread. E02, the rule “flag priority 1”: 0.178, from 0.170 to 0.186. E03, logistic regression: 0.327, from 0.315 to 0.339, highlighted as the model to keep. E04, logistic regression with balanced class weights: 0.323, from 0.311 to 0.335; its bar overlaps the bar of E03. E05, the rule “flag priority 1 or team payment”: 0.172, from 0.166 to 0.178, a little below E02.

Guided practice: log Tomás’s experiment

Do this in the notebook. Each step has a check.

1. Run the setup cells, then the cells of the worked example. You should see the log with three rows, E01 to E03.
2. Run the cell that computes balanced. You should see CV AP 0.323 ± 0.012.
3. Write the row for E04 with log_experiment(). Use the change "E03 with class_weight='balanced'", the mean and the std of balanced, no valid AP (None), and the decision "reject".
4. Write a reason in one sentence, in your own words.
5. Run show_log(). You should see four rows.
6. Run the check cell. You should see Check passed: the row E04 is right.

Show an example answer

log_experiment("E04", "E03 with class_weight='balanced'", balanced.mean(), balanced.std(),
               None, "reject", "Lower than E03 on all 5 folds; one more setting for no gain.")
show_log()

Output

 id                                                change  cv_ap  cv_std valid_ap decision
E01 Baseline: one score for every ticket (the train rate)  0.117   0.000    0.155 baseline
E02  Baseline: the rule 'flag priority 1', as a 0/1 score  0.178   0.008    0.228 baseline
E03              Logistic regression, the 10 C04 features  0.327   0.012    0.353     keep
E04                      E03 with class_weight='balanced'  0.323   0.012            reject

The valid AP stays empty. A change that loses on every fold is not a candidate, so there is nothing to confirm.

Your turn: a wider rule

Change one condition: not the model, but the baseline. Grace asks whether a wider rule would do better: “flag priority 1, or any ticket of the payment team”. The payment team has the highest escalation rate in train, 20.7%.

Compute the CV AP of this rule with rule_scores(), on the same folds. Before you run it, predict: will it beat the rule “flag priority 1” (0.178)? Then write the row E05 in the log, with your decision and reason.

wider_rule = rule_scores(lambda tickets: ((tickets["priority"] == 1) | (tickets["team"] == "payment")).astype(int))
print(wider_rule.round(3), f"CV AP {wider_rule.mean():.3f} ± {wider_rule.std():.3f}")

Show the answer

Output

[0.171 0.172 0.166 0.166 0.182] CV AP 0.172 ± 0.006

The wider rule is a little lower than the rule “flag priority 1” (0.178 ± 0.008). The payment tickets escalate more often than average, but most of them still do not escalate (79.3%). The wider rule flags many more tickets, and the share of real escalations among the flags falls. Decision: reject. A possible reason: “Lower than the simpler rule E02.”

log_experiment("E05", "Baseline: the rule 'flag priority 1 or team payment'", wider_rule.mean(), wider_rule.std(),
               None, "reject", "Lower than the simpler rule E02.")

A gain that came from the folds

Common mistake

A gain that came from the folds

Symptom: Tomás tried the balanced model before Priya did. He reports: “Balanced weights add 0.029 of AP. Logistic regression got 0.310, and the balanced model got 0.339.” His numbers are real:

print("Tomás's check of logistic regression, fold 2:", logistic[1].round(3))
print("Tomás's check of the balanced model, fold 3: ", balanced[2].round(3))

Output

Tomás's check of logistic regression, fold 2: 0.31
Tomás's check of the balanced model, fold 3:  0.339

Cause: he scored the two models on different tickets. Fold 3 is easier than fold 2: logistic regression itself gets 0.342 on fold 3 and 0.310 on fold 2. The whole “gain” comes from the tickets, not from the change. On the same folds, the balanced model is lower on all five.

Fix: score every experiment with the same cv object (the same 5 folds) and the same metric, and compare the mean ± std, or fold by fold. Write the result in the log only when it was measured in the agreed way.

Consequence in AI work: an unfair comparison makes a team adopt changes that do nothing, or that even make the model worse. Each one adds complexity. After a few months, nobody knows which parts of the model really help.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study keeps an experiment log through four model families, including rejected ideas.

Knowledge check

Check your understanding

Tomás changes two things in one experiment: he adds a new feature and switches to a random forest. The CV AP goes up from 0.327 to 0.345 ± 0.012. What can Priya conclude, and what should she do next?

Show the answer

She can conclude that the combination may be better: 0.018 is about one and a half standard deviations. But she cannot say which change helped. The feature alone could explain all of it, and the forest could even be worse. She should run two experiments, each with one change: logistic regression with the new feature, and the forest without it. Then she logs both.

Check your understanding

Experiment A has CV AP 0.331 ± 0.015 and the current best, E03, has 0.327 ± 0.012, on the same folds. A is more complex. What do you write in the decision and reason columns?

Show the answer

Decision: reject. Reason, for example: “A tie with E03: the difference, 0.004, is smaller than one standard deviation; A is more complex.” A difference smaller than the noise is not a difference, so the simpler model stays. Look at the folds too: if A were higher on all five folds, the case for A would be stronger.

Check your understanding

Omar asks why the log keeps the rejected experiments. 'Only the final model matters.' Give two reasons to keep them.

Show the answer

First, memory: the team does not repeat an experiment that already failed, such as the balanced weights. Second, honesty: a reader sees how many things were tried. If only the best of many tries is shown, its score looks better than it is, because some tries win by luck. A third reason: the log explains to Grace or a reviewer why the final model was chosen.

Recap

• No algorithm is the best for every problem. You find the best model for your data by trying several, measured in the same way.
• The loop: baseline, a first model, evaluate, look and log, then change one thing and evaluate again. Stop when the target is met, the gains are smaller than the noise, or the time is used up.
• Five kinds of change: data, features, algorithm, hyperparameters, threshold.
• Keep the loop honest: the same folds and metric, the baseline every time, choose on train, confirm on valid once, keep the test set sealed, one change at a time.
• The experiment log has one row per experiment: id, date, change, cv_ap, cv_std, valid_ap, decision, reason. Yours has five rows: logistic regression (0.327 ± 0.012) is the best so far.

Next, you try a different family of models: the decision tree.

Next: Decision trees