Thresholds and calibration

What you will be able to do

Turn the model’s scores into flags that the senior team can handle, and find out if people can read a score as a chance. Then give the model its one final exam on the test set. The default threshold flags almost nothing, and a score that was right last year can be too low now. Neither raises an error.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, the module practice and the module check, with the instructions. Its setup cells install pandas and scikit-learn, and download the practice data, ticket_model.py and train.py. You write evaluate.py in the notebook. So it works without the local project. Use the explorer and the chart here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 75 minutes

What you need to know

• The confusion matrix, precision and recall, from Classification metrics: The confusion matrix and Precision and recall.
• How a threshold trades precision for recall, from The precision–recall trade-off. This lesson uses it on the trained model.
• Why the test set is used once, at the end, from Train, validation and test sets: Three sets, three jobs.
• The chosen model and the one-standard-error rule, from Tune hyperparameters: The one-standard-error rule.
• predict and predict_proba, from Classification intuition.
• Calibration and the calibration diagram, from Calibration and slices, and the cost curve, from Thresholds and costs, both in Essential Mathematics and Statistics for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its active .venv and the data in data/, from Define a baseline: Set up your project. If a data file is missing or changed, run python get_data.py --reset.
• ticket_model.py with 11 features, from Select useful features: Change ticket_model.py.
• experiment_log.csv with the rows E01 to E21, from Tune hyperparameters: Write it in the experiment log. This lesson reads it and adds no row.
• This lesson adds train.py (downloaded), evaluate.py (you write it), and the folders model/ and reports/, in Test once with evaluate.py.
• If a file is missing or broken, follow Reset to a known state at the end of this lesson.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1, joblib 1.6.0 and matplotlib 3.11.2. The Windows commands are not tested. The notebook was tested in Jupyter.

From a score to a flag

A small clinic has one doctor free this afternoon, with time for three patients. Ten people are waiting. A nurse looks at each person and writes an urgency score. The doctor then sees the three with the highest scores. The nurse does not need a “correct” cut-off score: the doctor’s time decides it. With time for three, the cut-off is the score of the third most urgent person.

Grace’s senior team works in the same way, and you already know the tools from The precision–recall trade-off. A score sorts the tickets. A threshold turns the score into a flag: flag the ticket when its score is at least the threshold. A lower threshold raises recall and lowers precision. Grace’s team can review at most 20% of tickets, and this limit is her capacity. In that lesson, you chose the lowest cut-off of the points score that fits it: 3 points or more.

One thing is new in this lesson: the score now comes from the trained model, a number from 0 to 1 with many different values. So the threshold can move in small steps, and no tie rule is needed. The rule stays the same: the best threshold is the lowest one that stays within the capacity. It catches the most escalations that the team can review. Here it is on ten scores, with a capacity of 3 tickets in 10 (30%):

Threshold	Scores at or above it	Flagged	Within 30%?
0.43	0.81, 0.62, 0.55, 0.43	4 (40%)	No
0.44	0.81, 0.62, 0.55	3 (30%)	Yes: the lowest that fits
0.56	0.81, 0.62	2 (20%)	Yes, but it misses the ticket at 0.55

choose_threshold() does this in steps of 0.01. It is the function in the course’s train.py, which you download later in this lesson:

train.py (part)

def choose_threshold(scores, capacity):
    """The lowest threshold (in steps of 0.01) that flags no more than
    `capacity` of the tickets. Lower thresholds flag more tickets and catch
    more escalations, so the lowest one that fits catches the most."""
    for step in range(1, 100):
        threshold = step / 100
        if (scores >= threshold).mean() <= capacity:
            return threshold
    return 1.0

As in every notebook of this course, np is NumPy and pd is pandas; the notebook’s setup cells import them.

scores_10 = np.array([0.81, 0.62, 0.55, 0.43, 0.38, 0.30, 0.22, 0.15, 0.09, 0.04])
threshold_10 = choose_threshold(scores_10, 0.3)
print("threshold:", threshold_10, "| flagged:", (scores_10 >= threshold_10).sum())

Output

threshold: 0.44 | flagged: 3

Worked example: the threshold on the validation set

The threshold is a choice, so it is made on the validation set, never on the test set. Priya fits the final model on train, scores the 1,179 May tickets, and prints the four counts at six thresholds:

from sklearn.metrics import confusion_matrix
from ticket_model import FEATURES, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
for threshold in [0.5, 0.3, 0.2, 0.19, 0.15, 0.1]:
    flags = (scores >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(valid[TARGET], flags).ravel()
    print(f"{threshold:4}: flagged {tp + fp:3d} ({(tp + fp) / len(valid):5.1%})  TP {tp:3d}  FP {fp:3d}  FN {fn:3d}  TN {tn:3d}"
          f"  precision {tp / (tp + fp):.3f}  recall {tp / (tp + fn):.3f}")

Output

 0.5: flagged  25 ( 2.1%)  TP  17  FP   8  FN 166  TN 988  precision 0.680  recall 0.093
 0.3: flagged  92 ( 7.8%)  TP  46  FP  46  FN 137  TN 950  precision 0.500  recall 0.251
 0.2: flagged 203 (17.2%)  TP  81  FP 122  FN 102  TN 874  precision 0.399  recall 0.443
0.19: flagged 227 (19.3%)  TP  86  FP 141  FN  97  TN 855  precision 0.379  recall 0.470
0.15: flagged 314 (26.6%)  TP 106  FP 208  FN  77  TN 788  precision 0.338  recall 0.579
 0.1: flagged 475 (40.3%)  TP 128  FP 347  FN  55  TN 649  precision 0.269  recall 0.699

This is the trade-off of Module 3, now with the real model. Each step down catches more escalations and wastes more reviews. Which step is right is not a question for the data. Grace answers it: at most 20% of tickets.

threshold = choose_threshold(scores, 0.20)
flags = (scores >= threshold).astype(int)
print("threshold:", threshold, f"| flagged {flags.sum()} of {len(valid)} ({flags.mean():.1%})")

Output

threshold: 0.19 | flagged 227 of 1179 (19.3%)

The threshold is 0.19. At 0.18, the model flags 248 tickets (21.0%), which is over the capacity. At 0.19, it catches 86 of 183 escalations (recall 0.470) with 227 reviews. The priority-1 rule catches 61 with 163 reviews (recall 0.333). So the improvement target of Define a baseline, at most 20% flagged and at least 40% caught, is met on the validation set.

Three confusion matrices

Each column is the confusion matrix of the same model on the same 1,179 May tickets, at one threshold. Only the threshold changes:

	Threshold 0.5 (the default)	Threshold 0.19 (chosen)	Threshold 0.1
Caught: escalated, flagged (TP)	17	86	128
Missed: escalated, not flagged (FN)	166	97	55
False alarm: not escalated, flagged (FP)	8	141	347
Correctly left: not escalated, not flagged (TN)	988	855	649
Flagged	25 (2.1%)	227 (19.3%)	475 (40.3%)
Precision	0.680	0.379	0.269
Recall	0.093	0.470	0.699

• At 0.5, the team gets 25 reviews in a month, and 166 escalations wait in the normal queue. Precision is high because only the clearest cases pass.
• At 0.1, the team gets 475 reviews, twice its capacity. 347 of them, almost three in four, are not needed.
• At 0.19, the team is at its capacity, and it catches almost half of the escalations. About 4 in 10 reviews find an escalation, and a review costs about 10 minutes.

Explore the threshold on the real scores

The explorer has the scores of the final model for the 1,179 May tickets: 996 not escalated (upper panel) and 183 escalated (lower panel). These are the same scores as in the table, so every count matches.

Explore

The threshold for the senior team's capacity

Solid bars are flagged (score at or above the threshold). Pale bars are not flagged. You can also drag the round handle on the line, or focus it and press the arrow keys (Shift for steps of 0.10). Reviews per week assume 266 tickets a week, with the same mix as these 1,179.

At threshold 0.50, the model flags 25: 17 escalated and 8 false alarms. It misses 166 of the 183 escalated tickets.

Confusion matrix at threshold 0.50

Real class	Flagged	Not flagged	Total
Escalated	17true positive	166false negative (missed)	183
Not escalated	8false positive (false alarm)	988true negative	996
Total	25	1154	1,179

Flagged

25

of 1,179 tickets

Precision

68.0%

17 of 25 flags

Recall

9.3%

17 of 183

False-positive rate

0.8%

8 of 996

Reviews per week

6

about 1 a day

Scores of the final model for the 1,179 May tickets of the validation set, with the threshold, the confusion matrix and the reviews per week. May had about 266 tickets a week, so the capacity of 20% is about 53 reviews a week.

Text description of the explorer

The upper panel is a histogram of the scores of the 996 tickets that did not escalate. Most of them are low: two in three are below 0.10, the highest bar is from 0.02 to 0.04 (225 tickets), and the median is 0.065. The highest is 0.91. The lower panel is a histogram of the 183 escalated tickets. Their scores are spread wider, from 0.02 to 0.80, with a median of 0.18. About 3 in 10 escalated tickets have a score below 0.10, so no threshold can catch them without flagging most tickets. The two panels overlap over most of the range. A vertical line marks the threshold.

At the starting threshold of 0.50, the model flags 25 tickets: 17 escalated and 8 not. It misses 166 escalations. Precision is 68.0%, recall 9.3%, and the team gets about 6 reviews a week. At 0.30: 92 flagged, recall 25.1%. At 0.19: 227 flagged (86 escalated, 141 not), 97 missed, precision 37.9%, recall 47.0%, and about 51 reviews a week, about 7 a day. At 0.18, 248 tickets are flagged, which is above 235. At 0.10: 475 flagged, recall 69.9%, about 107 reviews a week.

Move the threshold from 0.50 down to 0.19, and watch the reviews per week. The explorer assumes 266 tickets a week (1,179 May tickets ÷ 31 days × 7). The capacity of 20% is about 53 reviews a week. At 0.19 the team gets about 51; at 0.18, about 56, which is too many.

Calibration: can a score be read as a chance?

A weather forecaster says “70% chance of rain”. How do you know if she is good? Collect all the days when she said 70%. If it rained on about 7 of every 10 of those days, her 70% means what it says. If it rained on only 4 of 10, her numbers are too high, even if she ranks wet days above dry days correctly.

Grace has a similar plan. She wants to show each agent “chance of escalation: 24%” next to a ticket. That is only honest if, of the tickets with a score near 0.24, about 24% really escalate. A model with this property is calibrated: its scores match how often things happen. In Calibration and slices in the mathematics course, you read a calibration diagram. Now you check a real model, on train and on the month after the policy change.

One number summarises it: the Brier score, the mean of (score − outcome)². The outcome is 1 for an escalation and 0 for none. Lower is better. Here are four tickets:

Ticket	Score	Escalated	(score − outcome)²
A	0.9	1	(0.9 − 1)² = 0.01
B	0.2	0	0.04
C	0.6	0	0.36
D	0.1	0	0.01

Brier score = (0.01 + 0.04 + 0.36 + 0.01) ÷ 4 = 0.105. Ticket C adds the most: a confident score that was wrong. The Brier score mixes two things: how well the scores rank the tickets, and how well they match the rates. So to see calibration alone, you also need the picture below.

For the real model, Priya groups the scores into bins, and compares the mean score with the observed escalation rate in each bin:

train_scores = pipeline.predict_proba(train[FEATURES])[:, 1]
bins = [0, 0.05, 0.1, 0.2, 0.3, 0.5, 1]

def reliability(part, part_scores):
    """Tickets, mean score and observed escalation rate per score bin."""
    table = pd.DataFrame({"score": part_scores, "escalated": part[TARGET].to_numpy()})
    return table.groupby(pd.cut(part_scores, bins), observed=True).agg(
        tickets=("escalated", "size"), mean_score=("score", "mean"), rate=("escalated", "mean")).round(3)

print("train\n", reliability(train, train_scores))
print("valid\n", reliability(valid, scores))

Output

train
               tickets  mean_score   rate
(0.0, 0.05]     5576       0.031  0.030
(0.05, 0.1]     4713       0.072  0.072
(0.1, 0.2]      3843       0.142  0.140
(0.2, 0.3]      1493       0.244  0.249
(0.3, 0.5]      1025       0.376  0.389
(0.5, 1.0]       289       0.614  0.574
valid
               tickets  mean_score   rate
(0.0, 0.05]      414       0.031  0.053
(0.05, 0.1]      290       0.071  0.114
(0.1, 0.2]       272       0.145  0.173
(0.2, 0.3]       111       0.241  0.315
(0.3, 0.5]        67       0.388  0.433
(0.5, 1.0]        25       0.625  0.680

On train, each bin’s mean score is close to its rate: scores from 0.2 to 0.3 have a mean of 0.244, and 24.9% of those tickets escalated. In May, every rate is higher than the mean score: the same bin has a mean of 0.241, and 31.5% escalated.

The reliability chart

A reliability chart is the calibration diagram that you read in the mathematics course. It draws the same bins: one dot per bin, at its mean score and its observed rate. A dot on the dashed diagonal means “score = rate”, like a forecaster whose 70% days are rainy 7 times in 10.

• Train (2024-07 to 2026-04)
• Valid (May 2026)
• Perfect calibration (dashed)

Reliability chart of the final model: mean score against the observed escalation rate in six score bins, on train and on valid. Train lies close to the diagonal; May lies above it in every bin.

Text description of the chart

A square chart. The horizontal axis is the mean score in a bin, and the vertical axis is the observed escalation rate, both from 0 to 0.7. A dashed diagonal shows perfect calibration. The train line has six dots, all close to the diagonal: (0.031, 0.030), (0.072, 0.072), (0.142, 0.140), (0.244, 0.249), (0.376, 0.389) and (0.614, 0.574); only the last bin, with 289 tickets, is a little below it. The valid line has six dots, all above the diagonal: (0.031, 0.053), (0.071, 0.114), (0.145, 0.173), (0.241, 0.315), (0.388, 0.433) and (0.625, 0.680). A marked dot shows the bin from 0.2 to 0.3 in May: mean score 0.24, rate 31.5%. The last valid bin has only 25 tickets.

The model is calibrated on the months it learned from, and it under-estimates the risk in May. The Brier score shows the same:

from sklearn.metrics import brier_score_loss

print(f"train: model {brier_score_loss(train[TARGET], train_scores):.4f}, constant 0.117 {brier_score_loss(train[TARGET], np.full(len(train), 0.117)):.4f}")
print(f"valid: model {brier_score_loss(valid[TARGET], scores):.4f}, constant 0.117 {brier_score_loss(valid[TARGET], np.full(len(valid), 0.117)):.4f}")
print(f"valid: mean score {scores.mean():.3f}, escalation rate {valid[TARGET].mean():.3f}")

Output

train: model 0.0902, constant 0.117 0.1034
valid: model 0.1155, constant 0.117 0.1326
valid: mean score 0.120, escalation rate 0.155

The model beats a constant score of 0.117 on both sets. Its Brier score is worse on valid (0.1155 against 0.0902). Part of that is expected: with more escalations, every model has a larger Brier score. The rest is the gap: the model says 12.0% on average, and 15.5% escalate.

What to do about the gap

First find where the gap is. The policy change was about warranties, so Priya compares by team:

by_team = valid.assign(score=scores).groupby("team").agg(
    tickets=(TARGET, "size"), mean_score=("score", "mean"), rate=(TARGET, "mean"))
print(by_team.round(3))

Output

          tickets  mean_score   rate
team
account       166       0.057  0.072
delivery      348       0.092  0.121
payment       206       0.214  0.233
returns       291       0.098  0.113
warranty      168       0.163  0.286

Warranty tickets have the largest gap: a mean score of 0.163 and a rate of 0.286, about 12 percentage points. The model learned the warranty rate from before the policy.

Then ask what the gap breaks. The capacity rule uses only the order of the scores: it flags the riskiest tickets until the capacity is full, like the nurse’s list. If every score is too low by the same amount, the order stays the same. Priya tests it: she adds 0.035 to every score, so the mean moves from 0.120 to the real rate.

from sklearn.metrics import average_precision_score

shifted = np.clip(scores + 0.035, 0, 1)
print(f"mean score: {scores.mean():.3f} -> {shifted.mean():.3f}")
print("average precision:", round(average_precision_score(valid[TARGET], scores), 3), "->",
      round(average_precision_score(valid[TARGET], shifted), 3))

Output

mean score: 0.120 -> 0.155
average precision: 0.391 -> 0.391

The order did not change, so the average precision did not change. That gives three conclusions:

• The threshold of 0.19 still works. It flags the riskiest 19.3% of May tickets, and that ranking holds after the policy change.
• The scores must not be shown to people as “the chance of escalation”. A warranty ticket with a score of 0.16 escalates almost twice as often as 16%. A dashboard with “16%” would mislead Grace’s agents.
• The fix is new data, not a new threshold. One way is to recalibrate the scores on recent labelled tickets. That means: learn a mapping from score to observed rate, for example with scikit-learn’s CalibratedClassifierCV, and check it on a later month. The other way is to retrain with May and June included. Module 8 makes that the retraining plan.

Guided practice: choose and check the threshold

Use the notebook, sections 1 to 4.

1. Run the setup cells. You should see the Ready: line.
2. Run section 1. You should see threshold: 0.44 | flagged: 3.
3. Run the threshold table in section 2. Check the line for 0.19: 227 flagged, recall 0.470.
4. Predict the threshold for a capacity of 20%, then run the next cell. You should see threshold: 0.19.
5. In section 4, run the reliability table and the chart. The valid line must lie above the diagonal.
6. Compute brier_valid by hand: the mean of (score − outcome)², rounded to 4 decimals.
7. Run the check cell. It prints Check passed: brier_valid is right. when your value is right.
8. Run the brier_score_loss cell. Your value must equal the valid line.

Show the answer

brier_valid = round(float(np.mean((scores - valid[TARGET].to_numpy()) ** 2)), 4)
print(brier_valid)

Output

0.1155

It equals brier_score_loss on valid: 0.1155.

Your turn: a busy week at 15%

Change one condition. Grace says that in a busy week her team can review only 15% of tickets. Before you run it, predict: does the threshold go up or down? Does recall go up or down? Then compute the threshold with choose_threshold(scores, 0.15), and count the flagged May tickets (notebook, section 2).

Show the answer

threshold_15 = choose_threshold(scores, 0.15)
flagged_15 = int((scores >= threshold_15).sum())
print(threshold_15, flagged_15)

Output

0.22 174

The threshold goes up to 0.22: 174 tickets flagged (14.8%). With fewer reviews, recall falls to 0.410 (75 of 183 caught), and precision rises to 0.431, because only the clearer cases pass. The model still catches more than the priority-1 rule (61 escalations) with almost the same number of reviews (174 against 163).

Common mistake

The default threshold of predict()

Symptom: Tomás uses pipeline.predict(), as most tutorials do:

flags_default = pipeline.predict(valid[FEATURES])
print("predict() flags:", flags_default.sum(), f"({flags_default.mean():.1%})")

Output

predict() flags: 25 (2.1%)

The senior team gets 25 reviews in a month and has free time. Meanwhile 166 of the 183 escalations wait in the normal queue. There is no error and no warning.

Cause: predict() uses a threshold of 0.5. Only 25 May tickets have a score that high, because only 15.5% of tickets escalate and the model is careful. 0.5 is a default of the library, not a decision of the business.

Fix: use predict_proba(), and choose the threshold on the validation set for the decision: choose_threshold(scores, 0.20) gives 0.19. Save the threshold with the model, so every user of the model applies the same one. train.py writes it into model_info.json.

Consequence in AI work: a model with a good ranking (ROC AUC 0.752) does almost nothing, because one number is missing. A team that only checks “the model is deployed” never sees it. Check the flag rate against the capacity every week (Module 8).

Optional: a cost view

This part is optional. Grace’s rough costs: one review costs about 10 minutes of a senior agent. Say that a missed escalation costs about 60 minutes of later work (and an unhappy customer). Then the total cost at a threshold is 10 × flagged + 60 × missed, in minutes. Thresholds and costs in the mathematics course explains the cost curve.

Explore

Optional: what the cheapest threshold would be

Solid bars are flagged (score at or above the threshold). Pale bars are not flagged. You can also drag the round handle on the line, or focus it and press the arrow keys (Shift for steps of 0.10).

At threshold 0.19, the model flags 227: 86 escalated and 141 false alarms. It misses 97 of the 183 escalated tickets. The cheapest threshold for these costs is 0.14 (7,470).

Confusion matrix at threshold 0.19

Real class	Flagged	Not flagged	Total
Escalated	86true positive	97false negative (missed)	183
Not escalated	141false positive (false alarm)	855true negative	996
Total	227	952	1,179

Flagged

227

of 1,179 tickets

Precision

37.9%

86 of 227 flags

Recall

47.0%

86 of 183

False-positive rate

14.2%

141 of 996

Total cost

8,090

minutes per 1,179 tickets

The same 1,179 May tickets, with the total cost in minutes at every threshold, for the costs that you enter. The capacity of 20% still limits the choice.

Text description of the explorer

The upper chart shows the same two score panels as the first explorer, with the threshold at 0.19. Below it are the confusion matrix, precision, recall, the false-positive rate and the total cost. Two number boxes set the minutes for one review (start: 10) and for one missed escalation (start: 60).

The lower chart shows the total cost of the 1,179 May tickets at every threshold. With the starting costs, it is 10,980 minutes when nothing is flagged (183 missed escalations × 60). At 0.19, it is 8,090 minutes. The cheapest threshold is 0.14, with 7,470 minutes; it flags 345 tickets (29.3%), more than the capacity. With a miss at 30 minutes, the cheapest threshold moves up to 0.23 (163 flagged). With a miss at 120 minutes, it moves down to 0.06 (686 flagged, 58.2%).

With these costs, the cheapest threshold is 0.14. It flags 29.3% of tickets, but the team can review only 20%. So the cost view does not change the threshold. It gives Grace an argument instead: more senior time would catch more escalations, and with these rough costs it would pay for itself. Note one more thing. With calibrated scores, “flag when the chance is above 10 ÷ 60 = 0.17” would be close to the cheapest rule. Here the cheapest is lower, 0.14, because the May scores are too low.

Test once with evaluate.py

In Train, validation and test sets, the test set was the final exam: new questions in a sealed envelope, opened once at the end. The exam score is the one that other people trust, because nothing was tuned to it. If a student sees the exam, practises on it and takes it again, the second score means nothing.

Now every choice is made. Lesson 1 chose the features, Lesson 2 chose the model and its settings, and this lesson chose the threshold, 0.19. So it is time to open the envelope: the 1,002 tickets of June 2026 that no step has used. Priya opens it once, with a script that refuses to run a second time. Tomás asks: “Why a script? You could just compute the numbers in the notebook.” A script makes the test the same every time, writes the result to a file, and can stop a second look. A notebook cell can be run again by accident.

First, train.py fits the pipeline on train, chooses the threshold on valid with choose_threshold(), and saves the model and the threshold in model/. You write it yourself in Package artifacts in Module 8. For now, download the course’s file into your project folder, then run it:

macOS and Linux

Terminal

curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/train.py
python train.py

Windows

PowerShell

curl.exe -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/train.py
python train.py

Output

Trained on 16939 tickets. Threshold 0.19 for a review capacity of 20%.
Valid: ROC AUC 0.752, average precision 0.391, flagged 227 of 1179 (19.3%), precision 0.379, recall 0.47.
Saved model/escalation_model.joblib, model_info.json and check_sample.csv.

If you see ModuleNotFoundError: No module named 'ticket_model', the terminal is not in ~/projects/ticket-model. Go to that folder, and run the command again.

Now write evaluate.py in the project folder, next to train.py:

evaluate.py

"""Report the saved model's results on the test set, once.

Nextia Learning, C05, Module 6, lesson 3 (Thresholds and calibration). Module 8
uses the report again.

    python evaluate.py

Uses the threshold that train.py chose on valid. Writes
reports/test_report.json. If the report exists already, it stops: look at
the test set once, and do not change the model after you look.
"""

import json
import sys
from pathlib import Path

import joblib
from sklearn.metrics import average_precision_score, brier_score_loss, confusion_matrix, roc_auc_score

from ticket_model import FEATURES, TARGET, load

REPORT = Path("reports/test_report.json")


def main():
    if REPORT.exists() and "--again" not in sys.argv:
        sys.exit(f"{REPORT} exists: the test set was used already. Read the report. "
                 "Run with --again only to reproduce it, never to choose a change.")
    pipeline = joblib.load("model/escalation_model.joblib")
    info = json.loads(Path("model/model_info.json").read_text(encoding="utf-8"))
    test = load("test")
    scores = pipeline.predict_proba(test[FEATURES])[:, 1]
    flags = (scores >= info["threshold"]).astype(int)
    tn, fp, fn, tp = (int(v) for v in confusion_matrix(test[TARGET], flags, labels=[0, 1]).ravel())
    rule = (test["priority"] == 1).astype(int)
    rule_tp = int((rule & test[TARGET]).sum())
    report = {
        "rows": len(test), "escalated": int(test[TARGET].sum()), "threshold": info["threshold"],
        "roc_auc": round(roc_auc_score(test[TARGET], scores), 3),
        "average_precision": round(average_precision_score(test[TARGET], scores), 3),
        "brier": round(brier_score_loss(test[TARGET], scores), 4),
        "mean_score": round(float(scores.mean()), 3), "escalated_rate": round(float(test[TARGET].mean()), 3),
        "confusion": {"tp": tp, "fp": fp, "fn": fn, "tn": tn},
        "flag_rate": round((tp + fp) / len(test), 3),
        "precision": round(tp / (tp + fp), 3), "recall": round(tp / (tp + fn), 3),
        "baseline_priority_1": {"flagged": int(rule.sum()), "flag_rate": round(float(rule.mean()), 3),
                                "precision": round(rule_tp / rule.sum(), 3),
                                "recall": round(rule_tp / test[TARGET].sum(), 3)},
    }
    REPORT.parent.mkdir(exist_ok=True)
    REPORT.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

Each part has a job:

Lines	Why
if REPORT.exists() and "--again" not in sys.argv	The guard. A second run stops with a message, so nobody runs the test “just once more” after a change. --again reproduces the same report; it is not for choosing.
joblib.load(...), model_info.json	The model and the threshold come from train.py. evaluate.py chooses nothing.
info["threshold"]	The threshold chosen on valid (0.19). The test set never sets it.
confusion_matrix(..., labels=[0, 1])	The four counts in a fixed order, even if one count is 0.
baseline_priority_1	The rule on the same test tickets. A number alone means nothing.
brier, mean_score, escalated_rate	Calibration on test, as in this lesson.
REPORT.write_text(...)	The report is a file in reports/, so the result is kept and can be checked.

Read the warning, then run it.

Run it once

The next command uses the test set. Run it once. Read the report, and do not change the model, the features or the threshold after you read it.

Terminal

python evaluate.py

Output

{
  "rows": 1002,
  "escalated": 157,
  "threshold": 0.19,
  "roc_auc": 0.749,
  "average_precision": 0.345,
  "brier": 0.1207,
  "mean_score": 0.123,
  "escalated_rate": 0.157,
  "confusion": {
    "tp": 70,
    "fp": 126,
    "fn": 87,
    "tn": 719
  },
  "flag_rate": 0.196,
  "precision": 0.357,
  "recall": 0.446,
  "baseline_priority_1": {
    "flagged": 150,
    "flag_rate": 0.15,
    "precision": 0.367,
    "recall": 0.35
  }
}

On test, the model flags 196 of 1,002 tickets (19.6%, within the capacity) and catches 70 of 157 escalations: recall 0.446. The rule catches 55: recall 0.350. So the improvement target (at most 20% flagged, at least 40% caught) is met on test too. Average precision is 0.345 against 0.391 on valid: a little lower. That is normal. Valid helped to choose, so it is a little optimistic, and June is a new month. The calibration gap is still there too: mean score 0.123, rate 0.157. We do not change anything after we look.

Now run it again, as Tomás would after “one small change”:

Terminal

python evaluate.py

Output

reports/test_report.json exists: the test set was used already. Read the report. Run with --again only to reproduce it, never to choose a change.

That is the guard. The report stays in reports/test_report.json. The module check below and Module 8 use it again.

Does the test go into the experiment log? No. The log records experiments: tries that compete to become the model, each scored with the same CV. The threshold and the test choose no model, so neither gets a row. The threshold is saved with the model in model/model_info.json, and the test result in reports/test_report.json. The log ends at E21, with E15 as the model to keep. A new row needs new data, for example a retrain with May and June (Module 8).

See it end to endPredict credit-card defaultCase study

The credit case study chooses its threshold from what each error costs, and checks that the scores are calibrated.

Knowledge check

Check your understanding

Why is the threshold chosen on the validation set, and why is it the lowest threshold that fits the capacity?

Show the answer

The threshold is a choice. You make choices on the validation set, so the test set stays an honest final check. May is also the month after the policy change, so it looks more like the future than train does. A lower threshold flags more tickets and so catches at least as many escalations. So the lowest threshold within the capacity catches the most escalations that the team can review. 0.19 flags 227 tickets (19.3%); 0.18 would flag 248 (21.0%).

Check your understanding

On valid, tickets with scores from 0.2 to 0.3 have a mean score of 0.241, and 31.5% of them escalated. Is the model overconfident or underconfident there, and does it break the 20% capacity rule?

Show the answer

Underconfident: the real rate (31.5%) is higher than the score (0.241), so the model under-estimates the risk. It does not break the capacity rule, because that rule uses only the order of the scores. When every score is too low by a similar amount, the same tickets are still the riskiest. Adding 0.035 to every score leaves the average precision at 0.391. It does break any use of the score as a chance, for example “24% likely” on a dashboard.

Check your understanding

Omar reads the test report (AP 0.345, recall 0.446 at 19.6% flagged) and asks: valid had AP 0.391 and recall 0.470, so should we lower the threshold to get back to 0.470 on June? What do you answer?

Show the answer

No. The threshold was chosen on valid, and the test report is the honest estimate for new tickets. If we change the threshold because of the test result, the test set becomes a second validation set, and its numbers are no longer honest. A lower test score than valid is normal: valid helped to choose, and June is a new month. The model still meets the target on test (19.6% flagged, recall 0.446 against the rule’s 0.350). We report these numbers as they are. A change needs new data: labelled tickets from after June.

Module practice

The module practice asks you to tune a small search space and to choose a threshold for a review capacity. You do the whole workflow of Tune hyperparameters and this lesson again, for a new capacity, without the test set.

Worked example: Priya’s workflow for 20%

Step	Data	Decision	Evidence
Features	CV on train	Add prior_escalations_90d; never the two leaks	CV AP 0.327 to 0.339, higher on every fold
Metric	(the decision)	Recall at a flag rate of at most 20%; average precision to compare models	Grace’s capacity: 1 in 5 tickets
Tune	CV on train	Logistic regression, C = 1	Best 0.3424 ± 0.0080; logistic 0.3393 is within one standard error
Threshold	valid (May)	0.19	227 flagged (19.3%), recall 0.470, precision 0.379
Calibration	train and valid	Show flags, not scores, to agents	Train calibrated; May under-estimated (0.120 against 0.155); warranty worst
Test	test (June), once	No change after it	AP 0.345, 19.6% flagged, recall 0.446; rule 0.350

Your task: a capacity of 25%

Grace may add a part-time senior agent. Then the team can review 25% of tickets. Prepare the recommendation, in the notebook, section 10:

1. Tune this small search space with GridSearchCV, the course’s CV and average precision: C 0.1 or 1, and class_weight None or "balanced". ("balanced" gives the escalated tickets more weight in the fit.)
2. Apply the one-standard-error rule, and choose one setting. Write one sentence on why.
3. Choose the threshold for a capacity of 25% on valid with choose_threshold().
4. Write the flagged count, the flag rate, precision and recall on valid.
5. Write what you do not do, and why.

Show the model answer

Output

                                              params  mean_test_score  std_test_score     se  rank_test_score
      {'model__C': 0.1, 'model__class_weight': None}           0.3391          0.0154 0.0069                2
{'model__C': 0.1, 'model__class_weight': 'balanced'}           0.3364          0.0157 0.0070                4
        {'model__C': 1, 'model__class_weight': None}           0.3393          0.0156 0.0070                1
  {'model__C': 1, 'model__class_weight': 'balanced'}           0.3365          0.0162 0.0072                3

Setting. The best is C = 1 without class weights (0.3393, standard error 0.0070). The limit is 0.3393 − 0.0070 = 0.3323, and all four settings are above it: a tie. Keep the simplest, the default model, which is the pipeline you already have. Its scores also stay close to the rates, which the balanced model’s do not.

Output

default  mean score 0.120  threshold 0.16  flagged 291 (24.7%)  recall 0.557
balanced mean score 0.415  threshold 0.59  flagged 287 (24.3%)  recall 0.552

Threshold. choose_threshold(scores, 0.25) gives 0.16: 291 of 1,179 May tickets flagged (24.7%), 102 of 183 escalations caught (recall 0.557), precision 0.351. Against 0.19 at 20%: 64 more reviews a month, and 16 more escalations caught. The balanced model needs a different threshold (0.59) for almost the same tickets. That confirms it: a threshold belongs to one model.

What you do not do. You do not run evaluate.py again for 25%, and you do not look at the test set to “check” 0.16. The test report stays the report for 0.19. If Grace adopts 25%, the honest evidence is the valid result above, plus the next labelled month (Module 8).

Module check

Check your understanding

Retrieval: name the job of the train set, the validation set and the test set in Module 6, with one number from each.

Show the answer

The train set fits the models and, with 5-fold stratified CV, chooses the features and the settings: CV AP 0.3393 for logistic regression with C = 1, within one standard error of the best (0.3424). The validation set (May) confirms the choices after the policy change and sets the threshold: 0.19 for the 20% capacity (227 flagged, recall 0.470). The test set (June) gives one final report and nothing else: AP 0.345, 19.6% flagged, recall 0.446.

Check your understanding

Explain or predict: in July, the warranty rate rises again, but the share of each team stays the same. The model and the threshold of 0.19 do not change. Predict the flag rate and the precision, and say which one you can check before the July labels arrive.

Show the answer

The flag rate stays about the same (near 19%). It depends only on the scores, and the scores depend only on the features, which do not change. Precision goes up, because more of the flagged tickets escalate. The model’s scores are now even more under-estimated. You can check the flag rate on the day the tickets arrive. Precision and recall need the labels, which come 72 hours or more later. That is why Module 8 monitors the flag rate first.

Check your understanding

Completion check: write the held-out report for Grace in four or five sentences. Use the test report, and say what you will not do with it.

Show the answer

A model answer: “The escalation model is a logistic regression with 11 features, chosen with 5-fold cross-validation on the tickets from July 2024 to April 2026. A gradient-boosting model was not better by more than one standard error. Its threshold, 0.19, was chosen on May 2026 for your capacity of 20%. On the June tickets, used once, it flags 19.6% of tickets and catches 44.6% of escalations (70 of 157), with precision 0.357. The priority-1 rule catches 35.0%. Its scores under-estimate the risk since the warranty policy (mean score 0.123 against a rate of 0.157 in June), so show the flags, not the scores, to agents. We will not change the threshold or the model because of these numbers. The next change uses newly labelled tickets.”

Module 6 recap

You can now tune and finish a model honestly:

• Select features with three questions. Known at prediction time, new information, a gain larger than the noise. prior_escalations_90d passed (CV AP 0.327 to 0.339); two leaks looked too good (0.445 and 0.851) and were never used.
• Tune hyperparameters with CV on train. The fit learns the parameters; you choose the hyperparameters before it. Use a grid search or a random search, then the one-standard-error rule. Logistic regression with C = 1 ties with a boosting model that was 0.0031 better.
• Choose the threshold on valid, for the decision. The lowest threshold within the capacity: 0.19, with 227 flagged and recall 0.470. predict() uses 0.5 and flags 2.1%.
• Separate ranking from probabilities. The ranking holds after the policy change; the scores, read as chances, do not.
• Look at the test set once. evaluate.py reported AP 0.345, 19.6% flagged, recall 0.446. Choosing on a set inflates its score (0.411 against 0.389 in the simulation).
• Write every experiment in the log, with its numbers and its reason, including the ideas that you rejected. Module 6 added E15 to E21; E15 is the model to keep.

Reset to a known state

If your project files or results differ from this module, go back to the course’s state. Run these commands in ~/projects/ticket-model, with .venv active.

macOS and Linux

Terminal

python get_data.py --reset
for f in ticket_model.py train.py evaluate.py; do curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/$f; done
python train.py
python evaluate.py --again

Windows

PowerShell

python get_data.py --reset
foreach ($f in "ticket_model.py", "train.py", "evaluate.py") { curl.exe -fsSL -O "https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/$f" }
python train.py
python evaluate.py --again

1. get_data.py --reset downloads the data again and checks every file. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
2. The download line replaces your three project files with the course’s files.
3. train.py must print Trained on 16939 tickets. Threshold 0.19 for a review capacity of 20%.
4. evaluate.py --again reproduces the test report: average precision 0.345, recall 0.446. It is allowed here because nothing is chosen with it.

Do not delete reports/test_report.json to run the test “fresh” after a change. A reset brings back the course’s model; it does not make the test set unseen again. In a notebook, a new session (or a new folder) starts from the downloaded files, so it needs no reset.

Recap

• The operating threshold is chosen on valid for the decision: the lowest threshold within the capacity. For 20%: 0.19, with 227 flagged and recall 0.470. predict() uses 0.5 and flags 2.1%.
• A threshold belongs to one model. A balanced model needs 0.59 for 25%; the default model needs 0.16.
• Calibration: on train, scores match the rates (Brier 0.0902). In May, the model under-estimates (mean 0.120, rate 0.155, Brier 0.1155), most of all for warranty tickets. The ranking still holds, so the capacity threshold still works; do not show the scores as chances without recalibration.
• evaluate.py opened the test set once, after every choice: AP 0.345, 19.6% flagged, recall 0.446 against the rule’s 0.350. Nothing changes after that look.
• Module 6 chose the features, the settings and the threshold with train and valid. The experiment log (E15 to E21) records each experiment and its reason.

Next, Module 7 looks inside the errors: which tickets the model misses, and for whom it works less well. It also asks when a simple model is the better choice.

Next: Slice analysis