Final assignment

What you will be able to do

Use every module of the course on one new decision, and deliver a classifier that another person can run, check and trust. The data is the same as in the lessons, but the team’s capacity is new, so you must make each choice again instead of copying it.

Before you startWhat to know and to have readyAbout 260 minutes

• You completed Modules 1 to 8. Each part below names the lessons that it uses.
• You do not need the case studies of Module 9 for this assignment. (The certificate asks for at least one of them.) To see a similar procedure on real data first, open Choose a model for hospital readmissions. It compares model families under one protocol and chooses a simple model with evidence.
• Your ticket-model project works: data/ from get_data.py, the virtual environment with requirements.txt installed, and ticket_model.py with the 11 features from Select useful features: Change ticket_model.py. If a file is broken, run python get_data.py --reset, or copy the file from the finished project folder, C05/project.
• Make a new folder in your project for this assignment: escalation/. Put every new script, file and report for the assignment in it.
• You can do Parts 1 to 6 in a notebook (local, Colab or Kaggle). Part 7 needs files and a terminal.
• Tested with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1 on macOS.

The decision: a bigger senior team

From July, Larkfield has two more senior agents. Grace says: “We can now review 1 in 4 new tickets (25%), not 1 in 5. I want the tickets that will escalate in that 25%.” Omar asks the question that decides the project: “Is a model worth it? A simple rule costs nothing to run.”

Priya asks you to answer with evidence. You build the classifier, compare it with simple baselines and two model families, and report the result on data that no choice has seen. You also keep a record of every experiment, and you explain why the final model is no more complex than it needs to be.

All the data is the synthetic Larkfield export. No real people.

Part 1: frame the problem

Uses Translate a need and Choose the learning task.

Write escalation/BRIEF.md, a one-page modelling brief. It must state:

1. The decision that a prediction changes, and who makes it.
2. The prediction unit, the target and the prediction time. Say exactly when the model runs and what is known then.
3. The features that are allowed, and the columns that are not, each with a one-sentence reason. Include priority_now and customer_escalation_rate.
4. The cost of each kind of error, in words.
5. The capacity (25%) and a measurable success target, for example “catch at least X% of escalations while flagging at most 25% of tickets”. Choose X before you train any model, and give the reason.

Part 2: baselines

Uses Define a baseline, Classification metrics and The precision–recall trade-off.

1. Start escalation/experiment_log.csv, an experiment log like the one of Machine learning is iterative, with the same columns. Add one row for each step below, as you do it.
2. On the validation set, compute the majority-class baseline: accuracy, precision and recall.
3. Compute the rule “flag priority 1”: the flag rate, precision and recall.
4. Write one more rule of your own that uses the new capacity better. Compute the same numbers. Check its flag rate against the capacity.
5. Write a table of the three baselines. Under it, write one sentence on why accuracy is the wrong metric for this decision.

Part 3: two model families, one procedure

Uses Preprocess with a pipeline, Machine learning is iterative, Decision trees, Control complexity, Ensembles, Train, validation and test sets: Cross-validation, Tune hyperparameters and When a simple model is better.

1. Choose two model families: logistic regression, and one tree ensemble (a random forest or gradient boosting).
2. Put each one in the same pipeline with build_pipeline(), so that the preprocessing is fitted on the train set only.
3. For each family, compare at least two settings with 5-fold stratified cross-validation on the train set (StratifiedKFold(5, shuffle=True, random_state=0), average precision).
4. Add one row to the experiment log for each setting: the mean and the standard deviation of the 5 folds, the decision and the reason. Give the standard error (the standard deviation ÷ √5) in the reason.
5. Choose one model. Use the one-standard-error rule. Do not use the validation set or the test set for this step.
6. Justify the simplicity of your choice in three to five sentences, under the heading Why this model. Answer these questions. Is the difference between the two families larger than one standard error? Who must understand or check the scores (Grace’s team, an auditor)? What would the more complex model cost: time, explanation, debugging, monitoring? If you chose the more complex model, show that its gain is larger than the noise and worth those costs.
7. Fit the chosen model on the train set. Write its average precision and ROC AUC on the validation set.

One change at a time

Use the 11 features of ticket_model.py. If you want to add a feature, add it in a separate step, decide with cross-validation on the train set, and show that it is known when the ticket is created. Give it its own row in the experiment log.

Part 4: the threshold and the calibration

Uses Thresholds and calibration.

1. On the validation set, choose the threshold for the 25% capacity: the lowest threshold, in steps of 0.01, that flags at most 25% of the tickets. Add it to the experiment log.
2. Write the confusion matrix, the flag rate, precision and recall at that threshold.
3. Compare them with the baselines of Part 2. Does the model meet your success target from Part 1?
4. Compare the mean score with the observed escalation rate on the validation set. Write one sentence about what the difference means for people who read the scores.

Part 5: the test result, once

Uses Thresholds and calibration: Test once with evaluate.py.

1. Freeze your choices: the model, its settings, the features and the threshold. Write them in your report before you look at the test set.
2. Run the evaluation on the test set once, with the threshold from Part 4.
3. Write the test confusion matrix, flag rate, precision, recall, average precision and ROC AUC, and the rule “flag priority 1” on the test set.
4. Compare the test results with the validation results. Explain any difference. Do not change the model after you look.

The test result is not a row of the experiment log. The log records choices, and the test set must not make any.

Part 6: errors and limits

Uses Slice analysis, Explain predictions and Unsupervised exploration.

1. On the validation set, compare recall and the flag rate across at least two slices (for example team and channel). Show the number of tickets next to every rate.
2. Look at ten missed escalations (false negatives) with the lowest scores. Write what they have in common, if anything.
3. Write the three features that the model relies on most (permutation importance), and one sentence on why this does not show what causes an escalation.
4. Name at least one data problem and one model limitation, each with the evidence that shows it.

Part 7: hand it over

Uses Package artifacts, Document use boundaries and Plan production checks.

1. Change train.py (in escalation/, or a copy of it) for your model and the 25% capacity. Run it. It saves the pipeline, model_info.json and the check sample.
2. In a new terminal (a fresh process), run python predict.py --check. It must report identical scores.
3. Score data/july_tickets.csv with predict.py. Write the flag rate for 1–14 July and for 15–31 July, and the warnings that predict.py prints.
4. Write escalation/MODEL_CARD.md for your model, with the intended use, the uses it is not for, why this model, the evaluation and the limits.
5. Write a retraining trigger in one or two sentences, with the numbers that you would watch.

What to submit

A folder escalation/ with:

escalation/
  BRIEF.md            Part 1
  experiment_log.csv  Parts 2 to 4: one row for each baseline, setting, feature change and threshold
  REPORT.md           Parts 2 to 6, and the commands to reproduce every number
  MODEL_CARD.md       Part 7
  train.py            and any other script or notebook that you used
  model/              escalation_model.joblib, model_info.json, check_sample.csv
  requirements.txt    exact versions

experiment_log.csv has the columns of The experiment log: id, date, change, cv_ap, cv_std, valid_ap, decision, reason. The decision is baseline, keep or reject. Write each row when you make the change, not at the end. Leave a number empty when it does not apply. A threshold has no CV score. For a rule, compute it with rule_scores() as in Machine learning is iterative, or leave it empty. valid_ap is only for a model that you check once on the validation set. Write other validation numbers, such as the flag rate and the recall, in the reason.

REPORT.md must contain these sections:

# Ticket escalation classifier: evaluation report

## Decision, target and prediction time      (short; the full text is in BRIEF.md)
## Baselines                                  (Part 2: table, and why not accuracy)
## Model selection                            (Part 3: CV table, the choice and the reason)
## Why this model                             (Part 3: three to five sentences on simplicity)
## Threshold for 25% capacity                 (Part 4: threshold, confusion matrix, comparison)
## Calibration                                (Part 4)
## Frozen choices                             (Part 5: written before the test)
## Test result                                (Part 5: once; comparison with validation)
## Errors and limits                          (Part 6: slices with counts, data problem, model limitation)
## July check and retraining trigger          (Part 7)
## How to reproduce                           (commands, versions, the fresh-process check)

The experiment log and Why this model are required, not extra. They add little work: the log records numbers that Parts 2 to 4 compute anyway, and Why this model replaces a short explanation that Part 3 asks for. But they show two things that a single score cannot. You tried, measured and decided in a fair order. And the final model is as simple as the evidence allows.

Check your work

There is no reviewer for this course yet. Review your work with this table, or ask a colleague to use it. Mark each item as demonstrated, partially demonstrated or not yet demonstrated, with a reason. The course is complete when every item is demonstrated. You can correct an item and review it again as often as you need.

Required evidence	Demonstrated when
Explicit prediction timing and target	The brief names the decision, the prediction unit, the target escalated_72h and the prediction time (ticket creation). Every feature is known at that time. priority_now and customer_escalation_rate are excluded, each with a correct reason.
Baseline comparison	The majority class, the priority-1 rule and your own rule are compared with the model on the same validation set, with flag rate, precision and recall. The report says whether the model is worth its cost, in Omar’s terms.
A recorded, honest model choice	The experiment log has a row for each baseline, each CV setting, any feature change and the threshold, each with its decision and reason. Why this model compares the difference between the families with one standard error. It also gives at least one reason that is not a score: who must understand the scores, cost, debugging or monitoring.
Leakage-safe validation and threshold selection	Preprocessing is fitted on the train set only (inside the pipeline). Settings and the model are chosen with cross-validation on the train set. The threshold is chosen on the validation set for the 25% capacity. The choices are frozen before the test, and the test is used once. predict.py --check passes in a fresh process.
Error analysis and limitations	Slices show counts next to rates, and small slices are treated with care. At least one data problem and one model limitation are named, each with evidence. Importance is not described as a cause. The model card states uses that the model is not for.

A model with a high score and one leak is not complete. For example, if a feature is computed from later tickets, the leakage item is not yet demonstrated, even when every other part is correct. In the same way, a result that you got after you changed the model because of the test set is not a held-out result.

Show the key numbers, to check your results

Open this only after you have your own numbers. Your rule and your second family can differ from these; then compare the method, not the numbers.

• Baselines (valid, 1,179 tickets, 183 escalated): never escalate: accuracy 0.845, recall 0. Priority 1: 163 flagged (13.8%), precision 0.374, recall 0.333. “Priority 1 or team payment”: 317 flagged (26.9%, over the capacity), precision 0.271, recall 0.470.
• Cross-validation (train, average precision): logistic regression 0.3393 ± 0.0156 (standard error 0.0070); gradient boosting with depth 2 and learning rate 0.1: 0.3424 ± 0.0178 (standard error 0.0080). The difference, 0.003, is smaller than one standard error, so the simpler logistic regression is a good choice. If you chose boosting with a clear reason, that is also correct.
• Threshold for 25% (valid): logistic regression: 0.16, 291 flagged (24.7%), TP 102, FP 189, FN 81, TN 807, precision 0.351, recall 0.557. Boosting (depth 2, rate 0.1): valid average precision 0.395, ROC AUC 0.754; at 0.16, 275 flagged (23.3%), precision 0.364, recall 0.546.
• Calibration (valid): mean score 0.120, observed rate 0.155. The scores are too low after the policy change of May, so they are not “the chance” of escalation.
• Test, once (logistic, 0.16; 1,002 tickets, 157 escalated): ROC AUC 0.749, average precision 0.345; 252 flagged (25.1%), TP 85, FP 167, FN 72, TN 678, precision 0.337, recall 0.541. Priority 1 on test: 150 flagged (15.0%), precision 0.367, recall 0.350. The flag rate is a little over 25% on new data: a threshold chosen on one month does not guarantee the capacity on the next. Say so in the report, and watch it (Part 7).
• July (logistic at 0.16): compute it with predict.py. Expect warnings for the social channel from 15 July, and a higher flag rate in the second half of the month.

A short example of the experiment log (the first rows; yours has your own dates and your own rule):

id	date	change	cv_ap	cv_std	valid_ap	decision	reason
E01	2026-08-01	Rule: flag priority 1				baseline	The rule to beat. Valid: 13.8% flagged, recall 0.333.
E02	2026-08-01	Rule: priority 1 or team payment				reject	Over the 25% capacity: 26.9% flagged on valid (recall 0.470).
E03	2026-08-02	Logistic regression, C = 1	0.339	0.016	0.391	keep	First model family; standard error 0.007.
E04	2026-08-02	Boosting, depth 2, learning rate 0.1	0.342	0.018		reject	0.003 higher: less than one standard error (0.008).
E05	2026-08-03	Threshold for 25% on valid				keep	0.16: 24.7% flagged, recall 0.557. The lowest threshold within the capacity.

Recap

You can now take a business need to a model that someone else can run. You frame the decision and its timing, and beat simple baselines with evidence. You try and record model families, and keep the model as simple as the evidence allows. You choose a threshold without leaking the test, explain where the model fails, and hand it over with a model card and production checks. The next courses build on this: Deep Learning and Transformers Explained teaches deep learning, and Deploying AI Models for Real Users deploys a model for real users.