Define a baseline

What you will be able to do

Measure the simplest predictions before you train any model, and write down the result that a model must reach. Without a baseline, a score of 84.5% sounds good, even when the model catches nothing.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes every baseline on this page, first on ten tickets and then on the 1,179 May tickets. You also test one more rule and fill in the modelling brief. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• The decision, the capacity of 20% and the two kinds of error. Recall and precision in words. Both from Translate a need: From the decision to the metric.
• Classification, and the answers that need no model (a rule), from Choose the learning task.
• Virtual environments and requirements.txt, from Isolate dependencies in Python for Practical AI Engineering. Notebooks and their kernel, from Notebook discipline in the Python course. You need them only for the local project.

What you need on your computer (only for the local route; the notebook needs none of it)

• Python 3.12 or later, from Install and verify Python in the Python course.
• VS Code, from Install and verify Python: Step 3: get a code editor.
• An internet connection, and about 500 MB of free disk space: the environment uses about 450 MB on macOS, and the data 2.5 MB. No account.
• This lesson makes the project folder ~/projects/ticket-model, in Set up your project. Every later lesson of the course can use it.

Tested on macOS. The setup used Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1, matplotlib 3.11.2 and ipykernel 7.4.0. The Windows and Linux commands are not tested. The notebook was tested in Jupyter.

Before you buy a weather app

Imagine a weather app that costs money and promises: “Our forecast for tomorrow is right 70% of the time.” Is that good? You cannot tell yet. Try the simplest forecast that anyone can make for free: “tomorrow will be like today”. In many places, that free forecast is right on most days, because the weather changes slowly. If it is right 65% of the time, the expensive app adds only a little. If it is right 80% of the time, the app is worse than doing nothing clever at all.

The same question comes before every model. A number such as 70%, or 84.5%, means nothing alone. It means something only next to what you get with no model at all. So before you train anything, you measure the simplest predictions on the same data and with the same measures. These simple predictions are the baselines, and every model in this course is compared with them.

What a baseline is

A baseline is the simplest prediction that anyone can make without a model. It needs no learning, or only one number learned from the train set. You measure it first, on the same data and with the same measures as the model. Then every model result has a reference: a model is useful only if it does clearly better than its baselines. A baseline that is hard to beat is also useful news, because then a simple rule may be the right answer.

For Larkfield’s yes/no question, two baselines make a flag for each ticket:

• Majority class: always predict the most common answer. Most tickets do not escalate, so this baseline never flags. It is the “tomorrow will be like today” of classification: it costs nothing and is often right.
• A rule: flag the tickets with priority 1. It uses one fact that the support team already trusts.

Ten tickets, counted by hand

Here are ten made-up tickets. Two of them escalated: T-1 and T-5.

Ticket	Priority	Team	Escalated
T-1	1	payment	1
T-2	3	delivery	0
T-3	2	returns	0
T-4	1	delivery	0
T-5	2	warranty	1
T-6	3	account	0
T-7	2	delivery	0
T-8	3	returns	0
T-9	1	account	0
T-10	2	payment	0

The majority class never flags, so it lets both escalations through. It is right on the 8 tickets that did not escalate, so it is right 8 times out of 10. The rule flags T-1, T-4 and T-9. T-1 escalated, so the rule caught it. T-4 and T-9 did not escalate, so they are two false alarms: unnecessary reviews. T-5 escalated without a flag, so the rule missed it. The other 6 tickets were correctly left alone. So the rule is right 7 times out of 10.

From these four counts come three shares, which you met in words in the last lesson:

• Recall: how many of the escalations did we catch? The rule caught 1 of the 2, so its recall is 0.5. The majority class caught none: 0.
• Precision: how many of the flagged tickets really escalated? 1 of the rule’s 3 flags did: 0.333. The majority class flags nothing, so it has no precision.
• Accuracy: on how many tickets was the flag (or the absence of a flag) right? 8 of 10 for the majority class, 7 of 10 for the rule.

The table puts the counts side by side:

	Majority: never flag	Rule: priority 1
Caught (flagged, escalated)	0	1 (T-1)
False alarms (flagged, did not escalate)	0	2 (T-4, T-9)
Missed (not flagged, escalated)	2 (T-1, T-5)	1 (T-5)
Correctly left (not flagged, did not escalate)	8	6
Recall: caught ÷ escalations	0 ÷ 2 = 0	1 ÷ 2 = 0.5
Precision: caught ÷ flagged	none flagged	1 ÷ 3 = 0.333
Accuracy: right ÷ all tickets	8 ÷ 10 = 0.8	7 ÷ 10 = 0.7

Look at the last row. The majority class never helps Grace, and it has the higher accuracy. The rule catches one of the two escalations, and it has the lower accuracy. Keep this in mind for the real data. Classification metrics gives these four counts their standard names and explains why accuracy behaves like this.

A constant score

The models of this course do not give a plain yes or no. They give each ticket a score from 0 to 1, where a higher score means “more likely to escalate”. The senior team then reviews the tickets with the highest scores. So a score needs its own baseline: the constant score, which gives every ticket the same score. A natural choice is the escalation rate of the train set.

Two measures judge a score before any cut into yes and no. The precision–recall trade-off explains both. For now, one sentence each is enough:

• ROC AUC: the chance that a random escalated ticket gets a higher score than a random other ticket. 0.5 is guessing, and 1.0 is a perfect ranking.
• Average precision (AP): how good the precision is as you go down the list of tickets, from the highest score. For a score that cannot rank the tickets at all, it equals the escalation rate.

On the ten tickets, the constant score gives every ticket 0.2, their escalation rate. With the same score for every ticket, no escalated ticket can be above another ticket. So ROC AUC is 0.5, and AP is 0.2, the rate. That is the floor for every model: ROC AUC above 0.5, and AP above the escalation rate.

Worked example: Priya’s baselines for May

Priya computes the baselines on the validation set, the 1,179 May tickets. (Train, validation and test sets explains why a result is measured on tickets that played no part in learning.) Each baseline learns only from train: the most common class and the escalation rate come from the train tickets, and the rule needs no data. Each code block is a cell of the lesson notebook, where the setup cells have already loaded pandas as pd and numpy as np.

First, she writes a function that counts the four kinds of result and computes the shares. It is the hand count above, in code:

def flag_report(y, flags):
    """Compare 0/1 flags with the real 0/1 answers: four counts and four shares."""
    y = np.asarray(y, dtype=bool)
    flags = np.asarray(flags, dtype=bool)
    caught, false_alarms = int((flags & y).sum()), int((flags & ~y).sum())
    missed, correctly_left = int((~flags & y).sum()), int((~flags & ~y).sum())
    return {"flagged": caught + false_alarms, "share flagged": round((caught + false_alarms) / len(y), 3),
            "caught": caught, "false alarms": false_alarms, "missed": missed, "correctly left": correctly_left,
            "precision": round(caught / (caught + false_alarms), 3) if caught + false_alarms else None,
            "recall": round(caught / (caught + missed), 3), "accuracy": round((caught + correctly_left) / len(y), 3)}

On the ten tickets, it gives the numbers of the table above. Then she loads the data. read() is the same function as in Translate a need:

TARGET = "escalated_72h"
train = read("data/train.csv")
valid = read("data/valid.csv")
majority_class = train[TARGET].mode()[0]
train_rate = round(train[TARGET].mean(), 3)
print("Most common class in train:", majority_class, "  train escalation rate:", train_rate)

The output:

Most common class in train: 0   train escalation rate: 0.117

Before she runs the next cell, Priya asks Omar to predict which baseline has the higher accuracy. He says: “The rule, of course. It finds escalations.”

y = valid[TARGET]
baselines = pd.DataFrame([flag_report(y, [majority_class] * len(valid)),
                          flag_report(y, valid["priority"] == 1)],
                         index=["majority: never escalate", "rule: flag priority 1"])
baselines

The output:

                          flagged  share flagged  caught  false alarms  missed  correctly left  precision  recall  accuracy
majority: never escalate        0          0.000       0             0     183             996        NaN   0.000     0.845
rule: flag priority 1         163          0.138      61           102     122             894      0.374   0.333     0.810

Read it row by row:

• Never escalate is right on 996 of the 1,179 tickets: an accuracy of 84.5%. It catches none of the 183 escalations. Its precision is NaN (not a number), because it flags nothing, and 0 ÷ 0 has no value.
• Flag priority 1 flags 163 tickets, 13.8%, which is inside the capacity of 20%. Of its flags, 61 escalated, so its precision is 0.374. It catches 61 of the 183 escalations, so its recall is 0.333: one in three. Its accuracy, 81.0%, is lower than the useless baseline’s.

Omar was wrong, as most people are the first time. The failure case below shows the trap, and Classification metrics explains it in full. Last, Priya tries the constant score, with the train rate 0.117 for every May ticket. The two metric functions come from sklearn.metrics; the notebook imports them in section 2:

constant_scores = [train_rate] * len(valid)
print("ROC AUC:", roc_auc_score(y, constant_scores))
print("Average precision:", round(average_precision_score(y, constant_scores), 3))
print("Escalation rate in valid:", round(y.mean(), 3))

The output:

ROC AUC: 0.5
Average precision: 0.155
Escalation rate in valid: 0.155

So a model’s ROC AUC must be above 0.5, and its AP above 0.155, on May. The three baselines, together:

Baseline (on valid, May 2026)	Flagged	Precision	Recall	Accuracy
Majority class: never escalate	0 (0%)	—	0.000	0.845
Rule: flag priority 1	163 (13.8%)	0.374	0.333	0.810
Constant score 0.117 (the train rate)	—	ROC AUC 0.5; AP 0.155 (the rate)

The improvement target

A baseline says where you start. An improvement target says what result makes a model worth using. Write it in the terms of the decision, and write it before you see any model’s score. If you write it afterwards, it is too easy to move the target until it fits the result.

Grace’s decision gives two conditions. The senior team can review at most 20% of new tickets. Inside that limit, the rule already catches 33% of the escalations. A model must catch clearly more, or the rule is enough. Grace and Priya agree on this target:

Flag at most 20% of new tickets (the capacity), and catch at least 40% of escalations (the rule catches 33%).

In the words of the last lesson: the share flagged must be at most 20%, and the recall must be at least 40%. Neither baseline meets the target:

	Share flagged (at most 20%)	Recall (at least 40%)	Meets the target
Never escalate	0%	0.000	No
Flag priority 1	13.8%	0.333	No: it catches too few

Every model in this course must meet this target. You check it on the validation set while you work, and on the test set once, at the end. In the notebook, the function meets_target() checks it:

def meets_target(report):
    """True if the flags fit the capacity (at most 20%) and catch at least 40%."""
    return report["share flagged"] <= 0.20 and report["recall"] >= 0.40


rule = flag_report(y, valid["priority"] == 1)
print("Rule meets the target:", meets_target(rule))

The output:

Rule meets the target: False

Set up your project

You can do the whole course in the lesson notebooks, in Colab or Kaggle, with no installation. In Module 8, you also need files and commands on your computer, because a model handoff is files and commands. This section makes that local project now, so that it is ready. It takes about 15 minutes. If you use only the notebooks for now, go on to Guided practice, and come back before Module 8.

At the end, you have this folder:

ticket-model/
  .venv/              the virtual environment
  requirements.txt    the packages and their versions
  get_data.py         downloads and checks the data
  data/               train.csv valid.csv test.csv and five more files

Check Python

Open a terminal, and check the Python version. You need 3.12 or later.

macOS

python3 --version

Windows

python --version

Linux

python3 --version

You should see Python 3.12 or a higher number, for example Python 3.14.6. If you see Python 3.11 or lower, or “command not found”, install Python as in Install and verify Python.

Make the project folder

1. Make the folder ~/projects/ticket-model, and go into it:

   macOS

   mkdir -p ~/projects/ticket-model
   cd ~/projects/ticket-model

   Windows

   mkdir ~\projects\ticket-model
   cd ~\projects\ticket-model

   If the folder already exists, mkdir shows an error. You can ignore it.

   Linux

   mkdir -p ~/projects/ticket-model
   cd ~/projects/ticket-model

2. Check where you are. Run pwd. The path ends with projects/ticket-model (on Windows, projects\ticket-model).

Run every command in the rest of the course in this folder.

Make the environment

A virtual environment keeps this project’s packages apart from every other project, as in Isolate dependencies in the Python course.

1. Make the environment:

   macOS

   python3 -m venv .venv

   Windows

   python -m venv .venv

   Linux

   python3 -m venv .venv

2. Activate it:

   macOS

   source .venv/bin/activate

   Windows

   .venv\Scripts\Activate.ps1

   If PowerShell says that running scripts is disabled, run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser once. Then activate again. Isolate dependencies in the Python course explains this error.

   Linux

   source .venv/bin/activate

   If the output says that ensurepip is not available, run sudo apt install python3-venv first. Then make the environment again.

3. Check that the prompt starts with (.venv). From now on, type python on every system.

Install the packages

1. In VS Code, open the folder ticket-model (File › Open Folder).

2. Make a new file requirements.txt in it, with these four lines:

   requirements.txt

   pandas==3.0.6
   scikit-learn==1.9.1
   matplotlib==3.11.2
   ipykernel

3. Save the file.

4. In the terminal, install the packages:

   python -m pip install -r requirements.txt

   It takes one or two minutes. The last line starts with Successfully installed, and the list includes pandas-3.0.6 and scikit-learn-1.9.1. If pip also says that a new version of pip is available, you can ignore it.

5. Check the versions:

   python -c "import pandas, sklearn; print(pandas.__version__, sklearn.__version__)"

   You should see 3.0.6 1.9.1.

Why each line: pandas reads and changes tables, and scikit-learn has the models and the metrics. matplotlib draws the charts in the notebooks. ipykernel lets VS Code run a notebook with this environment’s Python. The first three have an exact version (==), so your numbers match the lessons. In Module 8, you add two more exact versions, because a saved model needs them.

Download the data

Mei’s script get_data.py downloads the eight data files and checks each file’s checksum. It uses only Python’s standard library.

1. Download the script into the project folder:

   macOS

   curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/get_data.py

   Windows

   curl.exe -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/get_data.py

   Type curl.exe, not curl. In Windows PowerShell, curl is a different command.

   Linux

   curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/get_data.py

   If the output is curl: command not found, run sudo apt install curl (or sudo dnf install curl) first.

2. Run it:

   python get_data.py

   The output:

   Downloaded train.csv
   Downloaded valid.csv
   Downloaded test.csv
   Downloaded split_manifest.json
   Downloaded extra_features.csv
   Downloaded daily_tickets.csv
   Downloaded july_tickets.csv
   Downloaded july_labels.csv
   Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).

3. Run it a second time. Now it prints only the Ready: line, because every file is there and correct.

The last line is the check. Train has 16,939 tickets with 1,983 escalations. May has 1,179 with 183, and June has 1,002 with 157. If a file is broken or different, get_data.py downloads that file again.

Open a notebook in VS Code

The lesson notebooks also run on your computer, with the environment that you just made.

1. In VS Code, install the Python and Jupyter extensions from Microsoft (View › Extensions, then search for each name).
2. Make a folder notebooks in the project.
3. Download a lesson notebook into it. Use View on GitHub in the notebook box of a lesson, then the download button.
4. Open the notebook in VS Code.
5. Select Select Kernel › Python Environments, then the entry with .venv.
6. Run the cells from the top, as in Colab.

The setup cells of a lesson notebook work in their own folder ticket-model, next to the notebook, and download their own copy of the data there. This keeps each notebook self-contained, and it never changes your project’s data/.

To check the environment in a notebook, make a new notebook with File › New File, then Jupyter Notebook. Save it in the project folder, select the .venv kernel, and run this cell:

import pandas as pd, sklearn
print("pandas", pd.__version__, "  scikit-learn", sklearn.__version__)
train = pd.read_csv("data/train.csv")
print(len(train), "train tickets,", train["escalated_72h"].sum(), "escalated")

The output:

pandas 3.0.6   scikit-learn 1.9.1
16939 train tickets, 1983 escalated

If you see ModuleNotFoundError: No module named 'sklearn', the notebook uses another Python. Select the .venv kernel again (step 5). If you see FileNotFoundError for data/train.csv, the notebook is not in the project folder: save it in ticket-model, next to data/.

Or use Colab or Kaggle

You do not need any of the above for the lesson notebooks. In the notebook box of a lesson, select Open in Colab (a free Google account) or Open in Kaggle (a free Kaggle account). The notebook’s first cells install the same versions of pandas and scikit-learn and download the data. In Kaggle, turn on Internet in the notebook settings first; Kaggle asks for a verified phone number for this.

Reset the project

To get back to a known state:

• Data: run python get_data.py --reset. It deletes data/ and downloads every file again. The last line is the Ready: line above.
• Environment: close the terminal, delete the folder .venv, and do Make the environment and Install the packages again. requirements.txt stays.
• A project file from a later lesson: copy the course’s finished file from C05/project. Its README says which lesson writes each file.

Guided practice: the baselines in the notebook

1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Or open it in VS Code with the .venv kernel, as in Open a notebook in VS Code.
2. Run the setup cells. The data cell ends with the Ready: line.
3. In section 1, work out the accuracy of both baselines on the ten tickets by hand. Then run the cell. You see 0.8 for the majority and 0.7 for the rule.
4. In section 2, run the constant score on the ten tickets. You see ROC AUC: 0.5 and Average precision: 0.2.
5. In section 3, write your prediction, then run the real baselines. You see accuracy 0.845 and recall 0.000 for the majority, and 163 flagged with recall 0.333 for the rule.
6. Run the constant score on May. You see Average precision: 0.155.
7. In section 4, run meets_target(). You see Rule meets the target: False.

Your turn: Grace’s rule

Change one thing: the rule. Grace says: “Since the new warranty policy of 1 May, warranty tickets escalate more. Flag priority 1 or warranty tickets.” Before you run it, predict: will it catch more escalations than the priority-1 rule? Will it fit the capacity? In the notebook, section 5, make grace_flags, compute its flag_report, and run the check cell.

Show the answer

grace_flags = (valid["priority"] == 1) | (valid["team"] == "warranty")
grace = flag_report(y, grace_flags)
print(grace)
print("Meets the target:", meets_target(grace))

{'flagged': 312, 'share flagged': 0.265, 'caught': 98, 'false alarms': 214, 'missed': 85, 'correctly left': 782, 'precision': 0.314, 'recall': 0.536, 'accuracy': 0.746}
Meets the target: False

Grace’s rule catches 98 of 183 escalations: a recall of 0.536, more than the target of 40%. But it flags 312 tickets, 26.5%, more than the senior team can review. So it does not meet the target. A rule flags what it flags, and you cannot make it flag exactly 20%. A model gives a score, and you can cut a score at any level. The precision–recall trade-off shows how, and Thresholds and calibration chooses the cut that fits the 20%.

The notebook has an optional cell for your own rule. Try one or two at most. Each rule that you test on the May tickets, and keep because it scored well, makes the May numbers a little too good. Train, validation and test sets explains why.

Common mistake

Accuracy picks the useless baseline

Symptom: Tomás compares the three rules by accuracy, “the standard measure”, and recommends the first one:

                        flagged  recall  accuracy
never escalate                0   0.000     0.845
priority 1                  163   0.333     0.810
priority 1 or warranty      312   0.536     0.746

Accuracy of three rules on the 1,179 May tickets. The rule that never flags is first.

Text description of the chart

A column chart with three columns on an axis from 0 to 100%. Never escalate: 84.5%. Flag priority 1: 81.0%. Flag priority 1 or warranty: 74.6%. The rule that flags nothing has the highest accuracy.

Escalations caught by the same three rules. The order is the opposite.

Text description of the chart

A column chart of the escalations that each rule flags, out of 183 in May. Never escalate catches 0. Flag priority 1 catches 61. Flag priority 1 or warranty catches 98. The order is the opposite of the order by accuracy.

Cause: 84.5% of the May tickets did not escalate. A rule that always says “no” is right on all of them. Accuracy counts every correct “no”, but Grace’s decision is about the “yes”. When one answer is rare, accuracy mostly measures the common answer. Classification metrics explains this trap in full.

Fix: choose the measures from the decision, as in Translate a need. For Larkfield: the share flagged against the capacity, and the recall and precision of the flags; for scores, AP and ROC AUC. Always show the baselines next to a result.

Consequence in AI work: a team that optimises accuracy here ends with a model that flags almost nothing. It reports a high number, nobody sees an error, and the escalations wait in the normal queue as before. This is the mistake of “optimising a metric that does not match the decision”. It comes back in Module 3.

Knowledge check

Check your understanding

A colleague reports: our escalation model has 84% accuracy on May. What do you ask before you believe that it is useful?

Show the answer

Ask for the baselines and for the measures of the decision. On May, a rule that never flags has 84.5% accuracy and catches nothing, so 84% may be worse than doing nothing. Ask what share of tickets the model flags (at most 20%). Ask how many of the escalations it catches (its recall, at least 40%) and how many of its flags are right (its precision). Compare them with the rule “flag priority 1”: 13.8% flagged, recall 0.333, precision 0.374.

Check your understanding

Grace's rule catches 53.6% of the May escalations. Why does it not meet the improvement target? Why can a model with a score meet it?

Show the answer

It flags 26.5% of the tickets, more than the 20% that the senior team can review. The target has two conditions, and the rule breaks the capacity condition. A rule is fixed: it flags what matches. A model gives each ticket a score, so you can choose the cut that flags exactly the capacity. Then the question is how many escalations are in that top 20%.

Module practice

The module practice asks you to write a one-page modelling brief for ticket escalation, before any model exists. A brief puts every decision of this module on one page: the decision, the target, the data, the costs, the baselines and the improvement target. Grace, Omar and Tomás can then read it, and disagree, before anyone spends time on a model. A disagreement on one page costs minutes; the same disagreement after two weeks of modelling costs the two weeks.

Worked example: the template

Copy the template into a file reports/modelling-brief.md in your project, or into the last Markdown cell of the notebook. Each part needs one or two sentences, with the numbers of this module.

reports/modelling-brief.md

# Modelling brief: ticket escalation

| Part | Answer |
|---|---|
| Decision and owner | Who acts on the prediction, and what do they do? |
| Prediction unit | One prediction is about one … |
| Prediction time | When must the prediction exist? |
| Target | The exact definition, with its time window. |
| Features | The columns known at the prediction time. |
| Never features | The columns written after the prediction time. |
| Data and split | Which rows, which split, how many in each part. |
| Cost of a missed escalation | What happens to the customer? |
| Cost of an unnecessary review | What does it cost the team? |
| Capacity | How many flags can the team handle? |
| Baselines (on valid) | The numbers of the simple predictions. |
| Measures | Which measures, and why not accuracy. |
| Improvement target | The result that makes a model worth using. |
| Why not only a rule | What the best rule cannot do. |
| Risks | What can make the model worse in real use. |

Your task: the brief for ticket escalation

Fill in every part for Larkfield’s escalation model. Use the numbers from Translate a need and from this lesson. Then compare your brief with Priya’s. Yours can use other words; check that it has the same decisions and the same numbers.

Show Priya’s brief

Part	Answer
Decision and owner	Grace (support lead) owns it. Her team sends a flagged ticket straight to the senior team, instead of the normal queue.
Prediction unit	One ticket.
Prediction time	When the ticket is created.
Target	escalated_72h: 1 if a senior agent took the ticket within 72 hours after created_at, otherwise 0.
Features	The 10 columns from the SQL course: channel, team, priority, segment, region, order_value, word_count, customer_tenure_days, prior_tickets_90d, created_hour. To test in Module 6: prior_escalations_90d (known at creation).
Never features	priority_now (the escalation sets it), customer_escalation_rate (uses later targets), and every value written after creation: the first reply, the closing time, outcome events.
Data and split	The full export, split by time as in the SQL course. Train: 16,939 tickets, July 2024 to April 2026, 11.7% escalated. Valid: 1,179, May 2026, 15.5%. Test: 1,002, 1 to 27 June 2026, 15.7%. The test set is used once, at the end.
Cost of a missed escalation	The customer waits in the normal queue, often a day more. Some customers leave Larkfield.
Cost of an unnecessary review	About 10 minutes of a senior agent, and one review place that a real escalation could have had.
Capacity	At most 20% of new tickets: about 7.6 tickets a day in May.
Baselines (on valid)	Never escalate: accuracy 0.845, recall 0. Constant score 0.117: ROC AUC 0.5, AP 0.155. Flag priority 1: 13.8% flagged, precision 0.374, recall 0.333.
Measures	The share flagged (against the capacity), and recall and precision at that share. For scores: AP and ROC AUC. Not accuracy: never escalating has the highest accuracy.
Improvement target	Flag at most 20% of new tickets and catch at least 40% of escalations. Check on valid; confirm once on test.
Why not only a rule	Priority 1 catches 33%. Priority 1 or warranty catches 53.6%, but flags 26.5%, above the capacity. A rule cannot be cut at exactly 20%; a score can.
Risks	The escalation rate rose after the warranty policy of 1 May 2026 (11.7% in train, 15.5% in May). A model from older tickets may give scores that are too low. Guests have no history. Some customers tend to escalate, and no column shows it.

A complete brief has three things that are easy to miss. The target has its time window (72 hours). The capacity is a number. The baselines are numbers, not only names.

Module check

Check your understanding

Retrieval: name the three baselines of this module and give one number for each on the May tickets.

Show the answer

• Majority class, never escalate: accuracy 0.845, recall 0. It catches none of the 183 escalations.
• Constant score, the train rate 0.117 for every ticket: ROC AUC 0.5, average precision 0.155 (the May escalation rate).
• Rule, flag priority 1: 163 tickets flagged (13.8%), precision 0.374, recall 0.333.

Check your understanding

Predict and explain: a first model flags 19% of the May tickets with a recall of 0.38 and a precision of 0.37. Does it meet the improvement target? Is it better than the rule flag priority 1?

Show the answer

It does not meet the target. It fits the capacity (19% is at most 20%), but its recall, 0.38, is below 0.40. It is a little better than the rule on recall (0.38 against 0.333). But it also flags more tickets (19% against 13.8%), with about the same precision. So the gain is small, and part of it comes only from flagging more. Priya would not use it yet. She would look for a model that catches at least 40% within the 20%.

Check your understanding

Completion check: explain to Omar what action a prediction enables at Larkfield and what it costs to be wrong in each direction. Use the capacity and the numbers of this module.

Show the answer

The action: when a ticket is created, a flag sends it straight to the senior team. A customer with a serious problem then gets a senior agent on the first day, instead of waiting in the normal queue.

The cost of a wrong “no” (a missed escalation): the customer waits, often a day more, and some customers leave Larkfield. The rule “flag priority 1” misses 122 of the 183 May escalations.

The cost of a wrong “yes” (an unnecessary review): about 10 minutes of a senior agent. It also uses a review place that a real escalation could have had. The senior team can review at most 20% of new tickets, about 7.6 a day in May. So unnecessary reviews push real escalations out.

What it means for the model: a miss costs more than a review, but the capacity is fixed. So the model must catch as many escalations as possible within 20% flagged: at least 40%, against 33% for the rule. Accuracy does not show this, because never escalating has the best accuracy (84.5%).

A complete answer names the action, both errors with their costs, and the capacity, and links them to the measures.

Recap

In this module, you turned Grace’s need into a prediction problem that a model can be measured against:

• The problem: a flagged ticket goes to the senior team, at most 20% of new tickets. One ticket, at creation, target escalated_72h, with the 10 features from the SQL course. Never priority_now or customer_escalation_rate.
• The baselines on May: never escalating is 84.5% accurate and catches nothing. A constant score has ROC AUC 0.5 and AP 0.155. The rule “flag priority 1” flags 13.8%, with precision 0.374 and recall 0.333.
• The improvement target: flag at most 20% and catch at least 40% of escalations.
• The project: ~/projects/ticket-model with .venv, requirements.txt, get_data.py and data/. To reset it, run python get_data.py --reset, or make .venv again, as in Reset the project.

Next, in Module 3, you learn how a model is judged before you build one. The first question: why must a model be tested on data that it did not learn from?

Next: Train, validation and test sets