Classification intuition

What you will be able to do

Turn the facts of a ticket into a score from 0 to 1, and the score into a flag for the senior team. Then see why the default decision of scikit-learn flags almost no ticket, so you never ship that mistake.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every piece of code, task and check of this lesson, with the instructions. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. Read the explanations and the charts here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Linear regression, the intercept and the coefficient, fit and predict, from Regression intuition.
• Classification and labels, from Supervised learning: Features and labels.
• Accuracy, precision and recall, from Classification metrics: Precision and recall.
• Scores and thresholds, average precision and ROC AUC, from The precision–recall trade-off.
• The target escalated_72h, the baselines and the improvement target, from Define a baseline. On the validation set, “never escalate” has an accuracy of 0.845 and catches nothing. The rule “flag priority 1” flags 163 tickets, with precision 0.374 and recall 0.333.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model, with its .venv environment and the data in data/. You made them in Define a baseline: Set up your project.
• data/train.csv (16,939 rows) and data/valid.csv (1,179 rows). If they are missing or changed, run python get_data.py --reset in the project folder. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux steps are not tested. The notebook was tested in Jupyter.

A forecast gives a chance, you make the decision

Before you leave home, your weather app says “70% chance of rain”. It does not say “rain” or “no rain”. It gives a number from 0% to 100%, and you make the decision. A careful person takes an umbrella at 30%. Another person takes it only at 80%. The forecast is the same for both; only the limit differs.

Many classification models work the same way. The question has a yes-or-no answer, a category. But the model does not answer yes or no directly. It gives each case a score from 0 to 1: the higher the score, the more likely a “yes”. A threshold then turns the score into a decision, as you saw in The precision–recall trade-off. Keeping the two apart is useful, because the person who owns the decision can choose the threshold.

Larkfield’s question is yes or no: will this ticket be escalated to a senior agent within 72 hours? The target escalated_72h is 1 or 0. Grace, the support lead, wants a score for each new ticket. Then she chooses a threshold that fits her senior team’s capacity: it can review at most 1 in 5 tickets. If the score is at least the threshold, the ticket is flagged and goes to the senior team.

So you need a model that turns the facts of a ticket into a number between 0 and 1. You already know one model that turns facts into a number: the linear regression of the last lesson.

Why not a straight line?

Tomás, the AI engineer, suggests a shortcut: “The target is 0 or 1. Fit a linear regression to it, and call its prediction the chance.” Priya tries it with one feature, word_count, the number of words in the customer’s message. She predicts six message lengths:

import pandas as pd
from sklearn.linear_model import LinearRegression

TARGET = "escalated_72h"
train = pd.read_csv("data/train.csv")
line = LinearRegression().fit(train[["word_count"]], train[TARGET])
print(f"line: {line.intercept_:.4f} + {line.coef_[0]:.5f} x word_count")
words = pd.DataFrame({"word_count": [0, 100, 200, 500, 1000, 2000]})
words["line"] = line.predict(words[["word_count"]]).round(3)
print(words.to_string(index=False))

The output:

Output

line: 0.0488 + 0.00064 x word_count
 word_count  line
          0 0.049
        100 0.113
        200 0.177
        500 0.370
       1000 0.692
       2000 1.335

Up to a few hundred words, the numbers look like chances. But each word adds the same 0.00064, without a limit, like the taxi line. A message of 2,000 words gets 1.335: a “chance” of 134%. The line passes 1 at 1,479 words. The longest train message has 883 words, but nothing stops a longer message from arriving tomorrow.

The line also goes below 0. In the notebook, you fit the same kind of line with four features. Then 570 train tickets get a value below 0, a negative “chance”. One of them is a priority-3 ticket with no words, created at 21:00: −0.038.

There is a second problem, which the chart shows. The real escalation rate does not rise by the same amount for each word. It rises quickly for short messages, more slowly from about 150 words, and then it jumps for the few very long ones. A straight line cannot follow that shape.

• Escalation rate in train, by word count
• Straight line
• 0 and 1, the limits of a chance

A straight line fitted to a 0/1 target. It keeps rising and passes 1, the largest possible chance, at 1,479 words.

Text description of the chart

The horizontal axis is the number of words, from 0 to 2,000. Dashed lines mark 0 and 1, the limits of a chance. The dots are the escalation rates in train for groups of tickets. They are 0.059 for about 37 words, 0.096 for 74, 0.138 for 122, 0.178 for 172, 0.191 for 238, 0.214 for 338 and 0.262 for 461. The last dot is 0.600 for about 680 words, from only 15 tickets. The straight line starts at 0.049 for 0 words and rises steadily. It passes 1 at 1,479 words and reaches 1.335 at 2,000 words.

So a chance needs a different shape: one that can rise, but never goes below 0 or above 1.

The S-curve: the logistic function

Logistic regression keeps the useful part of linear regression and changes the shape. It makes the score in two steps:

1. A weighted sum, as in linear regression: z = intercept + weight × feature. With several features, it adds one weight × feature term for each. This z can be any number, from very negative to very positive.
2. The logistic function bends z into a number from 0 to 1. The result has the shape of a stretched S.

Think of the S-curve as a careful translator from z to a chance. When z is very negative, the score is close to 0, and making z even smaller changes almost nothing. When z is very positive, the score is close to 1, and it never passes 1. In the middle, the score changes fastest. Here are seven values of z and their scores:

z	−4	−2	−1	0	1	2	4
score	0.018	0.119	0.269	0.500	0.731	0.881	0.982

z = 0 always gives 0.5. A negative z gives a score below 0.5, and a positive z a score above it. The score comes close to 0 and 1, but never reaches them. The formula is:

$$\text{score} = \frac{1}{1 + e^{-z}}$$

Check one value with a calculator. For z = −1: e¹ = 2.718, so the score is 1 ÷ (1 + 2.718) = 1 ÷ 3.718 = 0.269.

• Logistic function
• Threshold 0.5

The logistic function, with the three priorities of Priya's one-feature model from the worked example below. All three are far below the threshold of 0.5.

Text description of the chart

The horizontal axis is z, from −8 to 6. The curve is an S shape: close to 0 on the left, 0.5 at z = 0, and close to 1 on the right. It is steepest around z = 0. A dashed line marks the threshold 0.5. Three points show the one-feature model of the worked example. Priority 3 is at z = −2.718, with the score 0.062. Priority 2 is at z = −1.854, with 0.135. Priority 1 is at z = −0.989, with 0.271. Every ticket has a z below 0, so no ticket reaches 0.5.

Why is it called a regression, when it answers a classification question? Because inside, it fits a weighted sum, just like linear regression. “Logistic” is the name of the S-curve. So the name describes how the model works, not the task it does.

Back to the message lengths. Fit a logistic regression on word_count. Its S-curve gives 0.070 for 0 words and 0.464 for 500 words. It gives 0.909 for 1,000 words and 0.999 for 2,000 words. It stays between 0 and 1 for every message:

• Escalation rate in train, by word count
• S-curve (logistic regression)
• Straight line
• 0 and 1

The same data with the S-curve of a logistic regression. It bends toward 1 and never passes it.

Text description of the chart

The axes and the dots are the same as in the previous chart. The dashed straight line rises from 0.049 to 1.335. The S-curve starts at 0.070 for 0 words, is 0.167 at 200 words and 0.464 at 500 words. Then it bends: 0.909 at 1,000 words and 0.999 at 2,000 words. It never passes 1. Up to about 300 words, the line and the S-curve are close to each other and to the dots.

The S-curve fixes the limits. It does not make word_count a good feature: the score still rises with every extra word. The independent variation below shows what that does.

Worked example: one feature, priority

Priya now uses a stronger feature, priority (1 is the most urgent, 3 the least). She loads the train and validation sets. The validation set is the data she keeps to check models while she works, as in Train, validation and test sets:

valid = pd.read_csv("data/valid.csv")
print(len(train), "train tickets, escalation rate", round(train[TARGET].mean(), 3))
print(len(valid), "valid tickets, escalation rate", round(valid[TARGET].mean(), 3))

The output:

Output

16939 train tickets, escalation rate 0.117
1179 valid tickets, escalation rate 0.155

Then she fits a logistic regression. An estimator is a scikit-learn object that learns from data: LinearRegression and LogisticRegression are estimators. fit learns the estimator’s numbers (here the intercept and the weight) from the train set. After fit, the estimator is a model.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(train[["priority"]], train[TARGET])
print("intercept:", model.intercept_.round(4))
print("weight of priority:", model.coef_.round(4))

The output:

Output

intercept: [-0.1249]
weight of priority: [[-0.8644]]

The weight is negative: a higher priority number (a less urgent ticket) gives a lower z, so a lower score. Before you read on, compute the score of a priority-1 ticket by hand: z = −0.1249 + (−0.8644 × 1).

import math


def logistic(z):
    """Bend any number into the range 0 to 1."""
    return 1 / (1 + math.exp(-z))


for priority in [1, 2, 3]:
    z = model.intercept_[0] + model.coef_[0][0] * priority
    rate = train.loc[train["priority"] == priority, TARGET].mean()
    print(f"priority {priority}: z = {z:.3f}, score {logistic(z):.3f}, train rate {rate:.3f}")

The output:

Output

priority 1: z = -0.989, score 0.271, train rate 0.291
priority 2: z = -1.854, score 0.135, train rate 0.120
priority 3: z = -2.718, score 0.062, train rate 0.067

The same numbers in a table:

Priority	z = −0.1249 − 0.8644 × priority	Score	Escalated in train
1	−0.989	0.271	29.1%
2	−1.854	0.135	12.0%
3	−2.718	0.062	6.7%

The scores are close to the escalation rates in train, but not equal. The model must use one straight line for z through all three priorities, and the S-curve then bends it. A line cannot match three rates exactly. That is the cost of a simple model.

predict_proba and predict

A classification estimator answers in two ways:

• predict_proba gives the scores, one column per class. Column 1 ([:, 1]) is the score for “escalated”.
• predict gives a decision, 0 or 1. It flags a ticket when its score is above 0.5.

The highest score of this model is 0.271, for priority 1. Predict before you run it: how many validation tickets will predict flag?

scores = model.predict_proba(valid[["priority"]])[:, 1]
flags = model.predict(valid[["priority"]])
print("highest score:", round(scores.max(), 3))
print("flagged by predict:", flags.sum())
print("accuracy:", round((flags == valid[TARGET]).mean(), 3))

The output:

Output

highest score: 0.271
flagged by predict: 0
accuracy: 0.845

No score reaches 0.5, so predict flags nothing. The result is the majority baseline of Define a baseline: 84.5% accuracy, and no escalation caught. This is the accuracy trap of Classification metrics. The scores are fine: they still rank the tickets. Only the threshold is wrong for Larkfield.

With a threshold of 0.2, the model flags every priority-1 ticket (score 0.271) and no other ticket (0.135 and 0.062). That is the rule “flag priority 1”: 163 tickets, 61 of them escalated. With one feature, logistic regression is the rule with a number attached. More features can rank the tickets better.

Several features

Four columns have a value for every ticket and are already numbers: priority, word_count, prior_tickets_90d and created_hour. Priya fits a model with all four:

NUMBERS4 = ["priority", "word_count", "prior_tickets_90d", "created_hour"]
model4 = LogisticRegression().fit(train[NUMBERS4], train[TARGET])
print("intercept:", model4.intercept_.round(4))
print(pd.Series(model4.coef_[0], index=NUMBERS4).round(4))

The output:

Output

intercept: [-0.7279]
priority            -0.8938
word_count           0.0052
prior_tickets_90d    0.0233
created_hour        -0.0096
dtype: float64

Now z = −0.7279 − 0.8938 × priority + 0.0052 × word_count + 0.0233 × prior_tickets_90d − 0.0096 × created_hour. The weights are in different units: 0.0052 per word, 0.0233 per earlier ticket. So a small weight does not mean a small effect. A ticket with 300 words adds 300 × 0.0052 = 1.56 to z, more than one step of priority. Preprocess with a pipeline puts all the numbers on one scale.

To compare the two models, Priya uses the two ranking metrics of The precision–recall trade-off. They measure how well the scores rank the tickets, for all thresholds at once. ROC AUC is the chance that a random escalated ticket gets a higher score than a random other ticket: 0.5 is guessing. Average precision (AP) is higher when the escalated tickets are near the top of the ranking; a constant score gets the escalation rate, 0.155.

from sklearn.metrics import average_precision_score, roc_auc_score

scores4 = model4.predict_proba(valid[NUMBERS4])[:, 1]
for name, s in [("priority only", scores), ("four numbers", scores4)]:
    auc = roc_auc_score(valid[TARGET], s)
    ap = average_precision_score(valid[TARGET], s)
    print(f"{name}: ROC AUC {auc:.3f}, AP {ap:.3f}, flagged at 0.5: {(s > 0.5).sum()}")

The output:

Output

priority only: ROC AUC 0.658, AP 0.251, flagged at 0.5: 0
four numbers: ROC AUC 0.703, AP 0.312, flagged at 0.5: 6

The same results next to the baseline:

Model (validation set)	ROC AUC	AP	Flagged at 0.5
Constant score (baseline)	0.5	0.155	—
priority only	0.658	0.251	0
Four numbers	0.703	0.312	6 of 1,179

The four-feature model ranks the tickets better. But at 0.5 it still flags only 6 of 1,179 tickets.

Only four features, for now

The ten features from the SQL course include four text categories (channel, team, segment, region) and two numbers with gaps (order_value, customer_tenure_days). A logistic regression needs numbers with no gaps, so it cannot use these six features yet. The next lesson shows each of these problems on real tickets, and then solves them all with one tool.

When to use logistic regression

Logistic regression is the usual first model for a yes-or-no question. It trains in a second, works with little data and gives a score that ranks the cases. Each weight has a direction that a person can check: here, a more urgent priority raises the score. A bank, a hospital or Grace’s team can follow how a score was made, which matters when people must trust it.

Its limits come from the weighted sum inside. Each feature pushes z by a fixed amount for each unit. So the model cannot learn an effect that rises and then falls, or one that depends on another feature, unless you add features for it. Other model families, such as the decision trees of Module 5, can learn such patterns. And like linear regression, it needs numbers with no gaps, on similar scales: that is the next lesson.

Guided practice: a threshold of 0.2

The lesson notebook has every cell of this page. The box at the top has the Colab and Kaggle links.

1. Open the notebook in Colab or Kaggle, and run the setup cells. You should see the Ready: line.
2. In section 1, run the straight line on word_count. You should see 1.335 for 2,000 words. Then run the four-feature line. You should see 570 train tickets below 0.
3. In section 2, predict the score for z = 0. Then run the cell. You should see z = 0 -> score 0.500.
4. In section 3, compute the priority-1 score by hand before you run the cell. You should see score 0.271.
5. In section 4, predict how many tickets predict flags. You should see flagged by predict: 0.
6. Do the Your turn task: count the tickets with a score of at least 0.2, and how many of them were escalated.
7. Run the check cell. It prints Check passed when both numbers are right.

Show the answer to step 6

flags_02 = scores >= 0.2
flagged_02 = int(flags_02.sum())
caught_02 = int((flags_02 & (valid[TARGET] == 1)).sum())
print(flagged_02, caught_02)

The output:

Output

163 61

163 flagged tickets, 61 of them escalated: the same as the rule “flag priority 1”. Precision is 61 ÷ 163 = 0.374, and recall is 61 ÷ 183 = 0.333.

Run the notebook on your computer

Download the notebook with View on GitHub, then the download button. Save it in ~/projects/ticket-model/notebooks/. In VS Code, open it and select Select Kernel › Python Environments › .venv. The notebook makes its own folder ticket-model next to it and works there, so it never changes your project files.

Independent variation: word_count instead of priority

Change one thing: fit the one-feature model with word_count instead of priority. Before you run it, predict: will it rank better or worse than priority (ROC AUC 0.658)? Will predict flag anything? In the notebook, section 6 has the task and a check.

Show the answer

model_words = LogisticRegression()
model_words.fit(train[["word_count"]], train[TARGET])
scores_words = model_words.predict_proba(valid[["word_count"]])[:, 1]
auc_words = round(roc_auc_score(valid[TARGET], scores_words), 3)
flagged_words = int(model_words.predict(valid[["word_count"]]).sum())
print(auc_words, flagged_words)

The output:

Output

0.598 1

word_count alone ranks worse than priority: ROC AUC 0.598 against 0.658. Long messages escalate a little more often, but the signal is weak. predict flags 1 ticket, and that ticket was not escalated.

Why one? The weight is +0.0049 per word and the intercept −2.5871, so z reaches 0 at about 529 words. Only one validation ticket is that long: 573 words. The S-curve keeps the score below 1. But the weight still adds up with every word, so a few extreme tickets get a high score. The flag goes to the longest message, not to the most urgent problem.

Common mistake

predict as the decision

Symptom: Tomás reports the four-feature model with predict. The accuracy looks good:

flags4 = model4.predict(valid[NUMBERS4])
caught = int((flags4 & (valid[TARGET] == 1)).sum())
print("flagged:", flags4.sum(), "of", len(valid))
print("caught:", caught, "of", valid[TARGET].sum(), "escalations")
print("accuracy:", round((flags4 == valid[TARGET]).mean(), 3))

The output:

Output

flagged: 6 of 1179
caught: 2 of 183 escalations
accuracy: 0.843

84.3% sounds good. But “never escalate” has 84.5%, and the model catches only 2 of 183 escalations.

Cause: predict uses a threshold of 0.5. That default suits two classes of about the same size. Here, 15.5% of the tickets escalate, and very few scores reach 0.5. Even the final model of this course flags only 25 of 1,179 tickets (2.1%) at 0.5.

Fix: use predict_proba, and choose the threshold from the decision. Grace’s senior team can review at most 1 in 5 tickets (20%). At 0.2, this model flags 192 tickets (16.3%) and catches 73. Thresholds and calibration chooses the threshold properly, on the validation set.

Consequence in AI work: predict raises no error and gives a high accuracy. A team that deploys it ships a model that almost never flags. Customers with serious problems wait in the normal queue, and the report says 84% accuracy.

See it end to endPredict credit-card defaultCase study

The credit case study builds a logistic-regression pipeline on 30,000 real clients.

Knowledge check

Check your understanding

Tomás wants to predict escalation with a linear regression on the 0/1 target. Give two reasons, with numbers from this lesson, why its output is not a chance.

Show the answer

First, a straight line has no limits. On word_count, the line gives 1.335 for a 2,000-word message, a “chance” of 134%. With four features, 570 train tickets get a value below 0. A chance must stay between 0 and 1. Second, the line rises by the same amount for every word. But the real escalation rate rises quickly for short messages and slowly after about 150 words. Logistic regression keeps the weighted sum but bends it with the S-curve, so every score stays between 0 and 1.

Check your understanding

A ticket has z = 0.6. What score does the logistic function give, and is it flagged at the default threshold?

Show the answer

e^−0.6 = 0.549. So the score is 1 ÷ 1.549 = 0.646. A positive z always gives a score above 0.5, so predict flags this ticket. At a threshold of 0.7, it would not be flagged.

Check your understanding

Predict and explain: Omar, the product manager, says: the four-feature model has 84.3% accuracy, so it is ready. What do you tell him?

Show the answer

The accuracy comes from predict, which uses 0.5. At 0.5 the model flags 6 of 1,179 tickets and catches 2 of 183 escalations. “Never escalate” has a higher accuracy, 84.5%, and catches nothing. Accuracy is high because most tickets do not escalate, so it does not measure the decision. The scores do rank better than the rule (ROC AUC 0.703). But the model needs a threshold chosen for the 20% capacity, and a comparison with the rule at that workload.

Recap

You can now explain how logistic regression turns ticket facts into a decision:

• A straight line cannot give a chance. On word_count it reaches 1.335 at 2,000 words. With four features, 570 train tickets get a value below 0.
• Logistic regression keeps the weighted sum z and bends it with the logistic function, an S-curve, into a score from 0 to 1. z = 0 gives 0.5. For priority 1, z = −0.989 and the score is 0.271.
• predict_proba gives the scores. predict flags a ticket when its score is above 0.5.
• On Larkfield’s data, 0.5 flags almost nothing: 0 tickets with priority alone, 6 with four features. At 0.2, the priority model is the rule “flag priority 1”.
• Four features rank better than one: ROC AUC 0.703 and AP 0.312, against 0.658 and 0.251.

Text categories and gaps still stop the model. The next lesson shows each problem on real tickets, and then solves them all with a pipeline.

Next: Preprocess with a pipeline