Slice analysis

What you will be able to do

Find where the model fails, and for whom. One overall number, such as a recall of 0.470, can hide a team where the model catches almost nothing. A slice table shows it, and the count next to each rate tells you if you can trust it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every cell, task and check of this lesson, with the instructions. Its setup cells install the packages, download the data and the course’s ticket_model.py, so it needs no local setup. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The final model (logistic regression with 11 features) and the threshold 0.19 for the review capacity of 20%, from Thresholds and calibration.
• Precision and recall, from Classification metrics: Precision and recall.
• Why the validation set is for decisions and the test set is for one final report, from Train, validation and test sets: Three sets, three jobs.
• What a slice is, and why one average can hide a weak group, from Calibration and slices in Essential Mathematics and Statistics for AI.
• The standard error of a share, from Estimate uncertainty in the mathematics course.
• groupby and agg in pandas, from Transform with Python in SQL and Data Preparation for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment and the data in data/. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows). If a data file is missing or changed, run python get_data.py --reset.
• ticket_model.py in its final form, with 11 features. You wrote it in Preprocess with a pipeline: Write ticket_model.py and added prior_escalations_90d in Select useful features: Change ticket_model.py. If it is missing or broken, download the course’s copy: curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py (curl.exe on Windows).
• Open a notebook in the project folder, in VS Code or JupyterLab, as in Define a baseline: Open a notebook in VS Code. Run the cells of this lesson in it.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows commands are not tested. The notebook was tested in Jupyter.

Module 7: from a score to an understanding

At the end of Module 6, the model met Larkfield’s target on the validation month. It flags 19.3% of the tickets and catches 47.0% of the escalations. The rule “flag priority 1” catches 33.3%. That is a good result, but it is one number for 1,179 tickets. Before Grace’s team relies on the model, someone must look behind that number. This module does that, in four steps:

1. This lesson: where are the errors, and for which groups of tickets?
2. Explain predictions: which features push a score up or down, and what that does not mean.
3. When a simple model is better: why Larkfield keeps a simple model, and where the law or good practice asks for one.
4. Unsupervised exploration: do customers fall into groups that the model does not see?

Every decision in this module uses the validation set (May 2026), the data we keep to check and improve the model. You may look at the test set once for the final report, but you never change the model because of it. The data is made up for this course; it has no real people.

An average that hides a problem

Think of a restaurant with an average review of 4.5 stars. That sounds safe. But suppose that all the bad reviews come from people who ordered the vegetarian dishes. If you eat no meat, the 4.5 stars tell you almost nothing about your evening. The average mixes two groups, and you belong to the weak one.

A model’s overall score has the same problem. Recall, the share of escalations that the model catches, is 0.470 on May. That number mixes payment tickets, account tickets, urgent and less urgent tickets, phone and email. If the model works well for some groups and badly for others, the overall number hides it. Grace would then tell her team “the model catches about half of all escalations”, and for some customers that is false.

The fix is simple: cut the data into groups by one property, and compute the same metric for each group. Each group is called a slice, like a slice cut from a cake. Slice analysis compares the slices. It answers a question that one overall number cannot answer: “Does the model work for everyone it serves?”

One number on 12 tickets

Here are 12 made-up tickets from two teams. flag is the model’s decision (1 = flagged for the senior team), and escalated_72h is the truth. A ticket is caught when it is flagged and escalated.

tiny = pd.DataFrame({
    "team": ["payment"] * 8 + ["account"] * 4,
    "escalated_72h": [1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0],
    "flag":          [1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0],
})
tiny["caught"] = tiny["flag"] * tiny["escalated_72h"]
print("recall, all tickets:", tiny["caught"].sum() / tiny["escalated_72h"].sum())
print(tiny.groupby("team")[["escalated_72h", "caught"]].sum())

The output:

recall, all tickets: 0.5
         escalated_72h  caught
team
account              2       0
payment              4       3

Check it by hand. There are 6 escalations, and the model catches 3: a recall of 3 ÷ 6 = 0.5. For payment, it is 3 ÷ 4 = 0.75. For account, it is 0 ÷ 2 = 0. The overall 0.5 is a mix of a good team and a team where the model catches nothing. It describes neither of them.

Now look at the count. Account has only 2 escalations. One more caught ticket changes its recall from 0 to 0.5. So a slice result is always a pair: the rate and the count it comes from. You will see later in this lesson how to judge whether a count is large enough.

Four kinds of error

Before you slice, sort the errors. Think of a mechanic who finds that a car failed its inspection. “It failed” is not enough to fix it. The mechanic needs to know which part failed, and why. Errors of a model are the same: they are not all one kind.

A missed escalation and a false alarm have different costs. A missed escalation means that a customer with a serious problem waits in the normal queue. A false alarm costs about 10 minutes of a senior agent. And a missed escalation with a score of 0.18 is a different problem from one with a score of 0.03. The first one was close; a slightly lower threshold would catch it. For the second one, the model saw no risk at all. So Priya uses four outcomes for the model’s decisions, and two questions for every error. The diagram shows them for the 1,179 tickets of May.

The error taxonomy for the final model on the validation month, at threshold 0.19. Two questions apply to every error: a data problem or a label problem.

Text description of the diagram

The validation month has 1,179 tickets. The first question is: is the ticket flagged, with a score of 0.19 or more? 227 tickets are flagged and 952 are not.

• Flagged and escalated: caught, 86 tickets.
• Flagged and not escalated: false alarm, 141 tickets. Each one costs about 10 minutes of a senior agent.
• Not flagged and not escalated: correct pass, 855 tickets.
• Not flagged and escalated: a missed escalation, 97 tickets. 42 are near the threshold, with a score from 0.10 to 0.19. 55 have a low score, below 0.10.

For every error, ask two more questions. Data problem: is an input unknown or missing, so the model could not see the ticket correctly? Label problem: does the target mean what we think, or did its meaning change?

The two questions need words:

• A data problem is an error in the inputs: a value is missing, unknown or wrong. In this data, channel and region can be unknown, and order_value is sometimes missing although the ticket has an order.
• A label problem is an error in the target. The label (the target value) may be wrong, or its meaning may change. Larkfield had a change: the new warranty policy of 1 May makes warranty tickets escalate more often. The same ticket can get a different label in May than in April.

Worked example: Priya sorts the errors of May

Priya fits the final model on the train set and scores the validation month. The code is the same as in Module 6. ticket_model.py gives the columns, load() and the pipeline.

from ticket_model import FEATURES, TARGET, build_pipeline, load

train, valid = load("train"), load("valid")
pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
threshold = 0.19
valid["score"] = pipeline.predict_proba(valid[FEATURES])[:, 1]
valid["flag"] = (valid["score"] >= threshold).astype(int)
print(len(valid), "tickets,", valid["flag"].sum(), "flagged,", valid[TARGET].sum(), "escalated")

The output:

1179 tickets, 227 flagged, 183 escalated

These are the numbers of Module 6: 227 flagged (19.3%). Now she gives each ticket its outcome. Before you read the result, predict: of the 97 missed escalations, are most near the threshold, or far below it?

escalated = valid[TARGET] == 1
flagged = valid["flag"] == 1
valid["outcome"] = "correct pass"
valid.loc[flagged & escalated, "outcome"] = "caught"
valid.loc[flagged & ~escalated, "outcome"] = "false alarm"
valid.loc[~flagged & escalated & (valid["score"] >= 0.10), "outcome"] = "missed, near threshold"
valid.loc[~flagged & escalated & (valid["score"] < 0.10), "outcome"] = "missed, low score"
print(valid["outcome"].value_counts())

The output:

outcome
correct pass              855
false alarm               141
caught                     86
missed, low score          55
missed, near threshold     42
Name: count, dtype: int64

The counts match Module 6: 86 caught (TP), 141 false alarms (FP) and 55 + 42 = 97 missed (FN). The new information is the split of the misses. 55 of the 97 missed escalations have a score below 0.10. A threshold of 0.10 would flag 40.3% of the tickets, twice the capacity, and it would still miss these 55. For these tickets, the model sees no risk at all.

Next, the data problems. Priya marks a ticket when channel or region is unknown, or when order_value is missing outside the account team. Account tickets have no order, so a missing value is normal there. pd.crosstab counts the tickets for each pair of values.

valid["data_problem"] = ((valid["channel"] == "unknown") | (valid["region"] == "unknown")
                         | (valid["order_value"].isna() & (valid["team"] != "account")))
print(pd.crosstab(valid["outcome"], valid["data_problem"]))

The output:

data_problem            False  True
outcome
caught                     80      6
correct pass              785     70
false alarm               134      7
missed, low score          50      5
missed, near threshold     40      2

90 tickets have a data problem: 7.6% of all tickets. The errors are 141 + 55 + 42 = 238 tickets, and 7 + 5 + 2 = 14 of them have a data problem: 5.9%. So the errors do not come mostly from bad inputs. Priya writes this down as a result, too. A check that finds nothing is still evidence: it tells her to look for the cause elsewhere.

Slices with counts

Now the slices. Priya writes one small function and uses it for every column. For each value of a column, it counts the tickets, the escalations, the flags and the caught escalations. Then it computes the escalation rate (esc_rate), the recall, the precision and the mean score.

def slice_table(df, column):
    """Tickets, escalations, flags and recall for each value of one column."""
    df = df.assign(caught=df["flag"] * df[TARGET])
    table = df.groupby(column).agg(tickets=(TARGET, "size"), escalated=(TARGET, "sum"),
                                   flagged=("flag", "sum"), caught=("caught", "sum"),
                                   mean_score=("score", "mean"))
    table["esc_rate"] = table["escalated"] / table["tickets"]
    table["recall"] = table["caught"] / table["escalated"]
    table["precision"] = table["caught"] / table["flagged"]
    return table.round(3)


slice_table(valid, "team")

The output:

          tickets  escalated  flagged  caught  mean_score  esc_rate  recall  precision
team
account       166         12        7       1       0.057     0.072   0.083      0.143
delivery      348         42       39      13       0.092     0.121   0.310      0.333
payment       206         48       91      37       0.214     0.233   0.771      0.407
returns       291         33       40      10       0.098     0.113   0.303      0.250
warranty      168         48       50      25       0.163     0.286   0.521      0.500

Recall of the final model on the validation month, for each team. The recall for all tickets is 0.47. The counts are in the table above: account has only 12 escalations.

Text description of the chart

A column chart of recall at threshold 0.19 on the validation month, for each team. For comparison, the recall for all tickets is 0.47 (86 of 183 escalations caught). Payment: 0.77 (37 of 48). Warranty: 0.52 (25 of 48). Delivery: 0.31 (13 of 42). Returns: 0.30 (10 of 33). Account: 0.08 (1 of 12).

Read it one column at a time.

• Recall goes from 0.771 (payment) to 0.083 (account). The overall 0.470 describes no team well.
• Count. Account has only 12 escalations, so its recall is uncertain. But every column agrees that the model sees little risk there: the mean score is 0.057, the lowest of all teams. Account tickets rarely escalate (7.2% in May), so the model gives them low scores, and the few that escalate look like the others.
• Mean score against the rate. For most teams, the two are close. Warranty is the exception: 28.6% of its tickets escalate, but the mean score is only 0.163. In the train set, warranty’s rate was 15.9%. The model learned the old meaning of “escalated” for warranty. That is the label problem from the diagram, caused by the policy of 1 May. It is also the calibration gap of Thresholds and calibration, seen in one slice.

Omar asks: “So is the model bad for account customers?” Priya answers with both numbers: “It catches 1 of their 12 escalations in May. Twelve is a small count, but the scores agree. Grace should know that account escalations mostly reach the senior team by the normal route.”

A small slice: 22 tickets

The unknown channel is rare. The channel is unknown when the help desk did not record it, and the SQL course kept it as its own category. Priya slices by channel:

slice_table(valid, "channel")

The output:

          tickets  escalated  flagged  caught  mean_score  esc_rate  recall  precision
channel
chat          373         54       68      24       0.113     0.145   0.444      0.353
email         467         71       86      34       0.114     0.152   0.479      0.395
phone         168         33       52      18       0.172     0.196   0.545      0.346
unknown        22          5        5       3       0.153     0.227   0.600      0.600
web_form      149         20       16       7       0.090     0.134   0.350      0.438

The unknown channel has the highest escalation rate (0.227) and the best recall (0.600). Both numbers come from 22 tickets and 5 escalations. How much can you trust them?

Think of a coin. If you toss it 4 times and get 3 heads, you do not say that the coin prefers heads. Four tosses are too few: 3 of 4 happens often with a fair coin. If you toss it 400 times and get 300 heads, something is wrong with the coin. The share is the same, 75%, but the count changes what it can tell you. A slice of 22 tickets is like the 4 tosses. Two simple checks show it.

Add or remove two escalations. If only two tickets had a different outcome, the rate would be 3 ÷ 22 = 0.136 or 7 ÷ 22 = 0.318. For email, with 467 tickets, the same change gives 0.148 or 0.156. Two tickets move the small slice by 9 points, and the large one by less than half a point.

The standard error. The mathematics course gave a short formula for the standard error (SE) of a share p from n tickets, the typical size of its chance error:

$$\text{SE} = \sqrt{\frac{p\,(1-p)}{n}}$$

unknown = valid[valid["channel"] == "unknown"]
k, n = int(unknown[TARGET].sum()), len(unknown)
p = k / n
se = (p * (1 - p) / n) ** 0.5
print(f"{k} of {n} escalated: rate {p:.3f}, SE {se:.3f}, 95% interval {p - 1.96 * se:.3f} to {p + 1.96 * se:.3f}")
print("two escalations fewer or more:", round((k - 2) / n, 3), round((k + 2) / n, 3))

The output:

5 of 22 escalated: rate 0.227, SE 0.089, 95% interval 0.052 to 0.402
two escalations fewer or more: 0.136 0.318

Slice	Tickets	Escalated	Rate	SE	95% interval
channel = unknown	22	5	0.227	0.089	0.052 to 0.402
channel = email	467	71	0.152	0.017	0.119 to 0.185
All of May	1,179	183	0.155	0.011	0.135 to 0.176

The interval of the unknown channel goes from 5% to 40%. It includes the rate of every other channel. So the data says nothing reliable about whether these tickets escalate more. The recall is worse: 3 of 5. With 5 escalations, the interval is wider than the whole range of useful values. (With so few tickets, the formula is only rough. The lesson is the width, not the exact ends.)

Guests are rarer still: 9 tickets in May, with 1 escalation, caught. A recall of “1 of 1” means nothing. Priya puts both slices in her notes as “too small to judge”, with the count, and does not hide them.

Tomás asks: why not just leave the small slices out?

A small slice can still matter. The unknown channel could grow, and guests are real customers. Leave it in the table with its count, and say “too small to judge”. Then the reader decides if more data is needed. Leaving it out hides that the model was never checked for those tickets.

Guided practice: slices by priority

Grace sets the priority of every new ticket, so she wants to know how the model does for each one.

1. Run the cells of the worked example, so that valid has the columns score and flag, and slice_table() exists.
2. Before you run the next step, write a prediction: which priority has the lowest recall?
3. Run slice_table(valid, "priority"). You should see three rows: priority 1, 2 and 3.
4. Read the recall of priority 3. You should see 0.102.
5. Read the escalations of each priority. You should see 61, 63 and 59.
6. Count where the 97 missed escalations are. Use escalated − caught for each priority.

Show the output and the answer

          tickets  escalated  flagged  caught  mean_score  esc_rate  recall  precision
priority
1             163         61      118      50       0.282     0.374   0.820      0.424
2             399         63       90      30       0.148     0.158   0.476      0.333
3             617         59       19       6       0.059     0.096   0.102      0.316

The missed escalations are 61 − 50 = 11 (priority 1), 63 − 30 = 33 (priority 2) and 59 − 6 = 53 (priority 3). More than half of all missed escalations are priority 3 tickets. The model catches 6 of their 59 escalations: a recall of 0.102.

The slice is large: 617 tickets and 59 escalations. So 0.102 is reliable, with an SE of about 0.04. This is a model limitation, not chance. Priority 3 tickets escalate at 9.6%, below the average of 15.5%, so the model gives them low scores (mean 0.059). Only 19 of the 617 reach the threshold. With room for 20% of the tickets, the senior team sees the riskier priority 1 and 2 tickets first.

Grace’s reading: “So a serious problem in a priority 3 ticket comes to us late, by the normal route. That was true before the model, too. But I must not tell my team that the model catches half of all escalations. For priority 3, it catches one in ten.”

Your turn: slice by week

Change one condition: slice by time instead of by group. A week of May is days 1–7, 8–14, 15–21, 22–28, and 29–31. The last “week” has only three days.

Make the column with valid["week"] = (valid["created_at"].dt.day - 1) // 7 + 1, then run slice_table(valid, "week"). Before you run it, predict: is the recall about the same each week? Which week has the fewest tickets?

Show the answer

      tickets  escalated  flagged  caught  mean_score  esc_rate  recall  precision
week
1         257         49       49      24       0.117     0.191   0.490      0.490
2         262         42       54      23       0.122     0.160   0.548      0.426
3         283         39       56      17       0.124     0.138   0.436      0.304
4         295         44       57      19       0.118     0.149   0.432      0.333
5          82          9       11       3       0.112     0.110   0.333      0.273

The recall moves from 0.432 to 0.548 in the four full weeks. Each week has only 39 to 49 escalations, so one ticket moves its recall by about 2 points, and the SE is about 0.08. These differences are what chance gives. Week 5 (82 tickets, 9 escalations) looks worst, at 0.333, but one more caught ticket would make it 0.444.

The flag rate stays close to the capacity: 49 to 57 flagged per 257 to 295 tickets, about 19% to 21%. So the threshold of 0.19 held for the whole month. A time slice is how you see a change that starts in the middle of a month, such as the policy change of 1 May. Module 8 uses the same idea to watch the model after it starts work.

Failure case: acting on a test slice

Common mistake

A rule from six test tickets

Symptom: Tomás runs the slice table on the test month (June) for the final report:

test = load("test")
test["score"] = pipeline.predict_proba(test[FEATURES])[:, 1]
test["flag"] = (test["score"] >= threshold).astype(int)
slice_table(test, "channel")

          tickets  escalated  flagged  caught  mean_score  esc_rate  recall  precision
channel
chat          305         41       57      17       0.119     0.134   0.415      0.298
email         355         52       65      25       0.119     0.146   0.481      0.385
phone         155         32       51      21       0.172     0.206   0.656      0.412
unknown        23          6        4       0       0.120     0.261   0.000      0.000
web_form      164         26       19       7       0.090     0.159   0.269      0.368

The unknown channel: 6 escalations, 0 caught. Tomás says: “The model fails on unknown-channel tickets. Let us flag all of them, or lower their threshold.”

Cause: two mistakes in one decision. First, 6 escalations are far too few to judge. In May, the same slice caught 3 of 5; one month says 0.600, the next 0.000, and both are noise. Second, the test set is for the final report. A rule made because of a test result turns the test set into a second validation set. Then the final score is no longer an honest estimate. That is the misconception “tuning on the test set” from Tune hyperparameters, in a new form.

Fix: report the slice, with its count: “unknown channel: 0 of 6 escalations caught in June; too few to judge”. Do not change the model or the threshold. If the slice matters, collect more data, and decide on validation data with an interval. Here, May and June together give 3 of 11 caught, still too few.

Consequence in AI work: a rule from a few test tickets moves work to the senior team for no proven gain. And the team’s report now has a test score that is too good. Small slices produce the most extreme rates, so they produce the most of these “findings”.

See it end to endPredict credit-card defaultCase study

The credit case study finds that young clients are flagged more often, although age is not an input.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study compares ranking quality across age groups, and checks race, gender and payer, which are not inputs.

Knowledge check

Check your understanding

The model's recall on May is 0.470. Grace says: the model catches about half of all escalations, for every team. What does the slice table say, and which numbers do you give her?

Show the answer

The recall is 0.470 overall, but it is very different by team. Payment has 0.771 (37 of 48), warranty 0.521 (25 of 48), delivery 0.310 (13 of 42), returns 0.303 (10 of 33) and account 0.083 (1 of 12). By priority, the model catches 0.820 of priority 1 escalations and 0.102 of priority 3 (6 of 59). Give her each recall with its count. The overall number is a mix of slices, and it describes none of them well.

Check your understanding

Predict: the unknown channel has 22 tickets and 5 escalations in May. If two of those tickets had a different outcome, how far could the escalation rate move? What do you write in the report?

Show the answer

From 0.227 to 0.136 (two fewer) or 0.318 (two more): about 9 points in each direction. The SE is 0.089, and the 95% interval is roughly 0.05 to 0.40, which includes every other channel’s rate. In the report, write the rate with its count and say “too small to judge”. For example: “unknown channel: 5 of 22 escalated (23%); too few tickets to compare with other channels”. Do not leave the slice out, and do not act on it.

Check your understanding

Of the 97 missed escalations in May, 55 have a score below 0.10. Is this a data problem, a label problem or a model limitation? Give the evidence.

Show the answer

Mostly a model limitation. Only 14 of the 238 errors have a data problem (an unknown channel or region, or a missing order value). That is fewer than the share in all tickets. So bad inputs do not explain the misses. 40 of the 55 low-score misses are priority 3 tickets. Their features look like those of ordinary low-risk tickets, so the model sees no risk. There is one label problem on top. Warranty tickets escalate at 28.6% in May, but the model, trained before the policy change, gives them a mean score of 0.163.

Recap

• A slice is a part of the data chosen by one property. Slice analysis computes the same metric for each slice, because one overall number can hide a weak group.
• Sort errors first: caught, false alarm, missed near the threshold, missed with a low score. Then ask of every error: is it a data problem or a label problem?
• In May, 55 of the 97 missed escalations have a score below 0.10. Recall is 0.771 for payment, 0.083 for account and 0.102 for priority 3. Warranty escalates at 28.6% but has a mean score of 0.163: the policy change is a label problem.
• Put the count next to every rate. A rate from 22 tickets moves 9 points when two tickets change; its 95% interval goes from 5% to 40%.
• Decide on the validation set. Report test slices, but never act on them.

Next, you look inside the model: which features push a score up or down, and why that is not the cause of an escalation.

Next: Explain predictions