Slices and fairness
- Lesson 32
- 30 min
- Module 7: Evaluate with few labels
Where we are
We chose a detector and a threshold from the budget, and we learned to check its precision without a full key. But one average can hide a group of honest customers who get stopped again and again. In this lesson, we check the false alerts by account type, country, device, region and customer age, and we see why that matters for the people behind the payments. A shop's door alarm shows the problem.
Lesson 32 of 38 · Anomaly Detection: Find What Is UnusualSlices and fairness
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Check a detector’s false alerts group by group, and decide what to do when one group of honest customers gets far more of them. A good average can hide customers who are stopped again and again.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It computes the false alerts of Isolation Forest for each account type, country, device, region and customer age, and counts the customers behind them. Then it compares Isolation Forest and kNN distance for the module practice. Its setup cells install pandas and scikit-learn and download the course data (about 7 MB). About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The alert budget of 50 a day and the cost per week, from The alert budget.
- The 95% interval of a share, from Evaluate without a full answer key.
- The behaviour features, from Features decide more than the algorithm, and the velocity rule, from Rules, and when a rule is the right answer.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1; Colab, Kaggle and Windows were not tested.
A door alarm that rings for the wrong people
Section titled “A door alarm that rings for the wrong people”Think of a shop with an alarm at its door. It rings when a customer walks out with an item that has not been paid for. Sometimes it rings by mistake. The shop counts these mistakes: 1 false ring in every 100 customers. The manager thinks that is fine.
Now look at who hears the false rings. The garden centre next door puts the same kind of tag on its plants. So almost every customer who carries a plant from next door makes the alarm ring. For most customers, the alarm never rings by mistake. For the garden centre’s customers, it rings on almost every visit. Each time, a guard stops them and checks their bags in front of everybody.
The average, 1 in 100, is true. It still hides the real problem: a few customers carry almost all the mistakes, and they pay with their time and their dignity.
The problem: one average hides a group
Section titled “The problem: one average hides a group”A fraud detector can do the same. In the last two lessons, we described Isolation Forest with averages over every payment. Here is one more: the share of honest payments that get an alert. An honest payment is one that is not planted, and a false alert is an alert on an honest payment.
A tiny example: ten customers
Section titled “A tiny example: ten customers”Ten customers each made 100 honest payments. Eight are personal customers and two are business customers. This is a constructed example. The table counts each group’s false alerts:
tiny = pd.DataFrame({ "customer": ["personal"] * 8 + ["business"] * 2, "honest payments": [100] * 10, "false alerts": [1, 0, 1, 1, 0, 1, 0, 1, 12, 9],})total = tiny[["honest payments", "false alerts"]].sum()print(f"all customers: {total['false alerts'] / total['honest payments'] * 10_000:.0f} false alerts per 10,000 honest payments")by_group = tiny.groupby("customer")[["honest payments", "false alerts"]].sum()by_group["per 10,000"] = by_group["false alerts"] / by_group["honest payments"] * 10_000by_groupThe output:
all customers: 260 false alerts per 10,000 honest payments
honest payments false alerts per 10,000customerbusiness 200 21 1050.0personal 800 5 62.5All customers together get 26 false alerts in 1,000 payments, so 260 per 10,000. But the business customers get 1,050 per 10,000, and the personal customers get 62.5. The business customers have about 17 times as many. The average of 260 describes neither group.
This lesson counts false alerts per 10,000 honest payments. It is a rate, so groups of different sizes can be compared. It uses only honest payments, so a group with more fraud does not look worse because of its real frauds.
The idea: look at the slices
Section titled “The idea: look at the slices”A slice is one group of the data, such as the payments of business customers, or the payments made abroad. To slice the results means to compute the same measure for each group, side by side. It is the door alarm’s second question: not “how often does it ring by mistake?”, but “for whom does it ring by mistake?”.
Suppose the mistakes of a system fall much more often on one group of people. Then the system treats those people unfairly, even when nobody planned it. This is a question of fairness. AI Security, Privacy and Responsible Design teaches it in depth. Bias and accessibility tests a system group by group. Impact and error costs decides who must review a decision that can harm someone. This lesson does not repeat that course. It shows how to find such a group in a detector’s alerts, and what Larkfield does about it.
Why a false alert costs a customer
Section titled “Why a false alert costs a customer”A false alert is not only 4 minutes of Sofia’s time. It is also a real customer whose payment is held or blocked:
Text description of the diagram
The diagram starts with an honest payment, for example a business customer who orders 12 garden chairs. An arrow leads to a false alert: the payment’s score is in the top 50 of the day. Two arrows leave the false alert. One goes to “a reviewer reads it”: 4 minutes, about 3 dollars. The other goes to “the payment is held or blocked”. From there, an arrow leads to the customer, who loses time and may lose the basket and some trust. A dashed arrow labelled “the same group, again and again” goes from the customer back to the honest payment. The same kind of customer makes the same kind of payment next week.
The course’s cost per week counts the 3 dollars of review time. It does not give the customer’s cost a money value: a lost basket, a call to support, a customer who goes to another shop. So the cost per week alone cannot show this harm. A slice table can.
Larkfield: Isolation Forest, slice by slice
Section titled “Larkfield: Isolation Forest, slice by slice”Priya, Larkfield’s data scientist, takes Isolation Forest at 50 alerts a day, the detector of the last lessons. Over the evaluation months, 2,274 of the 166,986 honest payments got an alert: 136.2 per 10,000. The function false_alerts_by() in the notebook gives the rate for each group of a column, with its counts:
def false_alerts_by(groups, flags=None): """False alerts per 10,000 honest payments, for each group; with the counts, so small groups show. flags: one True/False per evaluation payment (an alert or not); by default, Isolation Forest's top 50 a day.""" flags = alert if flags is None else flags t = pd.DataFrame({"group": np.asarray(groups)[honest], "alert": flags[honest]}) out = t.groupby("group").alert.agg(false_alerts="sum", honest_payments="size") out["per 10,000"] = (out.false_alerts / out.honest_payments * 10_000).round(1) return out.sort_values("per 10,000", ascending=False)
false_alerts_by(evaluation.account_type)The output:
false_alerts honest_payments per 10,000groupbusiness 1658 37337 444.1personal 616 129649 47.5The business customers made 22% of the honest payments, and they got 73% of the false alerts (1,658 of 2,274). Per 10,000 honest payments, they get 444.1 false alerts, against 47.5 for personal customers: 9.3 times as many.
Text description of the chart
A column chart of false alerts per 10,000 honest payments, for four groups. Business customers: 444.1. Personal customers: 47.5. Payments made abroad: 661.8, from only 272 honest payments. Payments in Canada: 135.3. The average over all honest payments is 136.2.
The device matters less: desktop payments get 199.5 false alerts per 10,000, mobile 117.0 and tablet 75.9. This slice partly repeats the last one: 59.4% of business payments come from a desktop, against 25.7% of personal payments.
Why the business customers
Section titled “Why the business customers”The behaviour features compare each payment with the customer’s own past and count the payments of the last hour. A business that buys for a project makes many large payments in one hour, from one desktop. To these features, that looks like card testing or a stolen card. Module 2 saw the same with the velocity rule: 1,785 of its 1,788 false alerts were business bulk orders.
Are the business alerts at least useful? Isolation Forest finds 243 of the 268 planted business payments (90.7%), and 2,033 of the 2,549 planted personal payments (79.8%). So it finds business frauds well, and it pays for that with many false alerts on honest businesses.
The same customers again and again
Section titled “The same customers again and again”A rate per payment does not show whether the false alerts are spread over many customers or fall on the same few. So Priya counts the customers behind them:
false = evaluation[alert & honest]per_customer = false.groupby("customer_id").size()account = evaluation.groupby("customer_id").account_type.first()pd.DataFrame({"customers": account.value_counts(), "with a false alert": account[per_customer.index].value_counts(), "with 3 or more": account[per_customer[per_customer >= 3].index].value_counts(), "most for one customer": per_customer.groupby(account[per_customer.index]).max()})The output:
customers with a false alert with 3 or more most for one customeraccount_typebusiness 2105 420 245 19personal 21289 558 7 3In 13 weeks, 420 of the 2,105 business customers got at least one false alert: one business in five. 245 of them got three or more, and one business got 19. Among 21,289 personal customers, only 7 got three or more. Like the garden centre’s customers, the same honest businesses are stopped again and again.
A small slice: payments made abroad
Section titled “A small slice: payments made abroad”Payments made abroad have the highest rate of all: 661.8 per 10,000. But look at the counts first. Only 272 honest payments were made abroad in three months, and 18 of them got an alert. A small slice moves a lot by chance, so Priya adds the 95% interval of the last lesson:
false_alerts honest_payments per 10,000 95% interval per 10,000groupabroad 18 272 661.8 366 to 957in Canada 2256 166714 135.3 130 to 141The rate abroad is somewhere between 366 and 957 per 10,000. Even its low end is well above Canada’s 135.3, so the difference is real. But “661.8” has a false precision: it rests on 18 alerts. Report the counts with the rate, and an interval when a slice is small. Customers who travel are honest, and the detector stops them often, because a new country is a strong sign of a stolen card.
Slices that are lower than expected
Section titled “Slices that are lower than expected”Priya also expected two groups to be stopped more: new customers, who have little past, and the new region east, live since 1 September 2025.
region false_alerts honest_payments per 10,000groupnorth 950 64227 147.9coast 470 32677 143.8south 610 46778 130.4east 244 23304 104.7
customer age false_alerts honest_payments per 10,000groupolder 2056 146872 140.0first 30 days 218 20114 108.4Both are lower than average. Customers in their first 30 days get 108.4 false alerts per 10,000, against 140.0 for older customers. The east region gets 104.7, the lowest of the four regions. A slice table that finds no problem is useful too: it replaces a guess with evidence.
What Larkfield does about the business slice
Section titled “What Larkfield does about the business slice”Priya, Sofia and Omar discuss the business slice. Several fixes are possible, and each one has a cost:
- Do not block on the score alone. An alert sends the payment to a reviewer, not to an automatic block. For a business alert, the first step is a call or a message to the customer. This costs review time, and it protects the customer.
- Give the detector the missing context. The features treat a bulk order like card testing. A feature such as “this business usually orders in bulk” could help. It must be measured again in the comparison table, and it can also hide a real takeover of a business account.
- Report the slice with the other numbers. Every report of the detector shows precision at the budget and the cost per week. It also shows the false alerts per 10,000 honest payments for business and for personal customers.
A separate quota of alerts for each group is risky: it can hide real fraud in one group to make the numbers look equal. The AI security course discusses such trade-offs.
When to slice, and what it costs
Section titled “When to slice, and what it costs”Slice every detector whose alerts touch customers, and slice by the groups that the business can act on: account type, country, device, region, customer age. Slice again after every change of the detector or its features.
Slicing is cheap: the alerts and one column for each group. The traps are in the reading. A small slice moves by chance, so always show its count and an interval. Many slices mean many comparisons, and some will look high by chance, as in the drift tests of Module 3. And some groups need data that you should not collect without a good reason, such as a person’s age or origin. Ask before you add such a column; the AI security course covers this.
Guided practice: slice the false alerts
Section titled “Guided practice: slice the false alerts”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, predict whether the business customers are close to the average. Then run the tiny example. You see
1050.0and62.5. - In section 2, run the cell that fits Isolation Forest. You see
136.2 per 10,000. - In section 3, run the account-type slice. You see
444.1for business customers. - In section 4, run the abroad slice. You see
366 to 957. - In section 5, run the device, region and age slices. You see
108.4for the first 30 days. - Run the customer count. You see
245business customers with 3 or more false alerts.
Your turn: two more slices
Section titled “Your turn: two more slices”The notebook’s section 6 has two tasks. First, slice the false alerts by merchant_category, and write the category with the highest rate as worst_category. Second, compute the false alerts per 10,000 honest night payments (from 00:00 to 05:59, hour from 0 to 5) as night_per_10000. Before you run them, predict which category is the worst.
Show the answers
by_category = false_alerts_by(evaluation.merchant_category)worst_category = by_category.index[0]night = np.where(evaluation.hour.between(0, 5), "night", "day")night_per_10000 = float(false_alerts_by(night).loc["night", "per 10,000"])worst_category is furniture, with 350.2 false alerts per 10,000 honest payments, then tools with 234.7. Electronics has 40.5, the lowest. Large orders of furniture and tools are what business customers buy. night_per_10000 is 841.5, against 112.6 by day: honest customers who pay at night are stopped about 7.5 times as often. The night slice has 5,395 honest payments, so it is not a tiny slice.
Common mistake
A good average, and one group that pays
Symptom: Tomás reports the detector to Omar: “Only 136 false alerts per 10,000 honest payments, about 1.4%. Customers will hardly notice.” Two weeks later, support has many calls from business customers whose orders were held. Some had three or four orders held.
Cause: the average mixes 129,649 honest personal payments, with 47.5 false alerts per 10,000, and 37,337 honest business payments, with 444.1. 245 businesses got three or more false alerts in 13 weeks. The average is right, and it describes nobody.
Fix: slice the false alerts by the groups that matter, with their counts, before a detector goes live. Report the worst slice beside the average. Then change the action for that slice, such as a call before any block, or change the features, and measure again.
Consequence in AI work: a system that is fine on average can treat one group badly, again and again, without any error in a log. The harm shows up late, as complaints or lost customers, and it can also be a legal and reputation problem.
Module practice: defend a detector to Sofia
Section titled “Module practice: defend a detector to Sofia”This practice joins the four lessons of the module. The task: choose between Isolation Forest and kNN distance for Sofia’s team, at 50 alerts a day. Defend the choice in three sentences, with precision at the budget, the cost per week and one slice result.
Guided: compare the two detectors
Section titled “Guided: compare the two detectors”In the notebook’s section 7, Priya scores both detectors with compare.score_payments() and slices kNN’s false alerts by account type:
from sklearn.neighbors import NearestNeighborsX_ref = scaler.transform(behaviour.loc[reference.index])sample = X_ref[np.random.default_rng(0).choice(len(X_ref), 20_000, replace=False)] # kNN, as in Module 4dist, _ = NearestNeighbors(n_neighbors=10).fit(sample).kneighbors(scaler.transform(behaviour.loc[evaluation.index]))knn = dist[:, -1]table = compare.Table()table.add(compare.score_payments(evaluation, score, "Isolation Forest", "behaviour features, 50 a day"))table.add(compare.score_payments(evaluation, knn, "kNN distance", "behaviour features, 50 a day"))table.show()knn_alert = compare.daily_alerts(evaluation, knn, 50).set_index("payment_id").reviewed.reindex(evaluation.payment_id).to_numpy()false_alerts_by(evaluation.account_type, knn_alert)The output:
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)Isolation Forest behaviour features, 50 a day 50.0 0.500 0.808 3218 None kNN distance behaviour features, 50 a day 50.0 0.505 0.816 2739 None
false_alerts honest_payments per 10,000groupbusiness 1056 37337 282.8personal 1196 129649 92.2kNN distance has almost the same precision (0.505 against 0.500), and it costs 479 dollars a week less (2,739 against 3,218). Its false alerts fall less on businesses: 282.8 per 10,000 honest business payments, against 444.1 for Isolation Forest. They fall more on personal customers: 92.2 against 47.5. So kNN spreads its false alerts more evenly: business customers get 3.1 times the personal rate, not 9.3.
Priya’s three sentences for Sofia:
We recommend kNN distance at 50 alerts a day: 50.5% of its alerts are planted frauds. It costs 2,739 dollars a week, against 3,218 for Isolation Forest (missed frauds plus 3 dollars per false alert, our assumptions). Business customers still get 282.8 false alerts per 10,000 honest payments, three times the personal rate, so every business alert starts with a call, not a block. The precision comes from planted frauds; in the first month, we review a random sample of 200 alerts to check it on real payments.
Your task: a different slice
Section titled “Your task: a different slice”Write the same three sentences for a fraud lead who cares most about customers who travel. Use the notebook to slice kNN’s false alerts by “paid abroad” (np.where(evaluation.country == "CA", "in Canada", "abroad"), with knn_alert as the flags), and add the 95% interval as in section 4. Say whether the slice is large enough to trust, and what Larkfield should do for a payment made abroad. There is no single right answer: a good answer uses the numbers that you computed, with their counts.
Module check
Section titled “Module check”Defend a detector choice to a fraud lead, as in the module practice. Your answer must include the detector, its budget and its precision at the budget. It must include its cost per week, with the costs that you assumed. And it must include one slice result with its counts, and what you will do about it. Say which number comes from planted cases, and how you will check it on real payments.
Knowledge check
Section titled “Knowledge check”Check your understanding
Isolation Forest sends 136.2 false alerts per 10,000 honest payments. Business customers get 444.1 and personal customers 47.5. Explain how the average can be low while one group gets so many, and why the slice matters more than the average to a business customer.
Show the answer
The average weights each group by its number of payments. Personal customers made 129,649 of the 166,986 honest payments, so their low rate pulls the average down. The business customers’ high rate counts for only 37,337 payments. To one business customer, the average is not their experience. One business in five got a false alert in 13 weeks, 245 got three or more, and one got 19. Each false alert can hold a real order, so the harm falls again and again on the same customers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Payments made abroad get 661.8 false alerts per 10,000 honest payments, from 18 false alerts among 272 honest payments. How should you report this number, and what does it tell you?
Show the answer
Report it with its counts and its 95% interval: 18 of 272, about 366 to 957 per 10,000. The slice is small, so the exact rate is uncertain. But even the low end of the interval is well above Canada’s 135.3, so the difference is real. The detector stops honest customers who pay abroad much more often. A new country is a strong sign of a stolen card, so the detector treats travellers as suspects.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás expected new customers (first 30 days) and the new region east to get the most false alerts. The slices show 108.4 and 104.7 per 10,000, below the average of 136.2. What do you learn from this result?
Show the answer
A slice that shows no problem is still evidence: it replaces a guess with a measurement. Here the guess was wrong, and the groups at risk were others: business customers, customers who pay abroad, night payments. So measure every group that the business can act on, and do not decide in advance which ones are at risk. Slice again after every change of the detector, because a new feature can move the false alerts onto another group.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- One average of false alerts can hide a group that gets many more. Count false alerts per 10,000 honest payments, slice by slice: business customers get 444.1, personal customers 47.5.
- Count the customers too: 245 of 2,105 business customers got three or more false alerts in 13 weeks; one got 19.
- Report every slice with its counts, and a 95% interval when it is small: payments abroad, 18 of 272, 366 to 957 per 10,000.
- Some expected slices are fine: new customers (108.4) and the new region (104.7) are below average. Measure, do not assume.
- A false alert can block an honest customer. Count that apart from the money, and change the action for the slice: a call before any block. The fairness of such decisions is the topic of the AI security course.
In this module, you measured a detector by its precision at the budget, its cost per week and its slices, with and without a full answer key. But every review that Sofia’s team makes is a label, and the detector ignores all of them. Next, you use the reviewers’ labels to re-rank the alerts.