Card fraud with a review budget
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we found Larkfield's planted frauds with rules, robust scores, isolation, an autoencoder and a re-ranker, all under a review budget. Now we try the same method on a published set of card payments with its own labels. In this case study, we compare baselines, detectors without labels and a supervised model, and choose by the cost per week at a daily budget.
Case study 1 of 4 · Anomaly Detection: Find What Is UnusualCard fraud with a review budget
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Take a year and a half of card payments to a daily review queue that you can defend with numbers. You will see why accuracy says nothing here, and build features from each card’s own past. You will compare baselines, two detectors without labels and a supervised model, at one budget and with one measure. Then you choose one, test it once and run it on a month that it has never seen.
The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install pandas and scikit-learn and download the data (21.5 MB). It needs no account and no local setup. It runs in about one minute after the install.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- Why rare labels make accuracy useless, from Labels are rare and Why accuracy lies here.
- Z-scores and robust scores, from Z-scores and why the mean is fooled and Median, MAD and IQR.
- kNN distance and Isolation Forest, from Far from the neighbours: kNN distance and Isolation Forest.
- Features from a customer’s own past, from Features decide more than the algorithm.
- The budget, precision at the budget and cost per week, from The alert budget.
- A model trained on labels, from From unsupervised to semi-supervised.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, NumPy 2.5.4 and scikit-learn 1.9.1. The setup cell makes a folder anomaly-course. To start again, delete that folder.
Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”Imagine a small card issuer: a bank that gives cards to about a thousand customers. Each day, its customers make about 2,500 payments with their cards. A few of those payments are fraud: a criminal uses a stolen card number.
Farah leads the review team. Her team can check 20 alerts a day with care: call the customer, block the card if needed. Kofi, a data scientist, must decide which 20 payments the team sees each day. We invented Farah, Kofi and the bank for this case study. The payments come from a published simulation, described below.
Every alert has a cost, and so does every fraud that nobody reviews. The case study uses the course’s costs, which are assumptions:
- A missed fraud costs its amount, plus a 15-dollar chargeback fee: the bank’s cost when a customer disputes a payment.
- A false alert costs 3 dollars, about 4 minutes of a reviewer’s time.
Farah asks Kofi: “Which 20 payments should my team read each day? And how much does your choice save us?”
What you will build
Section titled “What you will build”You will build a daily review queue: each day, the 20 payments that the team should read first. You will compare a simple rule, two detectors that need no labels, and a supervised model trained on past frauds. All of them are compared at the same budget, with the same measure: the cost per week of missed frauds and false alerts. You then test the choice once, and run it on December 2020, a month with twice the usual payments.
The data
Section titled “The data”The data is the Credit Card Transactions Fraud Detection Dataset by Kartik Shenoy on Kaggle. It has 1,852,394 card payments from 1 January 2019 to 31 December 2020, with a label for each payment: fraud or not.
The payments are simulated. Kartik Shenoy made them with the Sparkov generator by Brandon Harris. The generator invents customers, shops and spending habits, and adds fraud in a fixed way. No real person or card is behind any row.
Why not the famous fraud dataset?
Section titled “Why not the famous fraud dataset?”Most fraud tutorials use the ULB credit-card fraud data: real payments from European cardholders. We checked its licence: the Open Database License with the Database Contents License, which adds share-alike terms. The data of this course is under simpler terms: public domain, CC0, CC BY 4.0, MIT and Apache-2.0. Some other Nextia courses use share-alike data (CC BY-SA) or CC BY 3.0 data, each under its own terms. It also covers only two days and has no card ID, so it has no daily budget over weeks and no card history. We chose a simulated set with an open licence, and we say clearly where a simulation can mislead.
These are the columns that the case study uses:
| Column | What it means |
|---|---|
ts |
The date and time of the payment. |
card_id |
The card, from K-0001 to K-0999. We replaced the simulated card numbers. |
category |
The kind of shop, for example grocery_pos (in a shop) or shopping_net (online). There are 14. |
amount |
The amount, in dollars. |
is_fraud |
1 for fraud, 0 for an honest payment. |
gender, birth_year |
Simulated facts about the cardholder. We use them only to check that the alerts are fair, never as model inputs. |
First checks
Section titled “First checks”tx = pd.read_csv("data/card_transactions.csv.gz", parse_dates=["ts"])day = tx.ts.dt.floor("D")print(f"{len(tx):,} transactions, {tx.card_id.nunique()} cards, {day.min():%Y-%m-%d} to {day.max():%Y-%m-%d}")print(f"fraud: {tx.is_fraud.sum():,} ({tx.is_fraud.mean():.2%}) | cards with fraud: {tx[tx.is_fraud == 1].card_id.nunique()}")print(f"missing values: {tx.isna().sum().sum()} | repeated rows: {tx.duplicated().sum()} | amount from {tx.amount.min()} to {tx.amount.max():,}")print("days with no payment:", [f"{d:%Y-%m-%d}" for d in pd.date_range(day.min(), day.max()).difference(day.unique())])1,852,394 transactions, 999 cards, 2019-01-01 to 2020-12-31fraud: 9,651 (0.52%) | cards with fraud: 976missing values: 0 | repeated rows: 0 | amount from 1.0 to 28,948.9days with no payment: ['2020-02-29'] mean min 50% maxtransactions 2537.5 1073.0 2306.0 6530.0fraud 13.2 0.0 12.0 50.0What this tells us. The data is clean: no missing values and no repeated rows. One day is missing, 29 February 2020: the simulator skips the leap day. A day has about 2,540 payments and 13 frauds. So the budget of 20 is more than the mean number of frauds. But on some days there are 50 frauds, and on others none.
How fraud looks in this data
Section titled “How fraud looks in this data”fraud episodes: 976 | payments per episode: median 10 (from 2 to 19) | days per episode: median 1.9fraud between 22:00 and 03:59: 84.6% | all payments: 23.5%median amount ($): {'honest': 47.24, 'fraud': 390.0}An episode is all the frauds on one card. Here, each card with fraud has exactly one episode: about 10 payments in about 2 days. Most frauds happen late at night, and they are larger than honest payments.
- Frauds: share in each hour
- Honest payments: share in each hour
Text description of the chart
Two lines over the 24 hours of the day. The honest payments are almost flat: about 3.3% in each hour from midnight to 11:00, and about 5% in each hour from noon to 23:00. The frauds are low in the day, under 1.2% in each hour from 04:00 to 21:00. Then they jump to 25.7% at 22:00 and 25.3% at 23:00, and stay between 8.2% and 8.6% in each hour from midnight to 03:00.
What this tells us. This is a simulator, and its fraud is tidy. Real fraud is less regular. A detector can learn the simulator’s habits, for example “late night means fraud”. So the numbers on this page show how to compare and choose. They do not show how well a method works at a real bank.
The base-rate trap
Section titled “The base-rate trap”Think of a smoke alarm that is right 99% of the time. It sounds that good. But in a year, a kitchen has thousands of moments when someone cooks and only one real fire. If the alarm goes off at 1% of the cooking moments, it rings dozens of times for every fire. After a month, nobody believes it.
The same thing happens with fraud. The base rate is the share of payments that are fraud: here, 0.6% in the first half of 2020. When the base rate is small, even a tiny share of false alerts on the honest payments is many alerts. This is the base-rate trap: a detector that looks accurate can still send mostly false alerts.
Text description of the diagram
The diagram starts with 10,000 payments, with a fraud rate of 0.6%, as in January to June 2020. It splits them into 60 frauds and 9,940 honest payments. From the frauds, an arrow says “the detector finds 90%” and leads to 54 real alerts, with 6 frauds missed. From the honest payments, an arrow says “it wrongly flags only 1%” and leads to 99 false alerts. Both lead to the last box: 153 alerts, of which 54 are fraud. The precision is 35%. The accuracy is 98.95%, but most alerts are wrong.
The notebook does the same on the real months:
accuracy of 'nothing is fraud': 99.40%per day: 13.4 frauds, 2,207 honest paymentsalerts per day: 34.1 (12.0 real, 22.1 false) | precision 0.353 | accuracy 98.95%What this tells us. “Nothing is fraud” is 99.40% accurate, and it catches nothing. The good-looking detector sends 34 alerts a day to a team that can read 20, and two of every three are false. So we never judge a detector by accuracy. We judge it at the team’s budget: of the 20 payments read each day, how many are fraud, and what does the rest cost?
Frame the problem
Section titled “Frame the problem”Kofi writes down his answers before he builds any model:
- Unit: one payment. The team reviews payments, and the costs are per payment.
- Budget: 20 alerts a day, an assumption. Each day, the team reads the 20 highest scores. Ties keep the time order.
- One measure for the choice: the cost per week. Missed frauds cost their amount plus 15 dollars, and each false alert 3 dollars. Precision at the budget and recall stand next to it, as in The alert budget.
- What is known at the time of a payment: only the card’s earlier payments. Every feature uses the past only. The label of a payment comes weeks later, when the cardholder complains.
- Sensitive columns:
genderand the age stay out of every detector. They are used only at the end, to check the alerts group by group.
Split by time
Section titled “Split by time”A detector will always score the future with what it learned in the past. So the split follows time, as in Train, validation and test sets:
Text description of the diagram
Four boxes from left to right. The first is 2019, where the detectors learn: 924,850 payments with 5,220 frauds. The detectors without labels learn what normal looks like, and the supervised model learns from the labels. The second is January to June 2020, where Kofi compares the detectors and chooses: 401,883 payments with 2,419 frauds. The third is July to November 2020, the test, used once: 386,123 payments with 1,754 frauds. Nothing changes after it. The last is December 2020, the new input: 139,538 payments with 258 frauds. The model for it is retrained on the labels to 31 October.
Features from the card’s own past
Section titled “Features from the card’s own past”A payment of 400 dollars is normal for one card and strange for another. So, as in Features decide more than the algorithm, each payment is compared with its card’s past. Kofi uses four features:
| Feature | What it measures |
|---|---|
log_amount |
The size of the payment (the logarithm, so 10, 100 and 1,000 dollars are equal steps). |
hour |
The hour of the day. |
log_spend_24h_vs_usual_day |
The card’s spend in the last 24 hours, compared with its usual day: the median of its last 30 days with payments. |
night_payments_24h |
The card’s payments between 22:00 and 05:59 in the last 24 hours. |
The function card_features() in the notebook builds them, plus three columns for the rule and the baselines. Each feature uses only the card’s payments before this one, or this one itself:
log_amount hour log_spend_24h_vs_usual_day night_payments_24hhonest 3.86 14.0 0.34 1.0fraud 5.97 22.0 2.70 4.0What this tells us. These are the medians. A typical fraud is about 8 times larger (5.97 against 3.86 on the log scale). It comes on a day when the card spends about 15 times its usual (2.70 on the log scale). And it is the fourth night payment in 24 hours. A typical honest payment is none of these.
Baselines first
Section titled “Baselines first”Kofi starts with four simple scores. Each one gives a number to every payment, and the team reads the 20 highest each day:
- z-score: how far the amount is from the mean of all 2019 payments, one baseline for all cards (Z-scores).
- Robust z per card: how far the amount is from the card’s own median, in units of the card’s spread (the interquartile range ÷ 1.349, as in Median, MAD and IQR).
- Rule: a payment at night that is 3 times the card’s usual amount. It gives 1 or 0, so its ties keep the time order.
- Rule, ranked: the same rule, but the payments most over the limit come first.
On the validation months, January to June 2020:
| Detector | Alerts a day | Precision at 20 | Recall | Cost per week ($) |
|---|---|---|---|---|
| No detector | 0 | 0 | 0 | 51,430 |
| z-score of the amount | 20 | 0.288 | 0.431 | 14,071 |
| Robust z per card | 20 | 0.144 | 0.215 | 33,095 |
| Rule | 114.6 | 0.063 | 0.094 | 47,627 |
| Rule, ranked | 114.6 | 0.284 | 0.425 | 18,789 |
What this tells us. Doing nothing costs 51,430 dollars a week: frauds are large here. Simply reading the 20 largest amounts of each day cuts that to 14,071. The rule fires 115 times a day, so its first 20 by time are mostly honest. Ranked, it is close to the z-score on precision but more expensive. The robust z per card is the worst. A card’s own amounts vary a lot, so one honest travel booking looks extreme: 1,138 of its 3,099 false alerts are in the travel category.
Detectors without labels
Section titled “Detectors without labels”Next, Isolation Forest and the kNN distance, with the course’s settings from Isolation Forest and Far from the neighbours. They learn what normal looks like from 2019, and they never see a label:
forest_raw = IsolationForest(n_estimators=200, random_state=0).fit(X.loc[reference.index, ["log_amount", "hour"]])forest = IsolationForest(n_estimators=200, random_state=0).fit(X.loc[reference.index, FEATURES])scaler = StandardScaler().fit(X.loc[reference.index, FEATURES])sample = np.random.default_rng(0).choice(len(reference), 20_000, replace=False)neighbours = NearestNeighbors(n_neighbors=10).fit(scaler.transform(X.loc[reference.index, FEATURES])[sample])| Detector | Precision at 20 | Recall | Cost per week ($) |
|---|---|---|---|
| Isolation Forest, raw (amount and hour) | 0.264 | 0.396 | 17,288 |
| Isolation Forest, card features | 0.370 | 0.554 | 16,203 |
| kNN distance, card features | 0.302 | 0.451 | 29,971 |
What this tells us. The card features help Isolation Forest, as in Module 4: precision rises from 0.264 to 0.370. It finds more frauds than the z-score. But it costs more per week: 16,203 dollars against 14,071. In the six months, Isolation Forest’s queue holds 1,341 frauds worth 897,586 dollars, and 157 of them are under 100 dollars. The z-score reads the 20 largest amounts of each day: its queue holds fewer frauds, 1,043, but they are worth more, 958,086 dollars. So, without labels, sorting by amount wins on cost. The measure that you choose decides the winner.
A supervised model, trained on the past
Section titled “A supervised model, trained on the past”Kofi has labels: the bank knows which payments of 2019 were fraud. So he can train a model that learns what fraud looks like. He uses gradient boosting, an ensemble of small decision trees (scikit-learn’s HistGradientBoostingClassifier). Its inputs are the four card features and the shop category. It learns from 2019 only, and it never sees a label of 2020:
def train(part): model = HistGradientBoostingClassifier(max_iter=200, learning_rate=0.1, early_stopping=False, categorical_features="from_dtype", random_state=0) return model.fit(model_inputs(part), part.is_fraud)
model = train(reference) # labels of 2019 onlyp_fraud = model.predict_proba(model_inputs(validation))[:, 1]| Detector | Precision at 20 | Recall | Cost per week ($) |
|---|---|---|---|
| Gradient boosting, ranked by probability | 0.506 | 0.757 | 9,960 |
| Perfect (knows the answers, largest first) | 0.595 | 0.891 | 1,423 |
What this tells us. With labels, the model is far better than every detector without labels: 0.506 against 0.370 for Isolation Forest, and 9,960 dollars a week against 14,071 for the amount. The perfect row shows the ceiling. On days with more than 20 frauds, even a perfect queue cannot reach them all, and on quiet days it cannot fill 20 places with fraud.
Why not a threshold of 0.5?
Section titled “Why not a threshold of 0.5?”The model gives a probability. Why not send every payment above 0.5?
threshold 0.5: 13.6 alerts a day (from 0 to 38), precision 0.758, recall 0.768days with more than 20 alerts: 29 of 181accuracy at 0.5: 99.71%A fixed threshold sends a different number of alerts every day: none on some days, 38 on others. On 29 days, the team gets more than it can read. On many other days, its places stay empty. The budget keeps the queue at 20, and the threshold moves instead, as in The alert budget.
Common mistake
The new cards that looked normal
Symptom: in an earlier version, a new card’s first payment of 1,100 dollars online at 23:31 got a fraud probability of 0. The model trusted every new card.
Cause: a card with no past has no “usual day”. The first version compared the payment with itself, so the ratio was exactly 1: “usual”. In 2019, a card’s first day was almost always honest, because the data starts on 1 January 2019 and every card starts then. In 2020, every card that appeared for the first time began with a fraud:
2019: 2,880 payments on a card's first day, 294 of them fraudvalidation: 107 payments on a card's first day, 107 of them fraudFix: compare a card with no past with a typical card: the median payment and the median card-day of 2019. Then a new card’s 1,100-dollar night looks unusual, as it should. The precision at 20 rises from 0.455 to 0.506, and the cost per week falls from 15,902 to 9,960 dollars.
Consequence in AI work: “no history” is a value, not a gap to fill with anything. Decide on purpose what a new customer, card or device is compared with. Real fraud teams know this risk well: new accounts are where fraud often starts.
Text description of the diagram
On the left, a box: a new card’s first day, 1,100 dollars online at 23:31, with no earlier payments. An arrow leads to a question: compare with what? One branch says “with itself”: the usual day is this spend, the ratio is 1, and the payment looks usual. The model learned from 2019 that first days are honest. The other branch says “with a typical card of 2019”: the usual day is a typical card-day, the ratio is far above 1, and the payment looks unusual. The model can flag it.
Rank by expected loss
Section titled “Rank by expected loss”The model says how likely a payment is to be fraud. But Farah’s cost depends on how much each fraud takes. Think of a security guard at a shop door with time to check 20 bags. A bag that is likely to hold a stolen pen matters less than a bag that may hold a stolen laptop.
The expected loss of a payment is what it costs, on average, if nobody reviews it: the probability of fraud × (amount + 15). A tiny example with two payments:
| Payment | P(fraud) | Amount ($) | Expected loss ($) |
|---|---|---|---|
| A | 0.90 | 20 | 0.90 × (20 + 15) = 31.50 |
| B | 0.50 | 900 | 0.50 × (900 + 15) = 457.50 |
By probability, the team reads A first. By expected loss, it reads B first, because B is likely to cost about 15 times more. Kofi ranks the same model’s payments by expected loss:
def expected_loss(part, p): """The money that a payment is likely to cost if nobody reviews it: P(fraud) x (amount + fee).""" return p * (part.amount.to_numpy() + CHARGEBACK_FEE)| Gradient boosting | Precision at 20 | Recall | Cost per week ($) |
|---|---|---|---|
| ranked by probability | 0.506 | 0.757 | 9,960 |
| ranked by expected loss | 0.500 | 0.749 | 6,077 |
What this tells us. The same model, the same 20 places a day, almost the same precision. But the cost falls by 39%, from 9,960 to 6,077 dollars a week, because the team now reads the large frauds first. This ranking needs a probability, so only the supervised model can do it well. A detector without labels gives a score, not a probability.
The comparison and the choice
Section titled “The comparison and the choice”All the detectors on the validation months, at 20 a day:
Text description of the chart
A column chart with five columns, the cost per week at 20 alerts a day in January to June 2020. “Model”, gradient boosting ranked by expected loss: about 6,000 dollars, the lowest, in the first colour. The amount z-score: about 14,000. Isolation Forest on the card features: about 16,000. The ranked rule: about 19,000. kNN distance: about 30,000.
Kofi chooses gradient boosting on the card features, ranked by expected loss. It has the lowest cost per week, and a precision close to the best. He writes down two runners-up for later. The z-score of the amount is the fallback if labels ever stop arriving. Isolation Forest finds more frauds than the amount, but smaller ones.
The test, once
Section titled “The test, once”Now Kofi runs every detector on July to November 2020, once, and changes nothing after:
| Detector | Precision at 20 | Recall | Cost per week ($) |
|---|---|---|---|
| Perfect (the ceiling) | 0.528 | 0.921 | 802 |
| Gradient boosting, expected loss | 0.431 | 0.753 | 4,421 |
| Gradient boosting, probability | 0.432 | 0.753 | 7,307 |
| z-score of the amount | 0.245 | 0.428 | 11,224 |
| Isolation Forest, card features | 0.330 | 0.575 | 12,543 |
| Isolation Forest, raw | 0.221 | 0.386 | 14,417 |
| Rule, ranked (127.2 alerts a day) | 0.224 | 0.390 | 16,689 |
| kNN distance, card features | 0.256 | 0.446 | 25,696 |
| Robust z per card | 0.121 | 0.211 | 27,728 |
| Rule (127.2 alerts a day) | 0.048 | 0.084 | 40,200 |
| No detector | 0 | 0 | 43,245 |
Text description of the chart
A column chart with five columns, the cost per week at 20 alerts a day in July to November 2020. “Model”, gradient boosting ranked by expected loss: about 4,000 dollars, the lowest, in the first colour. The amount z-score: about 11,000. Isolation Forest: about 13,000. The ranked rule: about 17,000. kNN distance: about 26,000.
What this tells us. The choice holds. The order of the detectors is the same as on the validation months. Precision falls a little for every detector (0.500 to 0.431 for the choice), because the base rate falls from 0.60% to 0.45%. The chosen queue misses 434 frauds in five months. Their median amount is only 20.46 dollars: the queue lets the small frauds go, and catches the large ones (median 748.08 dollars).
What if the team could read more?
Section titled “What if the team could read more?”Farah asks whether more reviewers would pay. The notebook moves the budget on the validation months:
- Gradient boosting, expected loss
- z-score of the amount
- Perfect
Text description of the chart
Three lines of the cost per week against the budget, from 5 to 100 alerts a day. The chosen model: 27,568 dollars a week at 5, 15,222 at 10, 6,077 at 20, 3,348 at 30, 3,026 at 40 (the lowest), 3,322 at 60 and 4,033 at 100. The z-score of the amount: 45,083 at 5, 14,071 at 20, 6,372 at 50 and 3,013 at 100. The perfect queue, dashed: 1,423 at 20, 436 at 30 (its lowest) and 1,819 at 100.
What this tells us. Going from 20 to 30 alerts a day would cut the cost from 6,077 to 3,348 dollars a week, almost half. Beyond 40 a day, each extra alert is mostly false, and the cost rises again. One surprise: at 80 or 100 alerts a day, sorting by amount becomes cheaper than the model. With that many places, the team reads every large payment, and the large frauds hold the money. So the best detector depends on the budget. Choose them together.
December: the new input
Section titled “December: the new input”The last step runs the choice on new input: December 2020. The November labels are not complete yet, because chargebacks take weeks. So Kofi retrains on the labels up to 31 October, and scores every December payment:
detector setting alerts per day precision at budget recall cost per week ($) false alerts per week no detector nothing is reviewed 0.0 0.000 0.000 32744 0.0 z-score |z| of the amount 20.0 0.140 0.337 14101 120.4 Isolation Forest card features; fit on 2019 20.0 0.195 0.469 12839 112.7gradient boosting trained on 2019; expected loss 20.0 0.279 0.671 5653 100.9gradient boosting retrained to 2020-10-31; expected loss 20.0 0.281 0.674 4854 100.7 perfect knows the answers; largest first: the ceiling 20.0 0.400 0.961 573 84.0December: 4,501 payments a day (2020 before December: 2,359), 8.3 frauds a dayWhat this tells us. December has almost twice the usual payments, and fewer frauds. The base rate falls to 0.18%, a third of the validation months. So precision falls for every detector, even the perfect one (0.400). The retrained model still reviews 67% of the frauds, and costs 4,854 dollars a week, against 32,744 for doing nothing. Retraining helped the cost a little (5,653 to 4,854). This is the base-rate trap again, in production: normal moved, as in When normal moves.
One day of the queue
Section titled “One day of the queue”What does the team see? Here are the first 8 of the 20 alerts on 22 December 2020, ordered by expected loss, with the reasons that a reviewer reads:
2020-12-22: 6,123 payments, 7 frauds; the team's 20 alerts hold 5 ts card_id category amount amount_vs_usual night_payments_24h p_fraud expected_loss is_fraud1813534 2020-12-22 20:57:17 K-0288 travel 11586.34 470.42 6.0 1.00 11601.34 01814016 2020-12-22 22:31:48 K-0999 shopping_net 1039.42 1.20 6.0 1.00 1054.42 11811588 2020-12-22 14:30:10 K-0999 shopping_net 997.77 21.05 3.0 1.00 1012.77 11814246 2020-12-22 23:13:39 K-0999 misc_net 766.38 0.88 7.0 1.00 781.38 11813872 2020-12-22 22:05:48 K-0999 shopping_pos 1041.51 1.20 4.0 0.55 578.73 11808739 2020-12-22 01:49:59 K-0094 misc_net 719.80 14.55 4.0 0.59 436.82 01808796 2020-12-22 02:03:38 K-0224 shopping_pos 682.38 15.25 5.0 0.60 417.83 01814204 2020-12-22 23:06:03 K-0999 grocery_pos 289.27 0.31 6.0 1.00 304.27 1What this tells us. The first alert is honest: a travel booking of 11,586 dollars, 470 times the card’s usual. A reviewer clears it with one call, for 3 dollars. All 5 frauds in the queue are on one card, K-0999, which first appeared on 21 December. Its fraud started on its first days, and the typical-card default made it visible. A real team would group these into one alert per card, as in From score to alert.
Recommendation
Section titled “Recommendation”Kofi writes to Farah in her words:
- Each day, read the 20 payments with the highest expected loss from the gradient-boosting model. In the test months, this cost 4,421 dollars a week, against 43,245 for no review, 11,224 for “the 20 largest amounts” and 12,543 for Isolation Forest.
- Retrain every month on the labels that are complete, and watch the precision each week. In December, it fell from 0.431 to 0.281 because the base rate fell, not because the model broke.
- Ask for 10 more reviews a day. On the validation months, 30 alerts a day would cut the cost from 6,077 to 3,348 dollars a week. At 3 dollars an alert, 10 alerts cost 210 dollars a week.
- Treat new cards on purpose. Compare them with a typical card, and look at their first days with extra care.
- Keep the amount z-score as the fallback. It needs no labels and was the cheapest detector without them.
And what not to use it for:
- Not to block payments automatically. The queue is for people to review. At 20 a day, more than half of its alerts are honest customers.
- Not as proof that this works at a real bank. The data is simulated, with tidy fraud. Test again on real data before any decision.
Insights and limits
Section titled “Insights and limits”- Labels decided more than the algorithm. Isolation Forest and kNN never came close to the supervised model. When you have past labels, use them, and keep a detector without labels for frauds of a new kind.
- The measure decided the winner. By precision, Isolation Forest beat the amount z-score. By cost, the z-score won. Choose the measure with the business first.
- The simulator is tidy. One episode per card, fraud mostly at 22:00 and 23:00, every new card starting with fraud. A model can learn these patterns, and they would not hold at a real bank.
- The budget is an assumption. 20 a day is our choice. The best detector changed at 80 a day.
- No delay in the labels, except in December. In the comparison, we pretended that every label of 2019 was known on 1 January 2020. A real team gets labels weeks later.
- One payment is one alert. A team would group the alerts of one card. The course’s From score to alert shows how.
Responsible use
Section titled “Responsible use”- Check the alerts group by group. The gender and age of the simulated cardholders are never inputs. Still, the alerts can fall more on one group, as in Slices and fairness. On the test months, the chosen queue flags 47.9 honest payments per 10,000 for women and 42.1 for men. By age, it flags from 39.4 (ages 30 to 44) to 50.6 (60 and over). It finds 69.8% of the frauds on women’s cards and 81.6% on men’s. These differences come from the simulator’s spending profiles, but a real team must look for them, explain them and decide what is fair. Bias and accessibility in the AI security course shows how.
- A false alert is a real customer. It means a phone call, maybe a blocked card on a holiday. Keep a person in the decision, and count blocked honest customers apart from the money.
- Do not train on personal data that you do not need. We removed the simulated names, addresses and card numbers, although they are fake. Do the same with real data.
- Credit the sources. CC0 asks for nothing, but name Kartik Shenoy and the Sparkov generator, and say that the data is simulated.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can choose a detector by what it costs a business, and not by a single score. It also shows that you can find a hidden data trap and report a result honestly. Make it yours with one change, for example other costs, alerts grouped by card, or a second model. Then publish it so that someone can read it in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
comparison.csv: every detector on the validation months, andtest.csvapart, with the budget and the costs.- The data source and licence: the Kaggle page, CC0 1.0, “simulated with Sparkov”, and the SHA-256 of the file that you used.
A README template:
# A daily review queue for card fraud, chosen by cost
## ProblemA review team can check 20 card payments a day. A missed fraud costs its amount + $15,a false alert $3 (assumptions). Which 20 payments should it read?
## Data and licence"Credit Card Transactions Fraud Detection Dataset" (Kartik Shenoy, Kaggle), simulated withSparkov (Brandon Harris), CC0 1.0. 1.85 M payments, 2019-2020, 0.52% fraud. Simulated.
## ApproachFeatures from each card's past (spend vs usual day, night payments in 24 h). Time split:learn 2019, choose Jan-Jun 2020, test Jul-Nov 2020, run Dec 2020. Compared at 20 a day:amount z-score, robust z per card, a rule, Isolation Forest, kNN, gradient boosting.
## ResultsGradient boosting ranked by expected loss: $4,421 a week on the test months, against $11,224for the 20 largest amounts and $43,245 for no review. Without labels, the amount z-score wascheaper than Isolation Forest. December: the base rate fell to 0.18%, precision 0.43 -> 0.28.
## LimitsSimulated data with tidy fraud; budget and costs assumed; labels assumed on time.How to describe it in an interview, in two sentences: “I compared nine fraud detectors at a fixed review budget of 20 alerts a day, by the weekly cost of missed frauds and false alerts, on a time-ordered split. Ranking a gradient-boosting model by expected loss instead of probability cut the cost by about 40%, and I found and fixed a default that made every new card look safe.”
Go further:
- Group the alerts by card and day, and count one review per group. Does the cost change?
- Add Isolation Forest’s score as an input to the model, and test whether it helps on new kinds of fraud.
- Try the same steps on IBM’s TabFormer card data (Apache-2.0, 24 million rows), on a sample.
- Compare with the other case studies: Faults in machine or server metrics, Unusual readings in public open data and Control charts on a real production line.
Your turn: a smaller budget
Section titled “Your turn: a smaller budget”Before you run it, predict: at 10 alerts a day, is the chosen model’s precision on the validation months higher or lower than 0.500? In the notebook, section 12, compute it and put it in precision_at_10.
Show the answer
precision_at_10 = score_at_budget(validation, expected_loss(validation, p_fraud), "gradient boosting", budget=10)["precision at budget"]0.674Higher: 0.674. With fewer places, the team reads only the payments with the highest expected loss, and more of them are fraud. But the cost per week rises from 6,077 to 15,222 dollars, because many frauds wait with nobody to read them. Precision alone would choose the smaller budget, and the cost says it is the wrong choice.
Knowledge check
Section titled “Knowledge check”Check your understanding
'Nothing is fraud' is 99.40% accurate on the validation months, and a detector that finds 90% of the frauds and flags only 1% of the honest payments sends alerts that are 65% false. Explain both numbers, and say what Kofi measures instead.
Show the answer
Fraud is rare: 0.6% of the payments. So saying “no fraud” to every payment is right for the 99.4% that are honest, and catches nothing. The detector’s 1% of the many honest payments (22.1 a day) is more than its 90% of the few frauds (12.0 a day). So two of every three alerts are false, although its accuracy is 98.95%. This is the base-rate trap. Kofi measures the detector at the team’s budget. He looks at the precision of the 20 alerts read each day, the recall, and the cost per week of missed frauds and false alerts.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Without labels, Isolation Forest has a higher precision at 20 than the z-score of the amount, but the z-score costs less per week. Which one should Farah use if she had no labels, and why do the two measures disagree?
Show the answer
She should use the z-score of the amount. The decision is about money, and the cost per week measures that: 14,071 against 16,203 dollars a week on the validation months. The measures disagree because precision counts frauds, and every fraud counts the same. Isolation Forest finds more frauds, but many small ones. The z-score reads the largest payments, and the large frauds hold most of the money. With labels, ranking a model by expected loss, probability × (amount + 15), joins both ideas.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The first model gave a new card's 1,100-dollar payment at 23:31 a fraud probability of 0. What caused it, how did Kofi fix it, and what is the general lesson?
Show the answer
A new card has no past, so it has no usual day. The first version compared the payment with itself, so its spend ratio was 1, which means “usual”. In 2019, a card’s first day was almost always honest, because every card starts on 1 January 2019 when the data starts. So the model learned that first days are safe. In 2020, every new card started with fraud. Kofi compared cards with no past with a typical card of 2019 instead. Then the precision rose from 0.455 to 0.506. The lesson: a missing history needs a deliberate default, chosen for what it means, because a model learns from the default too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Accuracy says nothing when fraud is 0.5% of the payments. Compare detectors at the team’s budget, with one measure: here, the cost per week.
- Without labels, the 20 largest amounts (14,071 dollars a week) beat Isolation Forest (16,203) and kNN on cost, although Isolation Forest had a higher precision.
- With labels from the past, gradient boosting on the card features was the best. Ranked by expected loss, it cost 6,077 dollars a week on the validation months and 4,421 on the test months.
- A card with no past was compared with itself and looked safe. Compared with a typical card, new-card frauds became visible.
- In December, twice the payments and fewer frauds cut the base rate to 0.18%, and the precision fell for every detector.
Next, the second case study: faults in machine or server metrics, where the labels are windows in time, not payments.