Meet the Larkfield data
- Lesson 4
- 35 min
- Module 1: What "unusual" means
Where we are
We know the kinds of anomaly and the settings. So far, the Larkfield examples were only stories. In this lesson, we open the setup notebook, load every Larkfield table, read the rules of the anomalies we planted, and meet the comparison table that every module of the course fills.
Lesson 4 of 38 · Anomaly Detection: Find What Is UnusualMeet the Larkfield data
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Load every Larkfield table, check that your setup works, and score your first detectors in the comparison table that the whole course fills. A detector is only as good as the data under it and the way you score it.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the course data (about 17 MB), so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The alert budget, precision at the budget and the costs of the two mistakes, from The payment that did not fit.
- The kinds of anomaly, from Kinds of anomalies, and the base rate, from Labels are rare, and that changes everything.
- How to run a cell in a notebook, and how to read a pandas table. Machine Learning: From Problem to Reliable Model uses both from its first lessons.
What you need on your computer
-
The notebook route (recommended): a browser and a free Google account for Colab, or a free Kaggle account. Nothing to install.
-
The local route: Python 3.12 and VS Code with its Python and Jupyter extensions. Make a folder, then create a virtual environment and install the notebook kernel. On macOS and Linux:
Terminal window python3.12 -m venv .venvsource .venv/bin/activatepython -m pip install ipykernelOn Windows (PowerShell):
Terminal window py -3.12 -m venv .venv.venv\Scripts\Activate.ps1python -m pip install ipykernelDownload the notebook from the box above: the GitHub link has a download button. Open the notebook in VS Code, and choose the
.venvkernel. The notebook’s first cell installs the rest.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. The Windows commands and Colab were not tested.
Check the room before the match
Section titled “Check the room before the match”Before a football match, the referee walks onto the field. He checks the goals, the lines, the ball and the clock. None of this is the match. But if a goal is the wrong size or the clock is broken, every decision in the match is wrong, and nobody notices until later.
A detector is the same. If one day of data is missing, or the scores are in the wrong order, the detector still prints a number. Nothing raises an error. The number is just wrong. So before the course builds any detector, you check the data and the scoreboard once, carefully. That is this lesson.
Open the notebook and run the setup
Section titled “Open the notebook and run the setup”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the first setup cell. It installs the packages and prints one line per package, for example
pandas 3.1.1. - If you see other versions, restart the kernel, then run the cell again.
- Run the second setup cell. It downloads the data into a folder
anomaly-course/dataand checks each file’s checksum. It also downloadscompare.pyandfeatures.py, the course’s shared code. Then it makesanomaly-coursethe working folder, so the paths below start withdata/. - Check its last line. It starts with
Ready:and lists the files with their sizes. - Run the helpers cell. It loads pandas as
pd, NumPy asnpand matplotlib asplt, and defines the helpers of the check cells. It prints nothing.
The Ready: line looks like this:
Ready: payments.csv.gz (6.9 MB), customers.csv (1.2 MB), refunds_daily.csv (0.0 MB), refunds_agents.csv (0.5 MB), refunds.csv (0.6 MB), ticket_api_metrics.csv.gz (0.6 MB), ticket_api_logs.csv.gz (1.8 MB), ticket_api_events.csv (0.0 MB), orders.csv (0.0 MB), tickets_daily.csv (0.0 MB), calendar.csv (0.0 MB), sealed-key.txt (0.0 MB)A checksum is a short fingerprint of a file. If even one byte of the download is wrong, the fingerprint is different, and the cell stops with wrong checksum. Then delete the folder anomaly-course and run the cell again. If the cell says Could not download, check your internet connection. In Kaggle, also turn on Internet in the notebook settings.
The eleven tables
Section titled “The eleven tables”Section 1 of the notebook reads every table and prints its size:
files = ["payments.csv.gz", "customers.csv", "refunds_daily.csv", "refunds_agents.csv", "refunds.csv", "ticket_api_metrics.csv.gz", "ticket_api_logs.csv.gz", "ticket_api_events.csv", "orders.csv", "tickets_daily.csv", "calendar.csv"]tables = {name: pd.read_csv(f"data/{name}") for name in files}for name, table in tables.items(): print(f"{name:<28}{len(table):>9,} rows {', '.join(table.columns[:6])}{' ...' if len(table.columns) > 6 else ''}")The output:
payments.csv.gz 343,074 rows payment_id, ts, customer_id, account_type, region, amount ...customers.csv 24,038 rows customer_id, account_type, region, first_payment, signuprefunds_daily.csv 640 rows date, orders, refunds, refund_raterefunds_agents.csv 17,772 rows date, agent_id, orders, refunds, refund_amountrefunds.csv 9,923 rows refund_id, date, agent_id, order_id, region, amount ...ticket_api_metrics.csv.gz 86,400 rows ts, requests, error_rate, latency_p95_msticket_api_logs.csv.gz 137,013 rows ts, level, logger, messageticket_api_events.csv 22 rows ts, kind, detailorders.csv 2,181 rows date, channel, orderstickets_daily.csv 2,184 rows date, channel, ticketscalendar.csv 819 rows date, weekday, holiday, is_holidayThe tables form four groups. Each group belongs to one kind of question, and to the people who ask it:
Text description of the diagram
The diagram has four boxes on the left, each with an arrow to a question on the right. First, payments and customers: 343,074 card payments from June to November 2025. Their question: is this payment fraud? They are used for baselines, distance and isolation methods, deep methods, evaluation and the detection job. Second, refunds_daily, refunds_agents and refunds: 21 months of refunds. Their question: has the refund process changed? They are used for control charts, drift tests and business checks. Third, ticket_api_metrics, ticket_api_logs and ticket_api_events: 60 days of the help desk’s service, one row a minute. Their question: is the service failing? They are used for anomalies in time and in logs. Fourth, orders, tickets_daily and calendar: two years, one row a day. Their question: is today’s count unusual? They are used for anomalies in time.
| Tables | One row is | Owner at Larkfield |
|---|---|---|
payments, customers |
one card payment; one customer | Sofia, payments risk |
refunds_daily, refunds_agents, refunds |
one day; one agent on one day; one refund | Grace, support lead |
ticket_api_metrics, ticket_api_logs, ticket_api_events |
one minute; one log line; one release or change | Amira, developer |
orders, tickets_daily, calendar |
one channel on one day; one day | Mei, data engineer, owns the feeds; Grace reads the ticket counts |
The daily orders and tickets are the same numbers as in the forecasting course, so the two courses show the same Larkfield. The payments are a separate table at their own scale: about 1,875 payments a day. The orders table has about 676 orders a day. Never compare the two.
The payment that did not fit, in the data
Section titled “The payment that did not fit, in the data”Section 2 shows the last 12 payments of customer C-01234, the customer of the first lesson:
payments = pd.read_csv("data/payments.csv.gz", parse_dates=["ts"])hook = payments[payments.customer_id == "C-01234"]hook[["ts", "amount", "country", "device_id", "merchant_category"]].tail(12)The output:
ts amount country device_id merchant_category181950 2025-09-06 09:34:00 16.02 CA D-6674e29 kitchen194430 2025-09-13 09:29:08 16.75 CA D-6674e29 kitchen206782 2025-09-20 09:02:43 17.77 CA D-6674e29 kitchen219181 2025-09-27 09:17:38 15.16 CA D-6674e29 kitchen229857 2025-10-03 02:14:07 24.27 PT D-9f3e1a2 home229859 2025-10-03 02:15:05 19.26 PT D-9f3e1a2 gift_cards229860 2025-10-03 02:16:16 20.66 PT D-9f3e1a2 electronics229861 2025-10-03 02:17:17 18.88 PT D-9f3e1a2 electronics229862 2025-10-03 02:18:18 24.39 PT D-9f3e1a2 home229863 2025-10-03 02:19:13 19.16 PT D-9f3e1a2 electronics229864 2025-10-03 02:20:23 24.36 PT D-9f3e1a2 home229865 2025-10-03 02:21:31 19.25 PT D-9f3e1a2 homeThe story of the first lesson is in the data. Four Saturday baskets in Canada on one phone are followed by eight payments in Portugal on a new device, one a minute. The numbers on the left are the row numbers of the table. Row 229858 is missing from the list because it belongs to another customer.
What we planted: the rules, not the answers
Section titled “What we planted: the rules, not the answers”The data is synthetic, so we planted every problem on purpose and wrote them in an answer key. This section of the notebook tells you the rules of what we planted, not which rows. The counts come from the answer key:
| Table | Planted problems | How many |
|---|---|---|
| payments | Stolen cards: many small payments in a short time, often in a new country | 262 episodes, 2,558 payments |
| payments | Card testing: one device tries many customers’ cards with tiny payments | 52 episodes, 2,007 payments |
| payments | Account takeovers: a few expensive payments from a customer’s own account | 209 episodes, 406 payments |
| refunds | The refund rate rises by 0.3 points and stays; one agent refunds far more than the others; one agent’s refunds cluster just below the 100-dollar approval limit; some refunds are paid twice; the daily feed sends zeros for one day | see Module 3 |
| ticket-api | A slow memory leak; an error spike after a release; a configuration change | see Modules 5 and 6 |
In total, 4,971 payments are planted fraud, in 523 episodes. An episode is one fraud event: for example, the eight payments in Portugal are one episode.
Unusual, but normal
Section titled “Unusual, but normal”The data also contains events that look unusual and are honest. A good detector leaves them alone. We tagged them, so the course can count how many false alerts they cause:
| Normal but unusual | Payments tagged |
|---|---|
| Payments from the new region, east (live from 1 September 2025) | 21,795 |
| Split baskets: one order paid in several parts | 13,223 |
| The first payment in the Larkfield app (payment in the app from 15 October 2025) | 9,532 |
| Business customers’ bulk orders | 9,077 |
| Black Friday, 28 November 2025 | 5,087 |
| Borrowed devices: a device a customer used only once | 3,299 |
| Customers on a trip abroad | 703 |
| A customer’s new phone | 346 |
The answer key is sealed, not secret
Section titled “The answer key is sealed, not secret”The answer key is in the file data/sealed-key.txt, packed so that you do not read it by accident. The course’s compare.py unpacks it to score a detector. You can unpack it yourself with compare.reveal(): it is there to score your own work, not to keep a secret. But once you know the answers, your scores no longer say how a detector does on data that it has not seen. Treat it like the answers at the back of a book.
Daily orders, and a day that is missing
Section titled “Daily orders, and a day that is missing”Section 4 adds up the orders of the three channels for each day, in a series called daily, and draws it with the holidays:
orders = pd.read_csv("data/orders.csv", parse_dates=["date"])daily = orders.groupby("date").orders.sum() # one total per dayThe chart below shows the same daily totals:
Text description of the chart
The chart shows Larkfield’s orders on each of 727 days, from 1 April 2024 to 29 March 2026. The axis goes from 0 to 1,500. Labels mark the two Black Fridays and the heat wave. The line moves up and down every week, and it is higher in the summers. It averages 676 orders a day. The highest day is 5 July 2025, in a heat wave, with 1,372 orders. Black Friday makes a sharp, narrow peak each year: 1,059 orders in 2024 and 1,224 in 2025. The lowest days are at Christmas. One day, 19 October 2025, has no data, but you cannot see it. The line goes straight from 18 to 20 October.
The notebook’s chart prints the last Black Friday:
Black Friday 2025: 1224 ordersBlack Friday is one of the highest days of each year. Only a few days around the heat wave of early July 2025 are higher, with 1,372 orders on 5 July. Both are unusual, and both are normal: Larkfield plans for Black Friday every year, and hot weather sells parasols. In Module 5, a good detector does not report them.
The day that is not there
Section titled “The day that is not there”Before you run the next cell, the notebook asks you to predict: the orders feed dropped one day. Does the chart show a gap, a drop to zero, or nothing? The cell lists every day between the first and the last day that has no row:
all_days = pd.date_range(daily.index.min(), daily.index.max(), freq="D")missing = all_days.difference(daily.index)print("Days with no orders rows:", [str(d.date()) for d in missing])Days with no orders rows: ['2025-10-19']The chart shows nothing. A missing day is a data problem, and no detector that reads values can see it, because there is no value to read. A simple count of the rows per day finds it at once. Mei, who owns the feeds, adds this check before any detector runs. The same feed also left one day empty in the refunds table, as your second task shows.
The comparison table
Section titled “The comparison table”Every module of the course runs its detectors on the same data, scores them the same way, and adds one row to one comparison table. This is how you can compare a rule of Module 2 with a neural network of Module 6 fairly.
| Column | What it says |
|---|---|
| alerts per day | how many alerts the detector sends each day |
| precision at budget | of the 50 alerts that Sofia’s team reviews each day, the share that are planted anomalies |
| recall | the share of all planted anomalies that the reviewed alerts find |
| cost per week ($) | missed frauds (amount + 15 dollars each) and false alerts (3 dollars each): assumptions of the course |
| explain one alert (min) | how long a reviewer needs to understand one alert: an assumption that you state |
The same split for everyone
Section titled “The same split for everyone”compare.split() cuts the payments into two parts by date, the same way for every detector:
- The reference months, June to August 2025: 173,271 payments. A detector learns “normal” here.
- The evaluation months, September to November 2025: 169,803 payments over 91 days, with 2,817 planted, about 31 a day. Every row of the table is scored here.
The evaluation months are harder on purpose. They contain the new region, the app launch and Black Friday: normal changes that a detector must survive.
A first, silly detector
Section titled “A first, silly detector”Your first detector is silly on purpose: every day, alert on the 50 largest amounts. The notebook asks you to predict how many of these 50 are frauds before you run it. The first line imports compare.py, the shared code that the setup cell downloaded:
import comparereference, evaluation = compare.split(payments)table = compare.Table()table.add(compare.score_payments(evaluation, evaluation.amount, "biggest amounts", "the 50 largest amounts of each day"))table.show()score_payments takes the payments and one score per payment. Here the score is the amount, and a higher score means “more unusual”. It also takes a name and a short description of the setting. Each day, it sends the 50 payments with the highest scores to review. The output:
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 NoneA precision of 0.003 means that, on average, fewer than 1 of the 50 daily alerts is a planted fraud. The cost is 10,264 dollars a week. Doing nothing at all costs 10,367 dollars a week, so the largest amounts save almost nothing. Every one of its reviewed alerts was a business customer’s payment, and all of them were at least 478.28 dollars. Business customers buy in larger amounts, honestly. The last column says None, because nobody has stated how long one alert takes to explain. For your own detectors, write it in with explain_minutes=.
Change one thing: a random detector
Section titled “Change one thing: a random detector”Is “biggest amounts” at least better than chance? Section 6 adds a detector that picks 50 payments a day at random:
rng = np.random.default_rng(0)table.add(compare.score_payments(evaluation, rng.random(len(evaluation)), "random", "50 payments a day, chosen at random"))table.show() detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 None random 50 payments a day, chosen at random 50.0 0.019 0.031 11161 NoneRandom choice finds more fraud than the largest amounts: a precision of 0.019, close to the base rate of the evaluation months (1.66%). But the random detector costs more than doing nothing: 11,161 dollars a week. Its 343.2 false alerts a week cost 3 dollars each, and they catch very little fraud. A detector must earn its review time.
The chart puts these rows next to the third silly detector of your tasks, and next to the ceiling: a detector that knows the answers.
Text description of the chart
A column chart with four columns, on an axis from 0 to 0.6. The 50 biggest amounts of each day: 0.003. 50 payments at random: 0.019. The 50 smallest amounts: 0.192. A detector that knows the answers: 0.542. The ceiling is not 1, because on 78 of the 91 evaluation days fewer than 50 payments are planted.
The smallest amounts do much better than the largest: 0.192. Card testing uses payments of 1 to 5 dollars, so the smallest payments of a day contain many card tests. Still, a rule about the amount alone misses most frauds. Module 2 starts to compare each payment with the customer’s own usual amount.
Common mistake
Scores in a different order
Symptom: you sort the scores, but not the payments, before you score them. The notebook does it on purpose in section 7:
wrong = np.sort(evaluation.amount.to_numpy()) # sorted scores: no longer in the order of the paymentsrow = compare.score_payments(evaluation, wrong, "biggest amounts, wrong order", "scores sorted, payments not")print(row["precision at budget"], "(the right order gave", table.rows[0]["precision at budget"], ")")0.011 (the right order gave 0.003 )The row looks normal. Nothing warns you.
Cause: score_payments needs one score per payment, in the same order as the table that you give it. Sorting moved the scores to other payments. The detector is now a different, accidental detector, and its row means nothing.
Fix: compute the score as a column of the same table, such as evaluation.amount or the output of a model on the same rows. Never sort one without the other. If you must sort, sort the whole table, then compute the scores.
Consequence in AI work: a mismatch between rows and scores gives a believable number for the wrong detector. In a real comparison, it can make a weak method look good, or a good method look weak, and the team chooses the wrong one.
Your turn: three tasks
Section titled “Your turn: three tasks”Section 8 of the notebook has three tasks. Each one has a check cell that says Check passed or Not yet.
- Count the payments on Black Friday, 28 November 2025. Write the number as
bf_payments. - Count the rows of
refunds_daily.csvwhoseordersis empty. Write the number asmissing_orders. - Score a third silly detector: every day, the 50 smallest amounts. Keep its row as
smallest, and add it totable. - Run each check cell. Each one prints
Check passedwhen your answer is right.
Show the answers
bf_payments = int((payments.ts.dt.floor("D") == "2025-11-28").sum())refunds_daily = pd.read_csv("data/refunds_daily.csv", parse_dates=["date"])missing_orders = int(refunds_daily.orders.isna().sum())smallest = compare.score_payments(evaluation, -evaluation.amount, "smallest amounts", "the 50 smallest amounts of each day")table.add(smallest)table.show() detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min) biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 None random 50 payments a day, chosen at random 50.0 0.019 0.031 11161 Nonesmallest amounts the 50 smallest amounts of each day 50.0 0.192 0.310 9966 NoneThe check cells print Check passed: bf_payments is 5216., Check passed: missing_orders is 1. and Check passed: the precision at budget of the smallest amounts is right.
Black Friday has 5,216 payments, against about 1,875 on an average day: unusual, and normal. The refunds table has one day with no orders, the same day that the orders feed dropped. The minus sign in -evaluation.amount turns “smallest” into “highest score”, because score_payments always reviews the highest scores first.
If something breaks, delete the folder anomaly-course and run the notebook again from the top.
Knowledge check
Section titled “Knowledge check”Check your understanding
The orders feed dropped 19 October 2025. Why does the line chart of daily orders not show it, and what check finds it?
Show the answer
The day has no row at all, so there is no value to draw. The chart’s line joins 18 October and 20 October, and the gap is invisible. No detector that reads values can see it either. A simple check finds it at once: compare the days that have rows with the list of all days between the first and the last. It prints Days with no orders rows: ['2025-10-19']. Run this kind of check on the data before any detector.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: the random detector has a higher precision at the budget than “biggest amounts” (0.019 against 0.003), but it costs more per week than doing nothing (11,161 against 10,367 dollars). How can both be true?
Show the answer
A random choice finds fraud at about the base rate of the evaluation months (1.66%). The largest amounts are all business customers’ payments, almost all honest. So random wins on precision. But every alert costs review time. The random detector sends 343.2 false alerts a week at 3 dollars each, and catches very little fraud. The review time costs more than the fraud it saves. A detector is only worth running if its alerts save more than they cost.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does every detector in the course use the same split (reference months June to August, evaluation months September to November) and the same budget?
Show the answer
So that the rows of the comparison table can be compared. Suppose one detector is scored on a quiet month and another on the month of the app launch. Or one gets 50 alerts a day and another 200. Then a difference in the table could come from the data or the budget, not from the method. With the same split, the same budget and the same costs, the only difference between two rows is the detector.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The course has eleven tables in four groups: payments, refunds, the
ticket-apiservice, and the daily orders and tickets. - We planted 4,971 fraud payments in 523 episodes. We also tagged honest events that look unusual: trips, bulk orders, the new region, the app launch, Black Friday.
- The answer key is sealed, not secret:
compare.pyuses it to score, and you can unpack it, but then your scores mean less. - A missing day is invisible in a chart and to any detector that reads values. Count the rows per day.
- The comparison table scores every detector on the same evaluation months, at 50 alerts a day, with the same costs. The 50 largest amounts score 0.003, random 0.019, the smallest amounts 0.192, and the ceiling is 0.542.
You can now load the data and score a detector fairly. But so far, every detector looked at all payments together, and the largest amounts found almost nothing. Next, in Module 2, you measure how far each payment is from normal with a z-score. You also see how one huge payment fools the mean.