Skip to content
Français

Meet the Larkfield data

  • Lesson 4
  • 35 min
  • Module 1: What "unusual" means

Where we are

We know the kinds of anomaly and the settings. So far, the Larkfield examples were only stories. In this lesson, we open the setup notebook, load every Larkfield table, read the rules of the anomalies we planted, and meet the comparison table that every module of the course fills.

Lesson 4 of 38 · Anomaly Detection: Find What Is UnusualMeet the Larkfield data
Course overview

What you will be able to do

Load every Larkfield table, check that your setup works, and score your first detectors in the comparison table that the whole course fills. A detector is only as good as the data under it and the way you score it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the course data (about 17 MB), so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The notebook route (recommended): a browser and a free Google account for Colab, or a free Kaggle account. Nothing to install.

  • The local route: Python 3.12 and VS Code with its Python and Jupyter extensions. Make a folder, then create a virtual environment and install the notebook kernel. On macOS and Linux:

    Terminal window
    python3.12 -m venv .venv
    source .venv/bin/activate
    python -m pip install ipykernel

    On Windows (PowerShell):

    Terminal window
    py -3.12 -m venv .venv
    .venv\Scripts\Activate.ps1
    python -m pip install ipykernel

    Download the notebook from the box above: the GitHub link has a download button. Open the notebook in VS Code, and choose the .venv kernel. The notebook’s first cell installs the rest.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. The Windows commands and Colab were not tested.

Before a football match, the referee walks onto the field. He checks the goals, the lines, the ball and the clock. None of this is the match. But if a goal is the wrong size or the clock is broken, every decision in the match is wrong, and nobody notices until later.

A detector is the same. If one day of data is missing, or the scores are in the wrong order, the detector still prints a number. Nothing raises an error. The number is just wrong. So before the course builds any detector, you check the data and the scoreboard once, carefully. That is this lesson.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the first setup cell. It installs the packages and prints one line per package, for example pandas 3.1.1.
  3. If you see other versions, restart the kernel, then run the cell again.
  4. Run the second setup cell. It downloads the data into a folder anomaly-course/data and checks each file’s checksum. It also downloads compare.py and features.py, the course’s shared code. Then it makes anomaly-course the working folder, so the paths below start with data/.
  5. Check its last line. It starts with Ready: and lists the files with their sizes.
  6. Run the helpers cell. It loads pandas as pd, NumPy as np and matplotlib as plt, and defines the helpers of the check cells. It prints nothing.

The Ready: line looks like this:

Ready: payments.csv.gz (6.9 MB), customers.csv (1.2 MB), refunds_daily.csv (0.0 MB), refunds_agents.csv (0.5 MB), refunds.csv (0.6 MB), ticket_api_metrics.csv.gz (0.6 MB), ticket_api_logs.csv.gz (1.8 MB), ticket_api_events.csv (0.0 MB), orders.csv (0.0 MB), tickets_daily.csv (0.0 MB), calendar.csv (0.0 MB), sealed-key.txt (0.0 MB)

A checksum is a short fingerprint of a file. If even one byte of the download is wrong, the fingerprint is different, and the cell stops with wrong checksum. Then delete the folder anomaly-course and run the cell again. If the cell says Could not download, check your internet connection. In Kaggle, also turn on Internet in the notebook settings.

Section 1 of the notebook reads every table and prints its size:

files = ["payments.csv.gz", "customers.csv", "refunds_daily.csv", "refunds_agents.csv", "refunds.csv",
"ticket_api_metrics.csv.gz", "ticket_api_logs.csv.gz", "ticket_api_events.csv", "orders.csv",
"tickets_daily.csv", "calendar.csv"]
tables = {name: pd.read_csv(f"data/{name}") for name in files}
for name, table in tables.items():
print(f"{name:<28}{len(table):>9,} rows {', '.join(table.columns[:6])}{' ...' if len(table.columns) > 6 else ''}")

The output:

payments.csv.gz 343,074 rows payment_id, ts, customer_id, account_type, region, amount ...
customers.csv 24,038 rows customer_id, account_type, region, first_payment, signup
refunds_daily.csv 640 rows date, orders, refunds, refund_rate
refunds_agents.csv 17,772 rows date, agent_id, orders, refunds, refund_amount
refunds.csv 9,923 rows refund_id, date, agent_id, order_id, region, amount ...
ticket_api_metrics.csv.gz 86,400 rows ts, requests, error_rate, latency_p95_ms
ticket_api_logs.csv.gz 137,013 rows ts, level, logger, message
ticket_api_events.csv 22 rows ts, kind, detail
orders.csv 2,181 rows date, channel, orders
tickets_daily.csv 2,184 rows date, channel, tickets
calendar.csv 819 rows date, weekday, holiday, is_holiday

The tables form four groups. Each group belongs to one kind of question, and to the people who ask it:

Four boxes on the left, each with an arrow to a question on the right. payments and customers, 343,074 card payments from June to November 2025: is this payment fraud? Used for baselines, distance and isolation, deep methods, evaluation and the job. The three refunds tables, 21 months of refunds: has the refund process changed? Used for control charts, drift tests and business checks. The three ticket_api tables, 60 days with one row a minute: is the service failing? Used for anomalies in time and in logs. orders, tickets_daily and calendar, two years with one row a day: is today's count unusual? Used for anomalies in time.
The four groups of Larkfield tables, and the question that each group answers in the course.
Text description of the diagram

The diagram has four boxes on the left, each with an arrow to a question on the right. First, payments and customers: 343,074 card payments from June to November 2025. Their question: is this payment fraud? They are used for baselines, distance and isolation methods, deep methods, evaluation and the detection job. Second, refunds_daily, refunds_agents and refunds: 21 months of refunds. Their question: has the refund process changed? They are used for control charts, drift tests and business checks. Third, ticket_api_metrics, ticket_api_logs and ticket_api_events: 60 days of the help desk’s service, one row a minute. Their question: is the service failing? They are used for anomalies in time and in logs. Fourth, orders, tickets_daily and calendar: two years, one row a day. Their question: is today’s count unusual? They are used for anomalies in time.

Tables One row is Owner at Larkfield
payments, customers one card payment; one customer Sofia, payments risk
refunds_daily, refunds_agents, refunds one day; one agent on one day; one refund Grace, support lead
ticket_api_metrics, ticket_api_logs, ticket_api_events one minute; one log line; one release or change Amira, developer
orders, tickets_daily, calendar one channel on one day; one day Mei, data engineer, owns the feeds; Grace reads the ticket counts

The daily orders and tickets are the same numbers as in the forecasting course, so the two courses show the same Larkfield. The payments are a separate table at their own scale: about 1,875 payments a day. The orders table has about 676 orders a day. Never compare the two.

Section 2 shows the last 12 payments of customer C-01234, the customer of the first lesson:

payments = pd.read_csv("data/payments.csv.gz", parse_dates=["ts"])
hook = payments[payments.customer_id == "C-01234"]
hook[["ts", "amount", "country", "device_id", "merchant_category"]].tail(12)

The output:

ts amount country device_id merchant_category
181950 2025-09-06 09:34:00 16.02 CA D-6674e29 kitchen
194430 2025-09-13 09:29:08 16.75 CA D-6674e29 kitchen
206782 2025-09-20 09:02:43 17.77 CA D-6674e29 kitchen
219181 2025-09-27 09:17:38 15.16 CA D-6674e29 kitchen
229857 2025-10-03 02:14:07 24.27 PT D-9f3e1a2 home
229859 2025-10-03 02:15:05 19.26 PT D-9f3e1a2 gift_cards
229860 2025-10-03 02:16:16 20.66 PT D-9f3e1a2 electronics
229861 2025-10-03 02:17:17 18.88 PT D-9f3e1a2 electronics
229862 2025-10-03 02:18:18 24.39 PT D-9f3e1a2 home
229863 2025-10-03 02:19:13 19.16 PT D-9f3e1a2 electronics
229864 2025-10-03 02:20:23 24.36 PT D-9f3e1a2 home
229865 2025-10-03 02:21:31 19.25 PT D-9f3e1a2 home

The story of the first lesson is in the data. Four Saturday baskets in Canada on one phone are followed by eight payments in Portugal on a new device, one a minute. The numbers on the left are the row numbers of the table. Row 229858 is missing from the list because it belongs to another customer.

What we planted: the rules, not the answers

Section titled “What we planted: the rules, not the answers”

The data is synthetic, so we planted every problem on purpose and wrote them in an answer key. This section of the notebook tells you the rules of what we planted, not which rows. The counts come from the answer key:

Table Planted problems How many
payments Stolen cards: many small payments in a short time, often in a new country 262 episodes, 2,558 payments
payments Card testing: one device tries many customers’ cards with tiny payments 52 episodes, 2,007 payments
payments Account takeovers: a few expensive payments from a customer’s own account 209 episodes, 406 payments
refunds The refund rate rises by 0.3 points and stays; one agent refunds far more than the others; one agent’s refunds cluster just below the 100-dollar approval limit; some refunds are paid twice; the daily feed sends zeros for one day see Module 3
ticket-api A slow memory leak; an error spike after a release; a configuration change see Modules 5 and 6

In total, 4,971 payments are planted fraud, in 523 episodes. An episode is one fraud event: for example, the eight payments in Portugal are one episode.

The data also contains events that look unusual and are honest. A good detector leaves them alone. We tagged them, so the course can count how many false alerts they cause:

Normal but unusual Payments tagged
Payments from the new region, east (live from 1 September 2025) 21,795
Split baskets: one order paid in several parts 13,223
The first payment in the Larkfield app (payment in the app from 15 October 2025) 9,532
Business customers’ bulk orders 9,077
Black Friday, 28 November 2025 5,087
Borrowed devices: a device a customer used only once 3,299
Customers on a trip abroad 703
A customer’s new phone 346

The answer key is in the file data/sealed-key.txt, packed so that you do not read it by accident. The course’s compare.py unpacks it to score a detector. You can unpack it yourself with compare.reveal(): it is there to score your own work, not to keep a secret. But once you know the answers, your scores no longer say how a detector does on data that it has not seen. Treat it like the answers at the back of a book.

Section 4 adds up the orders of the three channels for each day, in a series called daily, and draws it with the holidays:

orders = pd.read_csv("data/orders.csv", parse_dates=["date"])
daily = orders.groupby("date").orders.sum() # one total per day

The chart below shows the same daily totals:

Larkfield's orders a day, all channels, from the shared daily series. Black Friday and the heat wave of July 2025 are the highest days, and both are normal. The missing day, 19 October 2025, is invisible: the line joins the day before and the day after.
Text description of the chart

The chart shows Larkfield’s orders on each of 727 days, from 1 April 2024 to 29 March 2026. The axis goes from 0 to 1,500. Labels mark the two Black Fridays and the heat wave. The line moves up and down every week, and it is higher in the summers. It averages 676 orders a day. The highest day is 5 July 2025, in a heat wave, with 1,372 orders. Black Friday makes a sharp, narrow peak each year: 1,059 orders in 2024 and 1,224 in 2025. The lowest days are at Christmas. One day, 19 October 2025, has no data, but you cannot see it. The line goes straight from 18 to 20 October.

The notebook’s chart prints the last Black Friday:

Black Friday 2025: 1224 orders

Black Friday is one of the highest days of each year. Only a few days around the heat wave of early July 2025 are higher, with 1,372 orders on 5 July. Both are unusual, and both are normal: Larkfield plans for Black Friday every year, and hot weather sells parasols. In Module 5, a good detector does not report them.

Before you run the next cell, the notebook asks you to predict: the orders feed dropped one day. Does the chart show a gap, a drop to zero, or nothing? The cell lists every day between the first and the last day that has no row:

all_days = pd.date_range(daily.index.min(), daily.index.max(), freq="D")
missing = all_days.difference(daily.index)
print("Days with no orders rows:", [str(d.date()) for d in missing])
Days with no orders rows: ['2025-10-19']

The chart shows nothing. A missing day is a data problem, and no detector that reads values can see it, because there is no value to read. A simple count of the rows per day finds it at once. Mei, who owns the feeds, adds this check before any detector runs. The same feed also left one day empty in the refunds table, as your second task shows.

Every module of the course runs its detectors on the same data, scores them the same way, and adds one row to one comparison table. This is how you can compare a rule of Module 2 with a neural network of Module 6 fairly.

Column What it says
alerts per day how many alerts the detector sends each day
precision at budget of the 50 alerts that Sofia’s team reviews each day, the share that are planted anomalies
recall the share of all planted anomalies that the reviewed alerts find
cost per week ($) missed frauds (amount + 15 dollars each) and false alerts (3 dollars each): assumptions of the course
explain one alert (min) how long a reviewer needs to understand one alert: an assumption that you state

compare.split() cuts the payments into two parts by date, the same way for every detector:

  • The reference months, June to August 2025: 173,271 payments. A detector learns “normal” here.
  • The evaluation months, September to November 2025: 169,803 payments over 91 days, with 2,817 planted, about 31 a day. Every row of the table is scored here.

The evaluation months are harder on purpose. They contain the new region, the app launch and Black Friday: normal changes that a detector must survive.

Your first detector is silly on purpose: every day, alert on the 50 largest amounts. The notebook asks you to predict how many of these 50 are frauds before you run it. The first line imports compare.py, the shared code that the setup cell downloaded:

import compare
reference, evaluation = compare.split(payments)
table = compare.Table()
table.add(compare.score_payments(evaluation, evaluation.amount, "biggest amounts", "the 50 largest amounts of each day"))
table.show()

score_payments takes the payments and one score per payment. Here the score is the amount, and a higher score means “more unusual”. It also takes a name and a short description of the setting. Each day, it sends the 50 payments with the highest scores to review. The output:

detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 None

A precision of 0.003 means that, on average, fewer than 1 of the 50 daily alerts is a planted fraud. The cost is 10,264 dollars a week. Doing nothing at all costs 10,367 dollars a week, so the largest amounts save almost nothing. Every one of its reviewed alerts was a business customer’s payment, and all of them were at least 478.28 dollars. Business customers buy in larger amounts, honestly. The last column says None, because nobody has stated how long one alert takes to explain. For your own detectors, write it in with explain_minutes=.

Is “biggest amounts” at least better than chance? Section 6 adds a detector that picks 50 payments a day at random:

rng = np.random.default_rng(0)
table.add(compare.score_payments(evaluation, rng.random(len(evaluation)), "random", "50 payments a day, chosen at random"))
table.show()
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 None
random 50 payments a day, chosen at random 50.0 0.019 0.031 11161 None

Random choice finds more fraud than the largest amounts: a precision of 0.019, close to the base rate of the evaluation months (1.66%). But the random detector costs more than doing nothing: 11,161 dollars a week. Its 343.2 false alerts a week cost 3 dollars each, and they catch very little fraud. A detector must earn its review time.

The chart puts these rows next to the third silly detector of your tasks, and next to the ceiling: a detector that knows the answers.

Precision at the budget (50 alerts a day) of three silly detectors in the evaluation months: the biggest amounts, random, the smallest amounts. Perfect is the ceiling, the comparison table's perfect row. Even the ceiling is 0.542, because most days have fewer than 50 planted payments.
Text description of the chart

A column chart with four columns, on an axis from 0 to 0.6. The 50 biggest amounts of each day: 0.003. 50 payments at random: 0.019. The 50 smallest amounts: 0.192. A detector that knows the answers: 0.542. The ceiling is not 1, because on 78 of the 91 evaluation days fewer than 50 payments are planted.

The smallest amounts do much better than the largest: 0.192. Card testing uses payments of 1 to 5 dollars, so the smallest payments of a day contain many card tests. Still, a rule about the amount alone misses most frauds. Module 2 starts to compare each payment with the customer’s own usual amount.

Common mistake

Scores in a different order

Symptom: you sort the scores, but not the payments, before you score them. The notebook does it on purpose in section 7:

wrong = np.sort(evaluation.amount.to_numpy()) # sorted scores: no longer in the order of the payments
row = compare.score_payments(evaluation, wrong, "biggest amounts, wrong order", "scores sorted, payments not")
print(row["precision at budget"], "(the right order gave", table.rows[0]["precision at budget"], ")")
0.011 (the right order gave 0.003 )

The row looks normal. Nothing warns you.

Cause: score_payments needs one score per payment, in the same order as the table that you give it. Sorting moved the scores to other payments. The detector is now a different, accidental detector, and its row means nothing.

Fix: compute the score as a column of the same table, such as evaluation.amount or the output of a model on the same rows. Never sort one without the other. If you must sort, sort the whole table, then compute the scores.

Consequence in AI work: a mismatch between rows and scores gives a believable number for the wrong detector. In a real comparison, it can make a weak method look good, or a good method look weak, and the team chooses the wrong one.

Section 8 of the notebook has three tasks. Each one has a check cell that says Check passed or Not yet.

  1. Count the payments on Black Friday, 28 November 2025. Write the number as bf_payments.
  2. Count the rows of refunds_daily.csv whose orders is empty. Write the number as missing_orders.
  3. Score a third silly detector: every day, the 50 smallest amounts. Keep its row as smallest, and add it to table.
  4. Run each check cell. Each one prints Check passed when your answer is right.
Show the answers
bf_payments = int((payments.ts.dt.floor("D") == "2025-11-28").sum())
refunds_daily = pd.read_csv("data/refunds_daily.csv", parse_dates=["date"])
missing_orders = int(refunds_daily.orders.isna().sum())
smallest = compare.score_payments(evaluation, -evaluation.amount, "smallest amounts", "the 50 smallest amounts of each day")
table.add(smallest)
table.show()
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
biggest amounts the 50 largest amounts of each day 50.0 0.003 0.005 10264 None
random 50 payments a day, chosen at random 50.0 0.019 0.031 11161 None
smallest amounts the 50 smallest amounts of each day 50.0 0.192 0.310 9966 None

The check cells print Check passed: bf_payments is 5216., Check passed: missing_orders is 1. and Check passed: the precision at budget of the smallest amounts is right.

Black Friday has 5,216 payments, against about 1,875 on an average day: unusual, and normal. The refunds table has one day with no orders, the same day that the orders feed dropped. The minus sign in -evaluation.amount turns “smallest” into “highest score”, because score_payments always reviews the highest scores first.

If something breaks, delete the folder anomaly-course and run the notebook again from the top.

Check your understanding

The orders feed dropped 19 October 2025. Why does the line chart of daily orders not show it, and what check finds it?

Show the answer

The day has no row at all, so there is no value to draw. The chart’s line joins 18 October and 20 October, and the gap is invisible. No detector that reads values can see it either. A simple check finds it at once: compare the days that have rows with the list of all days between the first and the last. It prints Days with no orders rows: ['2025-10-19']. Run this kind of check on the data before any detector.

Check your understanding

Predict and explain: the random detector has a higher precision at the budget than “biggest amounts” (0.019 against 0.003), but it costs more per week than doing nothing (11,161 against 10,367 dollars). How can both be true?

Show the answer

A random choice finds fraud at about the base rate of the evaluation months (1.66%). The largest amounts are all business customers’ payments, almost all honest. So random wins on precision. But every alert costs review time. The random detector sends 343.2 false alerts a week at 3 dollars each, and catches very little fraud. The review time costs more than the fraud it saves. A detector is only worth running if its alerts save more than they cost.

Check your understanding

Why does every detector in the course use the same split (reference months June to August, evaluation months September to November) and the same budget?

Show the answer

So that the rows of the comparison table can be compared. Suppose one detector is scored on a quiet month and another on the month of the app launch. Or one gets 50 alerts a day and another 200. Then a difference in the table could come from the data or the budget, not from the method. With the same split, the same budget and the same costs, the only difference between two rows is the detector.

Recap

  • The course has eleven tables in four groups: payments, refunds, the ticket-api service, and the daily orders and tickets.
  • We planted 4,971 fraud payments in 523 episodes. We also tagged honest events that look unusual: trips, bulk orders, the new region, the app launch, Black Friday.
  • The answer key is sealed, not secret: compare.py uses it to score, and you can unpack it, but then your scores mean less.
  • A missing day is invisible in a chart and to any detector that reads values. Count the rows per day.
  • The comparison table scores every detector on the same evaluation months, at 50 alerts a day, with the same costs. The 50 largest amounts score 0.003, random 0.019, the smallest amounts 0.192, and the ceiling is 0.542.

You can now load the data and score a detector fairly. But so far, every detector looked at all payments together, and the largest amounts found almost nothing. Next, in Module 2, you measure how far each payment is from normal with a z-score. You also see how one huge payment fools the mean.

Next: Z-scores, and why the mean is fooled