Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

In this course, we framed anomaly problems, built baselines, control charts and drift checks, compared four families of detectors, evaluated them under a budget and ran one as a daily job. In this assignment, you do it all on a new Larkfield data stream with problems we planted.

What you will be able to do

Use the whole course on one new data stream: Larkfield’s refunds from January to March. You frame the problem and watch the daily rate with a control chart. You check the new region for drift and compare detectors on one fair test. Then you run a small job over March and write a recommendation for Grace, with its cost and its limits.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

  • Modules 1 to 9. Each part below names the lessons that it uses.
  • The certificate also asks for at least one case study of Module 10. To see the same steps on real data first, open Control charts on a real production line.

What you need on your computer

  • Python 3.12, a terminal, and a notebook if you like one. Parts 1 to 4 and 6 work well in a notebook. Part 5 needs files and a terminal.
  • The course’s labs repository with the folder C34, as in Meet the Larkfield data. If you have it, run git -C nextia-learning-labs pull to get the latest files.
  • Your detection job from Module 9, or the snapshot C34/snapshots/end-of-m09. You reuse two ideas from it: the alert ID and the ledger.
  • About 400 MB of disk for the virtual environment. No account, key or money.

Tested on macOS (Apple silicon), with Python 3.12.13 and the package versions below. We wrote out the Windows (PowerShell) commands but did not run them.

Grace, the support lead, sends Priya, Larkfield’s data scientist, a new data stream. It has the refunds of January to March 2026, per agent, per region and per day. In the middle of March, Larkfield opens a fifth region, west. Grace writes:

“January and February were quiet. Please watch March for me. I want to know about real problems: an agent who refunds too much, refunds that avoid the approval limit, money paid twice, a broken feed. But west is new. I do not want an alarm every day just because it is new.”

We, the course team, planted some problems in March. We do not tell you what or where. The labs folder has no answer key, sealed or not. You find the problems with the course’s methods and show the evidence. The data is synthetic, like every Larkfield table.

Use these assumptions, so that your numbers compare with other learners’ numbers. If you want to try a second set, add it and say why.

Assumption Value
Alert budget Grace’s team checks at most 3 alerts a day.
Quiet months On a quiet month, Grace accepts about 1 false alert a week.
A false alert About an hour of checking: 45 dollars (at the course’s rate of 3 dollars for 4 minutes).
A missed problem The money it lets out: the extra refunds of an agent each day until the alert, or the amount paid twice.
Counted apart An agent questioned for nothing, and a wrong report from a broken feed. Do not give them a money value.

The data is in labs C34/final-assignment/data/. It has three files and a list of their checksums. A checksum is a long number computed from the bytes of a file. If one byte changes, the checksum changes.

File Rows One row Columns
final_refunds_agents.csv (63 KB) 2,431 one agent on one working day date, agent_id, orders, refunds, refund_amount
final_refunds_regions.csv (11 KB) 366 one region on one day date, region, orders, refunds, refund_amount
final_refunds.csv (88 KB) 1,550 one refund, from the payment system refund_id, date, agent_id, order_id, region, amount, reason

What the columns mean:

  • orders: the orders in the agent’s queue that day, or the region’s orders. It is the denominator of a refund rate.
  • refunds, refund_amount: the number of refunds and their total in dollars, as the daily feed reports them.
  • amount: one refund, in Canadian dollars. A refund of 100 dollars or more needs a team lead’s approval: that is the approval limit.
  • reason: the agent’s choice of damaged, late, wrong_item, missing or changed_mind.
  • There are 40 agents. Each one works about five days a week, with about 22 orders a day.

The agents’ and regions’ feeds come from one system. The refunds table comes from another, the payment system. When two systems should agree, you can check one against the other.

  1. Make a folder for the assignment, refund-pack, and copy the data into it:

    Terminal window
    mkdir refund-pack
    cd refund-pack
    cp -R ../nextia-learning-labs/C34/final-assignment/data data
    python3.12 -m venv .venv
    source .venv/bin/activate
    python -m pip install pandas==3.1.1 numpy==2.5.4 scipy==1.18.1 scikit-learn==1.9.1 matplotlib==3.11.2
  2. Save this script as check_data.py. It checks the checksums, then loads the three tables:

    import hashlib
    from pathlib import Path
    import pandas as pd
    for line in Path("data/SHA256SUMS").read_text().splitlines():
    digest, name = line.split()
    ok = hashlib.sha256(Path("data", name).read_bytes()).hexdigest() == digest
    print("OK " if ok else "BAD", name)
    agents = pd.read_csv("data/final_refunds_agents.csv", parse_dates=["date"])
    regions = pd.read_csv("data/final_refunds_regions.csv", parse_dates=["date"])
    refunds = pd.read_csv("data/final_refunds.csv", parse_dates=["date"])
    for name, t in [("agents", agents), ("regions", regions), ("refunds", refunds)]:
    print(f"{name:8} {len(t):6,} rows {t.date.min().date()} to {t.date.max().date()}")
    print("first day of each region:", regions.groupby("region").date.min().dt.strftime("%m-%d").to_dict())
  3. Run it with python check_data.py. You see:

    OK final_refunds.csv
    OK final_refunds_agents.csv
    OK final_refunds_regions.csv
    agents 2,431 rows 2026-01-01 to 2026-03-29
    regions 366 rows 2026-01-01 to 2026-03-29
    refunds 1,550 rows 2026-01-01 to 2026-03-29
    first day of each region: {'coast': '01-01', 'east': '01-01', 'north': '01-01', 'south': '01-01', 'west': '03-16'}

You never choose a setting on the data that grades it. So each month has a different job. A warm-up is the time at the start when a window of past days is not full yet.

Four boxes in time order. 1 to 14 January: warm-up, the 7-day and 28-day windows fill. 15 to 31 January: set each threshold for about 1 false alert a week. February: the test, a clean copy for the false alerts, and copies with your own planted shift for the days to the alert. January and February together are the reference months, phase I of the daily chart. Then March, 1 to 29: the replay, the job runs day by day, the hidden problems, west live on 16 March.
January sets the thresholds, February tests them, and March is the replay. January and February together are phase I of the daily chart.
Text description of the diagram

The diagram shows four boxes in time order. The first three are in a frame: “Reference months, phase I of the daily chart”. Box 1, 1 to 14 January: warm-up, the windows fill. Box 2, 15 to 31 January: set each threshold, about 1 false alert a week. Box 3, February: the test. A clean copy gives the false alerts. Copies with your own planted shift give the days to the alert. An arrow leads to box 4, March: the replay. The job runs day by day, with the hidden problems, and west goes live on 16 March.

Here is the daily refund rate of the whole shop, from the agents’ file. Look at it before you choose any method.

  • Refund rate of the day
  • Rate of January and February (2.35%)
  • End of February
  • West goes live (16 March)
Larkfield's daily refund rate. Until mid-March, the days move up and down around 2.35%. After west goes live, the rate climbs. No single day tells you if an agent has changed.
Text description of the chart

The chart shows one dot per day, from 1 January to 29 March 2026, for 88 days with orders. A solid line marks the rate of January and February together, 2.35%. A dashed vertical line marks the end of February, and another one marks 16 March, the day west goes live. Until 15 March, the dots move up and down around the line, mostly between 1.5% and 3.8%. The lowest day is 2026-02-04, at 1.2%. From 16 March, the dots climb, day after day, to 6.13% on 2026-03-28. March as a whole has a rate of 3.63%. Nothing on the chart shows one agent or one refund: that is why you need the checks below.

Uses The payment that did not fit, Kinds of anomalies and Labels are rare.

Write FRAMING.md, one page. For each problem that Grace names, write:

  1. The kind of anomaly: point, contextual, collective or change point.
  2. The unit and the time of a check. For example: one agent, each morning, over the last 7 working days.
  3. The table and the columns that show it.
  4. The cost of a miss and the cost of a false alert, with the assumptions above.
  5. The setting: a rule for an exact pattern, or a detector that learns normal from the reference months.

Then add the new region. In two sentences, explain why west is a change in normal, not a problem. Say what that means for each check.

Part 2: the daily chart and the drift check

Section titled “Part 2: the daily chart and the drift check”

Uses Control charts, Charts for rates and counts, Rules for patterns, Small, lasting shifts, Has the distribution changed? and Drift in many features.

  1. Compute the daily refund rate of the whole shop from the agents’ file. Use January and February as phase I.
  2. Choose the chart type: a p-chart or a Laney p′-chart. Measure the spread of the p-chart’s z in phase I, and give the reason for your choice in one sentence.
  3. Add a CUSUM or an EWMA for a small, lasting shift. Choose the shift that Grace must catch for the whole shop, for example 0.5 points.
  4. Set k and h, or λ and L, from that shift.
  5. Measure the false alarms per month of each chart. Count them in phase I, and simulate the average run length as in the lesson. Phase I is only 59 days, so the count alone is not enough.
  6. Run the charts on March. Write each alarm day, with the rate and the number of orders of that day.
  1. Compare the refund amounts of 16 to 29 March with the reference months: the KS test, PSI and the Wasserstein distance. Do it for all regions, then for the old regions only.
  2. Compare the mix of orders by region (PSI). Explain what happens to a category that the reference never saw.
  3. Compare west’s refund rate with the reference rate. Say if the sample is large enough to conclude anything.
  4. Run the same weekly drift check on each week of February against January, and count the alarms. That is your drift check’s false alarms per month.
  5. If you test many features or many days, control the false discovery rate.

Uses Median, MAD and IQR, Small, lasting shifts: one agent, Isolation Forest, Features decide more than the algorithm, Why accuracy lies here and Evaluate without a full answer key.

The table compares detectors for one problem: an agent who refunds more than before, or more than peers. The unit is one agent on one working day. Each detector scores each agent-day from that day and the days before only.

Compare a baseline and at least three families:

Family Example
Baseline No detector. Then a simple robust rule: the agent’s rate over the last 7 working days, as a robust z against all agents that day.
Statistical process control A p-chart per agent, and a CUSUM per agent on the p-chart’s z.
Isolation Isolation Forest on a few features of the agent’s past. For example: the rate over 7 and 28 working days, the share at 90 to 99.99 dollars, the mean refund.
Distance or density (optional) kNN distance or LOF on the same features.

The labs folder has no key, so you plant your own test problem. This is a constructed test: you know the answer because you made it.

  1. Set each detector’s threshold on 15 to 31 January, so that it gives about 1 false alert a week there. For the two charts, also keep the textbook limits (z above 3, h = 4) as extra rows.
  2. Apply the budget. An alert is an agent-day above the threshold. At most 3 a day go out, the highest first.
  3. On February as it is, count the alerts. Treat them all as false: Grace says February was quiet. Write the alerts a day and the false alerts a week.
  4. Make 10 copies of the data, with a fixed random seed. In each copy, choose a different agent.
  5. Add 3 points to that agent’s refund rate from 9 February. Draw the extra refunds with rng.binomial(orders, 0.03). Give them amounts drawn from the reference refunds.
  6. On each copy, find the first alert on the planted agent after 9 February. Write the shifts found, the shifts found within 14 days, and the median days to the alert.
  7. Write the precision at the budget: the share of the alerts after 9 February that are on the planted agent.
  8. Write the cost over 4 weeks with one shifted agent: 45 dollars per false alert, plus the extra refunds before the alert. Each working day, a 3-point shift costs about 0.03 × the agent’s orders × the mean refund. A shift that is never found costs every day of the test.

Put the results in one table, comparison.csv. Its columns are detector, setting, threshold, alerts_per_day, false_alerts_per_week, shifts_found, found_within_14_days, median_days_to_alert, precision_at_budget, cost_4_weeks. Add one row for “no detector”. Do not add an accuracy column.

Part 4: choose the detector and its threshold

Section titled “Part 4: choose the detector and its threshold”

Uses The alert budget, Slices and fairness and When not to use machine learning.

  1. Choose one agent detector from your table. Give the reason in three to five sentences. Use the cost over 4 weeks, the days to the alert and how easily Grace can explain an alert to an agent.
  2. Write its threshold and how the budget set it. Write its precision at the budget and its cost per week.
  3. Compare its cost with “no detector”. If the saving is small, say so. A small saving is an honest result, not a failure.
  4. Do one slice check on February’s false alerts. Split them by queue size: fewer than 20 orders a day, and 20 or more. Then split them by agent.
  5. Show the counts next to the rates. Say if a few agents get most of the false alerts.

Uses From score to alert, When “normal” moves, Alert fatigue and ownership and Benford’s law and other business checks.

Write refund_job.py, a small job in the shape of the course’s detection job. It has two commands: run --day <date> and replay --from <date> --to <date>. Each run reads only the data up to its day.

  1. Each day, run the daily chart of Part 2 and your agent detector of Part 4.
  2. Also run the exact checks of the business-checks lesson. Find a refund paid twice: the same order and amount. Find a feed total that does not match the refunds table.
  3. Once a week, test each agent’s share of refunds just below the approval limit.
  4. Add one check with a threshold fixed on the reference months. For example: a refund above the 99th percentile of the reference refunds.
  5. Make each alert’s ID from the day, the check and the key only. Copy alert_id() from detect/alerts.py.
  6. Keep a ledger of the sent IDs, as detect/ledger.py does.
  7. Write each alert as one JSON line in alerts/<day>.jsonl, with its reasons in words and a first step.
  8. Write a rule for the new region, with its owner and its end date. For example: no west in the chart, the drift check and the fixed-threshold check for 28 days, and a weekly note to Grace instead.
  9. Replay 1 to 29 March. Write a table of alerts per day, by check. No day may have more agent alerts than the budget.
  10. Run one day again. It must send 0 new alerts.
  11. In a new folder, replay March without the new-region rule. The extra alerts are the flood that your rule prevents.
  12. Run the job in a fresh virtual environment from your requirements.txt. Write the command and its output.

An alert line has this shape. This example is constructed: the values are placeholders.

{"alert_id": "3f9c0a1b2c3d4e5f", "day": "2026-03-xx", "check": "agent", "key": "A-xx", "score": 4.1,
"reasons": ["A-xx: 9 refunds on 150 orders in the last 7 working days (6.0%; all agents' median 2.4%)"],
"first_step": "Read the agent's refunds of the last 7 working days with the agent's team lead. Do not blame first."}

Uses The alert budget and Alert fatigue and ownership.

Write GRACE.md, one page, for a reader who does not code. It must say:

  1. What you found in March: each alert that you think is a real problem, with its evidence and the first step.
  2. Which alerts you think are false, and why.
  3. Which checks to run every day, and who owns each one (Grace, Mei for the feeds, finance for money paid twice).
  4. The cost per week of your choice, against doing nothing.
  5. What the detectors cannot find. Give at least one size of problem that your evaluation missed or found late, with the number.
  6. When to set new limits: for example, when west has 28 days of data, or after a change of carrier.
Six boxes in a line after the data: 1. Framing note: kinds, unit, costs, budget. 2. Daily chart and drift check: false alarms a month. 3. Comparison table: a baseline and 3 families, one evaluation. 4. The choice: threshold from the budget, cost, one slice. 5. The job over March: no alert twice, no flood from west. 6. Note for Grace: the decision and what it cannot find.
The pack, in the order that you build it. Each part uses the one before.
Text description of the diagram

The diagram starts with the data: the agents, regions and refunds tables. Six boxes follow in a line. 1, Framing note: kinds, unit, costs, budget. 2, Daily chart and drift check: false alarms a month. 3, Comparison table: a baseline and 3 families, one evaluation. 4, The choice: threshold from the budget, cost, one slice. 5, The job over March: no alert twice, no flood from west. 6, Note for Grace: the decision and what it cannot find.

refund-pack/
FRAMING.md Part 1
REPORT.md Parts 2 to 5, with the commands that make every number
comparison.csv Part 3
refund_job.py Part 5, and any notebook or script that you used
alerts/ Part 5: one file per day of March
state/ Part 5: the ledger
GRACE.md Part 6
requirements.txt exact versions

REPORT.md has these sections:

# Refunds, January to March 2026: anomaly-detection report
## Data checks (rows, dates, checksums)
## Daily control chart (Part 2: chart type and reason, settings, false alarms a month, March alarms)
## Drift check on the new region (Part 2: tests, sizes, false alarms a month, what you conclude)
## Comparison table (Part 3: the table, the planted test, the costs)
## Chosen detector (Part 4: the reason, threshold, precision at the budget, cost, the slice)
## Detection job (Part 5: alerts per day, the rerun, the new-region rule and the flood it prevents,
the fresh-environment run)
## Labels and limits (which numbers are measured, constructed or assumed)

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. Each item has points: all of them when demonstrated, half when partially demonstrated, none when not yet demonstrated. The points show where your pack is weakest. But the course is complete only when all six items are demonstrated: 100 points. You can improve an item and send it again.

Required item Points Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Framing note (Kinds of anomalies) 10 Each problem has its kind, unit, time of check, both costs and the budget. West is named as a change in normal. The costs are missing for one problem. No unit or no time of check.
2. Daily chart and drift check, with false alarms a month (Module 3) 20 The chart type has a reason from phase I. The settings come from the shift that matters. False alarms a month are measured for the chart and the drift check. The drift check compares old regions and all regions, and says if the sample is large enough. False alarms are counted in phase I only, not simulated. A p-value alone is called “drift”. Or no false-alarm rate.
3. Comparison table (Modules 2, 3, 4 and 7) 20 A baseline and at least three families, with thresholds from 15 to 31 January, tested on the same February copies, with the columns above. One family has a different test. No baseline. Or thresholds set on February or March.
4. Chosen detector (The alert budget) 15 The threshold comes from the budget. Precision at the budget, cost per week against doing nothing, and one slice with counts. No slice, or rates without counts. The choice is by accuracy, or by the name of the method.
5. Detection job over March (Module 9) 20 Each day reads only the past. A rerun sends 0 new alerts. Agent alerts stay within the budget. The new-region rule is written, and the flood that it prevents is counted. Each alert has reasons and a first step. A fresh-environment run works. No count of the flood without the rule. A rerun sends alerts again. Or west is left out for ever, with no end date.
6. Recommendation for Grace (Modules 7 and 9) 15 One page in plain words. The real and false alerts with evidence, owners, the cost, and at least one thing that the detectors cannot find, with a number. The limits are general, with no number. No limits. Or a claim that the job finds every problem.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • A detector judged by accuracy alone. On rare events, a detector that never alerts has a high accuracy. (Items 3 and 4.)
  • A threshold tuned on the data that grades it. Thresholds come from 15 to 31 January. You never change them after you look at February’s test or at March. (Items 3 and 5.)
  • A method chosen without a baseline in the comparison. (Item 3.)
  • A job that sends the same alert twice. (Item 5.)

A good pack is honest before it is impressive. It has three things.

  • Honest measures under the course’s costs. Precision at the budget, days to the alert, false alerts a week and cost in dollars. The assumptions are next to them, and each number has a label: measured, constructed or assumed.
  • A simple baseline in the comparison. The robust z of a 7-day rate is a few lines of pandas. If a complex detector does not beat it, the pack says so and keeps the simple one.
  • A decision with its cost. “We run these checks every morning. They cost about this much a week. They save about this much. They miss this.”
Show our results, to check your method

Open this only after you have your own numbers. We ran the assignment once, from the public data only, with the settings of the steps above. If your settings differ, your numbers differ. Then compare the method, not the numbers. These are not the answers to the planted problems.

  • Daily chart (phase I, 59 days): the rate is about 2.35%, on about 600 orders a day. The spread of the p-chart’s z is about 1.09, and the Laney σz is about 1.00. So phase I shows no extra spread to correct. No chart alarms in phase I. The simulated false alarms a month are about 0.2 for the Laney p′-chart and 0.1 for the EWMA (λ 0.1, L 2.6). The CUSUM (k about 0.40, h 8) has under 0.1. A 0.5-point shift of the whole shop takes about 9 days to an alarm with the EWMA, 18 with the CUSUM and 23 with the Laney p′-chart (simulated). On all regions, the Laney p′-chart alarms on every day from 21 to 29 March.
  • Drift check: compare the refund amounts of 16 to 29 March with the reference months. On all regions, the KS p-value is far below 0.001, and the PSI is about 0.49. The median refund doubles, from about 40 to about 80 dollars. On the old regions only, the KS p-value is about 0.12 and the PSI about 0.05: no drift. West alone has about 250 refunds, with a median of about 106 dollars and a refund rate of about 15.5%. The order mix by region has a PSI of about 1.9, only because west is new. The weekly check of February against January had 0 alarms in 4 weeks at PSI 0.25. The highest week reached 0.18 on about 90 refunds: PSI is noisy on small weeks.
  • Comparison (thresholds from 15 to 31 January, 10 copies, +3 points from 9 February): a 3-point shift costs about 39 dollars each working day. With no detector, the cost over 4 weeks is about 560 dollars. The p-chart per agent, at the matched threshold, was the cheapest: about 470 dollars. It gave about 0.75 false alerts a week and found 7 of 10 shifts, after a median of 7 days. Isolation Forest on 4 features gave no false alert but found only 3 of 10 (about 490 dollars). The robust z baseline found 8 of 10, after a median of 8 days. But its 1.75 false alerts a week cost more than doing nothing: about 610 dollars. The CUSUM found 8 of 10 soonest (median 6 days), with 2.5 false alerts a week (about 730 dollars). The textbook limits gave 2.75 (p-chart) and 9.5 (CUSUM) false alerts a week.
  • February is not March: when west raised every agent’s rate, the checks against the reference rate moved with it. In March, the CUSUM per agent sent about 1.6 alerts a day. The robust z against peers sent about 0.34 a day, because all the agents moved together. That is a reason to prefer a peer check in March, even though it cost more in February. Say which you choose, and why.
  • Job over March: about one alert a day and never more than 4 on one day. Agent alerts stay within the budget, and a rerun sends 0 new alerts. Without the new-region rule, the same job sent about 2.8 times as many alerts, up to 12 on one day. Most of the extra alerts were the fixed-threshold check on large refunds, the drift check and the daily chart.

Common mistake

A daily alert for a region that is only new

Symptom: from the second day of west, the drift check alarms every day.

Cause: the order mix has a category that the reference never saw. PSI grows with every new order there, and that will not stop.

Fix: write a rule for a new region, with an owner and an end date. Leave it out of the drift check and the chart for 28 days, and send a weekly note instead. Then make new limits with west in them.

Common mistake

Zero refunds is called a broken feed

Symptom: your check alarms on every region-day with no refunds.

Cause: a region with about 100 orders and a rate of 2.4% has zero refunds on many quiet days by chance. In January and February, 11 region-days with 85 or more orders had zero refunds.

Fix: compare the feed with the refunds table. A feed is broken when the two systems disagree, not when a number is small.

Common mistake

The rerun sends the alerts again

Symptom: you run 11 March a second time, and the same alerts arrive again.

Cause: the alert ID contains the score or the time of the run, or the job does not read its ledger.

Fix: build the ID from the day, the check and the key only. Check the ledger before you write, as in From score to alert.

Common mistake

A small test is called proof

Symptom: an agent has 3 of 8 refunds just below the limit (a constructed example), and your note says “this agent avoids the team lead”.

Cause: with 8 refunds, one refund more or less changes the share by more than 12 points. A small p-value says “unusual”, not “on purpose”.

Fix: write “unusual, to review”, give the counts, and let Grace’s team look at the orders. Watch the agent for longer before anyone draws a conclusion.

Recap

You can now take a new data stream to checks that a team can trust. You frame each problem with its kind, unit and costs. You watch a rate with a control chart and check a launch for drift, with a measured false-alarm rate. You compare detectors on one fair test, with a simple baseline. You choose by cost, and run the checks as a daily job that never sends an alert twice. And you tell the owner what the checks cannot see. To keep going, MLOps and LLMOps: Operating AI in Production runs jobs like this one in production, with dashboards and incidents. Forecasting and Time Series: Predict What Comes Next teaches the baselines of time series in depth.