Benford's law and other business checks
- Lesson 14
- 35 min
- Module 3: Statistical process control and distribution drift
Where we are
We have charts for processes and tests for distributions. Auditors also use simple checks on the amounts themselves. In this lesson, we test first digits with Benford's law, look for refunds just below the approval limit, and see what these checks prove and what they do not.
Lesson 14 of 38 · Anomaly Detection: Find What Is UnusualBenford's law and other business checks
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Check amounts the way auditors do: first digits, amounts just below a limit, duplicates, round numbers and totals that must agree. Each check is fast and easy to explain, and each one can point at an innocent person if you use it on the wrong data.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It tests Benford’s law on payments and refunds, then refunds just below the approval limit, duplicates, round amounts and the daily totals. Its setup cells install the packages and download the data (about 8 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Percentages, and a p-value with the Bonferroni correction, from Drift in many features, and what to do about it.
- The refunds tables: one row per refund, and one row per agent and day, from Meet the Larkfield data.
What you need on your computer
Nothing. The notebook runs in Colab, Kaggle or VS Code.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, NumPy 2.5.4 and SciPy 1.18.1. Colab and Kaggle were not tested.
Money that grows
Section titled “Money that grows”Put 10 dollars in a savings account that grows by 12% each year. Write down the balance every year, and look at its first digit. From 10 to 20 dollars, the balance must double, which takes about six years. From 90 to 100 dollars, it needs only 11% more: less than one year. So the balance spends a long time with a first digit of 1, and very little time with a first digit of 9.
Here are the 21 balances from 10 dollars to 96.50 dollars. This is a constructed example:
[10.0, 11.2, 12.5, 14.0, 15.7, 17.6, 19.7, 22.1, 24.8, 27.7, 31.1, 34.8, 39.0, 43.6, 48.9, 54.7, 61.3, 68.7, 76.9, 86.1, 96.5]first digits: {1: 7, 2: 3, 3: 3, 4: 2, 5: 1, 6: 2, 7: 1, 8: 1, 9: 1}Text description of the chart
A column chart with nine columns, for the first digits 1 to 9. The digit 1 has the tallest column, 7 of the 21 balances. The digits 2 and 3 have 3 each, 4 and 6 have 2 each, and 5, 7, 8 and 9 have 1 each.
A third of the balances start with 1. The same happens in many real collections of numbers: the populations of towns, the lengths of rivers, the amounts on invoices.
The problem: invented numbers look different
Section titled “The problem: invented numbers look different”People who invent numbers do not know this. When someone makes up the amounts of false expense claims, they tend to use every first digit about equally. They also like amounts just below a limit that needs approval, and round numbers. Auditors use these habits to choose which records to review.
But a check that is used on the wrong data points at innocent people. So for each check, we must know what it shows, and when it does not apply.
Benford’s law
Section titled “Benford’s law”The astronomer Simon Newcomb noticed the pattern in 1881, and the physicist Frank Benford tested it on thousands of numbers in 1938. Benford’s law says that, in data of this kind, the share of the first digit d is log₁₀(1 + 1/d):
| First digit | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Benford’s share | 30.1% | 17.6% | 12.5% | 9.7% | 7.9% | 6.7% | 5.8% | 5.1% | 4.6% |
To compare data with the law, compute the share of each first digit in the data. Then take the mean absolute deviation (MAD): the mean of the nine differences, without their signs. Be careful: this MAD is a mean, and it compares digit shares. It is not the median absolute deviation of Median, MAD and IQR. Benford tests use the same short name. The auditor Mark Nigrini gives limits for the first digit (Nigrini, Benford’s Law, Wiley, 2012). A MAD below 0.006 is a close fit, and 0.006 to 0.012 is acceptable. From 0.012 to 0.015 it is marginal, and above 0.015 it is a nonconformity.
Benford on Larkfield’s amounts
Section titled “Benford on Larkfield’s amounts”Before you read the output, predict: which of these follow Benford’s law? All payments, all refunds, the refunds of agent A-29, and the payments under 5 dollars.
amounts n MAD Nigrini orders of magnitude first digit 1 first digit 9 payments 343074 0.0215 nonconformity 2.24 0.2488 0.0323 refunds 9923 0.0132 marginal 1.61 0.2516 0.0424 refunds of A-29 249 0.0449 nonconformity 1.72 0.1727 0.1807payments under 5 dollars 2712 0.0905 nonconformity 0.67 0.1947 0.0000The orders of magnitude column is the range of the amounts: log₁₀ of the largest (the 99th percentile) divided by the smallest (the 1st percentile). A value of 2 means the amounts span a factor of 100.
- Benford's law
- All payments
- Refunds of A-29
Text description of the chart
The chart has three lines with dots, for the first digits 1 to 9. Benford’s law falls smoothly from 30.1% for the digit 1 to 4.6% for the digit 9. The payments start lower, at 24.9% for the digit 1. They are above the law for the digits 2 to 4 (21.1%, 16.3% and 11.8%), then below it, down to 3.2% for the digit 9. The refunds of A-29 are near 15% to 17% for the digits 1 to 3. They are between 4% and 7% for the digits 5 to 8. They jump to 18.1% for the digit 9.
No group is a close fit. Each result needs its own reading:
- The payments are a nonconformity (MAD 0.0215), and nobody planted anything in their first digits. Payments are a mixture of basket sizes: most are between 15 and 300 dollars. Their first digits do not follow Benford’s curve. The test is not wrong: the law does not apply to this data.
- The payments under 5 dollars span less than one order of magnitude (0.67). Their first digit can only be 1 to 4. Benford cannot apply at all.
- The refunds are marginal (MAD 0.0132). Refunds are capped by what the customer paid, and those over 100 dollars need approval.
- A-29’s refunds have 18.1% of first digits 9, against 4.6% in the law. This one result points at a person. It is worth a review, and the next check says why.
When Benford’s law applies
Section titled “When Benford’s law applies”Use it only on data that has all of these properties:
- The values come from many causes that multiply, such as growth, prices times quantities, or sums of many items.
- They span several orders of magnitude: from tens to thousands of dollars, for example.
- They have no built-in minimum or maximum, such as a price list, a cap or a fee.
- They are measured, not assigned: invoice numbers, phone numbers and product codes do not follow it.
- There are enough of them: hundreds at least, and better thousands.
Even then, a deviation is a reason to review, not a proof of fraud.
Just below the approval limit
Section titled “Just below the approval limit”At Larkfield, a refund of 100 dollars or more needs a team lead’s approval. Refunds that cluster just below the limit can mean that someone avoids the approval. This check asks the business question directly. For each agent, the notebook computes the share of refunds between 90 and 99.99 dollars. Then it compares that share with all agents, with a binomial test, and with the Bonferroni correction for 40 agents:
refunds["just_below"] = (refunds.amount >= 90) & (refunds.amount < 100) # the approval limit is 100 dollarsshare_all = refunds.just_below.mean()per_agent = refunds.groupby("agent_id").just_below.agg(["mean", "sum", "size"])per_agent["p"] = [stats.binomtest(int(k), int(n), share_all, alternative="greater").pvalue for k, n in zip(per_agent["sum"], per_agent["size"])]per_agent["after Bonferroni"] = per_agent.p < 0.05 / len(per_agent)all agents: 3.5% of refunds at 90 to 99.99 dollars mean sum size p after Bonferroniagent_idA-29 0.161 40 249 0.000 TrueA-28 0.054 11 202 0.105 FalseA-26 0.053 13 245 0.097 FalseA-15 0.050 11 222 0.166 FalseA-30 0.049 12 245 0.162 FalseText description of the chart
A column chart with six columns. A-29 has 16.1% of its refunds between 90 and 99.99 dollars. The next four agents, A-28, A-26, A-15 and A-30, have between 4.9% and 5.4%. The last column, all agents together, is 3.5%.
A-29 has 40 of 249 refunds just below the limit: 16.1%, against 3.5% for all agents. A-29 is the only agent after the Bonferroni correction. We planted this from 2024-09-02. The other agents in the top five are a little above 3.5%, and their p-values are near 0.1. With 40 agents, a few always look a little high by chance.
This check is clearer than Benford’s law, because it asks the real question: does anyone avoid the approval? The first-digit test only showed a pile of 9s, which the limit explains.
A review would ask A-29 and their manager. There can be an honest reason, such as a product that costs 95 dollars and often arrives broken. The check finds where to look, and a person finds why.
Duplicates, round numbers and totals
Section titled “Duplicates, round numbers and totals”Three more checks take one line each.
Duplicates. The same order refunded twice with the same amount:
14 pairs: the same order refunded twice with the same amount date agent_id order_id amount reason2024-09-28 A-03 O-17912763 72.54 missing2024-09-28 A-03 O-17912763 72.54 missing2024-04-21 A-11 O-21434121 35.04 late2024-04-23 A-11 O-21434121 35.04 lateEach pair is a refund paid twice (planted). Some pairs are two days apart, so a check that looks only within one day misses them. A real refund can be split into two parts, so a duplicate is a reason to review, not a proof.
Round numbers. Only 1.01% of Larkfield’s refunds are whole dollars. A team that types amounts by hand, or invents them, shows far more round amounts, such as 50.00 or 100.00.
Totals that do not reconcile. The daily feed gives each agent’s total refund amount. The refunds table has each refund. The two must agree:
21 agent-days where the feed total does not match the refunds size sumdate2025-03-16 1 1000.002025-04-06 1 99.002025-07-04 1 -250.002025-11-10 18 -1692.39On 2025-11-10, the feed sent zeros: all 18 agents who refunded that day show a negative difference. This is the broken feed that the control chart of the first lesson showed as a day at 0%. The other three agent-days are single differences, which we planted. A reconciliation finds broken data as often as fraud.
What these checks show, and what they do not
Section titled “What these checks show, and what they do not”| Check | It shows | It does not show |
|---|---|---|
| First digits (Benford) | a group whose digits differ from the law | fraud, or anything at all if the law does not apply |
| Just below a limit | a person or a team that avoids an approval more often than others | the reason |
| Duplicates | the same refund paid twice | that the second payment was not allowed |
| Round numbers | amounts typed or chosen by hand | that they are false |
| Totals that do not reconcile | two records that disagree | which record is wrong |
These checks are cheap, fast and easy to explain to an auditor. They cost the time of the people who review their results. And a check used on the wrong data wastes that time, or harms someone’s reputation.
Guided practice: the checks in the notebook
Section titled “Guided practice: the checks in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, write your prediction. Then run the cell. The digit 1 comes 7 times.
- Run the Benford cell, the helpers cell and the data cell.
- In section 2, write your prediction, then run the Benford table. You see a MAD of
0.0215for the payments. - Run the plot cell. The payments’ line is below the law at 1 and above it at 2 to 4.
- In section 3, run the cell. A-29 is first, with
Trueafter Bonferroni. - In sections 4 and 5, run the cells. You see
14 pairsand21 agent-days. - In section 6, write the three answers and run each check cell. Each prints
Check passed.
Your turn: the second agent
Section titled “Your turn: the second agent”Change one thing: look at the second agent instead of the first. Which agent has the second-highest share of refunds just below the limit? Before you look, predict: will that agent pass the Bonferroni correction too? In the notebook, put the agent ID in second_agent.
Show the answer
second_agent = per_agent.sort_values("mean", ascending=False).index[1]Check passed: second_agent is right.The second agent is A-28, with 11 of 202 refunds (5.4%) just below the limit. Its p-value is 0.105, far above the Bonferroni limit of 0.05 / 40 = 0.00125. A-28 is the highest of the agents who look normal. With 40 agents, one of them must be the highest by chance. Reviewing A-28 would cost time for no reason.
Common mistake
Benford on the wrong data
Symptom: Tomás runs Benford’s law on all 343,074 payments. The MAD is 0.0215: a nonconformity by Nigrini’s limits. He asks Sofia’s team to audit the payments of the last six months.
payments 343074 0.0215 nonconformity 2.24 0.2488 0.0323Cause: payments are a mixture of basket sizes, mostly between 15 and 300 dollars, so their first digits do not follow Benford’s curve. Nobody planted anything in the first digits of the payments. The law does not fit this data, so its result says nothing about fraud.
Diagnose: check the conditions before you trust the test. Do the values multiply, span several orders of magnitude, and have no minimum or maximum? Is there a known, honest reason for the shape, such as a price list?
Fix: use a direct check of the business question instead: amounts just below a limit, duplicates, totals that do not reconcile. For fraud in payments, use the detectors of the next modules, which compare each payment with the customer’s usual behaviour.
Consequence in AI work: a test that does not apply still prints a number, and a number looks like evidence. A review team that chases it loses days, and the people under review lose trust.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain in your own words why the first digit 1 is more common than 9 in data like savings balances, and name three conditions for Benford's law to apply.
Show the answer
A value that grows by multiplication must double to go from 10 to 20, but needs only 11% more to go from 90 to 100. So it spends much longer with a first digit of 1 than with a 9: in the example, 7 of 21 balances started with 1. The conditions: the values come from causes that multiply, and they span several orders of magnitude. They have no built-in minimum or maximum, they are measured (not assigned numbers), and there are enough of them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield's payments give a Benford MAD of 0.0215, a nonconformity, and A-29's refunds give 0.0449 with 18.1% of first digits 9. What does each result tell you?
Show the answer
The payments do not tell you anything about fraud. They are a mixture of basket sizes, mostly between 15 and 300 dollars, so Benford’s law does not apply. Nothing was planted in their first digits. A-29’s pile of 9s is worth a look, and a direct check explains it. 16.1% of A-29’s refunds are between 90 and 99.99 dollars, just below the 100-dollar approval limit. For all agents, the share is 3.5%. That is a reason to review A-29’s refunds with a person, not a proof of fraud.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Mei finds 21 agent-days where the feed's total refund amount does not match the refunds table. 18 of them are on 2025-11-10. What do you conclude, and what do you do next?
Show the answer
The 18 agent-days on 2025-11-10 are one problem: that day, the daily feed sent zeros, so every agent who refunded shows a negative difference. It is broken data, not fraud. Mei should fix or re-send the feed for that day. The other three agent-days (2025-03-16, 2025-04-06 and 2025-07-04) are single differences, which need a review of those agents’ refunds. A reconciliation shows that two records disagree, not which one is wrong.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Benford’s law fits data that grows by multiplication across several orders of magnitude. Larkfield’s payments do not fit it (MAD 0.0215), and nothing is wrong with them.
- A direct check of the business question is clearer: A-29 has 16.1% of refunds just below the 100-dollar limit, against 3.5% for all agents.
- Duplicates, round numbers and totals that do not reconcile find refunds paid twice and broken feeds: 14 pairs, 1.01% whole dollars, 21 agent-days.
- Each check is a reason to review, not a proof.
In this module, you watched processes with control charts, compared distributions and checked amounts. But each method looks at one number or one feature at a time. A stolen card can look normal on every feature alone and strange on all of them together. Next, you find payments that are far from their neighbours across many features at once.