Skip to content
Français

Benford's law and other business checks

  • Lesson 14
  • 35 min
  • Module 3: Statistical process control and distribution drift

Where we are

We have charts for processes and tests for distributions. Auditors also use simple checks on the amounts themselves. In this lesson, we test first digits with Benford's law, look for refunds just below the approval limit, and see what these checks prove and what they do not.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 14 of 38 · Anomaly Detection: Find What Is UnusualBenford's law and other business checks
Course overview

What you will be able to do

Check amounts the way auditors do: first digits, amounts just below a limit, duplicates, round numbers and totals that must agree. Each check is fast and easy to explain, and each one can point at an innocent person if you use it on the wrong data.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It tests Benford’s law on payments and refunds, then refunds just below the approval limit, duplicates, round amounts and the daily totals. Its setup cells install the packages and download the data (about 8 MB). About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing. The notebook runs in Colab, Kaggle or VS Code.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, NumPy 2.5.4 and SciPy 1.18.1. Colab and Kaggle were not tested.

Put 10 dollars in a savings account that grows by 12% each year. Write down the balance every year, and look at its first digit. From 10 to 20 dollars, the balance must double, which takes about six years. From 90 to 100 dollars, it needs only 11% more: less than one year. So the balance spends a long time with a first digit of 1, and very little time with a first digit of 9.

Here are the 21 balances from 10 dollars to 96.50 dollars. This is a constructed example:

[10.0, 11.2, 12.5, 14.0, 15.7, 17.6, 19.7, 22.1, 24.8, 27.7, 31.1, 34.8, 39.0, 43.6, 48.9, 54.7, 61.3, 68.7, 76.9, 86.1, 96.5]
first digits: {1: 7, 2: 3, 3: 3, 4: 2, 5: 1, 6: 2, 7: 1, 8: 1, 9: 1}
The first digits of a balance that grows 12% a year from 10 to 96.50 dollars. The digit 1 comes 7 times; the digits 5, 7, 8 and 9 come once each.
Text description of the chart

A column chart with nine columns, for the first digits 1 to 9. The digit 1 has the tallest column, 7 of the 21 balances. The digits 2 and 3 have 3 each, 4 and 6 have 2 each, and 5, 7, 8 and 9 have 1 each.

A third of the balances start with 1. The same happens in many real collections of numbers: the populations of towns, the lengths of rivers, the amounts on invoices.

The problem: invented numbers look different

Section titled “The problem: invented numbers look different”

People who invent numbers do not know this. When someone makes up the amounts of false expense claims, they tend to use every first digit about equally. They also like amounts just below a limit that needs approval, and round numbers. Auditors use these habits to choose which records to review.

But a check that is used on the wrong data points at innocent people. So for each check, we must know what it shows, and when it does not apply.

The astronomer Simon Newcomb noticed the pattern in 1881, and the physicist Frank Benford tested it on thousands of numbers in 1938. Benford’s law says that, in data of this kind, the share of the first digit d is log₁₀(1 + 1/d):

First digit 1 2 3 4 5 6 7 8 9
Benford’s share 30.1% 17.6% 12.5% 9.7% 7.9% 6.7% 5.8% 5.1% 4.6%

To compare data with the law, compute the share of each first digit in the data. Then take the mean absolute deviation (MAD): the mean of the nine differences, without their signs. Be careful: this MAD is a mean, and it compares digit shares. It is not the median absolute deviation of Median, MAD and IQR. Benford tests use the same short name. The auditor Mark Nigrini gives limits for the first digit (Nigrini, Benford’s Law, Wiley, 2012). A MAD below 0.006 is a close fit, and 0.006 to 0.012 is acceptable. From 0.012 to 0.015 it is marginal, and above 0.015 it is a nonconformity.

Before you read the output, predict: which of these follow Benford’s law? All payments, all refunds, the refunds of agent A-29, and the payments under 5 dollars.

amounts n MAD Nigrini orders of magnitude first digit 1 first digit 9
payments 343074 0.0215 nonconformity 2.24 0.2488 0.0323
refunds 9923 0.0132 marginal 1.61 0.2516 0.0424
refunds of A-29 249 0.0449 nonconformity 1.72 0.1727 0.1807
payments under 5 dollars 2712 0.0905 nonconformity 0.67 0.1947 0.0000

The orders of magnitude column is the range of the amounts: log₁₀ of the largest (the 99th percentile) divided by the smallest (the 1st percentile). A value of 2 means the amounts span a factor of 100.

  • Benford's law
  • All payments
  • Refunds of A-29
The share of each first digit: Benford's law, all 343,074 payments, and the 249 refunds of agent A-29. The payments have fewer 1s and more 2s to 4s than the law; A-29 has 18.1% of first digits 9.
Text description of the chart

The chart has three lines with dots, for the first digits 1 to 9. Benford’s law falls smoothly from 30.1% for the digit 1 to 4.6% for the digit 9. The payments start lower, at 24.9% for the digit 1. They are above the law for the digits 2 to 4 (21.1%, 16.3% and 11.8%), then below it, down to 3.2% for the digit 9. The refunds of A-29 are near 15% to 17% for the digits 1 to 3. They are between 4% and 7% for the digits 5 to 8. They jump to 18.1% for the digit 9.

No group is a close fit. Each result needs its own reading:

  • The payments are a nonconformity (MAD 0.0215), and nobody planted anything in their first digits. Payments are a mixture of basket sizes: most are between 15 and 300 dollars. Their first digits do not follow Benford’s curve. The test is not wrong: the law does not apply to this data.
  • The payments under 5 dollars span less than one order of magnitude (0.67). Their first digit can only be 1 to 4. Benford cannot apply at all.
  • The refunds are marginal (MAD 0.0132). Refunds are capped by what the customer paid, and those over 100 dollars need approval.
  • A-29’s refunds have 18.1% of first digits 9, against 4.6% in the law. This one result points at a person. It is worth a review, and the next check says why.

Use it only on data that has all of these properties:

  1. The values come from many causes that multiply, such as growth, prices times quantities, or sums of many items.
  2. They span several orders of magnitude: from tens to thousands of dollars, for example.
  3. They have no built-in minimum or maximum, such as a price list, a cap or a fee.
  4. They are measured, not assigned: invoice numbers, phone numbers and product codes do not follow it.
  5. There are enough of them: hundreds at least, and better thousands.

Even then, a deviation is a reason to review, not a proof of fraud.

At Larkfield, a refund of 100 dollars or more needs a team lead’s approval. Refunds that cluster just below the limit can mean that someone avoids the approval. This check asks the business question directly. For each agent, the notebook computes the share of refunds between 90 and 99.99 dollars. Then it compares that share with all agents, with a binomial test, and with the Bonferroni correction for 40 agents:

refunds["just_below"] = (refunds.amount >= 90) & (refunds.amount < 100) # the approval limit is 100 dollars
share_all = refunds.just_below.mean()
per_agent = refunds.groupby("agent_id").just_below.agg(["mean", "sum", "size"])
per_agent["p"] = [stats.binomtest(int(k), int(n), share_all, alternative="greater").pvalue
for k, n in zip(per_agent["sum"], per_agent["size"])]
per_agent["after Bonferroni"] = per_agent.p < 0.05 / len(per_agent)
all agents: 3.5% of refunds at 90 to 99.99 dollars
mean sum size p after Bonferroni
agent_id
A-29 0.161 40 249 0.000 True
A-28 0.054 11 202 0.105 False
A-26 0.053 13 245 0.097 False
A-15 0.050 11 222 0.166 False
A-30 0.049 12 245 0.162 False
The five agents with the highest share of refunds just below the 100-dollar approval limit, and all agents together. A-29 has 16.1%, more than four times the 3.5% of all agents.
Text description of the chart

A column chart with six columns. A-29 has 16.1% of its refunds between 90 and 99.99 dollars. The next four agents, A-28, A-26, A-15 and A-30, have between 4.9% and 5.4%. The last column, all agents together, is 3.5%.

A-29 has 40 of 249 refunds just below the limit: 16.1%, against 3.5% for all agents. A-29 is the only agent after the Bonferroni correction. We planted this from 2024-09-02. The other agents in the top five are a little above 3.5%, and their p-values are near 0.1. With 40 agents, a few always look a little high by chance.

This check is clearer than Benford’s law, because it asks the real question: does anyone avoid the approval? The first-digit test only showed a pile of 9s, which the limit explains.

A review would ask A-29 and their manager. There can be an honest reason, such as a product that costs 95 dollars and often arrives broken. The check finds where to look, and a person finds why.

Three more checks take one line each.

Duplicates. The same order refunded twice with the same amount:

14 pairs: the same order refunded twice with the same amount
date agent_id order_id amount reason
2024-09-28 A-03 O-17912763 72.54 missing
2024-09-28 A-03 O-17912763 72.54 missing
2024-04-21 A-11 O-21434121 35.04 late
2024-04-23 A-11 O-21434121 35.04 late

Each pair is a refund paid twice (planted). Some pairs are two days apart, so a check that looks only within one day misses them. A real refund can be split into two parts, so a duplicate is a reason to review, not a proof.

Round numbers. Only 1.01% of Larkfield’s refunds are whole dollars. A team that types amounts by hand, or invents them, shows far more round amounts, such as 50.00 or 100.00.

Totals that do not reconcile. The daily feed gives each agent’s total refund amount. The refunds table has each refund. The two must agree:

21 agent-days where the feed total does not match the refunds
size sum
date
2025-03-16 1 1000.00
2025-04-06 1 99.00
2025-07-04 1 -250.00
2025-11-10 18 -1692.39

On 2025-11-10, the feed sent zeros: all 18 agents who refunded that day show a negative difference. This is the broken feed that the control chart of the first lesson showed as a day at 0%. The other three agent-days are single differences, which we planted. A reconciliation finds broken data as often as fraud.

What these checks show, and what they do not

Section titled “What these checks show, and what they do not”
Check It shows It does not show
First digits (Benford) a group whose digits differ from the law fraud, or anything at all if the law does not apply
Just below a limit a person or a team that avoids an approval more often than others the reason
Duplicates the same refund paid twice that the second payment was not allowed
Round numbers amounts typed or chosen by hand that they are false
Totals that do not reconcile two records that disagree which record is wrong

These checks are cheap, fast and easy to explain to an auditor. They cost the time of the people who review their results. And a check used on the wrong data wastes that time, or harms someone’s reputation.

Guided practice: the checks in the notebook

Section titled “Guided practice: the checks in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, write your prediction. Then run the cell. The digit 1 comes 7 times.
  4. Run the Benford cell, the helpers cell and the data cell.
  5. In section 2, write your prediction, then run the Benford table. You see a MAD of 0.0215 for the payments.
  6. Run the plot cell. The payments’ line is below the law at 1 and above it at 2 to 4.
  7. In section 3, run the cell. A-29 is first, with True after Bonferroni.
  8. In sections 4 and 5, run the cells. You see 14 pairs and 21 agent-days.
  9. In section 6, write the three answers and run each check cell. Each prints Check passed.

Change one thing: look at the second agent instead of the first. Which agent has the second-highest share of refunds just below the limit? Before you look, predict: will that agent pass the Bonferroni correction too? In the notebook, put the agent ID in second_agent.

Show the answer
second_agent = per_agent.sort_values("mean", ascending=False).index[1]
Check passed: second_agent is right.

The second agent is A-28, with 11 of 202 refunds (5.4%) just below the limit. Its p-value is 0.105, far above the Bonferroni limit of 0.05 / 40 = 0.00125. A-28 is the highest of the agents who look normal. With 40 agents, one of them must be the highest by chance. Reviewing A-28 would cost time for no reason.

Common mistake

Benford on the wrong data

Symptom: Tomás runs Benford’s law on all 343,074 payments. The MAD is 0.0215: a nonconformity by Nigrini’s limits. He asks Sofia’s team to audit the payments of the last six months.

payments 343074 0.0215 nonconformity 2.24 0.2488 0.0323

Cause: payments are a mixture of basket sizes, mostly between 15 and 300 dollars, so their first digits do not follow Benford’s curve. Nobody planted anything in the first digits of the payments. The law does not fit this data, so its result says nothing about fraud.

Diagnose: check the conditions before you trust the test. Do the values multiply, span several orders of magnitude, and have no minimum or maximum? Is there a known, honest reason for the shape, such as a price list?

Fix: use a direct check of the business question instead: amounts just below a limit, duplicates, totals that do not reconcile. For fraud in payments, use the detectors of the next modules, which compare each payment with the customer’s usual behaviour.

Consequence in AI work: a test that does not apply still prints a number, and a number looks like evidence. A review team that chases it loses days, and the people under review lose trust.

Check your understanding

Explain in your own words why the first digit 1 is more common than 9 in data like savings balances, and name three conditions for Benford's law to apply.

Show the answer

A value that grows by multiplication must double to go from 10 to 20, but needs only 11% more to go from 90 to 100. So it spends much longer with a first digit of 1 than with a 9: in the example, 7 of 21 balances started with 1. The conditions: the values come from causes that multiply, and they span several orders of magnitude. They have no built-in minimum or maximum, they are measured (not assigned numbers), and there are enough of them.

Check your understanding

Larkfield's payments give a Benford MAD of 0.0215, a nonconformity, and A-29's refunds give 0.0449 with 18.1% of first digits 9. What does each result tell you?

Show the answer

The payments do not tell you anything about fraud. They are a mixture of basket sizes, mostly between 15 and 300 dollars, so Benford’s law does not apply. Nothing was planted in their first digits. A-29’s pile of 9s is worth a look, and a direct check explains it. 16.1% of A-29’s refunds are between 90 and 99.99 dollars, just below the 100-dollar approval limit. For all agents, the share is 3.5%. That is a reason to review A-29’s refunds with a person, not a proof of fraud.

Check your understanding

Mei finds 21 agent-days where the feed's total refund amount does not match the refunds table. 18 of them are on 2025-11-10. What do you conclude, and what do you do next?

Show the answer

The 18 agent-days on 2025-11-10 are one problem: that day, the daily feed sent zeros, so every agent who refunded shows a negative difference. It is broken data, not fraud. Mei should fix or re-send the feed for that day. The other three agent-days (2025-03-16, 2025-04-06 and 2025-07-04) are single differences, which need a review of those agents’ refunds. A reconciliation shows that two records disagree, not which one is wrong.

Recap

  • Benford’s law fits data that grows by multiplication across several orders of magnitude. Larkfield’s payments do not fit it (MAD 0.0215), and nothing is wrong with them.
  • A direct check of the business question is clearer: A-29 has 16.1% of refunds just below the 100-dollar limit, against 3.5% for all agents.
  • Duplicates, round numbers and totals that do not reconcile find refunds paid twice and broken feeds: 14 pairs, 1.01% whole dollars, 21 agent-days.
  • Each check is a reason to review, not a proof.

In this module, you watched processes with control charts, compared distributions and checked amounts. But each method looks at one number or one feature at a time. A stolen card can look normal on every feature alone and strange on all of them together. Next, you find payments that are far from their neighbours across many features at once.

Next: Far from the neighbours: kNN distance