Bandits
- Lesson 28
- 35 min
- Module 8: Experiments for AI features and in production
Where we are
An A/B test keeps half of the traffic on the worse version until it ends. A bandit moves traffic to the better version while it learns. In this lesson, we run Thompson sampling on a headline choice and see what it gains, and what evidence it does not give.
Lesson 28 of 30 · Experiments and Causal Impact: Did Our Change Work?Bandits
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Use a bandit when you want to earn while you learn: it moves traffic to the best option during the test. Learn how Thompson sampling chooses, what it gains over a fixed split, and why its comparisons are weaker evidence than an A/B test.
NotebookDo this lesson in a notebook
In the notebook, you write Thompson sampling in a few lines. You run it against a fixed split on four subject lines, and count the clicks that each one loses. It simulates the emails, so it needs no practice data and no key. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- An A/B test with a fixed split, and its 95% interval, from The result and its uncertainty.
- Why looking at a test every day is a trap, from Peeking.
- What a probability distribution is. For a refresher, see Distributions in practice in the mathematics course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m08, from Test a model or a prompt online, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m08 snapshot. Windows was not tested.
Lunch in a new town
Section titled “Lunch in a new town”You start a new job in a new town. There are four lunch places near the office. In the first week, you try each of them. One is clearly good. Do you eat there every day from now on?
Most people do something in between. They eat at the good place most days. Now and then, they try another place, because one bad day may have been bad luck. If a place disappoints again and again, they stop trying it.
That is the whole idea of a bandit. Each day, you exploit: you use what you know to choose the best option. Sometimes you explore: you try an option that you know less about, to learn more. A good bandit explores less as it learns more.
The name comes from slot machines, which were called “one-armed bandits”. A gambler with several machines must decide which arm to pull next. Each option is an arm.
The problem: a fixed split keeps sending the worst line
Section titled “The problem: a fixed split keeps sending the worst line”Rafael, the email manager, has four subject lines for the next reminder email. He wants the one that gets the most clicks. The click rates are simulated, so we know the truth:
| Line | A | B | C | D |
|---|---|---|---|---|
| True click rate | 4.0% | 4.6% | 5.2% | 4.4% |
Rafael sends 40,000 emails. A normal A/B/n test splits them equally: 10,000 per line, from the first email to the last. The test finds the best line in the end. But it sent 10,000 emails with line A, the worst one, all the way through.
If every email had used line C, Rafael would expect 0.052 × 40,000 = 2,080 clicks. The fixed split expects (4.0 + 4.6 + 5.2 + 4.4) / 4 = 4.55% on average, so 1,820 clicks. It loses 260 clicks on average. These are the costs of learning.
For a decision that lasts a year, such as a free-delivery threshold, 260 clicks do not matter. For an email that runs once, the test is the whole campaign. Then every click lost in the test is lost forever.
The idea: Thompson sampling
Section titled “The idea: Thompson sampling”Thompson sampling is one way to run a bandit. It keeps, for each arm, a range of believable click rates. It is simple to say in four steps.
A range of believable click rates
Section titled “A range of believable click rates”Say line A got 5 clicks from 100 emails, and line B got 9 clicks from 100. Which click rates are believable for each line? With only 100 emails, many rates are. The chart shows how believable each rate is. This shape is a Beta distribution: the usual choice for a rate from clicks and non-clicks.
- Line A: 5 clicks of 100
- Line B: 9 clicks of 100
Text description of the chart
Two hill-shaped curves over click rates from 0 to 25%. Line A, 5 clicks of 100, peaks near 5%; its middle 95% goes from about 2.2% to 11.2%. Line B, 9 clicks of 100, peaks near 9%; its middle 95% goes from about 4.9% to 16.2%. The curves overlap between about 5% and 11%.
To choose the next email, Thompson sampling draws one rate at random from each curve. Most draws for B are higher than for A, so B gets most emails. But sometimes A’s draw is higher, so A is still tried. As emails come in, the curves get narrower. A weak line then almost never wins the draw.
A tiny run with every value visible
Section titled “A tiny run with every value visible”Here is a real run of 12 emails, with two lines. The true rates are 20% for A and 50% for B, chosen large so that you can see the clicks. The draws are random numbers, rounded.
| A so far: clicks / sent | B so far: clicks / sent | Draw for A | Draw for B | Sends | Click? | |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | 0 / 0 | 0.01 | 0.52 | B | yes |
| 2 | 0 / 0 | 1 / 1 | 0.47 | 0.6 | B | no |
| 3 | 0 / 0 | 1 / 2 | 0.21 | 0.59 | B | no |
| 4 | 0 / 0 | 1 / 3 | 0.67 | 0.55 | A | no |
| 5 | 0 / 1 | 1 / 3 | 0 | 0.36 | B | no |
| 6 | 0 / 1 | 1 / 4 | 0.86 | 0.17 | A | yes |
| 7 | 1 / 2 | 1 / 4 | 0.55 | 0.18 | A | no |
| 8 | 1 / 3 | 1 / 4 | 0.5 | 0.18 | A | no |
| 9 | 1 / 4 | 1 / 4 | 0.48 | 0.56 | B | yes |
| 10 | 1 / 4 | 2 / 5 | 0.59 | 0.56 | A | no |
| 11 | 1 / 5 | 2 / 5 | 0.44 | 0.37 | A | no |
| 12 | 1 / 6 | 2 / 5 | 0.51 | 0.13 | A | no |
Follow the first emails. Before email 1, nothing is known, so both draws are random between 0 and 1. B’s draw is higher, B is sent, and the customer clicks. Now B has 1 click of 1 email, so its draws tend to be high. Emails 2 and 3 go to B too, with no click. At email 4, A’s draw (0.67) beats B’s (0.55): A is explored.
After 12 emails, A has 1 click of 7, and B 2 clicks of 5. The bandit does not yet favour B, which is truly better. Twelve emails are far too few. A bandit needs thousands of emails to settle, as the TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield run shows.
Larkfield: four subject lines
Section titled “Larkfield: four subject lines”xp bandit runs both ways on the same four lines: a fixed equal split and Thompson sampling, 40,000 emails each. It sends emails in batches of 100 and updates the ranges after each batch.
-
With the project’s
.venvactive, run the bandit.Terminal window xp banditSubject lines A 4.0%, B 4.6%, C 5.2%, D 4.4% (true click rates, simulated); 40,000 emails.fixed equal split (A/B/n test): 1,849 clicks; expected clicks lost against always the best: 261traffic share by tenth of the emails, best line C: 24% 25% 24% 25% 26% 25% 24% 25% 26% 25%chance each line is best: A 0.00, B 0.00, C 1.00, D 0.00C − A: +0.0112 (95% interval +0.0052 to +0.0171; true +0.0120); emails: 9,941 and 10,038Thompson sampling: 1,941 clicks; expected clicks lost against always the best: 44traffic share by tenth of the emails, best line C: 55% 80% 84% 87% 93% 93% 92% 93% 87% 89%chance each line is best: A 0.01, B 0.02, C 0.92, D 0.05C − A: +0.0217 (95% interval +0.0069 to +0.0365; true +0.0120); emails: 34,161 and 497 -
Repeat both ways 100 times, to see the average and not one lucky run.
Terminal window xp bandit --repeats 100100 repeats, fixed: clicks lost 260 on average; the best line has the highest observed rate in 97% of them100 repeats, thompson: clicks lost 73 on average; the best line has the highest observed rate in 100% of themWe show only the last two lines. The first eight lines are the same as in step 1.
Traffic moves to the best line
Section titled “Traffic moves to the best line”- Line C, Thompson sampling
- Line C, fixed split
- Line A, Thompson sampling
Text description of the chart
A line chart over the ten tenths of the campaign. Fixed split, line C: about 25% in every tenth, from 23.5% to 26.3%. Thompson sampling, line C: 54.6% in the first tenth, 80.2% in the second, then 84% to 93%, ending at 89.4%. Thompson sampling, line A: 3.6% in the first tenth, then almost 0%, with a small rise to 4.4% and 3.1% in the last two tenths.
In this run, the fixed split loses 261 clicks and Thompson sampling 44. Over 500 repeats, the course measured 260 clicks lost for the fixed split and 80 for Thompson sampling. Your 100 repeats gave 73. Both ways find the best line at the end about equally often: 97.4% and 97.8% of the 500 repeats.
Text description of the chart
A column chart. Fixed equal split: 260 clicks lost on average. Thompson sampling: 80 clicks lost on average. Both averages are over 500 simulated campaigns of 40,000 emails.
What a bandit cannot tell you
Section titled “What a bandit cannot tell you”TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. says: “So a bandit gives the same evidence as an A/B test, but faster and cheaper.” It does not. Look at the last line of each block in step 1: the comparison of the best line C with the worst line A.
Text description of the chart
A panel with a line at 0 and a line at the true difference, +1.2 points. Fixed split, 9,941 and 10,038 emails: +1.12 points (+0.52 to +1.71). Thompson sampling, 34,161 and 497 emails: +2.17 points (+0.69 to +3.65).
The weak lines get too few emails
Section titled “The weak lines get too few emails”Thompson sampling stopped sending line A early. So it knows little about A. The difference C − A is +2.17 points, against a true +1.2. Its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI is wide. A bandit is good at finding the best arm. It is bad at saying how much better the best arm is.
The weak lines look worse than they are
Section titled “The weak lines look worse than they are”This is a subtle problem. A line that is unlucky early gets fewer emails, so its bad luck is never corrected. A line that is lucky early gets more emails, and its luck is corrected. Over the 500 repeats, line A’s observed rate was 3.64% on average under Thompson sampling, against a true 4.0%. Under the fixed split, it was 4.00%.
So the intervals of a bandit are not as honest as they look. Over the 500 repeats, the 95% interval of C − A contained the truth in 89.6% of runs with Thompson sampling, and 94.4% with the fixed split.
When nothing differs, a bandit still picks favourites
Section titled “When nothing differs, a bandit still picks favourites”The worst case is four lines that are all the same. In 500 simulated campaigns with four lines at 5%, we compared the line with the most emails and the line with the fewest. Under the fixed split, the interval excluded zero in 5.0% of campaigns: the false-positive rate of a normal test. Under Thompson sampling, it excluded zero in 31.4%. A bandit’s numbers can show a “clear” difference that does not exist. You will see one in the Your turn.
Other limits
Section titled “Other limits”- A bandit optimises one metric, and it needs it fast. Clicks come in minutes. Orders, returns and complaints come in days or weeks: a bandit cannot wait for them.
- A bandit has no guardrail. It sends more traffic to whatever wins on its one metric, even if that line causes more unsubscribes.
- A bandit assumes the rates stay the same. If a line works better on Mondays, the bandit’s early choices can be wrong.
A bandit or an A/B test?
Section titled “A bandit or an A/B test?”| Question | Use an A/B test | Use a bandit |
|---|---|---|
| What do you want? | To know how big the effect is, with an interval | To get the most clicks now |
| How long does the decision last? | Long: a price rule, a page design | Short: one email, a headline for one day |
| When does the outcome come? | Any time, also weeks later | Within minutes or hours |
| Guardrails? | Yes, several | Only if you add them yourself |
| Many options? | Few | Many, such as 10 headlines |
For search results, a third way exists: interleaving mixes two rankers in one list, as When units affect each other showed. See Interleaving in Search and Ranking.
What a bandit costs: some code, a metric that comes in fast, and the loss of a clean comparison. What it saves: the traffic that a fixed split sends to the losing options.
Guided practice: Thompson sampling in a notebook
Section titled “Guided practice: Thompson sampling in a notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- In section 1, read the four true click rates and the
thompsonfunction. It has six lines. - In section 2, run 40,000 emails with a fixed split and with Thompson sampling. You see the clicks and the emails per line.
- In section 3, plot the share of emails that line C gets in each tenth.
- In section 4, compute the expected clicks lost by Thompson sampling against always line C. Put the value in
lost_thompson, and run the check cell. It printsCheck passed.
Show the answer to step 5
lost_thompson = round(float(rates.max() * EMAILS - (shown_t * rates).sum()), 1)print(lost_thompson)The expected clicks lost are the clicks of always line C, minus the clicks that each line’s emails are expected to get. The value is 97.6 clicks, against 260.7 for the fixed split. The notebook sends one email at a time, with other random numbers than xp bandit, so its numbers differ a little.
Your turn: four equal lines
Section titled “Your turn: four equal lines”Predict first. If all four lines have the same true click rate, 5%, how will Thompson sampling share the emails? And what will the comparison of two lines show? Then run:
xp bandit --rates 0.05,0.05,0.05,0.05Show the answer
Subject lines A 5.0%, B 5.0%, C 5.0%, D 5.0% (true click rates, simulated); 40,000 emails. fixed equal split (A/B/n test): 1,999 clicks; expected clicks lost against always the best: -0 traffic share by tenth of the emails, best line A: 26% 25% 25% 25% 24% 25% 26% 25% 25% 24% chance each line is best: A 0.37, B 0.06, C 0.47, D 0.09 A − B: +0.0025 (95% interval -0.0035 to +0.0086; true +0.0000); emails: 10,038 and 9,878 Thompson sampling: 1,945 clicks; expected clicks lost against always the best: 0 traffic share by tenth of the emails, best line A: 3% 0% 0% 0% 0% 1% 1% 1% 0% 0% chance each line is best: A 0.00, B 0.27, C 0.65, D 0.08 A − B: -0.0300 (95% interval -0.0453 to -0.0146; true +0.0000); emails: 323 and 11,128With equal lines, no line is “best”, so the tool labels line A. Nothing is lost either way (the “-0” is a rounding of zero).
Thompson sampling did not share the emails equally. Line A got only 323 emails, after an unlucky start, and B got 11,128. The comparison A − B is −3.00 points, with an interval from −4.53 to −1.46: a “clear” difference. The truth is zero. This is the false difference of the 31.4% above. With the fixed split, the interval contains zero. Never report a bandit’s comparison as if it came from an A/B test.
Common mistake
“A bandit gives the same evidence, faster”
Symptom: a team runs a bandit on four headlines for a week. They report: “Headline C beats headline A by 2.2 points (95% interval 0.7 to 3.7)”. The next A/B test of the same headlines finds 1.2 points.
Cause: the bandit sent almost no traffic to A after the first days. A’s estimate stayed low after its unlucky start, and its interval is wide and not honest. A bandit finds the best arm; it does not measure the differences between arms.
Fix: use a bandit to earn clicks, and report only which arm it chose. Do you need the size of the effect, for a decision that lasts? Then run an A/B test with a fixed split, or keep a fixed share of traffic for each arm.
Check your understanding
Completion check. Choose an A/B test or a bandit for each situation, with your reasons. (1) Larkfield's Black Friday email: one send of 200,000 emails, five subject lines, clicks within hours. (2) Hana's new free-delivery threshold: a rule that will stay for a year, measured on revenue and returns over four weeks.
Show the answer
(1) A bandit. The email is sent once, so the test is the whole campaign, and every click lost to a weak line is lost for good. Clicks come within hours, so the bandit can learn fast. Larkfield needs the best line, not the exact size of each difference. (2) An A/B test with a fixed split. The decision lasts a year, so Hana needs the size of the effect with an interval, to compare it with the cost. Revenue and returns come over weeks, too slowly for a bandit, and the return rate is a guardrail that a bandit would ignore.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the Larkfield run, Thompson sampling estimates line C − line A at +2.17 points, with a 95% interval from +0.69 to +3.65. The truth is +1.2. Explain in two or three sentences why a bandit's comparison of the best and the worst line is weak.
Show the answer
The bandit stopped sending line A early. So A got only 497 emails, its rate is measured badly, and the interval is wide. Also, an arm that is unlucky early gets fewer emails, so its low estimate stays low. Over 500 repeats, A’s observed rate was 3.64% against a true 4.0%. So the difference looks larger than it is, and the interval contains the truth less often than 95%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Explore or exploit? A bandit has sent 10,000 emails with line C (520 clicks) and 300 emails with line D (12 clicks). Why does Thompson sampling still sometimes send line D?
Show the answer
Line D has few emails, so its range of believable click rates is wide. 12 clicks of 300 is 4%, but the true rate could well be 5% or more. Sometimes a random draw from D’s wide range beats the draw from C’s narrow range near 5.2%. Then D gets the email. This is exploring: it checks that D is not better than it looks. As D gets more emails, its range narrows, and it is sent less often.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A bandit explores and exploits: it moves traffic to the best option while it learns. Thompson sampling draws a believable rate for each arm and sends the arm with the highest draw.
- On four subject lines (simulated), it lost 80 clicks against 260 for a fixed split, and found the best line as often.
- It is weak evidence for the size of differences. C − A: +2.17 points against a true +1.2. Interval coverage: 89.6%. A false “clear” difference: 31.4% of tests with equal lines.
- Use a bandit for short decisions with fast outcomes. Use an A/B test when you need the size of the effect, guardrails or a decision that lasts.
Next, we write the result of a test so that people trust it and act on it.