Skip to content
Français

Bandits

  • Lesson 28
  • 35 min
  • Module 8: Experiments for AI features and in production

Where we are

An A/B test keeps half of the traffic on the worse version until it ends. A bandit moves traffic to the better version while it learns. In this lesson, we run Thompson sampling on a headline choice and see what it gains, and what evidence it does not give.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Use a bandit when you want to earn while you learn: it moves traffic to the best option during the test. Learn how Thompson sampling chooses, what it gains over a fixed split, and why its comparisons are weaker evidence than an A/B test.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you write Thompson sampling in a few lines. You run it against a fixed split on four subject lines, and count the clicks that each one loses. It simulates the emails, so it needs no practice data and no key. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

Tested on macOS with Python 3.12 and the end-of-m08 snapshot. Windows was not tested.

You start a new job in a new town. There are four lunch places near the office. In the first week, you try each of them. One is clearly good. Do you eat there every day from now on?

Most people do something in between. They eat at the good place most days. Now and then, they try another place, because one bad day may have been bad luck. If a place disappoints again and again, they stop trying it.

That is the whole idea of a bandit. Each day, you exploit: you use what you know to choose the best option. Sometimes you explore: you try an option that you know less about, to learn more. A good bandit explores less as it learns more.

The name comes from slot machines, which were called “one-armed bandits”. A gambler with several machines must decide which arm to pull next. Each option is an arm.

The problem: a fixed split keeps sending the worst line

Section titled “The problem: a fixed split keeps sending the worst line”

Rafael, the email manager, has four subject lines for the next reminder email. He wants the one that gets the most clicks. The click rates are simulated, so we know the truth:

Line A B C D
True click rate 4.0% 4.6% 5.2% 4.4%

Rafael sends 40,000 emails. A normal A/B/n test splits them equally: 10,000 per line, from the first email to the last. The test finds the best line in the end. But it sent 10,000 emails with line A, the worst one, all the way through.

If every email had used line C, Rafael would expect 0.052 × 40,000 = 2,080 clicks. The fixed split expects (4.0 + 4.6 + 5.2 + 4.4) / 4 = 4.55% on average, so 1,820 clicks. It loses 260 clicks on average. These are the costs of learning.

For a decision that lasts a year, such as a free-delivery threshold, 260 clicks do not matter. For an email that runs once, the test is the whole campaign. Then every click lost in the test is lost forever.

Thompson sampling is one way to run a bandit. It keeps, for each arm, a range of believable click rates. It is simple to say in four steps.

A loop of four boxes. 1, for each line, keep the clicks and the emails sent. 2, for each line, draw one believable click rate from what we know. 3, send the line with the highest draw. 4, add the result, click or no click. An arrow labelled next email goes back to step 1. A note: a line we know little about has a wide range, so its draw is sometimes high and it is still tried; a line that is clearly weak has a low, narrow range, so it is almost never tried.
Thompson sampling, one email at a time. Lines that look good get more emails. Lines we know little about still get some, because their wide range sometimes gives a high draw.

Say line A got 5 clicks from 100 emails, and line B got 9 clicks from 100. Which click rates are believable for each line? With only 100 emails, many rates are. The chart shows how believable each rate is. This shape is a Beta distribution: the usual choice for a rate from clicks and non-clicks.

  • Line A: 5 clicks of 100
  • Line B: 9 clicks of 100
How believable each click rate is after 100 emails per line (a Beta distribution, computed). The two ranges overlap a lot: B looks better, but A could still be the better line.
Text description of the chart

Two hill-shaped curves over click rates from 0 to 25%. Line A, 5 clicks of 100, peaks near 5%; its middle 95% goes from about 2.2% to 11.2%. Line B, 9 clicks of 100, peaks near 9%; its middle 95% goes from about 4.9% to 16.2%. The curves overlap between about 5% and 11%.

To choose the next email, Thompson sampling draws one rate at random from each curve. Most draws for B are higher than for A, so B gets most emails. But sometimes A’s draw is higher, so A is still tried. As emails come in, the curves get narrower. A weak line then almost never wins the draw.

Here is a real run of 12 emails, with two lines. The true rates are 20% for A and 50% for B, chosen large so that you can see the clicks. The draws are random numbers, rounded.

Email A so far: clicks / sent B so far: clicks / sent Draw for A Draw for B Sends Click?
1 0 / 0 0 / 0 0.01 0.52 B yes
2 0 / 0 1 / 1 0.47 0.6 B no
3 0 / 0 1 / 2 0.21 0.59 B no
4 0 / 0 1 / 3 0.67 0.55 A no
5 0 / 1 1 / 3 0 0.36 B no
6 0 / 1 1 / 4 0.86 0.17 A yes
7 1 / 2 1 / 4 0.55 0.18 A no
8 1 / 3 1 / 4 0.5 0.18 A no
9 1 / 4 1 / 4 0.48 0.56 B yes
10 1 / 4 2 / 5 0.59 0.56 A no
11 1 / 5 2 / 5 0.44 0.37 A no
12 1 / 6 2 / 5 0.51 0.13 A no

Follow the first emails. Before email 1, nothing is known, so both draws are random between 0 and 1. B’s draw is higher, B is sent, and the customer clicks. Now B has 1 click of 1 email, so its draws tend to be high. Emails 2 and 3 go to B too, with no click. At email 4, A’s draw (0.67) beats B’s (0.55): A is explored.

After 12 emails, A has 1 click of 7, and B 2 clicks of 5. The bandit does not yet favour B, which is truly better. Twelve emails are far too few. A bandit needs thousands of emails to settle, as the TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield run shows.

xp bandit runs both ways on the same four lines: a fixed equal split and Thompson sampling, 40,000 emails each. It sends emails in batches of 100 and updates the ranges after each batch.

  1. With the project’s .venv active, run the bandit.

    Terminal window
    xp bandit
    Subject lines A 4.0%, B 4.6%, C 5.2%, D 4.4% (true click rates, simulated); 40,000 emails.
    fixed equal split (A/B/n test): 1,849 clicks; expected clicks lost against always the best: 261
    traffic share by tenth of the emails, best line C: 24% 25% 24% 25% 26% 25% 24% 25% 26% 25%
    chance each line is best: A 0.00, B 0.00, C 1.00, D 0.00
    C − A: +0.0112 (95% interval +0.0052 to +0.0171; true +0.0120); emails: 9,941 and 10,038
    Thompson sampling: 1,941 clicks; expected clicks lost against always the best: 44
    traffic share by tenth of the emails, best line C: 55% 80% 84% 87% 93% 93% 92% 93% 87% 89%
    chance each line is best: A 0.01, B 0.02, C 0.92, D 0.05
    C − A: +0.0217 (95% interval +0.0069 to +0.0365; true +0.0120); emails: 34,161 and 497
  2. Repeat both ways 100 times, to see the average and not one lucky run.

    Terminal window
    xp bandit --repeats 100
    100 repeats, fixed: clicks lost 260 on average; the best line has the highest observed rate in 97% of them
    100 repeats, thompson: clicks lost 73 on average; the best line has the highest observed rate in 100% of them

    We show only the last two lines. The first eight lines are the same as in step 1.

  • Line C, Thompson sampling
  • Line C, fixed split
  • Line A, Thompson sampling
The share of emails that went to each line, by tenth of the campaign (simulated, seed 40). The fixed split keeps 25% on the best line C to the end. Thompson sampling moves most emails to C within the first tenths, and the worst line A gets almost none.
Text description of the chart

A line chart over the ten tenths of the campaign. Fixed split, line C: about 25% in every tenth, from 23.5% to 26.3%. Thompson sampling, line C: 54.6% in the first tenth, 80.2% in the second, then 84% to 93%, ending at 89.4%. Thompson sampling, line A: 3.6% in the first tenth, then almost 0%, with a small rise to 4.4% and 3.1% in the last two tenths.

In this run, the fixed split loses 261 clicks and Thompson sampling 44. Over 500 repeats, the course measured 260 clicks lost for the fixed split and 80 for Thompson sampling. Your 100 repeats gave 73. Both ways find the best line at the end about equally often: 97.4% and 97.8% of the 500 repeats.

Clicks lost in a campaign of 40,000 emails, against sending line C to everyone: the average of 500 simulated repeats. Thompson sampling loses less than a third of what the fixed split loses.
Text description of the chart

A column chart. Fixed equal split: 260 clicks lost on average. Thompson sampling: 80 clicks lost on average. Both averages are over 500 simulated campaigns of 40,000 emails.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. says: “So a bandit gives the same evidence as an A/B test, but faster and cheaper.” It does not. Look at the last line of each block in step 1: the comparison of the best line C with the worst line A.

The difference in click rate between the best line C and the worst line A, with 95% intervals, in one simulated run (seed 40). The fixed split measures it well. Thompson sampling sent only 497 emails with line A, so its interval is two and a half times wider, and its estimate is far from the truth.
Text description of the chart

A panel with a line at 0 and a line at the true difference, +1.2 points. Fixed split, 9,941 and 10,038 emails: +1.12 points (+0.52 to +1.71). Thompson sampling, 34,161 and 497 emails: +2.17 points (+0.69 to +3.65).

Thompson sampling stopped sending line A early. So it knows little about A. The difference C − A is +2.17 points, against a true +1.2. Its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI is wide. A bandit is good at finding the best arm. It is bad at saying how much better the best arm is.

This is a subtle problem. A line that is unlucky early gets fewer emails, so its bad luck is never corrected. A line that is lucky early gets more emails, and its luck is corrected. Over the 500 repeats, line A’s observed rate was 3.64% on average under Thompson sampling, against a true 4.0%. Under the fixed split, it was 4.00%.

So the intervals of a bandit are not as honest as they look. Over the 500 repeats, the 95% interval of C − A contained the truth in 89.6% of runs with Thompson sampling, and 94.4% with the fixed split.

When nothing differs, a bandit still picks favourites

Section titled “When nothing differs, a bandit still picks favourites”

The worst case is four lines that are all the same. In 500 simulated campaigns with four lines at 5%, we compared the line with the most emails and the line with the fewest. Under the fixed split, the interval excluded zero in 5.0% of campaigns: the false-positive rate of a normal test. Under Thompson sampling, it excluded zero in 31.4%. A bandit’s numbers can show a “clear” difference that does not exist. You will see one in the Your turn.

  • A bandit optimises one metric, and it needs it fast. Clicks come in minutes. Orders, returns and complaints come in days or weeks: a bandit cannot wait for them.
  • A bandit has no guardrail. It sends more traffic to whatever wins on its one metric, even if that line causes more unsubscribes.
  • A bandit assumes the rates stay the same. If a line works better on Mondays, the bandit’s early choices can be wrong.
Question Use an A/B test Use a bandit
What do you want? To know how big the effect is, with an interval To get the most clicks now
How long does the decision last? Long: a price rule, a page design Short: one email, a headline for one day
When does the outcome come? Any time, also weeks later Within minutes or hours
Guardrails? Yes, several Only if you add them yourself
Many options? Few Many, such as 10 headlines

For search results, a third way exists: interleaving mixes two rankers in one list, as When units affect each other showed. See Interleaving in Search and Ranking.

What a bandit costs: some code, a metric that comes in fast, and the loss of a clean comparison. What it saves: the traffic that a fixed split sends to the losing options.

Guided practice: Thompson sampling in a notebook

Section titled “Guided practice: Thompson sampling in a notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. In section 1, read the four true click rates and the thompson function. It has six lines.
  3. In section 2, run 40,000 emails with a fixed split and with Thompson sampling. You see the clicks and the emails per line.
  4. In section 3, plot the share of emails that line C gets in each tenth.
  5. In section 4, compute the expected clicks lost by Thompson sampling against always line C. Put the value in lost_thompson, and run the check cell. It prints Check passed.
Show the answer to step 5
lost_thompson = round(float(rates.max() * EMAILS - (shown_t * rates).sum()), 1)
print(lost_thompson)

The expected clicks lost are the clicks of always line C, minus the clicks that each line’s emails are expected to get. The value is 97.6 clicks, against 260.7 for the fixed split. The notebook sends one email at a time, with other random numbers than xp bandit, so its numbers differ a little.

Predict first. If all four lines have the same true click rate, 5%, how will Thompson sampling share the emails? And what will the comparison of two lines show? Then run:

Terminal window
xp bandit --rates 0.05,0.05,0.05,0.05
Show the answer
Subject lines A 5.0%, B 5.0%, C 5.0%, D 5.0% (true click rates, simulated); 40,000 emails.
fixed equal split (A/B/n test): 1,999 clicks; expected clicks lost against always the best: -0
traffic share by tenth of the emails, best line A: 26% 25% 25% 25% 24% 25% 26% 25% 25% 24%
chance each line is best: A 0.37, B 0.06, C 0.47, D 0.09
A − B: +0.0025 (95% interval -0.0035 to +0.0086; true +0.0000); emails: 10,038 and 9,878
Thompson sampling: 1,945 clicks; expected clicks lost against always the best: 0
traffic share by tenth of the emails, best line A: 3% 0% 0% 0% 0% 1% 1% 1% 0% 0%
chance each line is best: A 0.00, B 0.27, C 0.65, D 0.08
A − B: -0.0300 (95% interval -0.0453 to -0.0146; true +0.0000); emails: 323 and 11,128

With equal lines, no line is “best”, so the tool labels line A. Nothing is lost either way (the “-0” is a rounding of zero).

Thompson sampling did not share the emails equally. Line A got only 323 emails, after an unlucky start, and B got 11,128. The comparison A − B is −3.00 points, with an interval from −4.53 to −1.46: a “clear” difference. The truth is zero. This is the false difference of the 31.4% above. With the fixed split, the interval contains zero. Never report a bandit’s comparison as if it came from an A/B test.

Common mistake

“A bandit gives the same evidence, faster”

Symptom: a team runs a bandit on four headlines for a week. They report: “Headline C beats headline A by 2.2 points (95% interval 0.7 to 3.7)”. The next A/B test of the same headlines finds 1.2 points.

Cause: the bandit sent almost no traffic to A after the first days. A’s estimate stayed low after its unlucky start, and its interval is wide and not honest. A bandit finds the best arm; it does not measure the differences between arms.

Fix: use a bandit to earn clicks, and report only which arm it chose. Do you need the size of the effect, for a decision that lasts? Then run an A/B test with a fixed split, or keep a fixed share of traffic for each arm.

Check your understanding

Completion check. Choose an A/B test or a bandit for each situation, with your reasons. (1) Larkfield's Black Friday email: one send of 200,000 emails, five subject lines, clicks within hours. (2) Hana's new free-delivery threshold: a rule that will stay for a year, measured on revenue and returns over four weeks.

Show the answer

(1) A bandit. The email is sent once, so the test is the whole campaign, and every click lost to a weak line is lost for good. Clicks come within hours, so the bandit can learn fast. Larkfield needs the best line, not the exact size of each difference. (2) An A/B test with a fixed split. The decision lasts a year, so Hana needs the size of the effect with an interval, to compare it with the cost. Revenue and returns come over weeks, too slowly for a bandit, and the return rate is a guardrail that a bandit would ignore.

Check your understanding

In the Larkfield run, Thompson sampling estimates line C − line A at +2.17 points, with a 95% interval from +0.69 to +3.65. The truth is +1.2. Explain in two or three sentences why a bandit's comparison of the best and the worst line is weak.

Show the answer

The bandit stopped sending line A early. So A got only 497 emails, its rate is measured badly, and the interval is wide. Also, an arm that is unlucky early gets fewer emails, so its low estimate stays low. Over 500 repeats, A’s observed rate was 3.64% against a true 4.0%. So the difference looks larger than it is, and the interval contains the truth less often than 95%.

Check your understanding

Explore or exploit? A bandit has sent 10,000 emails with line C (520 clicks) and 300 emails with line D (12 clicks). Why does Thompson sampling still sometimes send line D?

Show the answer

Line D has few emails, so its range of believable click rates is wide. 12 clicks of 300 is 4%, but the true rate could well be 5% or more. Sometimes a random draw from D’s wide range beats the draw from C’s narrow range near 5.2%. Then D gets the email. This is exploring: it checks that D is not better than it looks. As D gets more emails, its range narrows, and it is sent less often.

Recap

  • A bandit explores and exploits: it moves traffic to the best option while it learns. Thompson sampling draws a believable rate for each arm and sends the arm with the highest draw.
  • On four subject lines (simulated), it lost 80 clicks against 260 for a fixed split, and found the best line as often.
  • It is weak evidence for the size of differences. C − A: +2.17 points against a true +1.2. Interval coverage: 89.6%. A false “clear” difference: 31.4% of tests with equal lines.
  • Use a bandit for short decisions with fast outcomes. Use an A/B test when you need the size of the effect, guardrails or a decision that lasts.

Next, we write the result of a test so that people trust it and act on it.

Next: Report a result people trust