Predicting continuations

What you will be able to do

Understand how a model writes: one token at a time, each one chosen from a list of probabilities. Then you can explain why the same ticket can get two different replies, and set the one control that changes this.

NotebookTemperature on real probabilities, in a notebook (optional)Colab · Kaggle · GitHub

The notebook uses the recorded probabilities of this lesson. It re-weights them with the temperature and draws samples, so you can check the explorer’s numbers. It needs no model and no API key.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Tokens, the vocabulary and token IDs, from From text to tokens.
• A probability as a number from 0 to 1, from Events and uncertainty, and a distribution as every value with its chance, from Distributions in practice. Both are in Essential Mathematics and Statistics for AI.

What you need

• A browser. The optional notebook opens in Colab or Kaggle and needs no API key.

Hosted models change often: your own results will differ.

Your phone’s next word

Type “See you” on a phone, and the keyboard suggests three words: “tomorrow”, “soon”, “later”. It does not know what you mean. It knows what people often type after “See you”, and it shows the most likely words first. Tap one, and it suggests the next three.

A language model does the same thing, at a much larger scale. It has read a huge amount of text. Given the text so far, it says how likely each possible next token is. Then one token is chosen, added to the text, and the model is asked again.

The problem: there is no single right next word

Read this unfinished sentence: “The parcel arrived late, so the customer asked for a …”. What comes next? “Refund” is the obvious word. But “discount”, “replacement” and “full refund” are all good English and all possible. A program that stores one right answer for each sentence cannot handle this. The model needs a way to say “refund is very likely, discount is possible, banana is almost impossible”.

The idea: a probability for every token

A probability is a number from 0 to 1 that says how likely something is. The probabilities of all the possible outcomes add up to 1. For each step, the model gives a probability to every token in its vocabulary. Together they form a distribution over the vocabulary.

Start with a tiny vocabulary, so you can see every number. Pretend that the model knows only six things to say next: five words and “anything else”. These are the real probabilities that Gemma 3 4B gave for the sentence above. Gemma 3 is an open model; the course runs it on a small computer with Ollama, a free tool for local models.

Next token	Probability
·refund	91.3%
·full	3.4%
·discount	1.9%
·partial	1.8%
·return	0.5%
anything else	1.1%

The middle dot marks the space at the start of each token. The five words take 98.9% of the probability; the other 262,139 tokens of Gemma’s vocabulary share the last 1.1%. The model is very sure, but not certain.

Now the whole trip, from the text to the next word. Read it from 1 to 7.

From text to a new token, and around again. The numbers are Gemma 3 4B's real token IDs and probabilities for this sentence, from Ollama.

Text description of the diagram

Seven steps in a loop. Step 1, the text: “The parcel arrived late, so the customer asked for a”. Step 2, the tokenizer cuts it into eleven tokens: The, parcel, arrived, late, a comma, so, the, customer, asked, for, a. Step 3, the token IDs: 669, 38396, 12208, 5226, 236764, 834, 506, 5602, 4733, 573 and 496. Step 4, the model, Gemma 3 4B, computes one score for each of the 262,144 tokens of its vocabulary. Step 5, the scores become probabilities: refund 91.3%, full 3.4%, discount 1.9%, partial 1.8%, return 0.5%; the other 262,139 tokens share the last 1.1%. Step 6, one token is chosen: refund, token ID 19527, the most likely one at temperature 0. Step 7, the token is added to the text: “…asked for a refund”. A dashed arrow goes back from step 7 to step 1: the new text goes in again, for one more token.

In step 6, the most likely token wins. That is one way to choose; the sections on sampling and temperature below show the others. The model never writes a whole answer at once. It writes one token, and the next token depends on everything before it, including the tokens it wrote itself. This loop, one token after another, is called generation. Using a trained model in this way, to get outputs, is called inference. Training is how the model learned its probabilities; inference is every time you use it. Lesson 3 of this module covers training.

The loop, step by step

Here are the first ten steps of the loop for the same sentence. At each step, the model takes its most likely token. The table shows the three most likely tokens and the one that was chosen.

Step	Text so far (end)	Three most likely next tokens	Chosen
1	…o the customer asked for a	·refund 91.3%, ·full 3.4%, ·discount 1.9%	·refund
2	…ustomer asked for a refund	. 83.5%, ·or 4.7%, , 3.4%	.
3	…stomer asked for a refund.	·The 33.4%, (new paragraph) 25.9%, (new line) 20.0%	·The
4	…er asked for a refund. The	·company 37.2%, ·store 13.9%, ·customer 7.9%	·company
5	… for a refund. The company	·decided 11.6%, ' 10.8%, ·agreed 7.8%	·decided
6	…efund. The company decided	·to 96.4%, ·not 1.9%, ·that 0.9%	·to
7	…nd. The company decided to	·offer 32.9%, ·give 20.7%, ·honor 14.5%	·offer
8	…e company decided to offer	·a 75.7%, ·the 15.2%, ·him 4.2%	·a
9	…company decided to offer a	·partial 79.3%, ·discount 13.4%, · 4.3%	·partial
10	…decided to offer a partial	·refund 99.8%, ·reimbursement 0.1%, ·one <0.1%	·refund

Three things to see. First, the model is not always as sure as at step 1. At step 3 it gives “·The” only 33.4%, and a new paragraph 25.9%: many continuations are good. Second, each chosen token changes the next list. After “·offer a”, the model gives “·partial” 79.3%, and then “·refund” 99.8%. Third, nothing in the text says that the company offered a partial refund. The model wrote a likely story, not a fact. It continues text; it does not look anything up.

How to choose the token: greedy or sampling

The table took the most likely token every time. That is greedy choice. It is simple, and the same text always gives the same continuation. But it is not the only way. Always taking the most likely word can also make text dull or repetitive.

The other way is sampling: choose the token at random, with the model’s probabilities as the chances. Think of a spinner with one slice for each token, and the size of each slice is its probability. “·refund” gets 91.3% of the circle, so it wins most spins, but “·discount” wins about 2 spins in 100. Sampling makes the text more varied, and it also means that the same prompt can give a different answer each time.

Amira asked Gemma 3 to continue the same sentence seven times, 12 tokens each. Twice with greedy choice, and five times with sampling. A seed is a number that starts the random choices; the same seed gives the same choices.

Choice	Seed	Continuation
greedy	1	refund. The company decided to offer a partial refund instead of
greedy	2	refund. The company decided to offer a partial refund instead of
sampling, temperature 1	1	free repair. After reviewing the damage, our repair technician determined
sampling, temperature 1	2	refund?</h1>\n\nHere are some ways to respond to a
sampling, temperature 1	3	refund. The store owner, Mr. Smith, decided to
sampling, temperature 1	4	refund. \nThe customer was dissatisfied with the product,
sampling, temperature 1	5	refund.\n\nLet's break down how this situation is

The two greedy runs are the same, word for word. The five sampled runs all differ. Seed 1 chose “·free”, a token with a probability of only 0.05%. Seed 2 wrote “refund?” and then an HTML tag, </h1>. The model has read many web pages, and their code is part of what it learned.

Temperature: how adventurous the choice is

One setting controls how bold the sampling is: the temperature. The name comes from physics, where a hotter system moves more at random. At a low temperature, the most likely token gets an even bigger share, so the choice is almost always the same. At a high temperature, the shares become more even, so less likely tokens win more often. At temperature 0, the choice is greedy.

Predict first, then move the control. The explorer uses Gemma’s real probabilities for its 20 most likely tokens, and shares 100% among those 20 only. So at temperature 1, “·refund” has 91.7%, a little more than the 91.3% in the table above.

Explore

Re-weight the next token with the temperature

Text so far: The parcel arrived late, so the customer asked for a▢

Chance of the top token

91.7%

Chance of any other token

8.3%

Tokens with 1% or more

4

·refund91.7%

·full3.4%

·discount1.9%

·partial1.8%

·return0.5%

the other 150.6%

The thin dark mark on each bar is the share at temperature 1, the model's own probabilities. The last bar is the other 15 of the 20 most likely tokens together.

At temperature 1.0, "·refund" has a 91.7% chance, so another word comes about 1 run in 12. At 1, these are the model's own probabilities.

Gemma 3 4B's real probabilities for the next token (Ollama 0.40.1), shared among its 20 most likely tokens and re-weighted by the temperature.

Text description of the explorer

The explorer shows the text “The parcel arrived late, so the customer asked for a”. Below it are bars for the five most likely next tokens, and one bar for the other 15 of the 20 most likely tokens. One slider sets the temperature, from 0 to 2 in steps of 0.1. A thin mark on each bar shows its share at temperature 1. The readout shows the chance of the top token, the chance of any other token, and how many tokens have a chance of 1% or more.

At temperature 1: refund 91.7%, full 3.4%, discount 1.9%, partial 1.8%, return 0.5%, the other 15 together 0.6%; 4 tokens have 1% or more. At 0.5: refund 99.8%. At 0.7: refund 98.3%. At 1.5: refund 72.8%, full 8.1%, discount 5.6%, partial 5.3%, return 2.3%, the other 15 together 6.0%. At 2: refund 54.2%, full 10.4%, discount 7.9%, partial 7.6%, return 4.1%, the other 15 together 15.9%; 12 tokens have 1% or more. At 0, refund has 100%: the choice is greedy.

At temperature 2, “·refund” falls from 91.7% to 54.2%: almost half of the runs now start with another word. At 0.5, it rises to 99.8%. The sampler in these runs also kept only the 20 most likely tokens. Most samplers keep only the top tokens in this way, so that a high temperature cannot pick a nonsense token from the long tail.

The rule, on two tokens

Here is what the temperature does, on the tiniest vocabulary: only “·refund” and “·full”. Their probabilities are 0.913 and 0.034, so “·refund” is 0.913 ÷ 0.034 ≈ 27 times as likely. With only these two, the shares are 27 ÷ 28 = 96% and 1 ÷ 28 = 4%.

At temperature 2, take the square root of that ratio: √27 ≈ 5.2. Now “·refund” is only 5.2 times as likely, and the shares are 5.2 ÷ 6.2 = 84% and 1 ÷ 6.2 = 16%. At temperature 0.5, square the ratio instead: 27² = 729, so “·refund” gets 99.9%. In general, the temperature T raises every probability to the power 1/T. Then the shares are scaled again, so that they add up to 1:

$$p_i(T) = \frac{p_i^{1/T}}{\sum_j p_j^{1/T}}$$

Read it as: the new share of token i at temperature T is its probability to the power 1/T, divided by the same for all the tokens. T = 1 changes nothing. T below 1 sharpens the distribution; T above 1 flattens it.

Is that really what the sampler does? Amira checked. She drew 200 one-token samples at temperature 2, each with a different seed, and counted the tokens. The formula expected 108 “·refund”, 21 “·full”, 16 “·discount”, 15 “·partial” and 8 “·return”. She counted 103, 16, 23, 18, 9. With 200 draws, differences of a few counts are normal, so the sampler follows the formula.

The running example: one ticket, five replies

Labels and replies for Larkfield come out of the same loop. In the recorded runs of this course, Amira gave Gemma 3 the assistant’s instructions, the support policy and ticket S01: “My order L-20431 was supposed to arrive last week. It is now 7 business days late.” She ran it five times at temperature 0.8, with seeds 0 to 4. The policy says: more than 5 business days late, free shipping; more than 15, treated as lost.

The labels were the same in all five runs: English, delivery, no person needed. The replies were not. Seed 2 wrote the right answer: “As the order is now 7 business days late, we can offer you free shipping on the order.” The other four seeds wrote, for example: “According to our policy, orders over 15 business days late are considered lost, and we will process a replacement or full refund.” That rule is real, but it does not apply to a 7-day delay. A different random choice early in the reply led to a different, wrong reply.

Omar asks: “So we set the temperature to 0, and the problem is gone?” Partly. At temperature 0, the same model gives the same reply to the same text, so you can test it and repeat the test. But a wrong reply is then wrong every time. And some models do not let you choose: Claude models, for example, reject a temperature setting today. So their replies can differ between runs. Module 2 shows more ways that replies change.

When to use which temperature

• Low (0 to 0.3) for labels, extraction, JSON and anything that a program reads. You want the same answer to the same input, and a test that you can repeat.
• Medium (about 0.7 to 1) for drafts that a person reads and edits, such as reply drafts, when some variety in wording helps.
• High (above 1) for brainstorming many different ideas. Expect more mistakes and more strange tokens.

What it costs: a higher temperature costs no money and no time, but it costs predictability. A lower temperature makes the output repeatable, but it does not make it correct.

Worked example: Amira reads the probabilities

You do not need to run this. Amira asked Ollama, on her own computer, for the five most likely next tokens. "raw": true sends the text as it is, and "logprobs": true asks for the log-probabilities: the natural logarithm of each probability, which math.exp turns back into a probability.

next_token.py

import math
import requests

body = {
    "model": "gemma3:4b",
    "prompt": "The parcel arrived late, so the customer asked for a",
    "raw": True, "stream": False,
    "logprobs": True, "top_logprobs": 5,
    "options": {"temperature": 0, "num_predict": 1},
}
answer = requests.post("http://localhost:11434/api/generate", json=body).json()
for item in answer["logprobs"][0]["top_logprobs"]:
    print(f"{item['token']!r:12} {math.exp(item['logprob']):.3f}")

The output:

' refund'    0.913
' full'      0.034
' discount'  0.019
' partial'   0.018
' return'    0.005

These are the numbers of the tiny vocabulary at the start of this lesson. The notebook does the same steps with the recorded numbers, so you do not need Ollama.

Guided practice: read the explorer

1. Make your prediction in the explorer, then select I have made my prediction.
2. Move the temperature to 1.0. You should see “·refund” at 91.7% and 4 tokens with 1% or more.
3. Move it to 2.0. You should see “·refund” at 54.2% and 12 tokens with 1% or more.
4. Move it to 0.5. You should see “·refund” at 99.8%.
5. Move it to 0. You should see “·refund” at 100%: greedy choice.
6. Write one sentence for Grace: which temperature should the assistant use for the labels language, category and needs_human, and why?

Show an example answer

“Use temperature 0, or as low as the model allows, for the labels. Then the same ticket gets the same labels every time, and we can test them and repeat the test. A low temperature does not make a label right; only a test on real tickets shows that.”

Your turn: a French sentence

Here is the same sentence in French: “Le colis est arrivé en retard, alors le client a demandé un …”. Before you move the control, predict: is Gemma 3 more sure or less sure of its first choice in French than in English? At which temperature does the top token fall below 90%?

Explore

The same idea in French

Text so far: Le colis est arrivé en retard, alors le client a demandé un▢

Chance of the top token

99.4%

Chance of any other token

0.6%

Tokens with 1% or more

1

·remboursement99.4%

·retour0.3%

·remplacement<0.1%

·dé<0.1%

·reembolso<0.1%

the other 150.2%

The thin dark mark on each bar is the share at temperature 1, the model's own probabilities. The last bar is the other 15 of the 20 most likely tokens together.

At temperature 1.0, "·remboursement" has a 99.4% chance, so another word comes about 1 run in 156. At 1, these are the model's own probabilities.

Gemma 3 4B's real probabilities for the next token of the French sentence (Ollama 0.40.1), shared among its 20 most likely tokens.

Text description of the explorer

The same explorer for the French sentence “Le colis est arrivé en retard, alors le client a demandé un”. At temperature 1: remboursement 99.4%, retour 0.3%, remplacement 0.1%, the other tokens less. At 1.5: remboursement 93.2%. At 2: remboursement 78.9%, retour 4.7%, remplacement 1.8%; 9 tokens have 1% or more.

Show the answer

Gemma 3 is even more sure in French: “·remboursement” (refund) has 99.4% at temperature 1, against 91.7% for “·refund” in English. It stays above 90% up to temperature 1.5 (93.2%) and falls to 78.9% at 2. In French, “un remboursement” is the usual phrase after “a demandé un”, while English has “a refund”, “a full refund”, “a discount” and more. How sure a model is depends on the text, not only on the language.

Common mistake

One good reply taken as proof

Symptom: Tomás tries the assistant on ticket S01 once and reads a correct reply: free shipping for a 7-day delay. He tells Grace, “It handles late orders.” In the first week, customers with 7-day delays are told that their order counts as lost.

Cause: the reply is one sample from a distribution. In the recorded runs at temperature 0.8, Gemma 3 gave the correct reply to S01 in 1 run of 5. Tomás saw one lucky draw.

Fix: for anything you will judge, test many tickets, not one, at the lowest temperature the model allows, and record every reply. If a reply can still change between runs, run a few tickets several times. Judge the share of good replies, not one reply. Module 4 builds such a test: 24 tickets, each sent once to each model, and every reply read.

What it means for a real AI service: a demo shows what a model can say, not what it usually says. Decisions about a model need many runs on many inputs, read the same way each time.

Knowledge check

Check your understanding

In your own words: what does a language model produce at each step of generation, and how does a whole reply come out of it?

Show the answer

At each step, the model produces a probability for every token in its vocabulary. It says how likely each token is to come next, given all the text so far. One token is chosen (the most likely one, or a random draw with those probabilities), added to the text, and the model runs again. A reply is this loop repeated, one token at a time, until the model chooses an end token or reaches a limit.

Check your understanding

Predict: two tokens have probabilities 0.6 and 0.3, and the rest is tiny. At temperature 0.5, does the gap between them grow or shrink? At temperature 2?

Show the answer

At temperature 0.5, the ratio is squared: 0.6 ÷ 0.3 = 2 becomes 4, so the first token gets a bigger share (about 80% of the two together, from 67%). The gap grows. At temperature 2, take the square root: 2 becomes about 1.41, so the first token gets about 59% of the two. The gap shrinks, and the second token wins more often.

Check your understanding

Omar says: 'At temperature 0 the assistant is always right, because it always takes the most likely word.' What is wrong with this?

Show the answer

Temperature 0 makes the reply the same every time, not right. The most likely token is the one that fits the text best, according to what the model learned. That can still lead to a wrong fact or a wrong rule. A wrong reply at temperature 0 is wrong every time. Only a test on real tickets shows how often the replies are right.

Recap

• At each step, a language model gives every token in its vocabulary a probability. One token is chosen, added to the text, and the loop repeats: this is generation. Using a trained model in this way is inference.
• Greedy choice takes the most likely token: the same text gives the same continuation. Sampling draws a token with the model’s probabilities as chances, so the same prompt can give different replies.
• The temperature re-weights the probabilities: below 1 the top token gets more, above 1 the shares become more even, and 0 is greedy. For Gemma 3’s “·refund”: 99.8% at 0.5, 91.7% at 1, 54.2% at 2 (among the 20 most likely tokens).
• One reply is one draw. Judge a model on many runs, and remember that a repeatable reply is not a correct one.

Next, see where the probabilities come from: the stages of training that turn a text predictor into an assistant.

Next: Training stages