From text to tokens

What you will be able to do

See what a language model really reads: not words, but tokens. Then you can explain why the same ticket costs more, fills more room and takes longer in one language than in another.

NotebookCount tokens in a notebook (optional)Colab · Kaggle · GitHub

The notebook cuts the tickets of this course into tokens with the same two tokenizers as the explorer, and counts them. You do not need it to follow the lesson.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• How to read a count and a ratio (“1.3 times as many”), from Describe a dataset in Essential Mathematics and Statistics for AI.
• Nothing about language models. This is the first lesson of the course.

What you need

• A browser. The optional notebook opens in Colab or Kaggle and needs no API key. (An API is the way one program asks another program for something, such as an answer from a hosted model. An API key is the secret code that a provider gives you so that it can identify you and bill you.)

Hosted models change often: your own results will differ.

Reading a long word in pieces

A child who is learning to read meets the word “unbreakable” for the first time. She does not know it as one word. So she reads it in pieces that she knows: “un”, “break”, “able”. Each piece is familiar from other words, so the new word is not a problem.

An adult does the same with a long word in a foreign language. A language model reads every text in this way: in pieces that it has seen many times.

The problem: a model computes only with numbers

Larkfield, the home-and-garden shop, now sells in Quebec. Tickets arrive in English and in French. Grace, the support lead, wants an assistant that reads each ticket and drafts a reply. The model behind the assistant is a large mathematical function: numbers go in, and numbers come out. So the first step must turn the customer’s text into a list of numbers.

First idea: one number for each word. Make a list of all the words, and give each word a number. This fails quickly. Tickets contain order numbers such as L-20431, names such as Larkfield, and typing errors such as “arived”. French adds many forms of one word: cassé, cassée, cassés, cassées. A word that is not on the list cannot be read at all. And a list for English and French is not enough for a shop that later sells in Mexico.

Second idea: one number for each letter. Now nothing is unknown, because every text is made of letters. But the list of numbers becomes very long. Larkfield’s support sentence below has 90 characters, so it becomes 90 numbers. The model does a lot of work for each number that it reads or writes. A longer list is slower and costs more, and it uses more of the limited room that the model can read at once.

The idea: pieces that appear often

The answer is in the middle, as with the child’s pieces. A common word is one piece: “ refund”, “ order”, “ broken”. A rare word is cut into common pieces: “ Larkfield” becomes “ L” + “ark” + “field”. Nothing is unknown, because the smallest pieces are single letters, or even single bytes. And common text stays short.

Each piece is a token. The fixed list of all the pieces that a model knows is its vocabulary. Each token has a number, its position in the vocabulary: the token ID. Cutting a text into tokens is tokenization, and the program that does it is the tokenizer. Note that a token often starts with a space: “ refund” with its space is a different token from “refund” at the start of a line.

How does a tokenizer choose its pieces? Most tokenizers use a method called byte-pair encoding (BPE). It starts with single bytes: an English letter is one byte, and a letter such as é or a Hindi character is two or three. It then looks at a very large collection of text. It finds the pair of neighbouring pieces that appears most often, for example “t” and “h”. It joins them into a new piece, “th”, and adds it to the vocabulary. It repeats this, often 100,000 times or more. The name says what it does: it joins pairs, again and again.

This has a consequence that matters for Larkfield. The pieces come from the collection of text. A language with a lot of text in the collection gets many whole-word tokens. A language with less text stays in smaller pieces, so the same sentence needs more tokens.

A tiny example: eight words, two tokenizers

OpenAI publishes two of its tokenizers in an open-source library, tiktoken, so you can see every piece. o200k_base has 200,019 tokens and is used by newer models such as GPT-4o. cl100k_base has 100,277 tokens and is used by older models such as GPT-4. Other providers, such as Anthropic and Google, do not publish their tokenizers as a library, but they report the number of tokens of every request. Here are eight words, each with a space before it, as in a sentence. A vertical bar separates the tokens.

Word	o200k_base	cl100k_base
refund	refund (1)	refund (1)
remboursement	remboursement (1)	rem | b | ourse | ment (4)
ceramic	ceramic (1)	ceramic (1)
céramique	c | ér | amique (3)	cé | ram | ique (3)
broken	broken (1)	broken (1)
cassé	cass | é (2)	cass | é (2)
Larkfield	L | ark | field (3)	L | ark | field (3)
L-20431	L | - | 204 | 31 (4)	L | - | 204 | 31 (4)

The English words are one token each. The French words are often cut, and more so by the older tokenizer: “ remboursement” (refund) is one token in o200k_base but four in cl100k_base. Numbers are cut in groups of up to three digits.

The IDs belong to one vocabulary only. “ refund” is token ID 18376 in o200k_base and 21639 in cl100k_base. A model learns with one tokenizer and must always use the same one: the IDs of another tokenizer mean nothing to it.

Larkfield’s sentence in eight languages

Camille, Larkfield’s new bilingual support agent in Montreal, reads the French tickets. Camille asks a fair question: “Does a French ticket cost the same as an English one?” To find out, take one support sentence and its translations. The sentence and every ticket in this course are synthetic: written for the course, not taken from real customers. The translations other than French are machine-assisted and checked for French only.

Explore

One support sentence, eight languages, two tokenizers

Tokens

20

English, same tokenizer

20

Compared with English

1.00 ×

Characters

90

English20

English: 20 tokens with o200k_base. This is the reference for the other languages.

• one token (· marks a space; the number is its token ID)
• raw bytes: part of one character

My order arrived late and the ceramic pot was broken. Can I get a replacement or a refund?

1. My5444
2. ·order2569
3. ·arrived18157
4. ·late7844
5. ·and326
6. ·the290
7. ·ceramic48698
8. ·pot3208
9. ·was673
10. ·broken17162
11. .13
12. ·Can4101
13. ·I357
14. ·get717
15. ·a261
16. ·replacement19207
17. ·or503
18. ·a261
19. ·refund18376
20. ?30

The real tokens of one support sentence in eight languages, from OpenAI's tiktoken library (version 0.14.0). Each chip is one token, with its token ID under it.

Text description of the explorer

The explorer shows the sentence “My order arrived late and the ceramic pot was broken. Can I get a replacement or a refund?” and its translations, cut into tokens. Each token is a chip with its token ID; a middle dot marks a space. One control chooses the language (English, French, Spanish, German, Urdu, Arabic, Hindi, Chinese) and another the tokenizer (o200k_base or cl100k_base). The readout shows the count, the English count with the same tokenizer and the ratio.

With o200k_base: English 20 tokens, French 26 (1.30 times English), Spanish 26, German 28, Urdu 36, Arabic 32, Hindi 36 (1.80 times) and Chinese 21. With cl100k_base: English 20, French 34 (1.70 times), Spanish 33, German 36, Urdu 91, Arabic 66, Hindi 111 (5.55 times) and Chinese 31. Some cl100k_base tokens are raw bytes: parts of one character, shown in hexadecimal on a dashed chip. They are 58 of the 111 Hindi tokens, 30 of the 91 Urdu tokens, 8 of the 66 Arabic tokens and 15 of the 31 Chinese tokens. In French, both tokenizers cut “ céramique” and “ cassé” into pieces.

The chart shows the same counts for all eight languages, with a third tokenizer: the one of Gemma 3 4B. Gemma 3 4B is an open model that this course runs on a small computer: anyone can download it and run it on their own machine. (Module 3 calls this open weights.)

• o200k_base
• cl100k_base
• Gemma 3 4B

Tokens for the same support sentence in eight languages, with three tokenizers. The dashed line is the English count, 20 tokens. Gemma 3 counts from Ollama 0.40.1.

Text description of the chart

A bar chart with one row per language and three bars in each row: o200k_base, cl100k_base and Gemma 3 4B. A dashed vertical line marks 20 tokens, the English count. English: 20, 20, 20. French: 26, 34, 26. Spanish: 26, 33, 24. German: 28, 36, 28. Urdu: 36, 91, 31. Arabic: 32, 66, 32. Hindi: 36, 111, 27. Chinese: 21, 31, 20. The cl100k_base bars are the longest in every language except English, and much longer for Urdu, Arabic and Hindi.

Four things to see:

1. The same meaning, different counts. With o200k_base, the French sentence needs 26 tokens: 1.3 times the English one. Hindi and Urdu need 36: 1.8 times.
2. The tokenizer matters as much as the language. The older cl100k_base needs 34 tokens for French and 111 for Hindi, 5.5 times English. It has no piece for most Hindi characters, so it spells them byte by byte.
3. Each model family has its own tokenizer. Gemma 3 has a vocabulary of 262,144 tokens, with much text in many languages. It needs only 27 tokens for Hindi. Mistral 7B, another open model, has a vocabulary of 32,768 tokens and needs 38 for French and 121 for Hindi.
4. Characters do not predict tokens. The Chinese sentence has only 26 characters, but 21 tokens with o200k_base: most Chinese tokens are one or two characters.

Why the count matters

Three things are measured in tokens, so a language that needs more tokens pays three times:

• Cost. Hosted models charge for each token that goes in and each token that comes out. The same French ticket costs more than its English version.
• Room. A model can read only a fixed number of tokens at once. Module 2 calls this the context window. The support policy and a long French ticket fill it faster than their English versions.
• Time. A model writes its answer one token at a time, as Predicting continuations shows. More tokens in the reply mean a longer wait for the customer.

Omar, the product manager, asks the cost question. “Provider A charges less per token than provider B. So A is cheaper?” Not necessarily. Amira sent the same request to several models: the assistant’s instructions, the policy and one ticket. Claude counted 796 input tokens, Gemini 560, and the local models 549 to 609. A low price for each token can come with more tokens for each ticket. So compare the cost per ticket: tokens per ticket times the price per token. Module 4 measures it for every model.

For a first estimate, Larkfield’s English text has about 1.3 tokens per word: the 275 words of the support policy are 349 tokens with o200k_base. The French tickets have about 1.5. Use such a rule only for a rough estimate; for a budget, count with the tokenizer of the model you will use.

Worked example: Amira counts two tickets

Amira, Larkfield’s developer, compares two tickets about late orders: S01 in English and S07 in French. You do not need to run this code; it is here so that you see the steps. (It is also in the notebook.)

count_tokens.py

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tickets = {
    "S01 (English)": "My order L-20431 was supposed to arrive last week. It is now 7 business days late. What can you do?",
    "S07 (French)": "Bonjour, ma commande L-20877 devait arriver il y a 8 jours ouvrables et je n'ai toujours rien reçu. Que pouvez-vous faire?",
}
for name, text in tickets.items():
    ids = enc.encode(text)
    print(f"{name}: {len(text.split())} words, {len(ids)} tokens")

print(ids[:8])
print([enc.decode([i]) for i in ids[:8]])

encode turns the text into token IDs, and decode turns one ID back into its piece. The output:

S01 (English): 20 words, 27 tokens
S07 (French): 21 words, 31 tokens
[45751, 11, 831, 53567, 451, 12, 20021, 3732]
['Bonjour', ',', ' ma', ' commande', ' L', '-', '208', '77']

The two tickets have almost the same number of words, but the French one has four more tokens. The last two lines show what the model receives: not “Bonjour, ma commande”, but the numbers 45751, 11, 831 and 53567. The order number is four tokens, cut into “ L”, “-”, “208” and “77”.

Guided practice: read the explorer

Use the explorer above. Make your prediction first, then follow the steps.

1. Select English and o200k_base. You should see 20 tokens. Every word is one token, and the full stop and the question mark are tokens too.
2. Select French. You should see 26 tokens, 1.30 times English. Find the chips for “ c”, “ér” and “amique”.
3. Keep French and select cl100k_base. You should see 34 tokens. Find “ remboursement”: it is now four chips.
4. Select Hindi. You should see 111 tokens, 5.55 times English. Most chips are dashed: raw bytes, parts of one character.
5. Write one sentence for Camille that answers the question “Does a French ticket cost the same as an English one?”

Show an example answer

“No. With a recent tokenizer, the same French sentence needs about 1.3 times the tokens of the English one. With an older one, it needs 1.7 times. So a French ticket costs more, fills more of the model’s room and takes a little longer. The difference depends on the model, so we count tokens with the tokenizer of the model we choose.”

Your turn: a language with long words

German joins words into long compounds. “Ceramic pot” is one word, Keramiktopf. Before you look, predict: with o200k_base, does the German sentence need more or fewer tokens than the French one (26)? Then check in the explorer, and find the chips of Keramiktopf.

Show the answer

German needs 28 tokens with o200k_base, two more than French, and 36 with cl100k_base. Keramiktopf is not in the vocabulary, so it becomes four tokens: “ Ker”, “amik”, “top” and “f”. A long compound word is rare as a whole, even when its parts are common. So it is cut into pieces, like “ Larkfield”.

Common mistake

A budget that counts English words

Symptom: Omar plans the monthly budget for the assistant. He takes ten English tickets, counts about 20 words in each, and plans “20 tokens per ticket”. The first month’s bill is higher than the plan, and French tickets cost the most.

Cause: two mistakes. A word is not a token: S01 has 20 words and 27 tokens with o200k_base. And English is not the whole mix. S07, a French ticket of 21 words, has 31 tokens with o200k_base, and 46 with the tokenizer of Mistral 7B. A plan from English words was too low for every ticket, and much too low for French ones.

Fix: count tokens, not words, with the tokenizer of the model you plan to use. Or read the token counts that the provider reports for each request. Use a sample of tickets in each language, in the same mix as real traffic.

What it means for a real AI service: every budget, limit and time estimate of a language model is in tokens. A team that serves several languages must measure each one. Otherwise the customers who write in one language get a slower and more costly service, and nobody planned it.

Knowledge check

Check your understanding

Why do language models not use one token per word, or one token per letter?

Show the answer

One token per word fails on every word that is not in the list: names, order numbers, typing errors, new words and the many forms of a word in French or German. One token per letter can write anything, but it makes every text very long. The model does work for each token, so a long text is slow and costly. Tokens are the middle way: common words are one token, and rare words are cut into common pieces.

Check your understanding

Predict: a Hindi ticket and an English ticket have the same meaning. Which one uses more tokens? Does the answer depend on the model?

Show the answer

The Hindi ticket almost always uses more tokens, but how many more depends on the tokenizer. For Larkfield’s sentence: 1.8 times English with o200k_base, 5.55 times with cl100k_base, and only 1.35 times with Gemma 3 (27 tokens against 20). A tokenizer learns its pieces from its collection of text. If that text had little Hindi, Hindi stays in small pieces. So measure with the tokenizer of the model you choose.

Check your understanding

The token ' refund' has ID 18376 in o200k_base and 21639 in cl100k_base. A model was trained with o200k_base. What happens if a program sends it the cl100k_base IDs of a ticket?

Show the answer

The model reads the wrong pieces. To it, 21639 is whatever piece has that position in its own vocabulary, not “ refund”. The text becomes nonsense to the model. A token ID means something only inside one vocabulary, so a model always uses the tokenizer it was trained with.

The case studies of this course

The course ends with two case studies. Each one takes a real model decision from the question to the recommendation, with real model runs. You must complete at least one for the certificate.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Choose a model for bilingual government answersCompare hosted and local models on real English and French Government of Canada questions, with a rubric, measured cost and latency, and a decision memo.
• Choose a model that runs on your own hardwareA privacy rule forbids sending the text to a provider: compare open-weight models and quantizations on a laptop, measure memory, speed and quality, and recommend one.

Recap

• A language model reads tokens: common words are one token, and rare words are cut into common pieces. The vocabulary is the fixed list of pieces, and each piece has a token ID.
• Most tokenizers learn their pieces with byte-pair encoding. It joins the most frequent pair of pieces in a large collection of text, again and again.
• The same sentence needs different numbers of tokens in different languages and with different tokenizers. For Larkfield’s sentence: French 26 and Hindi 36 against English 20 with o200k_base; Hindi 111 with cl100k_base.
• Cost, room and time are all measured in tokens. Compare models by the cost per ticket, not by the price per token.

Next, see what the model does with the token IDs: it replaces each one with a list of learned numbers, a vector.

Next: From tokens to vectors