Training stages

What you will be able to do

Know what each stage of training adds to a language model. Then you can say what a model has learned and why two models answer the same question differently. You can also explain why a model is not a database of facts.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Tokens and token counts, from From text to tokens.
• Next-token probabilities, generation and temperature, from Predicting continuations.

What you need

• A browser. This lesson has no notebook: the outputs below are recorded.

Hosted models change often: your own results will differ.

How Camille learned the job

Camille did not start at Larkfield knowing how to answer tickets. First came many years of reading and listening, in French and English: school, books, conversations. That gave Camille the languages and a lot of general knowledge, but not the job.

Then came a few weeks of training with examples: here is a ticket, here is a good reply. Camille learned the shape of a reply, the tone, and when to stop. Finally, Grace read Camille’s drafts and said, again and again, “this reply is better than that one”. Camille’s replies moved toward what Grace and the customers prefer.

A language model is trained in the same three stages. Each stage adds something different, and each one explains a behaviour that you will see in this course.

The problem: a text predictor is not an assistant

The last lesson showed that a model predicts the next token. Is that enough for Larkfield’s assistant? Amira tested it with two versions of one open model, Meta’s Llama 3.2 3B. One version had only the first stage of training; the other had all three. She sent both the same text, at temperature 0, and stopped each answer after 80 tokens (… marks an answer that the limit cut).

Amira sent: Translate into French: Your refund is on its way.

After stage 1 only (base model): We will send you an email when your refund has been processed, including the date the refund will be issued. Translate into French: We will send you an email when your refund has been processed, including the date the refund will be issued. Translate into French: We will send you an email when your refund has been processed, including the date the refund will be issued.

After all three stages: Your refund is on its way. Here's the translation: "Votre remboursement est en route."

Amira sent: My order arrived late and the ceramic pot was broken. Can I get a replacement or a refund?

After stage 1 only (base model): I am very disappointed. I am sorry to hear that your order arrived late and the ceramic pot was broken. We will be happy to replace the broken pot for you. Please contact us at 1-800-227-3528 and we will be happy to assist you.

After all three stages: I'm so sorry to hear that your order arrived late and the ceramic pot was broken. I'd be happy to help you with a replacement or refund. To proceed, could you please provide me with the following information: 1. Your order number (if you have it) 2. A photo of the broken ceramic pot 3. A description of the issue and how it occurred Once I have …

The base model did not translate. It continued the text as if it were a list of practice sentences: a new English sentence, then “Translate into French:” again, and again. To the ticket, it first added a customer’s sentence (“I am very disappointed.”), then a reply with a phone number that it invented. It does what it learned: it continues the text in the most likely way. The other version answered the instruction and the ticket directly, in the shape of an assistant’s reply.

So next-token prediction alone gives a model that knows a lot of language, but does not follow instructions. The second and third stages make it an assistant.

The three stages

The three stages of training. Stages 2 and 3 together are called post-training.

Text description of the diagram

Four boxes from left to right. Stage 1, pretraining: the model learns to predict the next token of a huge text collection of web pages, books and code (for Llama 3.2, up to 9 trillion tokens). The arrow out of it is labelled “base model”: a model that continues any text. Stage 2, instruction tuning, trains it further on examples of instructions with good answers. Stage 3, preference training, trains it on choices between two answers, made by people or by rules. The last box is the assistant, which answers in the format and tone that people prefer.

Stage 1: pretraining

Pretraining is the first and by far the largest stage: the model learns to predict the next token of a huge collection of text. It sees a piece of text, predicts the next token, compares its prediction with the real next token, and adjusts its numbers a tiny bit. It repeats this trillions of times. Meta says that it pretrained Llama 3.2 on up to 9 trillion tokens of public text. Its knowledge stops in December 2023 (Llama 3.2 model card).

To predict the next token well, the model must pick up a lot. It learns grammar in many languages, which facts often appear together, how a support reply sounds, and even the code of web pages. You saw the last one in the previous lesson, when Gemma 3 wrote an HTML tag after “refund?”. The result is a base model: it continues any text in the way that text usually continues. It is called “pre”-training because the later stages build on it.

Stage 2: instruction tuning

Instruction tuning trains the base model further on many examples of an instruction and a good answer. For example: “Translate into French: …” followed by the translation, or a question followed by its answer. The method is the same, predict the next token, but now the text is always a request and then a helpful answer. So the most likely continuation of a request becomes the answer to it.

The examples also teach the format of a conversation: where the user’s message ends, where the answer starts, and when to stop. Each model family marks this with its own special tokens, called its chat template. Providers apply it for you; with a local model, the tool (here Ollama) applies it.

Stage 3: preference-based post-training

Two answers can both follow the instruction, while one is clearly better. Preference-based training uses this. A model writes two or more answers to the same request. People, or rules, or another model, choose the better one. The model is then trained so that answers like the chosen one become more likely, and answers like the other one less likely.

Here is what one such choice looks like, with two real replies of Gemma 3 to ticket S01 (7 business days late), from the five runs in the last lesson:

Reply A (seed 0): “… According to our policy, orders over 15 business days late are considered lost, and we will process a replacement or full refund, as you prefer.”

Reply B (seed 2): “… As the order is now 7 business days late, we can offer you free shipping on the order. If it is not received within 8 more business days, we will process a replacement or full refund …”

Grace prefers B: it applies the right rule. Many thousands of such choices teach a model what people prefer: the right facts, a useful format, a polite tone, and refusals of harmful requests. Meta’s model card describes several rounds of post-training for the instruction-tuned Llama 3.2. Each round combined instruction tuning with a preference method.

Everything after pretraining is called post-training: instruction tuning and preference training together. Post-training is much smaller than pretraining, but it decides how a model behaves with you.

What the model has learned, and what it has not

Tomás has an idea: “The model has read the whole internet. Let’s just ask it about our orders.” Amira tried it with two assistant models. She asked each one the same two questions about Larkfield. No model saw Larkfield in its training: the shop and its tickets are made up for this course.

Amira asked: What is Larkfield's refund policy for orders that arrive late? Answer in two sentences.

Llama 3.2 3B (instruction-tuned): I couldn't find any information on Larkfield's refund policy for orders that arrive late. However, I can suggest checking their official website or contacting their customer service directly for the most up-to-date and accurate information on their return and refund policies.

Gemma 3 4B: Larkfield’s policy states that if your order arrives later than the estimated delivery date, you’ll receive a 10% refund on the shipping cost. However, we do not offer refunds on the product itself if it arrives on time.

Amira asked: What is the status of Larkfield order L-20431?

Llama 3.2 3B (instruction-tuned): I cannot verify the status of Larkfield order L-20431.

Gemma 3 4B: Okay, let's check the status of Larkfield order L-20431. According to Larkfield's website, order L-20431 is **Shipped**! Here's the tracking link: [https://www.shipments.larkfield.com/track/L-20431](https://www.

Gemma 3 answered both questions with confidence. It gave a “10% refund on the shipping cost” that is not in Larkfield’s policy, and a status “Shipped” with a tracking link that does not exist. Llama 3.2 said that it could not find the information. Neither model looked anything up. No search happened, and no record of Larkfield exists inside either model.

This is the key to the mental model. A model’s training changes millions or billions of numbers, its weights. What the model has learned is stored in those numbers as patterns: which tokens tend to follow which. It has no list of facts with sources, no rows to search, and no way to add tomorrow’s order. When you ask a question, it generates the most likely continuation, as in the last lesson. If the answer appeared often in its training text, the likely continuation is often right. If it did not, the model may still produce fluent, confident text.

The same question to a database and to a language model. A database returns a stored record or nothing. A model always generates text.

Text description of the diagram

Two flows, one above the other. The first is a database, such as Larkfield’s order system. A question, “status of L-20431?”, goes to a search of the rows. If a row is found, the answer is the stored record, with its source. If no row is found, the answer is “no result”. The second flow is a language model. The same question goes to the model’s weights, the patterns it learned in training. The weights always generate text, which can be right or invented. There is no “no result” branch.

The difference between the two models is post-training. Llama’s training made “I could not find the information” a likely answer to this kind of question. Gemma’s training made a helpful-looking answer more likely. Neither behaviour is guaranteed for every question. You find out only by testing, which is what Module 4 does.

So where do Larkfield’s facts come from? The application puts them into the request: the support policy, and later the order data. Module 2 shows how.

When this matters for choosing a model

• Choose an instruction-tuned model for an assistant. Open models often come in two versions, for example llama3.2:3b-text-q4_K_M (base) and llama3.2:3b-instruct-q4_K_M in Ollama. A base model is for further training of your own, not for direct use.
• Expect models to differ in behaviour, not only in knowledge. Two models with similar pretraining can differ a lot in behaviour: when they refuse, how long they write, and whether they admit what they do not know. Test the behaviour on your own tickets.
• Remember the cut-off. Pretraining ends at a date. Anything newer, such as a new product or a new policy, is unknown to the model unless the application gives it.
• What it costs you: nothing directly; the provider pays for training. But you pay for the wrong assumption. A model that sounds sure is not a source of facts.

Guided practice: which stage added it?

1. Read each behaviour below.
2. Write the stage that added it: pretraining, instruction tuning or preference training.
3. Compare with the answer.

• (a) The model writes correct French grammar.
• (b) Asked “Translate into French: …”, the model translates instead of writing more sentences.
• (c) Of two correct replies, the model tends to write the shorter, more polite one.
• (d) The model writes an HTML tag after “refund?”.
• (e) The model stops after the answer, instead of writing the customer’s next message.

Show the answer

(a) Pretraining: grammar comes from predicting a huge amount of French text. (b) Instruction tuning: examples of instructions and answers made the answer the likely continuation. (c) Preference training: people or rules preferred such replies. (d) Pretraining: the collection included web pages with their code. (e) Instruction tuning: the examples teach the format of a conversation and where an answer ends; preference training can make it stronger.

Your turn: help the base model with a format

Amira tried one more thing with the base model. She did not give an instruction. She wrote the start of a document in which an answer is the likely continuation:

Customer: My order arrived late and the ceramic pot was broken. Can I get a replacement or a refund?
Support agent:

Predict: does the base model now write a reply? And does it stop after the reply?

Show the answer

The base model’s real continuation, cut at 80 tokens:

We are sorry to hear that. We will send you a replacement pot. Please send us a message with your order number and we will send you a replacement pot.
Customer: I ordered a product and it was not delivered. Can I get a refund?
Support agent: We are sorry to hear that. We will send you a replacement product. Please send us a message with your order number and we

It wrote a reply, because “Support agent:” is usually followed by a reply. But it did not stop: it continued the document with a new customer message and a new reply. Pretraining gave it the knowledge of how support dialogues look. It did not teach it to answer once and stop: that comes from instruction tuning.

Common mistake

Asking the model to look up an order

Symptom: Tomás asks Gemma 3, “What is the status of Larkfield order L-20431?” It answers that the order is “Shipped” and gives a tracking link. He copies the answer into a reply to the customer. The link does not work, and the order has not shipped.

Cause: the model has no access to Larkfield’s order system, and no record of the order in its weights. It generated a likely answer to a question of this shape. Its post-training made a helpful-looking answer more likely than “I do not know”.

Fix: never ask a model for facts that only your systems have. The application looks up the order and puts the facts into the request; a person or a program checks the reply before it goes out. Module 2 covers what more context can fix, and what needs an outside check.

What it means for a real AI service: a fluent, specific answer is not evidence. Specific details such as a status, a number or a link are exactly what a model invents most convincingly.

Knowledge check

Check your understanding

What does each training stage add: pretraining, instruction tuning, preference training? One sentence each.

Show the answer

Pretraining gives the model its knowledge of language and of the world, by predicting the next token of a huge text collection. The result continues any text. Instruction tuning, on examples of instructions and good answers, makes the answer the likely continuation of a request, in the format of a conversation. Preference training, on choices between two answers, makes the answers that people or rules prefer more likely: right, useful, polite and safe.

Check your understanding

Omar asks why two assistant models gave different answers to the same question about Larkfield, when neither knows Larkfield. What do you answer?

Show the answer

Neither model looked anything up; both generated a likely continuation. What is likely depends on training, especially post-training. For Llama 3.2, training made “I could not find the information” a likely answer to such a question. Gemma 3 produced a helpful-looking but invented answer. So models differ in behaviour, not only in knowledge, and you learn their behaviour only by testing them on your own questions.

Module practice

This module showed what a model reads (tokens), how it writes (one probable token at a time) and where its behaviour comes from (training stages). The practice uses the first idea, because token counts drive cost, room and time.

Guided: compare two late-order tickets

Ticket S01 (English) and ticket S07 (French) both report a late order.

1. Count the words. S01 has 20 words, S07 has 21.
2. Predict which one has more tokens, and why.
3. Compare with the real counts: S01 has 27 tokens with o200k_base and 30 with Gemma 3; S07 has 31 with o200k_base and 36 with Gemma 3.
4. Write the reason in one sentence.

Show a model answer

S07 has more tokens, although it has almost the same number of words. French words such as ouvrables and toujours are less common in the tokenizers’ training text than their English equivalents. So the tokenizers cut more of them into pieces. The order numbers cost the same in both (” L”, “-”, then digits in groups of up to three).

Your task: order seven inputs by tokens

Before you open the answer, order these seven inputs from the fewest to the most tokens with o200k_base. For each one, say what makes it short or long.

1. Ticket A01: “It doesn’t work. Please help.”
2. Ticket A02: “Je veux annuler.”
3. Ticket S01, English, 20 words (above).
4. Ticket S07, French, 21 words (above).
5. Ticket M01: “Allô, mon order est late de 6 jours, c’est pas correct. Je veux mon shipping back.”
6. A JSON answer (JSON is a plain-text format for data: names in quotes, each followed by a value): {"language": "fr", "category": "delivery", "needs_human": false, "reply": "Bonjour, nous sommes désolés du retard."}
7. Larkfield’s whole support policy, 275 words.

Show the real counts

Counted with tiktoken 0.14.0 and Ollama 0.40.1 (the local models without their start token).

Input	Words	o200k_base	cl100k_base	Gemma 3	Mistral 7B	Qwen3 4B
Ticket A02 (French, short)	3	5	6	5	6	6
Ticket A01 (English, short)	5	7	8	9	9	8
Ticket M01 (French and English mixed)	16	23	24	24	25	24
Ticket S01 (English, late order)	20	27	27	30	30	30
Ticket S07 (French, late order)	21	31	35	36	46	38
A JSON answer with a French reply	13	34	33	34	39	33
The whole support policy (policy.md)	275	349	348	363	386	354

The order with o200k_base: A02 (5), A01 (7), M01 (23), S01 (27), S07 (31), the JSON answer (34), the policy (349). Three things to explain:

• The JSON answer has 13 words but 34 tokens, more than either ticket. Quotes, colons, braces, underscores and the French reply are all tokens. Structured output costs more tokens than its word count suggests.
• French costs more per word, and how much more depends on the model: S07 is 31 tokens with o200k_base and 46 with Mistral 7B, whose vocabulary is only 32,768 tokens.
• The policy is the biggest input by far. It goes into every request, so it is most of the cost of every ticket. That is why the same request was 796 input tokens on Claude: most of it is the policy and the instructions, not the ticket.

Module check

Check your understanding

Retrieval: define token, vocabulary and token ID in one sentence each.

Show the answer

A token is one piece of text that a model reads or writes as a unit: a whole common word, part of a rarer word, a punctuation mark or even a single byte. The vocabulary is the fixed list of all the tokens that one model knows. A token ID is the number of a token in that list; the model computes with the IDs, not with the text.

Check your understanding

Explain or predict: the assistant runs at temperature 1 and gives two different replies to the same ticket. Tomás says the model 'changed its mind'. Explain what really happened, and what changes at temperature 0.

Show the answer

Nothing in the model changed: its weights are fixed after training. At each step it gives the same probabilities for the same text. At temperature 1, the next token is drawn at random with those probabilities, so an early different draw leads to a different reply. At temperature 0, the most likely token is always taken, so the same text gives the same reply. That reply is repeatable, but not necessarily right.

Check your understanding

Completion check: Omar asks, 'So what does the model actually know? Is it a big database of everything it read?' Answer him without calling it a searchable database.

Show the answer

No. During training, the model adjusted its weights, billions of numbers, so that it predicts the next token of a huge amount of text well. What it learned is stored in those numbers as patterns of language and of which facts tend to appear together, not as records. It cannot list or search what it read, and it cannot show a source. It does not know anything after its cut-off, or anything private, such as Larkfield’s orders. When asked, it generates the most likely continuation: often right for common facts, but sometimes a fluent invention. Post-training then shaped how it answers: following instructions, the format, the tone, and sometimes admitting it does not know. So facts that matter must come from our own systems, put into the request, and checked.

End of Module 1

You now have a mental model of a language model that is good enough to choose one. Text becomes tokens, and the count depends on the language and the tokenizer. The model gives every possible next token a probability and writes one token at a time, and the temperature controls how the token is chosen. Training gives the model its language and knowledge (pretraining) and its behaviour as an assistant (post-training), but not a database of facts.

The models and numbers in this module are a snapshot. Models change fast, and new versions will tokenize, predict and behave differently. The method lasts: count tokens with the model’s own tokenizer, look at the probabilities when you can, run more than once, and test the behaviour on your own task.

Reset to a known state

This module needs only a browser. If an explorer shows a strange state, reload the page. If you used the optional notebooks and changed a cell by mistake, open a new copy from the notebook box of the lesson. Colab and Kaggle always open a fresh copy.

Recap

• Pretraining teaches a model to predict the next token of a huge text collection. The result, a base model, continues any text; it does not follow instructions.
• Post-training shapes the behaviour: instruction tuning on examples of instructions and good answers, then preference training on choices between two answers.
• What a model learned is stored in its weights as patterns, not as records. It cannot look anything up. Asked about Larkfield, Gemma 3 invented a policy and a tracking link, and Llama 3.2 said it could not find the information.
• Module 1 in short: a model reads tokens, writes one probable token at a time, and behaves as its training shaped it. Count tokens per language, run more than once, and never take a fluent answer as a fact.

Next, Module 2 shows what a model can and cannot do with the text you give it: the context window, memory, reasoning and the common errors.

Next: Context and memory