From tokens to vectors

What you will be able to do

See what a model does with a token ID: it looks up a list of learned numbers, a vector. Then you can explain why “ refund” and “ reimburse” are close for a model, why “ late” and “ early” are close too, and what an embedding is when a later course uses one for search.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• What tokens and token IDs are, from From text to tokens.
• What a vector is, and how cosine similarity compares two vectors, from Similarity and distance in Essential Mathematics and Statistics for AI. This lesson repeats the short version.

What you need

• A browser. The code in this lesson is optional; you do not need to run it.

The numbers come from one small open model. Other models have other numbers, but the same ideas.

Colours as three numbers

A paint shop mixes colours with a machine. The machine does not know the names “sky blue” or “brick red”. It knows three numbers for each colour: how much red, green and blue to add. Sky blue is 135, 206, 235. Light blue is 173, 216, 230. Brick red is 178, 34, 34.

Look at the numbers, not the names. Sky blue and light blue have close numbers, and they look alike. Brick red has very different numbers, and it looks different. The three numbers put every colour at a place in a space, and in that space, close means alike.

A language model needs the same thing for tokens.

The problem: a token ID is only a position

The last lesson turned text into token IDs. This lesson uses SmolLM2-135M, a small open model that runs on a laptop. Its tokenizer gives “ refund” the ID 37393.

Now look at the IDs next to it. 37392 is “ Yin”. 37394 is “ elem”. 37395 is “ COPY”. The ID is only the position of the token in the vocabulary list. Two close IDs are not close in meaning, and two tokens with a close meaning can have IDs that are far apart: “ Monday” is 12397, and “ Tuesday” is 14801.

So the model cannot compute with the IDs directly. It is a mathematical function, and it needs numbers where close means alike, as with the paint colours.

The idea: a row of numbers for each token

The model keeps a large table. The table has one row for each token in the vocabulary, and each row is a list of numbers: a vector. The first thing the model does with a ticket is to replace each token ID with its row. Token ID 37393 becomes row 37393.

This table is the embedding table, and the row of a token is its embedding, or token vector. The name comes from the picture of the paint colours: each token is embedded, placed, at one point in a space. The space of the paint machine has 3 directions. The space of SmolLM2-135M has 576.

The real embedding of the token " refund" (token ID 37393) in SmolLM2-135M: 576 numbers, one thin bar each.

Text description of the chart

A strip of 576 thin bars, one for each number of the vector of “ refund”, above and below a zero line. Most bars are short: most numbers are between −0.3 and +0.3, and a few reach −0.97 and +0.77. Under the strip, the first six numbers are printed: 0.019, 0.022, −0.08, −0.147, −0.195, 0.177, then “570 more”.

No person chose these numbers. Before training, every row is random. During training, the model adjusts every number of every row, a little at a time, so that it predicts the next token better. (Training stages describes this training.) Tokens that appear in the same kind of places in text end up with close rows.

The table is a large part of the model. SmolLM2-135M has 49,152 tokens × 576 numbers: about 28 million numbers, 21% of the model’s 135 million.

A tiny example: three tokens, two numbers

Take three tokens with made-up vectors of only two numbers, so that you can compute by hand:

Token	Vector
“ refund”	(0.9, 0.2)
“ reimburse”	(0.8, 0.3)
“ Monday”	(−0.1, 0.9)

To compare two vectors, models use cosine similarity: it measures whether two vectors point in the same direction. It is 1 for the same direction, 0 for unrelated directions and −1 for opposite directions. You multiply the two vectors number by number and add (the dot product), then divide by the two lengths:

• “ refund” and “ reimburse”: (0.9 × 0.8 + 0.2 × 0.3) ÷ (0.92 × 0.85) = 0.78 ÷ 0.79 = 0.99.
• “ refund” and “ Monday”: (0.9 × −0.1 + 0.2 × 0.9) ÷ (0.92 × 0.91) = 0.09 ÷ 0.84 = 0.11.

The first pair points in almost the same direction, and the second pair does not. That is all “close” means in this lesson: a high cosine similarity.

What the real table learned

Now the real table of SmolLM2-135M, with 576 numbers per token. For one token, the explorer shows two lists. On the left: the tokens next to it by ID. On the right: the tokens nearest to it by vector, out of all 49,152 rows. The number on each bar is the cosine similarity of the two rows.

Explore

Neighbours by ID and by vector, in SmolLM2-135M

·Monday token ID 12397

Next to it by ID

1. 12394·retrie0.48
2. 12395·Isa0.41
3. 12396·disappear0.37
4. 12398·exert0.42
5. 12399·Hot0.34
6. 12400·weapon0.33

Nearest by vector

1. ·Tuesday0.92
2. ·Wednesday0.90
3. ·Thursday0.90
4. ·Saturday0.85
5. ·Friday0.84
6. ·Sunday0.82
7. ·Sundays0.69
8. Monday0.68

The other days of the week are its nearest tokens (0.82 to 0.92). Its neighbours by ID are unrelated (0.33 to 0.48).

The real rows of SmolLM2-135M's embedding table (Hugging Face, transformers 5.19.0). A middle dot marks a space at the start of a token.

Text description of the explorer

For each of seven tokens, two lists. The first list has the six tokens with the IDs just before and after it, and their similarity. The second list has the eight tokens nearest to it by vector, and their similarity.

“ Monday” (ID 12397): neighbours by ID “ retrie”, “ Isa”, “ disappear”, “ exert”, “ Hot” and “ weapon” (0.33 to 0.48); nearest by vector “ Tuesday” 0.92, “ Wednesday” 0.90, “ Thursday” 0.90, “ Saturday” 0.85, “ Friday” 0.84, “ Sunday” 0.82. “ late” (ID 3008): neighbours by ID 0.07 to 0.13 and one −0.01; nearest by vector “ early” 0.69, “ Late” 0.68, “Late” 0.67, “ Early” 0.57. “ broken” (ID 6792): by ID 0.12 to 0.27; by vector “broken” 0.74, “ Broken” 0.73, “ breaking” 0.71. “ cheap”: “ inexpensive” 0.80, “ cheaper” 0.74, “ affordable” 0.66, “ expensive” 0.59, “ costly” 0.55. “ France”: “France” 0.87, “ Spain” 0.71, “ Germany” 0.70, “French” 0.67. “ refund” (ID 37393): by ID 0.49 to 0.70; by vector “ reimburse” 0.84, then pieces of rare words such as “ authent” and “ remun” 0.80. “ parcel”: “ parcels” 0.90, then pieces of rare words 0.73 to 0.74.

Four things to see:

1. IDs say nothing. The tokens next to “ Monday” by ID are “ Isa”, “ Hot” and “ weapon”. The tokens next to it by vector are the other days of the week, at 0.82 to 0.92.
2. Vectors group tokens that are used alike. Forms of one word (” broken”, “broken”, “ Broken”), words with the same meaning (” cheap”, “ inexpensive”) and members of one family (” France”, “ Spain”, “ Germany”) are close.
3. Close is not the same meaning. The nearest token to “ late” is “ early”, its opposite, at 0.69. Both appear in the same places: “arrived late”, “arrived early”. The vectors learn where a token is used, and opposites are used in the same places. Keep this in mind for search: a question about late orders can find a text about early orders.
4. Rare tokens get vague vectors. “ refund” and “ parcel” appear little in SmolLM2’s training text, which is mostly English web pages, code and mathematics. Apart from “ reimburse” and “ parcels”, their nearest tokens are pieces of rare words. Even their neighbours by ID score 0.49 to 0.70: rare rows stay crowded together, because training moved them only a little.

Before and after training

The explorer below shows 20 tokens in five groups: days, countries, damage, price and delivery. A vector of 576 numbers cannot be drawn, so the map draws a shadow of it on a flat page, with a method called PCA (principal component analysis). The shadow keeps the two directions in which the tokens differ most. It loses the rest, so trust the nearest-neighbour score under the map more than the distances on the page.

Explore

20 tokens before and after training

• days
• countries
• damage
• price
• delivery

Random numbers, as in the table before training: the tokens are scattered, and no group sits together. Same-group score of these 20 tokens: 0.017 (chance 0.158).

2-D PCA of the unit vectors of 20 tokens of SmolLM2-135M. Before training: a table of random numbers with the same shape and the model's own starting scale. The same-group score: the share of each token's three nearest tokens (in all 576 numbers) that are in its group.

Text description of the explorer

A scatter plot of 20 tokens, each with the colour and shape of its group: Monday, Tuesday, Friday and Sunday (days); France, Spain, Germany and Canada (countries); broken, cracked, smashed and shattered (damage); cheap, expensive, costly and affordable (price); delivery, shipping, shipment and delivered (delivery). A control switches between two tables. Before training (random numbers): the tokens are scattered, and no group sits together; the same-group score is 0.017. After training (SmolLM2-135M): the four days sit together on the right and the four countries at the bottom; the damage, price and delivery words share the top left. The nearest token of every token is in its own group, and the same-group score is 0.85. Chance is 0.158.

Training made the groups. Nobody told the model that Monday and Friday are days: it saw them in the same kind of sentences, many times, and their rows moved closer. The same happened to “ cracked” and “ shattered”. The groups also leak a little, as the text does: “ shipment” is close to “ smashed” (0.67), because damaged shipments are a common topic.

The same token in different sentences

The table gives each token one row. But “ order” means different things in “My order arrived late” and “put the steps in the right order”. How does the model tell them apart?

The table is only the first step. SmolLM2-135M then has 30 layers, and each layer updates the vector of every token with information from the tokens before it in the text. So, after the first layers, the vector of “ order” is no longer the row from the table: it is the row changed by its sentence. Deep Learning and Transformers Explained shows how a layer does this, in The mathematics of attention. You do not need it here.

Amira compared the vector of “ order” in three sentences: A, “My order arrived late and the pot was broken.”; B, “I placed an order for two ceramic pots last week.”; and C, “Please put the steps in the right order before you start.”

• Layer 15 (the middle)
• Layer 30 (the last)

The vectors of the token " order" in three sentences, compared in pairs, in SmolLM2-135M. At the input (layer 0), every pair is 1.00: the same row. The axis starts at 0.5.

Text description of the chart

A bar chart with six bars: three pairs of sentences, each at layer 15 and layer 30. A and B (two orders in a shop): 0.90 at layer 15 and 0.85 at layer 30. A and C (a shop order and the order of steps): 0.77 and 0.79. B and C: 0.78 and 0.81. At layer 0, all three pairs are 1.00.

At the input, the three vectors are the same, because they are the same row: a similarity of 1.00. By the middle of the model, the two shop sentences are the most alike (0.90), and the sentence about steps has moved away (0.77 and 0.78). The vectors now carry the sentence, not only the token. The differences are not large, because these vectors also carry other things, such as the position in the text and what may come next.

The last vector chooses the next token

After the last layer, the model has one vector for each token of the text. To write the next token, it uses the vector of the last one. It compares that vector with every row of a table, and gives each token a score: the more alike, the higher the score.

In SmolLM2-135M, this is the same embedding table as at the input. The model reads with the table, and it writes with the table. Not every model does this: Gemma uses one table at both ends, but larger models such as Llama 3 8B learn a second table for the output. Amira gave SmolLM2-135M the start “My parcel arrived late, so the customer asked for a”. Its highest score was “ refund”.

The next lesson turns those scores into probabilities, and shows how the model picks one token and repeats the loop.

From token vectors to embeddings for search

Larkfield’s assistant will later need to find the right part of the support policy for a ticket. For that, a program needs one vector for a whole text, not one for each token. An embedding model gives it: a model that was trained to read a text and return one vector, so that texts with a similar meaning get close vectors. RAG: Building AI That Uses Your Data uses one that returns 384 numbers for any text, in Embeddings for search.

What this lesson showed about token vectors is also true for these text embeddings:

• Close means “used alike”, not “the same”. A text about late orders can be close to a text about early orders. Search must be tested on real questions.
• The numbers belong to one model. A vector from one model means nothing to another model, as a token ID from another tokenizer means nothing (see the last lesson). Compare vectors only when the same model made both.

Worked example: Amira looks up a row

Amira wants to see the table herself. You do not need to run this code; it is here so that you see the steps. It downloads the model (about 270 MB) from Hugging Face the first time.

token_vectors.py

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "HuggingFaceTB/SmolLM2-135M"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
table = model.get_input_embeddings().weight.detach().float()  # one row per token

print(table.shape)
refund = tokenizer.encode(" refund")[0]
print(refund, [round(v, 3) for v in table[refund][:4].tolist()])


def similarity(a, b):
    x, y = table[tokenizer.encode(a)[0]], table[tokenizer.encode(b)[0]]
    return round((x @ y / (x.norm() * y.norm())).item(), 2)


print(similarity(" broken", " cracked"))
print(similarity(" broken", " Monday"))
print(similarity(" late", " early"))

get_input_embeddings().weight is the embedding table. table[refund] is one row: the token’s vector. The function similarity is the cosine similarity of two rows. The output:

torch.Size([49152, 576])
37393 [0.019, 0.022, -0.08, -0.147]
0.5
0.23
0.69

The table has 49,152 rows of 576 numbers. “ broken” is closer to “ cracked” (0.5) than to “ Monday” (0.23). And “ late” is closer to “ early” (0.69) than “ broken” is to “ cracked”.

Guided practice: read the explorers

Use the two explorers above. Make your predictions first, then follow the steps.

1. In the first explorer, select ·Monday. Compare the two lists: the tokens by ID score 0.33 to 0.48, and the days of the week by vector score 0.82 to 0.92.
2. Select ·late. Find “ early” at the top of the list by vector, at 0.69.
3. Select ·refund. Find “ reimburse” (0.84), then the pieces of rare words at 0.80. Look at the scores of the neighbours by ID: 0.49 to 0.70, much higher than for “ Monday”.
4. In the second explorer, switch from Before training to After training. Find the four days and the four countries.
5. Write one sentence for Tomás, who says: “Two tokens with close vectors mean the same thing.”

Show an example answer

“Not always. Close vectors mean that the two tokens are used in the same kind of places in text. That is often a similar meaning, as with Monday and Tuesday, but ’ late’ and ’ early’ are close too, and they are opposites.”

Your turn: two price words

In the first explorer, “ cheap” has “ inexpensive” near it. Before you look, predict: is “ expensive”, its opposite, among its eight nearest tokens? Then check.

Show the answer

Yes. “ expensive” is sixth, at 0.59, and “ costly” is seventh, at 0.55. The two words are opposites, but both are about price, and both appear in the same places: “a cheap pot”, “an expensive pot”. It is the same pattern as “ late” and “ early”.

Common mistake

A search that mixes two models

Symptom: Larkfield’s search tool stores one vector for each section of the support policy. Amira changes the embedding model to a newer one, and embeds only the new sections with it. The next day, questions about refunds find sections about passwords.

Cause: the old sections still have vectors from the old model, and the questions now get vectors from the new model. The two models put meanings at different places, as two tokenizers give different IDs to “ refund”. A similarity between a vector of one model and a vector of the other is a number, but it means nothing.

Fix: embed every section and every question with one model. When the model changes, embed all the sections again. Store the name and the version of the model with the vectors, so that a program can check it.

What it means for a real AI service: a vector is only useful next to vectors of the same model. RAG: Building AI That Uses Your Data records the model with the index for this reason.

Knowledge check

Check your understanding

A token ID is a number. Why does the model not compute with the IDs directly?

Show the answer

An ID is only the position of the token in the vocabulary list. Close IDs are not close in meaning: next to “ Monday” (12397) are “ Isa” and “ weapon”. The model needs numbers where close means alike. So it replaces each ID with the token’s row in the embedding table, a vector of learned numbers.

Check your understanding

In SmolLM2-135M, ' late' and ' early' have close vectors (0.69). Does the model treat them as the same? What does it mean for search?

Show the answer

No. Close vectors mean that the two tokens are used in the same kind of places in text, such as “arrived late” and “arrived early”. Opposites are often used in the same places. For search with embeddings, this means that a question about late orders can find a text about early orders. Test search on real questions, and read what it finds.

Check your understanding

In 'My order arrived late' and 'put the steps in the right order', the token ' order' has the same ID. Does it have the same vector inside the model?

Show the answer

At the input, yes: the model takes the same row of the embedding table, so the two vectors are the same (similarity 1.00). Then each layer changes the vector with information from the other tokens of the sentence. By the middle of SmolLM2-135M, the vector of “ order” in a shop sentence and in the sentence about steps have a similarity of about 0.77, against 0.90 for two shop sentences.

Recap

• A token ID is only a position in a list. The model replaces it with the token’s row in the embedding table: a vector of learned numbers, the token’s embedding. In SmolLM2-135M, 49,152 rows of 576 numbers, 21% of the model.
• Training places tokens that are used alike close together: the days of the week at 0.82 to 0.92. Close means used alike, not the same meaning: “ late” is nearest to “ early”. Rare tokens get vague vectors.
• The table gives the same row in every sentence. The model’s layers then change each vector with the rest of the text, so “ order” in a shop sentence and in a sentence about steps move apart.
• An embedding model returns one vector for a whole text, for search. Its vectors follow the same rules: close means used alike, and vectors of different models never mix.

Next, see what the model does with the last vector: it gives every possible next token a probability, and picks one.

Next: Predicting continuations