Transformer blocks

What you will be able to do

Build the transformer block, the unit that every modern language model repeats many times. Then stack two blocks, train them on Larkfield’s tickets, and judge the result honestly against the routers you already have.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code of this lesson, with tasks and checks. It trains the tiny transformer for 8 epochs instead of 20, so that it finishes in a few minutes on a free CPU. About 40 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Queries, keys, values and one attention head, from Attention step by step. This lesson starts where that one ends.
• Tokens, token IDs, padding and the embedding table, from Tokens and embeddings.
• The five steps of training and train() in ticketnet.py, from The training loop; the best epoch on validation loss, from Batches, epochs and validation.
• Dropout, from Overfitting controls, and vanishing gradients, from Debug the experiment.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ticket-router with its active .venv, ticketnet.py and data/, from Set up PyTorch: Route 2. If it is missing or broken, follow Reset the setup.

Tested with Python 3.12 and PyTorch 2.14.1 on the CPU (macOS).

From one attention head to a block

In the last lesson, you computed one attention head. Each token sent a query, compared it with every key, and took a weighted average of the values. The result was a new vector for each token, mixed from the tokens that matter to it.

One head is not yet a model. A transformer is a stack of identical blocks, and each block has four parts around attention. This lesson adds them one at a time, and for each part you first see the problem that it solves:

1. Multi-head attention: several heads, each with its own question.
2. A feed-forward layer: a small network that works on each token alone.
3. Residual connections: keep the input and add the change.
4. Layer normalization: keep the numbers in a stable range.

Multi-head attention: several listeners

The familiar situation. In a meeting about a broken lamp, one colleague listens for what is broken. Another listens for when it happened. A third listens for what the customer wants. Each person hears the same words, but each pays attention to different ones.

The problem. One head gives each token one row of weights, and the weights add up to 1. Take “the lamp I returned arrived broken”. The token “broken” should read “lamp”, to know what is broken. It should also read “returned”, to know which team cares. One softmax row must split its weight between the two, so each signal gets weaker.

The idea. Use several smaller heads side by side. Each head has its own query, key and value weights, so each one learns its own question. Their outputs are joined, and one last linear layer mixes them. This is multi-head attention: “multi-head” because there are several heads, and each is a complete attention calculation.

Here are two heads on 3 tokens with 8 numbers each. nn.MultiheadAttention is PyTorch’s attention layer. batch_first=True says that the batch is the first dimension, as everywhere in this course. average_attn_weights=False keeps the weights of each head separate.

blocks.py

import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(1, 3, 8)              # 1 ticket, 3 tokens, 8 numbers per token
attention = nn.MultiheadAttention(embed_dim=8, num_heads=2, batch_first=True)
out, weights = attention(x, x, x, average_attn_weights=False)
print("input  ", tuple(x.shape))
print("output ", tuple(out.shape))
print("weights", tuple(weights.shape))
print(weights[0].detach().numpy().round(2))

The output:

input   (1, 3, 8)
output  (1, 3, 8)
weights (1, 2, 3, 3)
[[[0.4  0.3  0.3 ]
  [0.33 0.33 0.34]
  [0.19 0.41 0.4 ]]
 [[0.26 0.34 0.4 ]
  [0.21 0.28 0.51]
  [0.29 0.34 0.37]]]

Read the weights shape (1, 2, 3, 3) from the left. There is 1 ticket and there are 2 heads. Each head has a 3 × 3 table, with one row for each token that reads. Every row adds up to 1. The two heads read differently: token 3 gives 0.19 to token 1 in head 1, but 0.29 in head 2. The output has the same shape as the input, (1, 3, 8).

Each head works with a slice of the numbers: 2 heads × 4 numbers = 8. So more heads do not cost more parameters. They cut the same numbers into more, smaller slices:

for heads in (1, 4, 8):
    layer = nn.MultiheadAttention(embed_dim=64, num_heads=heads, batch_first=True)
    print(heads, "heads:", sum(p.numel() for p in layer.parameters()), "parameters")

The output:

1 heads: 16640 parameters
4 heads: 16640 parameters
8 heads: 16640 parameters

The 16,640 parameters are the query, key and value weights (3 × 64 × 64) and the output mix (64 × 64), with their biases. Larkfield’s tiny transformer uses 4 heads of 16 numbers each.

Go deeper

The optional lesson The mathematics of attention shows multi-head attention as formulas, with every shape, the count 4d² + 4d, and a check against nn.MultiheadAttention.

The feed-forward layer: work on what you heard

The familiar situation. After the meeting, each person goes back to their desk and works alone on what they heard.

The problem. Attention only mixes. Each output is a weighted average of value vectors, so a token can collect information but cannot do much with it. Recall from Why layers need activations that averages and other straight-line steps cannot bend.

The idea. Give each token a small network: a linear layer to 256 numbers, ReLU, and a linear layer back to 64. Every token uses the same weights, but here no token looks at another one. That is why it is called a feed-forward layer, per token: the numbers only go forward, through one token’s own small network.

torch.manual_seed(0)
feed_forward = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 64))
h = torch.randn(1, 5, 64)                       # 1 ticket, 5 tokens
together = feed_forward(h)                      # all 5 tokens at once
alone = feed_forward(h[:, 2])                   # token 2 on its own
print(tuple(together.shape), tuple(alone.shape))
print(torch.allclose(together[:, 2], alone, atol=1e-6))
print(sum(p.numel() for p in feed_forward.parameters()), "parameters")

The output:

(1, 5, 64) (1, 64)
True
33088 parameters

nn.Linear works on the last dimension and keeps every dimension before it. So the 5 tokens go through together, and token 2 gets the same result as when it goes through alone: True. The feed-forward layer holds 33,088 parameters, twice as many as attention.

Residual connections: keep the original, add the change

The familiar situation. You edit a colleague’s report with “track changes”. The original text stays, and your edits are added on top. If an edit is bad, the original is still there.

The problem. In a deep stack, each layer replaces its input with its output. Every layer must pass on everything that matters, and the gradient must travel back through every layer. In Debug the experiment, the first layer of a deep stack of sigmoid layers got a gradient of 1.45 × 10⁻⁸: it could not learn.

The idea. Do not replace the input: add the layer’s output to it. A tiny example with 3 numbers:

x = torch.tensor([1.0, -2.0, 0.5])
change = torch.tensor([0.1, 0.3, -0.2])
print(x + change)

The output:

tensor([ 1.1000, -1.7000,  0.3000])

This is a residual connection. “Residual” means “what is left over”: the layer only has to learn the difference between its input and a better version of it. The original numbers travel straight through the addition, and so does the gradient on the way back. Here is a stack of 20 sigmoid layers, with and without it:

def first_layer_gradient(residual, depth=20):
    torch.manual_seed(0)
    layers = [nn.Sequential(nn.Linear(64, 64), nn.Sigmoid()) for _ in range(depth)]
    h = torch.randn(32, 64)
    for layer in layers:
        h = h + layer(h) if residual else layer(h)   # keep h and add the change, or replace h
    h.pow(2).mean().backward()
    return layers[0][0].weight.grad.norm().item()

print(f"without residual connections: {first_layer_gradient(False):.2e}")
print(f"with residual connections:    {first_layer_gradient(True):.2e}")

The output:

without residual connections: 3.44e-18
with residual connections:    9.55e-01

Without residual connections, the first layer’s gradient is 3.44 × 10⁻¹⁸, so that layer never learns. With them, it is 0.955. The same layers, with one + added, can learn from the first layer to the last.

Layer normalization: keep the numbers in range

The familiar situation. Before a concert, a sound engineer sets each microphone to the same level. A quiet singer and a loud drum then reach the mixing desk in the same range.

The problem. Residual connections add a change in every layer, so the numbers can grow. Here are twenty additions of a random linear layer:

torch.manual_seed(0)
h = torch.randn(1, 64)
for step in range(1, 21):
    h = h + nn.Linear(64, 64)(h)                 # add a change, twenty times
    if step in (1, 5, 10, 20):
        print(f"after {step:2} additions: spread {h.std().item():10.2f}")

The output:

after  1 additions: spread       1.16
after  5 additions: spread       1.95
after 10 additions: spread       3.92
after 20 additions: spread      14.42

The spread (the standard deviation) grows from about 1 to 14. Large inputs push softmax and the next layers into ranges where learning is slow and unstable.

The idea. Before each part of the block, rescale each token’s numbers: subtract their mean and divide by their spread. Then a learned scale and shift give the model back control. This is layer normalization: it normalizes the numbers of a layer’s output, one token at a time. With the 4 numbers 2, 4, 6 and 8, the mean is 5 and the spread is √5 ≈ 2.236, so 2 becomes (2 − 5) ÷ 2.236 = −1.342:

row = torch.tensor([[2.0, 4.0, 6.0, 8.0]])
print(nn.LayerNorm(4)(row).detach())
big = row * 1000
print(nn.LayerNorm(4)(big).detach())

The output:

tensor([[-1.3416, -0.4472,  0.4472,  1.3416]])
tensor([[-1.3416, -0.4472,  0.4472,  1.3416]])

The row times 1,000 gives the same result: the level is gone, and the pattern stays. A new nn.LayerNorm starts with scale 1 and shift 0, so you see the plain normalized numbers.

The block: four parts, one shape

Now put the four parts together. The diagram shows the block of Larkfield’s tiny transformer, with the shape on every arrow and the parameters of every part.

One transformer block of the tiny transformer. Every arrow carries the same shape, (8, T, 64), so the output of one block can be the input of the next.

Text description of the diagram

The diagram is a chain from left to right. The block input has shape (8, T, 64): 8 tickets, T tokens and 64 numbers for each token. It goes through a LayerNorm (128 parameters) into multi-head attention with 4 heads of 16 numbers (16,640 parameters). A line carries the block input around the two boxes to a circle with a plus sign, which adds it to the attention output: the first residual connection. The sum goes through a second LayerNorm (128 parameters) into the feed-forward layer. That layer works on each token alone: 64 to 256 numbers, ReLU, and back to 64 (33,088 parameters). A second plus sign adds its output to its input, which the second residual connection carries around. The block output has the same shape, (8, T, 64). A note says that one block has 49,984 parameters, and that the tiny transformer stacks 2 blocks.

PyTorch has the whole block as one layer, nn.TransformerEncoderLayer. norm_first=True puts each LayerNorm before its part, as in the diagram. The first transformer, in 2017, put the norm after the addition (“add and norm”). Normalizing first is the usual choice today, because deep stacks train more steadily with it. dropout=0.1 switches off 10% of the numbers during training, as in Overfitting controls.

torch.manual_seed(0)
block = nn.TransformerEncoderLayer(d_model=64, nhead=4, dim_feedforward=256, dropout=0.1,
                                   batch_first=True, norm_first=True)
x = torch.randn(8, 120, 64)             # 8 tickets, 120 tokens, 64 numbers per token
print(tuple(block(x).shape))
for name, p in block.named_parameters():
    print(f"{name:<24} {str(tuple(p.shape)):<10} {p.numel():>6,}")
print("in all", f"{sum(p.numel() for p in block.parameters()):,}")

The output:

(8, 120, 64)
self_attn.in_proj_weight (192, 64)  12,288
self_attn.in_proj_bias   (192,)        192
self_attn.out_proj.weight (64, 64)    4,096
self_attn.out_proj.bias  (64,)          64
linear1.weight           (256, 64)  16,384
linear1.bias             (256,)        256
linear2.weight           (64, 256)  16,384
linear2.bias             (64,)          64
norm1.weight             (64,)          64
norm1.bias               (64,)          64
norm2.weight             (64,)          64
norm2.bias               (64,)          64
in all 49,984

(8, 120, 64) in, (8, 120, 64) out. Each part of the diagram is in the list. in_proj_weight holds the query, key and value weights together (3 × 64 = 192 rows), and out_proj mixes the heads. linear1 and linear2 are the feed-forward layer, and norm1 and norm2 hold the learned scales and shifts. Because the shape does not change, blocks stack like bricks. Large models stack more of them: SmolLM2-135M, in Module 7, has 30.

Go deeper

The 2017 paper put each norm after the addition. The original transformer, step by step builds that block from its formulas and follows one example through the whole encoder–decoder, in the optional mathematics module.

The tiny transformer on the tickets

The course’s ticketnet.py has TinyTransformer. It adds position vectors to the token vectors, runs 2 blocks and a final LayerNorm, and averages each ticket’s real tokens. A linear layer then gives 5 scores. The next lesson explains the position vectors and the padding mask. Here are the shapes for 8 validation tickets; the longest has 93 tokens, so T = 93:

from ticketnet import (TinyTransformer, Vocab, labels, load_split, set_seed, token_ids, train,
                       evaluate, count_parameters, TEAMS, PAD)

train_rows, valid_rows, test_rows = (load_split(f"data/{s}.csv") for s in ("train", "valid", "test"))
vocab = Vocab([row["text"] for row in train_rows])
ids_train, ids_valid = token_ids(train_rows, vocab), token_ids(valid_rows, vocab)
y_train, y_valid = labels(train_rows), labels(valid_rows)

set_seed(0)
model = TinyTransformer(len(vocab))
model.eval()
ids = token_ids(valid_rows[:8], vocab)
with torch.no_grad():
    positions = torch.arange(ids.shape[1])
    h = model.embed(ids);                    print("token vectors     ", tuple(h.shape))
    h = h + model.position(positions);       print("+ position vectors", tuple(h.shape))
    h = model.blocks(h, src_key_padding_mask=ids == PAD)
    print("after 2 blocks    ", tuple(h.shape))
    keep = (ids != PAD).unsqueeze(-1).float()
    mean = (model.norm(h) * keep).sum(1) / keep.sum(1)
    print("average per ticket", tuple(mean.shape))
    print("scores            ", tuple(model.out(mean).shape))

The output:

token vectors      (8, 93, 64)
+ position vectors (8, 93, 64)
after 2 blocks     (8, 93, 64)
average per ticket (8, 64)
scores             (8, 5)

The blocks keep (8, 93, 64). Only the average removes the token dimension, and the last layer turns 64 numbers into 5 team scores. Where do the parameters sit?

for name, part in model.named_children():
    print(f"{name:<9} {count_parameters(part):>8,}")
print(f"{'in all':<9} {count_parameters(model):>8,}")

The output:

embed      183,296
position     7,680
blocks      99,968
norm           128
out            325
in all     291,397

The embedding table, 2,864 tokens × 64 numbers, holds 63% of the 291,397 parameters. The two blocks, 2 × 49,984, hold most of the rest.

Train it

Training uses the same train() as the bag-of-words network: Adam, learning rate 0.001, batches of 64, 20 epochs and seed 0. log=None keeps it quiet, and the loop prints six of the epochs.

import time

set_seed(0)
model = TinyTransformer(len(vocab))
start = time.time()
history = train(model, ids_train, y_train, ids_valid, y_valid, epochs=20, lr=1e-3, log=None)
print(f"{time.time() - start:.0f} seconds")
for row in history:
    if row["epoch"] in (1, 2, 5, 10, 15, 20):
        print(row)
best = min(history, key=lambda row: row["valid_loss"])
print("best epoch", best["epoch"], "| valid loss", best["valid_loss"], "| valid accuracy", best["valid_acc"])

The output:

119 seconds
{'epoch': 1, 'train_loss': 0.9262, 'train_acc': 0.6767, 'valid_loss': 0.9591, 'valid_acc': 0.6717}
{'epoch': 2, 'train_loss': 0.4359, 'train_acc': 0.8524, 'valid_loss': 0.4562, 'valid_acc': 0.8533}
{'epoch': 5, 'train_loss': 0.1381, 'train_acc': 0.9678, 'valid_loss': 0.2343, 'valid_acc': 0.91}
{'epoch': 10, 'train_loss': 0.0317, 'train_acc': 0.9951, 'valid_loss': 0.2239, 'valid_acc': 0.925}
{'epoch': 15, 'train_loss': 0.0034, 'train_acc': 0.9998, 'valid_loss': 0.2091, 'valid_acc': 0.9417}
{'epoch': 20, 'train_loss': 0.0005, 'train_acc': 1.0, 'valid_loss': 0.2266, 'valid_acc': 0.9433}
best epoch 15 | valid loss 0.2091 | valid accuracy 0.9417

It takes about two minutes on a Mac mini CPU; the bag-of-words network needs a few seconds. Your numbers can differ slightly on another computer. The training loss falls to 0.0005, and the training accuracy reaches 100%. The validation loss is lowest at epoch 15, 0.209, with 94.2% of the validation tickets right.

The next lesson and Attention step by step load this trained model, so save its weights. torch.save() writes them to a file; Save and inspect results explains it in detail. The line prints nothing.

torch.save(model.state_dict(), "tiny_transformer.pt")

The honest comparison

Priya puts the new model next to the routers that Larkfield already has, on the same 600 validation tickets.

• Tiny transformer
• Bag-of-words network
• TF-IDF + logistic regression

Validation accuracy after each epoch, from real runs with seed 0. The tiny transformer starts lower, climbs more slowly and stays below both older routers.

Text description of the chart

A line chart with the epoch, 1 to 20, on the horizontal axis and the validation accuracy, 60% to 100%, on the vertical axis. The bag-of-words network starts at 97.2% after epoch 1, peaks at 99.2% at epoch 4 and ends at 97.3%. The TF-IDF and logistic regression model of the machine learning course is a flat dashed line at 98.8%. The tiny transformer starts at 67.2% and reaches 91.0% at epoch 5. It has 94.2% at its best epoch, 15, and ends at 94.3%. It never comes close to the other two lines.

Overall, the tiny transformer is right on 94.2% at its best epoch. The bag-of-words network reached 98.3% at its best epoch, and TF-IDF with logistic regression 98.8%. By style, on the final epoch of each run, the tiny transformer is weaker almost everywhere:

Style	Bag of words	Tiny transformer	TF-IDF + logistic
two_topics	90.4%	84.6%	88.5%
long	81.0%	76.2%	100%
negation	94.7%	92.1%	100%
short	97.9%	89.6%	100%
typos	100%	92.5%	97.5%
boundary	97.7%	96.9%	100%
plain	100%	97.9%	100%

Even on negation and two-topic tickets, where word order should help, the transformer does not win.

Is it the settings? Priya runs a fair tuning pass of 8 runs, 25 epochs each. The runs try 1 or 2 blocks, dropout 0.1 or 0.3, and learning rate 0.0005 or 0.001. Each run keeps its epoch with the lowest validation loss. The eight runs reach 92.8% to 94.7%. Tuning does not close the gap. Last, each model meets the test tickets once:

The four routers on the 600 test tickets, measured once at the end: TF-IDF with logistic regression, the bag-of-words network (Words), the average-embedding network (Embed.) and the tiny transformer. The linear model of the machine learning course is the best; the tiny transformer is the weakest.

Text description of the chart

A column chart of the test accuracy of four routers. TF-IDF with logistic regression: 99.2%. The bag-of-words network: 96.5%. The average-embedding network: 95.2%. The tiny transformer: 91.2%.

On the test tickets, the tiny transformer is right on 91.2%. That is below the bag of words (96.5%) and far below TF-IDF with logistic regression (99.2%). Larkfield keeps the linear model. That is a result, not a failure, and the reasons explain the rest of the course:

• Too little data to learn language. The 4,696 training tickets have about 269,000 tokens. From these alone, the model must learn what each word means and how word order works.
• More parameters to fit. 291,397 numbers start from random values and are fitted to 4,696 examples. The training accuracy of 100% and the rising validation loss after epoch 15 show that the model learns the training tickets by heart.
• Nothing is pretrained. The vector of “refund” starts as 64 random numbers. A model that has already read billions of words starts with useful vectors and useful blocks.

On these short tickets, the words decide the team, so word counts are enough. The blocks pay off when a model has read a lot of text first. That is pretraining, the topic of Training objectives. Module 7 runs a pretrained model, and the case study Fine-tune a small transformer measures what pretraining buys on a text task.

Guided practice: build and train the block in the notebook

1. Open the notebook from the box Do this lesson in a notebook at the top of this page, and run the setup cells. The data cell ends with Ready: train 4696 tickets, valid 600, test 600.
2. In section 1, run the two heads. You see the weights shape (1, 2, 3, 3).
3. In section 2, run the feed-forward layer and the residual test. You see True and 3.44e-18.
4. In section 3, build one block. In the Your turn cell, put its number of parameters in block_parameters, then run the check cell. It prints Check passed.
5. In section 4, train the tiny transformer for 8 epochs. It takes one to three minutes. You see a validation accuracy of about 92%.
6. Compute the accuracy by style. You see that long and two_topics are among the weakest styles.

Show an example answer to step 4

block_parameters = sum(p.numel() for p in block.parameters())
print(block_parameters)

49984

Attention has 16,640, the feed-forward layer 33,088, and the two LayerNorms 128 each.

Your turn: eight heads instead of four

Change one thing: TinyTransformer(len(vocab), heads=8). Before you run it, predict the number of parameters and the shape of the scores for the 8 tickets.

Show the answer

set_seed(0)
eight = TinyTransformer(len(vocab), heads=8)
print(count_parameters(eight))
with torch.no_grad():
    print(tuple(eight.eval()(ids).shape))

291397
(8, 5)

The parameters stay at 291,397 and the scores at (8, 5). Eight heads cut the 64 numbers into slices of 8 instead of 16, and the weights stay the same size. The number of heads changes what the model can learn, not its size in memory.

Common mistake

"PyTorch will figure out which dimension is the batch"

Symptom: Tomás builds a block without batch_first=True. It runs with no error and returns the right shape. But a ticket’s output changes when the other tickets in its batch change:

torch.manual_seed(0)
wrong = nn.TransformerEncoderLayer(64, 4, 256, dropout=0.0)          # batch_first is False
wrong.eval()
batch = torch.randn(8, 10, 64)                   # 8 tickets, 10 tokens
other = batch.clone()
other[1:] = torch.randn(7, 10, 64)               # the same ticket 0, seven other tickets
with torch.no_grad():
    a, b = wrong(batch)[0], wrong(other)[0]
print(tuple(wrong(batch).shape))
print("ticket 0 changed:", not torch.allclose(a, b, atol=1e-6))

(8, 10, 64)
ticket 0 changed: True

Cause: without batch_first=True, PyTorch reads the shape as (tokens, batch, numbers). It sees 10 sequences of 8 tokens. So attention mixes the 8 tickets with each other, and ticket 0 reads the other tickets. The shape (8, 10, 64) looks right, so no error appears.

Fix: set batch_first=True on every attention layer and block, as ticketnet.py does. Then test it: a ticket’s output must not depend on its neighbours.

torch.manual_seed(0)
right_layer = nn.TransformerEncoderLayer(64, 4, 256, dropout=0.0, batch_first=True)
right_layer.eval()
with torch.no_grad():
    a, b = right_layer(batch)[0], right_layer(other)[0]
print("ticket 0 changed:", not torch.allclose(a, b, atol=1e-6))

ticket 0 changed: False

What it means for a real AI system: in the help desk, Amira sends tickets in batches. With this bug, the same ticket could go to another team, depending on which tickets arrived with it. Check the meaning of every dimension, not only the shape.

See it end to endFine-tune a small transformerCase study

The fine-tuning case study answers this lesson’s question: does a pretrained transformer beat word counts when labelled data is small?

Knowledge check

Check your understanding

Name the four parts of a transformer block, and the problem that each one solves.

Show the answer

Multi-head attention lets each token collect information from the other tokens. It has several heads that each learn their own question, so one softmax row does not have to serve every purpose. The feed-forward layer works on each token alone, with ReLU, so the block can transform what attention collected; attention alone only takes weighted averages. Residual connections add each part’s output to its input. The original is kept, and the gradient reaches the first layers (3.44 × 10⁻¹⁸ without them, 0.955 with them, in the 20-layer test). Layer normalization rescales each token’s numbers to mean 0 and spread 1 before each part, so the numbers do not grow as the changes add up.

Check your understanding

Omar asks: the transformer is the newest model, so why does Larkfield keep the linear model? Answer him with the numbers and the reasons.

Show the answer

On the same tickets, the tiny transformer is the weakest router. It gets 94.2% on validation at its best epoch and 92.8% to 94.7% after a fair tuning pass. On the test tickets, it gets 91.2%. TF-IDF with logistic regression gets 98.8% on validation and 99.2% on test. The bag-of-words network gets 96.5% on test. The transformer must learn the meaning of every word, and how order works, from only 4,696 tickets. Its 291,397 parameters start random, and it learns the training tickets by heart (100% training accuracy). On these short tickets, the words decide the team, so word counts are enough. A transformer pays off when it starts from pretraining on far more text.

Recap

• Multi-head attention runs several smaller heads side by side, each with its own question. More heads do not add parameters.
• The feed-forward layer is a small network for each token alone: 64 → 256 → ReLU → 64.
• A residual connection keeps the input and adds the change, so gradients reach the first layers.
• Layer normalization keeps each token’s numbers at mean 0 and spread 1.
• A block has the same shape in and out, (batch, tokens, 64), so blocks stack. The tiny transformer has 2 blocks and 291,397 parameters.
• On Larkfield’s tickets, it is the weakest router: 91.2% on test, against 96.5% for the bag of words and 99.2% for the linear model. It has too little data, and nothing is pretrained.

Next: the two pieces that this lesson used without explaining them, the position vectors and the padding mask.

Next: Position and masking