The forward pass

What you will be able to do

Turn Larkfield’s tickets into tensors that a network can read. Send a batch of them through a first, untrained network, and check the shape after every layer. Then you can read any network’s shapes and fix a shape error from its message.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell of this lesson, the tasks with checks, and the module practice. Its setup cells download the tickets and ticketnet.py, so it needs no local setup. About 40 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Shapes, the batch dimension, broadcasting and float32, from Tensors and shapes.
• A layer as many neurons that read the same inputs, from Neurons and layers, and ReLU, from Why layers need activations.
• The seed, from Set up PyTorch.

What you need on your computer (only for the local route; the notebook needs none of it)

• The folder ticket-router with its active .venv, ticketnet.py and data/, from Set up PyTorch: Route 2. If python first_run.py does not print sum 2.4265, or data/ is missing, follow Reset the setup.

Tested with Python 3.12 and PyTorch 2.14.1 on the CPU (macOS).

A form with one box for each word

A survey team gives every visitor the same form: a fixed list of items, each with one box. A short answer and a long answer fill in the same form, so every form has the same boxes in the same places. The person who adds up the results never has to guess what a box means.

The problem: tickets have different lengths

A layer such as nn.Linear(2864, 64) reads exactly 2,864 numbers for each ticket. But tickets have different lengths. The median Larkfield ticket has 55 tokens, and the longest has 232. Here are three tiny made-up tickets, cut into tokens: lower-case words, numbers and single punctuation marks. re.findall() finds every piece of text that matches the pattern.

forward.py

import re
import torch

def tokenize(text):
    """Lower-case words, numbers and single punctuation marks."""
    return re.findall(r"[a-z]+|[0-9]+|[^\sa-z0-9]", text.lower())

tiny = ["Parcel not arrived.", "Card charged twice.", "Parcel arrived broken, broken lamp."]
for text in tiny:
    print(len(tokenize(text)), tokenize(text))

The output:

4 ['parcel', 'not', 'arrived', '.']
4 ['card', 'charged', 'twice', '.']
7 ['parcel', 'arrived', 'broken', ',', 'broken', 'lamp', '.']

Turn each token into its position in a sorted list, and try to make one tensor:

tiny_vocab = sorted({token for text in tiny for token in tokenize(text)})
ids = [[tiny_vocab.index(token) for token in tokenize(text)] for text in tiny]
print(ids)
print(torch.tensor(ids))

The output:

[[8, 7, 2, 1], [4, 5, 9, 1], [8, 2, 3, 0, 3, 6, 1]]
ValueError: expected sequence of length 4 at dim 1 (got 7)

A tensor is a box: every row must have the same length. Rows of 4, 4 and 7 numbers do not make a box, so PyTorch refuses. The network needs a fixed number of inputs for every ticket, whatever its length.

The idea: a vocabulary and a bag of words

Use the survey form. First, fix a list of tokens, the vocabulary. Then give every ticket the same form: one box for each vocabulary entry, with the count of that token in the ticket. Every ticket gets a vector of the same length, the size of the vocabulary. This is a bag of words. The name describes what happens: you tip the words of a ticket into a bag and count them, so their order is lost.

A bag of words for one tiny ticket. The vocabulary fixes the length of the vector; the order of the words is lost.

Text description of the diagram

The diagram has three steps from top to bottom. Step 1 is the ticket: “Parcel arrived broken, broken lamp.” Step 2 shows its 7 tokens in the order of the text: parcel, arrived, broken, a comma, broken, lamp and a full stop. Step 3 shows the counts, one box for each of the 10 vocabulary entries in sorted order. The counts are comma 1, full stop 1, arrived 1, broken 2, card 0, charged 0, lamp 1, not 0, parcel 1 and twice 0. The boxes with a count above 0 are shaded. Below: shape (10,), the same length for every ticket, short or long.

In code, counts() fills one form for one text:

print(tiny_vocab)

def counts(text, vocab):
    """One number per vocabulary entry: how often it appears in the text."""
    x = torch.zeros(len(vocab))
    for token in tokenize(text):
        x[vocab.index(token)] += 1
    return x

print(torch.stack([counts(text, tiny_vocab) for text in tiny]))

The output:

[',', '.', 'arrived', 'broken', 'card', 'charged', 'lamp', 'not', 'parcel', 'twice']
tensor([[0., 1., 1., 0., 0., 0., 0., 1., 1., 0.],
        [0., 1., 0., 0., 1., 1., 0., 0., 0., 1.],
        [1., 1., 1., 2., 0., 0., 1., 0., 1., 0.]])

torch.stack() puts the three vectors one under the other, so the three tickets become a batch of shape (3, 10). The third row has a 2 for “broken”. torch.zeros() makes the counts float32 from the start, so they can meet the network’s weights.

The Larkfield tickets

Now the real tickets. They are made up for this course by a language model, from a list of situations; no real customer wrote them. load_split() from the course’s ticketnet.py reads a CSV file into a list of rows, one dictionary for each ticket.

from collections import Counter
from ticketnet import load_split

train_rows = load_split("data/train.csv")
valid_rows = load_split("data/valid.csv")
print(len(train_rows), "training tickets,", len(valid_rows), "validation tickets")

The output:

4696 training tickets, 600 validation tickets

The vocabulary comes from the training tickets only, as every step that learns from data does. A token must appear at least twice to get a place: a token seen once tells the network little and makes every vector longer. Two special entries come first. <unk>, at position 1, counts every token that is not in the vocabulary, such as an order number. <pad>, at position 0, is for Module 5, where tickets become lists of token IDs.

word_counts = Counter(token for row in train_rows for token in tokenize(row["text"]))
itos = ["<pad>", "<unk>"] + sorted(t for t, c in word_counts.items() if c >= 2)   # position -> token
stoi = {token: i for i, token in enumerate(itos)}                                 # token -> position
print(len(word_counts), "different tokens in train,", len(itos), "in the vocabulary")
print(itos[:4], itos[1000:1004])

The output:

4964 different tokens in train, 2864 in the vocabulary
['<pad>', '<unk>', '!', '$'] ['explain', 'explained', 'explaining', 'explains']

itos (“integer to string”) turns a position into a token, and the dictionary stoi turns a token into its position. Now bag() fills the form for one ticket. It keeps the first 120 tokens, as ticketnet.py does; about 95% of the training tickets are shorter.

def bag(text):
    """The counts of one ticket: shape (vocabulary size,)."""
    x = torch.zeros(len(itos))
    for token in tokenize(text)[:120]:     # the first 120 tokens, as in ticketnet.py
        x[stoi.get(token, 1)] += 1          # 1 is <unk>: a token not in the vocabulary
    return x

ticket = valid_rows[0]
x = bag(ticket["text"])
print(ticket["ticket_id"], ticket["team"], "|", len(tokenize(ticket["text"])), "tokens")
print("shape", tuple(x.shape), "| non-zero entries", int((x > 0).sum()), "| sum", int(x.sum()))
for i in x.argsort(descending=True)[:5].tolist():
    print(f"  {itos[i]!r:8} position {i:4}  count {int(x[i])}")

The output:

T-64697 payment | 92 tokens
shape (2864,) | non-zero entries 70 | sum 92
  'the'    position 2540  count 7
  '.'      position   10  count 4
  'i'      position 1307  count 4
  ','      position    8  count 4
  'and'    position  336  count 3

Ticket T-64697 asks about a card charge that is higher than the listed price. Its 92 tokens fill 70 of the 2,864 boxes, so 2,794 boxes, 97.6%, are 0. The sum is 92: every token is counted once. Eight tickets stacked together make the batch:

x8 = torch.stack([bag(row["text"]) for row in valid_rows[:8]])
print(x8.shape)

The output:

torch.Size([8, 2864])

The network

The network has three steps, the layers of Module 1:

• nn.Linear(2864, 64) is a layer of 64 neurons. Each neuron reads all 2,864 counts and computes a weighted sum plus a bias. So the layer turns 2,864 numbers into 64.
• nn.ReLU() sets every negative number to 0, as in Why layers need activations.
• nn.Linear(64, 5) turns the 64 numbers into 5 scores, one for each team. A higher score means “more likely this team”.

nn.Sequential chains layers: the output of one layer is the input of the next. The seed makes the random starting weights the same as on this page.

from torch import nn

TEAMS = ["delivery", "returns", "payment", "warranty", "account"]
torch.manual_seed(0)
model = nn.Sequential(
    nn.Linear(len(itos), 64),   # 2,864 counts in, 64 numbers out
    nn.ReLU(),
    nn.Linear(64, len(TEAMS)),  # 64 numbers in, 5 scores out
)
print(model)

The output:

Sequential(
  (0): Linear(in_features=2864, out_features=64, bias=True)
  (1): ReLU()
  (2): Linear(in_features=64, out_features=5, bias=True)
)

The weights and biases are the network’s parameters: the numbers that training will change. Count them:

for name, p in model.named_parameters():
    print(f"{name:<9} shape {str(tuple(p.shape)):<11} {p.numel():>7,} numbers")
print("in all", f"{sum(p.numel() for p in model.parameters()):,}")

The output:

0.weight  shape (64, 2864)  183,296 numbers
0.bias    shape (64,)            64 numbers
2.weight  shape (5, 64)         320 numbers
2.bias    shape (5,)              5 numbers
in all 183,685

Check it by hand: 2,864 × 64 + 64 = 183,360, and 64 × 5 + 5 = 325, so 183,685 in all. Note the order of the weight’s shape: PyTorch stores it as (out, in), (64, 2864). A layer computes x @ weight.T + bias. So (8, 2864) @ (2864, 64) gives (8, 64), and the bias of shape (64,) is broadcast to every ticket.

One more step comes first. A long ticket repeats common tokens: “the” appears 7 times in T-64697. Large counts would push the weighted sums too far. torch.log1p() computes log(1 + x), which softens large counts and keeps 0 at 0:

print(torch.log1p(torch.tensor([0.0, 1.0, 2.0, 5.0, 20.0])))

The output:

tensor([0.0000, 0.6931, 1.0986, 1.7918, 3.0445])

The forward pass, shape by shape

Sending inputs through the layers, from the first to the last, is the forward pass. The name says the direction: the numbers flow forward, from the counts to the scores. Module 3 adds the backward pass, which training needs. Read the shapes first, then run each step and print its shape:

first, relu, last = model
h = torch.log1p(x8);  print("input     ", tuple(h.shape))
h = first(h);         print("Linear    ", tuple(h.shape))
h = relu(h);          print("ReLU      ", tuple(h.shape))
scores = last(h);     print("Linear    ", tuple(scores.shape))

The output:

input      (8, 2864)
Linear     (8, 64)
ReLU       (8, 64)
Linear     (8, 5)

The bag-of-words network with the shape on every edge, for a batch of 8 tickets. The batch dimension, 8, never changes; each layer changes only the last dimension.

Text description of the diagram

The diagram is a chain of boxes. It starts with 8 tickets as text, a list of 8. The next box, bag of words, gives 2,864 counts for each ticket: the arrow shows (8, 2864). The log1p box softens large counts; its output is still (8, 2864). The first Linear layer, 2,864 to 64, has a weight of shape (64, 2864) and a bias of shape (64,): 183,360 parameters. Its output is (8, 64). ReLU sets negatives to 0; its output is still (8, 64). The second Linear layer, 64 to 5, has a weight of shape (5, 64) and a bias of shape (5,): 325 parameters. Its output is (8, 5), 5 scores for each of the 8 tickets. A note says: 183,685 parameters in all, all random before training. A legend explains the numbers: 8 tickets in the batch, 2,864 vocabulary entries, 64 hidden numbers, 5 teams.

Each number of the output shape (8, 5) has a meaning. 8 is the batch, one row for each ticket. 5 is the teams, one score for each. The batch dimension never changes. Each layer changes only the last dimension, from its in-size to its out-size.

What the untrained network says

The weights are still random, so the scores mean nothing yet. Here are the 5 scores of T-64697, and the team with the highest score for each of the 8 tickets. argmax(dim=1) finds the position of the highest score in each row.

print(scores[0])
pred = scores.argmax(dim=1)
print([TEAMS[i] for i in pred.tolist()])
print([row["team"] for row in valid_rows[:8]])

The output:

tensor([-0.1288,  0.0228,  0.1072,  0.0242,  0.0647],
       grad_fn=<SelectBackward0>)
['payment', 'account', 'account', 'account', 'account', 'account', 'account', 'account']
['payment', 'delivery', 'delivery', 'delivery', 'payment', 'warranty', 'delivery', 'warranty']

grad_fn is a note that PyTorch keeps for training; Module 3 uses it. The 5 scores are close together, from −0.129 to 0.107. Payment has the highest score, 0.107, and T-64697 really is a payment ticket. That is luck. The network says “account” for the next 7 tickets, and none of them is an account ticket.

On all 600 validation tickets, torch.no_grad() tells PyTorch not to keep those notes, which saves memory:

x_valid = torch.stack([bag(row["text"]) for row in valid_rows])
y_valid = torch.tensor([TEAMS.index(row["team"]) for row in valid_rows])
with torch.no_grad():
    pred = model(torch.log1p(x_valid)).argmax(dim=1)
print(tuple(x_valid.shape), "->", tuple(pred.shape))
print("accuracy", round((pred == y_valid).float().mean().item(), 3))
print("predicted:", Counter(TEAMS[i] for i in pred.tolist()).most_common())

The output:

(600, 2864) -> (600,)
accuracy 0.182
predicted: [('account', 438), ('payment', 156), ('returns', 4), ('warranty', 2)]

Where the untrained network sends the 600 validation tickets. The real teams have 88 to 165 tickets each; the network sends 438 to account and none to delivery.

Text description of the chart

A column chart with one column for each team, on an axis from 0 to 500 tickets. The untrained network sends 0 tickets to delivery, 4 to returns, 156 to payment, 2 to warranty and 438 to account. The real validation set has 165 delivery tickets, 120 returns, 126 payment, 101 warranty and 88 account tickets.

The untrained network is right on 18.2% of the tickets. It gets 70 of the 88 account tickets and 39 of the 126 payment tickets, because it sends most tickets there. That is close to 20%, the chance of a blind guess among 5 teams. It is below the 27.5% of always saying “delivery”, the most common team. Random weights do not guess evenly: they happen to favour one or two teams. In Module 3, training changes the 183,685 parameters until the scores mean something. Module 3 also turns the scores into probabilities.

The same code in ticketnet.py

You wrote every piece yourself. The course’s ticketnet.py holds the same code, so that every later lesson uses the same numbers. Vocab builds the vocabulary, bag_of_words() makes the batch, and BagOfWordsNet is the network, with log1p inside. It has one more layer, dropout, which Module 4 explains; here it does nothing. Check that everything is the same:

import ticketnet

vocab = ticketnet.Vocab([row["text"] for row in train_rows])
print(len(vocab), vocab.itos == itos)
ticketnet.set_seed(0)
net = ticketnet.BagOfWordsNet(len(vocab))
x_same = ticketnet.bag_of_words(valid_rows[:8], vocab)
print(torch.equal(x_same, x8), torch.allclose(net(x_same), scores))

The output:

2864 True
True True

The same vocabulary, the same counts, and the same scores.

Shape errors in the network

Predict first, then try each mistake in the explorer. All the messages are real.

Explore

Shapes through the ticket network

Shape of x8

(8, 2864)

Shape of model[0].weight

(64, 2864)

Result

(8, 64)

Result: shape (8, 64). nn.Linear(2864, 64) reads 2,864 numbers per ticket and gives 64. It keeps the batch dimension: 8 tickets in, 8 rows out. It stores its weight as (out, in) = (64, 2864) and computes x8 @ weight.T + bias.

x8

(8, 2864) float32

8 × 2,864

model[0].weight

(64, 2864) float32

64 × 2,864

→

result

(8, 64) float32

8 × 64

PyTorch prints

>>> (model[0](torch.log1p(x8))).shape
torch.Size([8, 64])

Real shapes and error messages from PyTorch 2.14.1 on the CPU, for the untrained bag-of-words network of this lesson.

Text description of the explorer

The explorer has one row of buttons. First layer: x8 of shape (8, 2864) into nn.Linear(2864, 64), whose weight has shape (64, 2864), gives (8, 64). Whole network: (8, 2864) gives (8, 5). One ticket, no batch: shape (2864,) gives (5,), which works but has lost the batch dimension. Transposed batch: shape (2864, 8) stops with “mat1 and mat2 shapes cannot be multiplied (2864x8 and 2864x64)”. New vocabulary: counts from a vocabulary with every token, shape (8, 4966), stop with “mat1 and mat2 shapes cannot be multiplied (8x4966 and 2864x64)”. Second layer too small: the second layer is nn.Linear(32, 5), and the network stops with “mat1 and mat2 shapes cannot be multiplied (8x64 and 32x5)”.

The one-ticket case is the quiet one. The result (5,) works, but the next line fails: argmax(dim=1) on a tensor with one dimension stops with IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1). Add the batch dimension back with .unsqueeze(0), which gives (1, 2864) and then (1, 5).

When to use a bag of words, and what it costs

A bag of words is simple, fast and has a fixed length, and it works well when the words decide the team. The TF-IDF features of the machine learning course are a bag of words too, with weights, and they route these tickets very well. The costs are real, though:

• The order is lost. Take two made-up tickets: “damaged on arrival, not after use” and “damaged after use, not on arrival”. They have the same tokens, so they get the same counts, and the same scores. Module 5 shows where this fails on real tickets.
• The vocabulary is fixed. A new product name that was not in the training tickets only counts as <unk>.
• The vectors are mostly zeros, 97.6% for T-64697, and the first layer holds 183,360 of the 183,685 parameters. A larger vocabulary makes the first layer larger.

Go deeper

The optional deep dive From counts to meaning writes the bag of words, cosine similarity and TF-IDF as formulas, and measures what each one fixes.

Guided practice: the forward pass in the notebook

1. Open the notebook from the box Do this lesson in a notebook at the top of this page, and run the setup cells. The data cell ends with Ready: train 4696 tickets, valid 600, test 600.
2. In section 1, run the tiny tickets. You see the ValueError, then the (3, 10) tensor of counts.
3. In section 2, build the vocabulary. You see 2864 in the vocabulary.
4. Run bag() on T-64697. You see non-zero entries 70 | sum 92.
5. In section 3, build the network with seed 0, and print the shape after every layer. You see (8, 2864), (8, 64), (8, 64) and (8, 5).
6. In the Your turn cell, compute n_parameters, the number of parameters of model. Then run the check cell. It prints Check passed.
7. Compute the accuracy on the validation tickets. You see accuracy 0.182.

Show an example answer to step 6

n_parameters = sum(p.numel() for p in model.parameters())
print(n_parameters)

183685

2,864 × 64 weights and 64 biases in the first layer, 64 × 5 weights and 5 biases in the second.

Your turn: a wider hidden layer

Change one thing: the hidden layer gets 128 neurons instead of 64. Before you run it, write down the output shape of the first layer, the output shape of the network, and the number of parameters.

Show the answer

torch.manual_seed(0)
wide = nn.Sequential(nn.Linear(len(itos), 128), nn.ReLU(), nn.Linear(128, len(TEAMS)))
print(wide(torch.log1p(x8)).shape, wide[0](torch.log1p(x8)).shape)
print(f"{sum(p.numel() for p in wide.parameters()):,}")

torch.Size([8, 5]) torch.Size([8, 128])
367,365

The first layer now gives (8, 128), but the network still gives (8, 5): one row for each ticket and one score for each team. The parameters almost double: 2,864 × 128 + 128 = 366,720, and 128 × 5 + 5 = 645. The hidden size changes only the inside of the network, and its cost.

Common mistake

"PyTorch will figure out the shapes"

Symptom: Tomás wants the network to know every word, also the rare ones. He builds the vocabulary with every token of the training tickets, without the “at least twice” rule. He makes the counts in the same way, and sends them to the network:

itos_all = ["<pad>", "<unk>"] + sorted(word_counts)          # every token, also the rare ones
stoi_all = {token: i for i, token in enumerate(itos_all)}
x8_all = torch.zeros(8, len(itos_all))
for row_i, row in enumerate(valid_rows[:8]):
    for token in tokenize(row["text"])[:120]:
        x8_all[row_i, stoi_all.get(token, 1)] += 1
print(x8_all.shape)
print(model(torch.log1p(x8_all)).shape)

torch.Size([8, 4966])
RuntimeError: mat1 and mat2 shapes cannot be multiplied (8x4966 and 2864x64)

“PyTorch will figure out the shapes,” he says. It does not.

Cause: read the message as in Tensors and shapes. mat1 is the input: 8 tickets with 4,966 counts each. mat2 is the first layer’s weight, transposed: it expects 2,864. The size of the vocabulary is the in-size of the first layer. A new vocabulary makes a different input, and the old network cannot read it. Even if the sizes matched by chance, box 1,000 would mean another token, and the network would read nonsense.

Fix: build the first layer from the vocabulary, never from a number that you typed: nn.Linear(len(itos_all), 64). Then the new network reads the new counts:

torch.manual_seed(0)
model_all = nn.Sequential(nn.Linear(len(itos_all), 64), nn.ReLU(), nn.Linear(64, len(TEAMS)))
print(model_all(torch.log1p(x8_all)).shape)

torch.Size([8, 5])

What it means for a real AI system: the vocabulary is part of the model. A network must always get counts from the vocabulary that it was built and trained with. Amira must save the vocabulary next to the trained network, as Module 7 shows. A new vocabulary means a new network, and new training.

Knowledge check

Check your understanding

Why can a network not read the token lists of tickets directly, and how does a bag of words solve this? What does it lose?

Show the answer

A layer such as nn.Linear(2864, 64) needs the same number of inputs for every ticket, and a tensor must have rows of the same length. Tickets have different numbers of tokens (4, 4 and 7 in the tiny example), so their lists do not make a tensor. A bag of words fixes a vocabulary and gives every ticket one count for each entry. So every ticket becomes a vector of the vocabulary’s size, 2,864 for Larkfield. It loses the order of the words: two tickets with the same words in a different order get the same vector.

Check your understanding

Before training, Omar sees that the network is right on 18.2% of the validation tickets. He asks: is it slightly worse than guessing? Explain what the number means.

Show the answer

The number means only that the network has not learned anything yet. Its weights are random, so its scores are random too. A blind guess among 5 teams is right about 20% of the time, so 18.2% is chance level. Random weights do not spread their answers evenly: this network sends 438 of the 600 tickets to account. So it is right on most account tickets and on no delivery ticket. It is also below 27.5%, the share of the most common team, delivery. Training in Module 3 changes the weights until the scores mean something.

Module practice

In the module practice, you run a forward pass through a tiny network and inspect every intermediate shape. First follow the worked example, then do it on your own.

Worked example: a tiny network

Priya’s three tickets of Tensors and shapes go through a network with 4 inputs, 3 hidden neurons and 2 outputs. The loop sends the batch through one layer at a time, and prints the shape and the first ticket’s numbers after each one.

batch = torch.tensor([[2.0, 2.0, 0.0, 1.0],
                      [1.0, 0.0, 3.0, 0.0],
                      [3.0, 1.0, 1.0, 1.0]])
torch.manual_seed(0)
tiny_net = nn.Sequential(nn.Linear(4, 3), nn.ReLU(), nn.Linear(3, 2))
h = batch
for layer in tiny_net:
    h = layer(h)
    print(f"{layer.__class__.__name__:<6} {tuple(h.shape)}  {h[0].detach().numpy().round(3)}")

The output:

Linear (3, 3)  [-0.317 -0.052 -0.129]
ReLU   (3, 3)  [0. 0. 0.]
Linear (3, 2)  [ 0.479 -0.119]

h[0].detach() drops the training note (grad_fn) so that .numpy() can turn the first ticket’s numbers into a NumPy array, and .round(3) keeps three decimals. Read each line. The first layer turns the 4 features into 3 numbers per ticket. For the first ticket, all 3 are negative, so ReLU makes them 0. The last layer still gives 2 scores, 0.479 and −0.119: with only zeros coming in, these are its 2 biases. The batch dimension, 3, stays the same in every line.

Your task

Make a network with 4 inputs, 6 hidden neurons and 2 outputs, with seed 0. Send a batch of 5 tickets through it, five = torch.ones(5, 4). Before you run it, write down the shape after each layer and the number of parameters. Then run it and compare.

Show the model answer

torch.manual_seed(0)
task_net = nn.Sequential(nn.Linear(4, 6), nn.ReLU(), nn.Linear(6, 2))
five = torch.ones(5, 4)
h = five
for layer in task_net:
    h = layer(h)
    print(f"{layer.__class__.__name__:<6} {tuple(h.shape)}")
print("parameters", sum(p.numel() for p in task_net.parameters()))

Linear (5, 6)
ReLU   (5, 6)
Linear (5, 2)
parameters 44

The batch, 5, stays in the first place. The first layer changes the last dimension from 4 to 6, ReLU keeps the shape, and the last layer changes it from 6 to 2. The parameters are 4 × 6 + 6 = 30 in the first layer and 6 × 2 + 2 = 14 in the second: 44 in all.

Module check

Check your understanding

Retrieval: what is the shape of a tensor, and what is the broadcasting rule? Give one example that works and one that fails.

Show the answer

The shape lists the size of each dimension, the outer dimension first: a batch of 3 tickets with 4 features has shape (3, 4). Broadcasting compares two shapes from the right. Two sizes fit when they are equal, when one is 1, or when one is missing; a 1 or a missing size is stretched. (3, 4) − (4,) works and gives (3, 4): the row is used for every ticket. (3, 4) ÷ (3,) fails: from the right, 4 meets 3, they differ and neither is 1.

Check your understanding

Predict: a batch of 32 tickets goes through the bag-of-words network with a hidden layer of 64. Give the shape after the bag of words, after each layer, and of one ticket's scores.

Show the answer

After the bag of words: (32, 2864), one count for each vocabulary entry. After log1p: (32, 2864). After the first Linear layer: (32, 64). After ReLU: (32, 64). After the second Linear layer: (32, 5). One ticket’s scores, scores[0], have the shape (5,), or (1, 5) with scores[0:1]. Only the batch dimension changed from the lesson’s 8 to 32; the layers did not change.

Check your understanding

Completion check: explain each dimension of the output shape (8, 5). Then find and fix the mistake in this network, which stops with: mat1 and mat2 shapes cannot be multiplied (5x6 and 5x2). The code is: broken = nn.Sequential(nn.Linear(4, 6), nn.ReLU(), nn.Linear(5, 2)); broken(five).

Show the answer

The output (8, 5): 8 is the batch dimension, one row for each of the 8 tickets. 5 is the last dimension, one score for each team (delivery, returns, payment, warranty, account). scores[i, j] is ticket i’s score for team j.

The mistake: mat1, (5 × 6), is what reaches the second Linear layer: 5 tickets with 6 numbers each, from the first layer. mat2, (5 × 2), is that layer’s transposed weight: it expects 5 numbers per ticket. The first layer’s out-size, 6, is not the second layer’s in-size, 5.

The fix: the in-size of each layer must equal the out-size of the layer before it. Write nn.Linear(6, 2) as the last layer:

fixed = nn.Sequential(nn.Linear(4, 6), nn.ReLU(), nn.Linear(6, 2))
print(fixed(five).shape)

torch.Size([5, 2])

A complete answer names both dimensions of the output, reads both shapes in the message, and says which pair must be equal.

Recap

In this module, you learned to read and control the shapes of a network:

• A tensor is a box of numbers with a shape; a network’s tensors are usually (batch, features). Broadcasting stretches a size of 1 or a missing dimension, and it can do so by accident, with no error.
• A known-good first run with a seed proves your setup: torch.manual_seed(0) then torch.rand(2, 3) sums to 2.4265.
• A bag of words turns every ticket into a vector of the vocabulary’s size: 2,864 counts from the training tickets. The order of the words is lost.
• The forward pass of the untrained network: (8, 2864) → Linear → (8, 64) → ReLU → (8, 64) → Linear → (8, 5), with 183,685 random parameters. It is right on 18.2% of the validation tickets: chance level.
• A shape error names two shapes: find the pair that must be equal. Build each layer’s size from the data (len(vocab)) and from the layer before it.

Reset: in the notebook, select Runtime › Disconnect and delete runtime in Colab, then run all cells from the top. On your computer, run python get_data.py --reset for the data and ticketnet.py, or delete .venv and make it again as in Reset the setup. Then run python first_run.py and check sum 2.4265.

Next, in Module 3, you measure how wrong these scores are, and start to train the network.

Next: Loss: how wrong is the network?