How embeddings learn meaning: word2vec and GloVe

What you will be able to do

See how a network can learn word vectors from raw text alone, with no labels. You follow word2vec’s skip-gram step by step, then GloVe and a plain matrix factorisation. All three fit the same table of counts. Then you test pretrained vectors, including where they fail and the bias they carry.

NotebookSkip-gram, PPMI and pretrained GloVe, in a notebookColab · Kaggle · GitHub

The notebook builds the (centre, context) pairs of the tickets and writes skip-gram in about 15 lines. It trains it for 3 epochs, in under a minute on a free Colab CPU. Then it computes PPMI and an SVD of the co-occurrence matrix. Last, it loads a small subset of pretrained GloVe and tries analogies, including the ones that fail. About 35 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Embeddings and nearest words, from Tokens and embeddings.
• Cross-entropy as “the surprise at the right answer”, −ln p, from Loss.
• Gradients and one step downhill, from Loss and gradients in the mathematics course.
• The dot product and cosine similarity, from Vector operations and Similarity and distance in the mathematics course.
• A probability inside a group, from Conditional probability in the mathematics course.

What you need on your computer

• A browser and a calculator. The notebook opens in Colab or Kaggle with PyTorch already installed.

This lesson is optional: the certificate does not require it.

A word is known by the company it keeps

You saw the idea in Tokens and embeddings: each token gets a row of learned numbers, and words that act alike get rows that point the same way. There, the rows learned from the teams: they changed only to route tickets better. From counts to meaning ended with Firth’s sentence, “You shall know a word by the company it keeps”: the distributional hypothesis. This page shows the mathematics that turns that sentence into vectors.

The question this page answers

Can we learn word vectors from the text itself, without any labels? Text tells us one thing for free: which words appear near which other words. Count this for Larkfield’s 4,696 training tickets, with a window of 5 tokens on each side. The result is a table with one row and one column for each of the 2,862 tokens of the vocabulary: 8.2 million cells. Only 3.94% of them are not zero. Most pairs of words never meet.

The raw counts have a second problem. “refund” and “delivery” meet 25 times, but only because both are frequent. “password” and “reset” meet 127 times, and that pair really belongs together. A useful vector must separate these two cases.

This page shows three answers. Skip-gram (word2vec) learns to predict each word’s neighbours. GloVe fits the logarithm of the counts directly. SVD factorises a cleaned-up version of the counts, with no neural network at all. At the end, you see that the three answers are close relatives.

Words as points: latent dimensions

All three methods give each word a vector of 50 numbers. Think of it as a point in a space with 50 directions. Nobody chooses what the directions mean. Training places the points so that words used in the same contexts end up near each other. That is why the directions are called latent: they are hidden and learned, not designed.

Three facts follow, and the rest of the page shows each one with real numbers:

• Near means “used alike”. In the skip-gram vectors trained below, “password” and “reset” have a cosine similarity of 0.95.
• Directions can carry relations. In pretrained vectors, the step from “man” to “woman” points roughly the same way as the step from “king” to “queen”.
• A single dimension is usually not readable. The first 6 of the 50 numbers of “password” are −0.41, 0.31, 0.30, 0.16, 0.42, −0.77. No number on its own means “account” or “secret”. Only the whole vector, compared with other vectors, carries meaning.

Step 1: the window turns text into pairs

Skip-gram reads the text with a sliding window. Each token in turn is the centre word c. Every token within 5 positions of it, on either side, is a context word o. Each (c, o) is one training example. The window never crosses from one ticket into the next.

window.py

from ticketnet import tokenize

tokens = tokenize("Can't log in and the password reset email never arrived. Please help.")
WINDOW = 5
c = tokens.index("password")
context = tokens[max(0, c - WINDOW):c] + tokens[c + 1:c + 1 + WINDOW]
print(len(tokens), "tokens; centre:", tokens[c])
print("context:", context)

The output:

16 tokens; centre: password
context: ['t', 'log', 'in', 'and', 'the', 'reset', 'email', 'never', 'arrived', '.']

This short ticket gives 130 (centre, context) pairs; all the training tickets give about 1.36 million per pass. No labels, only the text.

The word2vec paper also subsamples frequent words: in each pass it drops a token w at random with probability $1 - \sqrt{t / f(w)}$. Here f(w) is the token’s share of all tokens and t = 0.001. “the” makes up 5.4% of the tokens, so it stays in only about 14% of the passes. A rare word always stays. This removes many pairs like (password, the), which say little.

Explore

The training pairs of one ticket

Ticket T-60140 (account): “Can't log in and the password reset email never arrived. Please help.”

• centre word
• context window (5 tokens each side)

can'tloginandthepasswordresetemailneverarrived.pleasehelp.

Centre word

password

True pairs

10

Surest context word

reset, 0.96

Kept by subsampling

always

Centre password: 10 true pairs from this ticket. The trained model is surest about reset (0.96) and least sure about the (0.10). Its 5 random negatives get 0.12 on average. One negative is also a real context word here: random draws can do that, and training copes with the noise.

True pairs: (password, context word). Training pulls these up toward 1.

Context word	Bar	σ(u·v)
t		0.22
log		0.77
in		0.35
and		0.16
the		0.10
reset		0.96
email		0.59
never		0.86
arrived		0.20
“.”		0.10

5 random negatives for one pair. Training pushes these down toward 0.

Random word	Bar	σ(u·v)
breeze drawn 0.008%		0.01
gone drawn 0.052%		0.08
“'” drawn 1.20%		0.14
thanks drawn 0.34%		0.15
arrived drawn 0.33%		0.20

Real ticket T-60140 from the training set. The probabilities σ(u·v) come from the skip-gram model trained on all the tickets; the negatives are one real draw from the unigram^0.75 distribution.

Text description of the explorer

The explorer shows ticket T-60140, “Can’t log in and the password reset email never arrived. Please help.”, as 16 token buttons. One token is the centre word; the 5 tokens on each side form the context window. For the centre word, a table lists every context word with the trained model’s probability σ(u·v) that it appears near the centre word, and a second table lists 5 random negative words with their probabilities. For the centre word password, the context probabilities are: reset 0.96, never 0.86, log 0.77, email 0.59, in 0.35, the letter t 0.22, arrived 0.20, and 0.16, the full stop 0.11 and the 0.10. The 5 negatives are breeze 0.01, gone 0.08, the apostrophe 0.14, thanks 0.15 and arrived 0.20. Subsampling keeps password in every pass. For the centre word the, subsampling keeps it in only 14% of the passes.

“reset”, “never” and “log” get high probabilities: they appear near “password” in many tickets. “the” and “.” get low ones, although they are in the window: they appear near every word, so they say nothing about “password”.

Step 2: two vectors per word and one probability

Skip-gram gives every word two vectors of 50 numbers. $v_w$ is the word’s vector when it is the centre word, and $u_w$ is its vector when it is a context word. They live in two tables, V and U, each 2,862 × 50. The model’s score for a pair is the dot product $u_o \cdot v_c$. The sigmoid σ turns the score into a probability that o really appears near c:

$$P(\text{o appears near c}) = \sigma(u_o \cdot v_c) = \frac{1}{1 + e^{-u_o \cdot v_c}}$$

In words: the more the two vectors point the same way, the larger the dot product, and the closer the probability is to 1. A score of 0 gives exactly 0.5.

Why two tables? A word rarely appears next to itself. With one table, the model would need a large $v_a \cdot v_b$ for two similar words and a small $v_a \cdot v_a$ for the word with itself. That is hard: a vector’s dot product with itself is its squared length, which is never small for a useful vector. Two tables avoid this. After training, word2vec keeps V as the word vectors.

A tiny example. Take $v_c = (0.5, 0.5)$ for “password” and $u_o = (1, 0)$ for “reset”. The score is 0.5 × 1 + 0.5 × 0 = 0.5, and σ(0.5) = 1 ÷ (1 + e^−0.5) = 1 ÷ 1.607 = 0.62.

Step 3: the loss, with negative sampling

Training should push σ(u_o·v_c) toward 1 for every true pair. But if that were the only goal, the model would cheat: make every vector huge and identical, and every score becomes large. It needs counter-examples. Negative sampling draws K random words for each true pair and teaches the model that those are not context words. We use K = 5.

For one true pair (c, o) and K negatives $k_1, \dots, k_K$, the loss is:

$$\ell = -\ln \sigma(u_o \cdot v_c) \;-\; \sum_{i=1}^{K} \ln \sigma(-\,u_{k_i} \cdot v_c)$$

The first term is the cross-entropy you know from Loss, with the right answer “yes, a neighbour”. The second term uses σ(−x) = 1 − σ(x), the probability of “no”. So each negative adds “the surprise if this random word were called a neighbour”. The loss is small when true pairs score high and random pairs score low.

Where do the negatives come from? From the word counts raised to the power 0.75, then scaled to add up to 1:

$$P_{\text{noise}}(w) = \frac{\text{count}(w)^{0.75}}{\sum_{w'} \text{count}(w')^{0.75}}$$

Why 0.75? Plain counts (the unigram distribution) would draw “the” and “.” very often, and the model would learn little from them. Equal chances would draw rare words, such as typos, far too often. The power 0.75 sits between the two: it lowers the frequent words and raises the rare ones. Mikolov and colleagues found that 0.75 worked best in their tests. It is a tuned choice, not a law.

noise.py

import numpy as np

# counts: how often each of the 2,862 tokens appears in the training tickets
p_unigram = counts / counts.sum()
p_noise = counts ** 0.75 / (counts ** 0.75).sum()
for w in ["the", "password", "warranty", "umbrella"]:
    i = W2I[w]
    print(f"{w:<9} seen {int(counts[i]):>6,} times   unigram {p_unigram[i]:.5f}   unigram^0.75 {p_noise[i]:.5f}"
          f"   x{p_noise[i] / p_unigram[i]:.2f}")

The output:

the       seen 14,432 times   unigram 0.05403   unigram^0.75 0.02378   x0.44
password  seen    214 times   unigram 0.00080   unigram^0.75 0.00101   x1.26
warranty  seen     53 times   unigram 0.00020   unigram^0.75 0.00035   x1.79
umbrella  seen      5 times   unigram 0.00002   unigram^0.75 0.00006   x3.23

The power more than halves the chance of “the” and triples the chance of “umbrella”. The 10 most frequent tokens fill 30.0% of the negatives with plain counts, and 15.1% with the power 0.75.

The diagram shows one training step for the centre word “password” with its real neighbour “reset” and one real draw of 5 negatives.

One skip-gram step for the centre word “password”. The probabilities are those of the trained model; the negatives are one real draw.

Text description of the diagram

On the left, the centre word password with its row v from table V, the centre vectors. In the middle, six rows of table U, the context vectors: first the true context word reset, then 5 random negatives drawn from the unigram^0.75 distribution: breeze, gone, an apostrophe, thanks and arrived. For each row, the trained model’s probability σ(u·v): reset 0.96, breeze 0.01, gone 0.08, the apostrophe 0.14, thanks 0.15, arrived 0.20. On the right, the target: 1 for reset, so training pulls u and v together; 0 for each negative, so training pushes u and v apart. At the bottom, the loss of this step: minus ln σ(u_reset · v) minus the sum over the 5 negatives of ln σ(−u_k · v). One negative, arrived, is also a real context word in this ticket: a random draw can do that.

Step 4: the gradient pulls and pushes

The gradient of the loss with respect to the centre vector has one part for each term:

$$\frac{\partial \ell}{\partial v_c} = -\big(1 - \sigma(u_o \cdot v_c)\big)\, u_o \;+\; \sum_{i=1}^{K} \sigma(u_{k_i} \cdot v_c)\, u_{k_i}$$

Read it in words. A step downhill subtracts the gradient. So $v_c$ moves toward $u_o$, by the amount the model still misses, 1 − σ. And it moves away from each negative $u_k$, by the probability the model wrongly gives it. A pair the model already gets right barely moves. The vectors u change in the same way: $u_o$ moves toward $v_c$, and each $u_k$ moves away from it. In short: pull the true context closer, push random words away.

Check one step by hand, with made-up 2-D vectors and one negative. Take $v_c = (0.5, 0.5)$, $u_o = (1, 0)$ and $u_k = (0, 1)$. Both scores are 0.5, so both sigmoids are 0.6225.

• Loss: −ln 0.6225 − ln(1 − 0.6225) = 0.474 + 0.974 = 1.448.
• Gradient: −(1 − 0.6225) × (1, 0) + 0.6225 × (0, 1) = (−0.3775, 0.6225).
• One step with learning rate 0.5: (0.5, 0.5) − 0.5 × (−0.3775, 0.6225) = (0.689, 0.189).

The true score rose from 0.5 to 0.689, the negative score fell from 0.5 to 0.189, and the loss fell to 1.199. PyTorch’s autograd agrees:

one_step.py

import torch
import torch.nn.functional as F

v_c = torch.tensor([0.5, 0.5], requires_grad=True)   # centre word "password"
u_o = torch.tensor([1.0, 0.0], requires_grad=True)   # true context word "reset"
u_k = torch.tensor([0.0, 1.0], requires_grad=True)   # one negative, a random word
loss = -F.logsigmoid(u_o @ v_c) - F.logsigmoid(-(u_k @ v_c))
loss.backward()
print("sigma(u_o.v_c) =", round(torch.sigmoid(u_o @ v_c).item(), 4), " loss =", round(loss.item(), 4))
print("gradient for v_c:", [round(g, 4) for g in v_c.grad.tolist()])
with torch.no_grad():
    v_new = v_c - 0.5 * v_c.grad                     # one step, learning rate 0.5
    new_loss = -F.logsigmoid(u_o @ v_new) - F.logsigmoid(-(u_k @ v_new))
print("new v_c:", [round(x, 4) for x in v_new.tolist()], " u_o.v_c:", round((u_o @ v_new).item(), 4),
      " u_k.v_c:", round((u_k @ v_new).item(), 4), " loss:", round(new_loss.item(), 4))

The output:

sigma(u_o.v_c) = 0.6225  loss = 1.4482
gradient for v_c: [-0.3775, 0.6225]
new v_c: [0.6888, 0.1888]  u_o.v_c: 0.6888  u_k.v_c: 0.1888  loss: 1.1989

F.logsigmoid(x) computes ln σ(x) in one step that stays accurate for large scores.

Training on Larkfield’s tickets

The whole model is two embedding tables and the loss above. It is short enough to read in full:

skipgram.py

class SkipGram(torch.nn.Module):
    def __init__(self, n, dim):
        super().__init__()
        self.v = torch.nn.Embedding(n, dim)   # centre words
        self.u = torch.nn.Embedding(n, dim)   # context words
        torch.nn.init.uniform_(self.v.weight, -0.5 / dim, 0.5 / dim)
        torch.nn.init.zeros_(self.u.weight)

    def forward(self, c, o, neg):            # c, o: (B,)  neg: (B, K)
        vc = self.v(c)                                              # (B, d)
        pos = (self.u(o) * vc).sum(-1)                              # (B,)   u_o . v_c
        negs = torch.bmm(self.u(neg), vc.unsqueeze(-1)).squeeze(-1)  # (B, K) u_k . v_c
        loss = -F.logsigmoid(pos) - F.logsigmoid(-negs).sum(-1)
        return loss.mean()

sg = SkipGram(2862, 50)
opt = torch.optim.Adam(sg.parameters(), lr=0.002)
# then 8 epochs: build the pairs (with subsampling), shuffle, batches of 512,
# draw 5 negatives per pair with torch.multinomial(p_noise, ...), and the five training steps

torch.bmm multiplies a batch of matrices: each pair’s 5 negative vectors, (5, 50), by its centre vector, (50, 1). As in word2vec’s original code, V starts with small random numbers and U with zeros.

The output of the training run:

loss before training: 4.1589   6 ln 2 = 4.1589
epoch 1: 1,357,310 pairs, mean loss 2.638
epoch 2: 1,356,890 pairs, mean loss 2.414
epoch 3: 1,360,876 pairs, mean loss 2.363
epoch 4: 1,358,556 pairs, mean loss 2.335
epoch 5: 1,359,294 pairs, mean loss 2.319
epoch 6: 1,360,146 pairs, mean loss 2.308
epoch 7: 1,359,106 pairs, mean loss 2.298
epoch 8: 1,359,596 pairs, mean loss 2.293
20 s

Check the first line by hand. With U all zeros, every score is 0 and every σ is 0.5. Each of the 6 terms of the loss (1 true pair + 5 negatives) is −ln 0.5 = ln 2. So the start is 6 ln 2 = 4.159, exactly. If your first loss is far from this, a sign or a sum is wrong.

Subsampling drops different tokens each time, so the number of pairs changes a little. The 8 epochs take 17–20 seconds on two CPU threads of a shared computer.

The skip-gram loss on Larkfield's 267,129 training tokens. Epoch 0 is the exact start, 6 ln 2; each later point is the mean loss over one epoch.

Text description of the chart

A line chart. At epoch 0, before training, the loss is 4.159, which is 6 ln 2. After epoch 1 the mean loss is 2.638, then 2.414, 2.363, 2.335, 2.319, 2.308, 2.298, and 2.293 after epoch 8. Most of the drop happens in the first epoch; the later epochs gain less and less.

The loss does not go near 0, and it should not: some random negatives are real neighbours, as “arrived” was in the explorer, and many true pairs, such as (password, the), carry little information.

What the vectors learned

Now take the trained table V and ask for the nearest words by cosine similarity, as in Tokens and embeddings.

neighbours.py

Vn = V / np.linalg.norm(V, axis=1, keepdims=True)   # every row to length 1

def nearest(word, k=5):
    sims = Vn @ Vn[W2I[word]]
    order = [i for i in np.argsort(-sims) if i != W2I[word]][:k]
    return [(words[i], round(float(sims[i]), 2)) for i in order]

for w in ["password", "card", "broken", "late"]:
    print(f"{w:<9} {int(counts[W2I[w]]):>5,}x", nearest(w))

The output:

password    214x [('reset', 0.95), ('resetting', 0.8), ('link', 0.79), ('forgot', 0.78), ('rejecting', 0.73)]
card      1,323x [('debit', 0.75), ('paid', 0.73), ('visa', 0.69), ('ending', 0.65), ('42', 0.63)]
broken      219x [('cracked', 0.83), ('faulty', 0.83), ('places', 0.77), ('bent', 0.77), ('join', 0.75)]
late         47x [('umbrella', 0.65), ('rescheduled', 0.63), ('autumn', 0.6), ('excuse', 0.6), ('august', 0.57)]

The first three lists work. “password” sits with “reset”, “link” and “forgot”; “card” with “debit” and “visa”; “broken” with “cracked” and “faulty”. No label told the model any of this. It came from the windows alone. “ending” and “42” come from tickets such as “my card ending in 42”: the model learned how customers write, not a dictionary.

The last list is noise. “late” appears only 47 times, so a handful of tickets decide where its vector lands. Its neighbours “umbrella”, “autumn” and “august” share little with it. This is the honest limit of 267,129 tokens (the tokens in the vocabulary; 269,231 in all): word2vec’s published vectors learned from about 100 billion words. Our vectors learn ticket topics, words that appear in the same kind of ticket. That is useful for routing, but it is not general English meaning.

The map shows the same idea as points. Each word has 50 numbers, so the map uses PCA to cast a 2-D shadow of them, as in Tokens and embeddings. The explorer also shows the two other ticket-trained methods of this page and the pretrained vectors from the end of the page.

Explore

27 ticket words under four sets of vectors

• mostly in delivery tickets
• mostly in returns tickets
• mostly in payment tickets
• mostly in warranty tickets
• mostly in account tickets

Skip-gram, trained on the tickets: the delivery words group at the top, the payment words in the middle, the warranty words at the bottom left, and password, reset and account together at the bottom right, with login close by. "late" sits next to "arrived", with the delivery words. "broken" sits with the warranty words, although more of its tickets are delivery tickets. Same-team score of all 904 frequent words: 0.585 (chance 0.251).

2-D PCA of the unit vectors of 27 chosen ticket words. Colour and shape: the team whose training tickets use the word most. The same-team score covers all 904 frequent words, not only this map.

Text description of the explorer

A scatter plot of 27 words, each marked with the team whose tickets use it most: parcel, courier, delivery, delivered, tracking, arrived, late, broken and login (delivery); return, label and exchange (returns); returned, refund, card, charged, invoice, payment, paid and email (payment); faulty, repair, warranty and stopped (warranty); password, account and reset (account). A control switches between four sets of 50-number word vectors. With skip-gram trained on the tickets, the words form groups by team: delivery words at the top, payment words in the middle, warranty words at the bottom left, and password, reset and account at the bottom right. The same-team score of all 904 frequent words is 0.585. With the small GloVe fit, the groups are looser; score 0.432. With the SVD of shifted PPMI, the groups are similar to skip-gram’s; score 0.492. With pretrained GloVe 6B, the words group by general English meaning rather than by Larkfield team: password stays next to login, but account moves next to payment, refund and paid; score 0.320. Chance is 0.251.

Compare the first and the last map. In the ticket vectors, “password”, “reset” and “account” form one group, because they share tickets. In the pretrained vectors, “password” stays next to “login”. But “account” moves to “payment”, “refund” and “paid”: in general English, an account is often a bank account. The vectors learned the meaning of their own text.

GloVe: fit the counts directly

Skip-gram slides over the text pair by pair. GloVe, from Pennington, Socher and Manning (2014), starts from the finished count table instead. Let $X_{ij}$ be how often word j appears in the window of word i. GloVe counts a pair that is 3 tokens apart as 1/3, and in general a pair r tokens apart as 1/r, so close neighbours count more. Then it fits one equation for every non-zero cell:

$$w_i \cdot \tilde{w}_j + b_i + \tilde{b}_j \approx \ln X_{ij}$$

Here $w_i$ and $\tilde{w}_j$ are two vectors per word, again one for each role. $b_i$ and $\tilde{b}_j$ are one number per word, a bias. The biases absorb how frequent each word is. The dot product is left with the rest: how much more the two words meet than their frequencies explain. GloVe fits all cells at once by weighted least squares:

$$J = \sum_{X_{ij} > 0} f(X_{ij}) \left( w_i \cdot \tilde{w}_j + b_i + \tilde{b}_j - \ln X_{ij} \right)^2 \qquad f(x) = \min\!\left(1, \left(\tfrac{x}{100}\right)^{0.75}\right)$$

The weight f stops two problems. Rare pairs are noisy, so a pair seen once gets only f(1) = 0.032 of the full weight. Very frequent pairs would dominate the sum, so f stops at 1 from 100 co-occurrences on. In between, f(10) = 0.178 and f(50) = 0.595. The final word vector is $w + \tilde{w}$.

One term, from the fit on the tickets. “password” and “reset” have a weighted count of 110.95, so ln X = 4.709 and f = 1. The fitted dot product is 4.173 and the two biases are 0.307 and 0.197: together 4.677. The error is 4.677 − 4.709 = −0.032, and its weighted square adds only 0.001 to J. The fit used the 322,363 non-zero cells and 50 epochs, in 2.2 seconds; J per cell fell from 0.198 to 0.0145.

Now the honest result: on the tickets, our GloVe fit is the weakest of the three methods. Its nearest words to “refund” are “had”, “a”, “got”, “received” and “want”. With x_max = 100, almost every pair in a small corpus is rare and gets a small weight. Frequent function words then stay close to everything. We kept the paper’s settings rather than tune them until GloVe won. GloVe was made for billions of tokens.

One matrix behind both: PMI

Both methods fight the same enemy: raw counts that mostly measure frequency. The standard fix is pointwise mutual information (PMI). It compares how often two words meet with how often they would meet if they were independent:

$$\text{PMI}(i, j) = \ln \frac{P(i, j)}{P(i)\,P(j)} = \ln \frac{X_{ij} \cdot T}{X_i \cdot X_j}$$

Here $X_i$ is the row sum of word i (all its window pairs) and T is the sum of all cells, 2,530,410 pairs. PMI is 0 for independent words, positive when they meet more often than chance, and negative when they meet less often. PPMI = max(PMI, 0) keeps only the positive part.

Work two cells by hand:

• password–reset: $X_{ij} = 127$, $X_i = 2{,}057$, $X_j = 1{,}923$. Then 127 × 2,530,410 ÷ (2,057 × 1,923) = 81.2, and ln 81.2 = 4.40. They meet 81 times more often than chance.
• refund–delivery: 25 × 2,530,410 ÷ (13,286 × 10,848) = 0.44, and ln 0.44 = −0.82. That is less often than chance, so PPMI = 0, although they meet 25 times.

Explore

Counts, PPMI and shifted PPMI on ten ticket words

password, reset

127

refund, card

191

refund, delivery

25

Raw counts favour frequent words: refund (1,358 times in the tickets) and card meet 191 times, and refund meets delivery 25 times, partly because both words are common. Only 3.94% of all word pairs ever meet.

• darker: larger value (relative to the largest cell in view)

How often the row word and the column word appear within 5 tokens of each other

	parcel	courier	delivery	arrived	refund	card	charged	password	reset	faulty
parcel	0	25	12	38	1	3	0	0	0	0
courier	25	0	16	6	5	1	0	0	0	0
delivery	12	16	10	73	25	7	1	2	0	1
arrived	38	6	73	0	37	4	1	11	21	4
refund	1	5	25	37	2	191	2	1	1	2
card	3	1	7	4	191	10	185	0	0	0
charged	0	0	1	1	2	185	0	0	1	0
password	0	0	2	11	1	0	0	2	127	0
reset	0	0	0	21	1	0	1	127	0	0
faulty	0	0	1	4	2	0	0	0	0	0

A 10 × 10 excerpt of the real co-occurrence matrix of the training tickets (window 5 on each side). Shading is relative to the largest cell in the current view.

Text description of the explorer

A heat table of 10 words: parcel, courier, delivery, arrived, refund, card, charged, password, reset and faulty. A control shows the cells as raw counts, as PPMI, or as shifted PPMI. In the counts view, the largest cells are refund with card, 191, card with charged, 185, password with reset, 127, and delivery with arrived, 73; refund with delivery is 25. In the PPMI view, password with reset is the largest, 4.40, then card with charged, 2.38, parcel with courier, 2.02, and refund with card, 1.06; refund with delivery is 0. In the shifted PPMI view, PMI minus ln 5, only three pairs stay above 0: password with reset, 2.79, card with charged, 0.77, and parcel with courier, 0.41.

Skip-gram fits shifted PMI. Levy and Goldberg (2014) proved a surprising result. Suppose the vectors have enough numbers to fit every pair freely. Then the best skip-gram solution with K negatives has

$$v_c \cdot u_o = \text{PMI}(c, o) - \ln K$$

for every pair. So skip-gram quietly factorises the PMI matrix, shifted down by ln K. With K = 5, the shift is ln 5 = 1.61. Our model has only 50 numbers per word, so it can only approximate this. Check it on the tickets: for password–reset, the trained $v \cdot u$ is 3.25 and PMI − ln 5 is 2.79. For 4,949 pairs of frequent words that meet at least 5 times, the two numbers have a Pearson correlation of r = 0.852.

• one pair of words
• v · u = PMI − ln 5, the Levy and Goldberg optimum

Each dot is a pair of frequent ticket words that meet at least 5 times (600 of the 4,949 pairs shown). The line is the Levy and Goldberg optimum. Correlation over all pairs: r = 0.852.

Text description of the chart

A scatter plot of 600 word pairs. The horizontal axis is PMI minus ln 5, from about −4.3 to 4.2. The vertical axis is the trained skip-gram dot product v · u, from about −3.5 to 5.6. The dots follow a rising band around the line v · u = PMI − ln 5. Over all 4,949 pairs the correlation is 0.852, 87.5% of the pairs are within 1 of the line, and the least-squares line is v · u = −0.39 + 0.70 × (PMI − ln 5): a little flatter than the ideal, because 50 numbers per word cannot fit every pair.

A third method: factorise the matrix yourself. If skip-gram approximates the shifted PPMI matrix, we can also factorise that matrix directly. The singular value decomposition (SVD) writes any matrix M as $U S V^\top$: two tables of directions and a list of strengths S, largest first. (These U and V are not skip-gram’s tables.) Keep the 50 strongest directions, and each word gets 50 numbers, $W = U_{50} \sqrt{S_{50}}$. There is no neural network and no learning rate: only one call to np.linalg.svd. The 50 directions keep 14.9% of the matrix’s squared singular values; much of the rest is noise from rare pairs. GloVe is a relative too: it fits the log counts, and its biases do the work of the division in PMI.

Three roads to word vectors on the tickets. All three end at 50 numbers per word, and all three fit the same co-occurrence statistics.

Text description of the diagram

The training tickets, 267,129 tokens, feed two paths. Path one: skip-gram reads the text pair by pair and predicts each word’s neighbours, with the sigmoid and 5 negatives. Path two: count the co-occurrences into a matrix X of 2,862 by 2,862 cells, window 5, of which 3.94% are non-zero. From X, GloVe fits a dot product and two biases to ln X by weighted least squares, and the PMI road turns X into shifted PPMI and factorises it with an SVD to 50 numbers. All three give 50 numbers per word. A dotted link between skip-gram and the PMI road says that skip-gram’s dot products approximate PMI minus ln 5, with a correlation of 0.852 on the tickets.

Which vectors serve Larkfield best?

Nearest-word lists are easy to cherry-pick, so measure instead. For every alphabetic word seen at least 20 times (904 words), find its dominant team: the team whose tickets use it most. Then check what share of its 10 nearest words have the same dominant team. Random vectors would score 0.251, the chance that two words share a dominant team. This same-team score is our own test for this task, not a standard benchmark.

Share of each word's 10 nearest words that belong to its team. The first three methods learned from Larkfield's 267,129 training tokens (SVD: of shifted PPMI); GloVe 6B learned from 6 billion tokens of Wikipedia and news.

Text description of the chart

A column chart. Skip-gram on the tickets scores 0.585, the SVD of shifted PPMI 0.492, the GloVe fit on the tickets 0.432, pretrained GloVe 6B 0.320, and chance 0.251.

The full table adds two more variants and a second test. For that test, average the word vectors of each ticket and train a logistic regression on the averages.

Vectors	same-team score	average vector + logistic regression, validation
skip-gram, tickets	0.585	94.5%
SVD of PPMI (no shift), tickets	0.547	90.5%
rows of ln(1 + counts), 2,862 numbers	0.533	96.2%
SVD of shifted PPMI, tickets	0.492	84.2%
GloVe fit, tickets	0.432	84.0%
pretrained GloVe 6B, 50 numbers	0.320	81.7%

Skip-gram is the best of the 50-number vectors on both tests. Pretrained GloVe, trained on about 22,000 times more text, comes last. General-purpose vectors learn general English: in them, “warranty” is near “payment”, “rental” and “paperwork”, and “late” is near “early”. Larkfield’s teams are a local distinction that Wikipedia never saw. And keep the whole course in view: TF-IDF with logistic regression routes 98.8% of the validation tickets. Word vectors averaged over a ticket lose the exact words that decide the team.

Pretrained GloVe and analogies

The notebook ships a small subset of GloVe 6B: 50 numbers for each of 20,367 words. Pennington, Socher and Manning (2014) trained the full vectors on 6 billion tokens of Wikipedia 2014 and Gigaword 5, with 400,000 words. Stanford publishes them under the Open Data Commons Public Domain Dedication and License (PDDL) 1.0, so we may share the subset. The notebook gives the full citation.

Vectors from a large corpus show the second fact of latent dimensions: directions can carry relations. To solve “man is to woman as king is to ?”, take a = king, b = man and c = woman. Compute the target vector a − b + c and find its nearest word by cosine, leaving out the three query words:

$$\text{answer} = \arg\max_{w \notin \{a, b, c\}} \cos\!\left(w,\; a - b + c\right)$$

analogies.py

d = np.load("glove6b-50d-subset.npz")
gwords, G = d["words"].tolist(), d["vectors"]
gi = {w: k for k, w in enumerate(gwords)}
Gn = G / np.linalg.norm(G, axis=1, keepdims=True)

def analogy(a, b, c, k=3):
    t = Gn[gi[a]] - Gn[gi[b]] + Gn[gi[c]]
    sims = Gn @ (t / np.linalg.norm(t))
    for w in (a, b, c):
        sims[gi[w]] = -np.inf                      # the three query words do not count
    top = np.argsort(-sims)[:k]
    return [(gwords[n], round(float(sims[n]), 3)) for n in top]

print(G.shape)
for a, b, c in [("king", "man", "woman"), ("paris", "france", "canada"), ("brother", "boy", "girl"),
                ("doctor", "man", "woman")]:
    print(f"{a} - {b} + {c} ->", analogy(a, b, c))

The output:

(20367, 50)
king - man + woman -> [('queen', 0.852), ('throne', 0.766), ('prince', 0.759)]
paris - france + canada -> [('vancouver', 0.802), ('toronto', 0.792), ('london', 0.781)]
brother - boy + girl -> [('cousin', 0.924), ('son', 0.904), ('father', 0.9)]
doctor - man + woman -> [('nurse', 0.84), ('child', 0.766), ('pregnant', 0.757)]

The picture shows why the first one works. Seen in a 2-D shadow, the step from “man” to “woman” and the step from “king” to “queen” point in a similar direction. So king − man + woman lands near queen.

King, queen, man and woman in pretrained GloVe, projected from 50 numbers to 2 with PCA of these four unit vectors (the two directions keep 95% of their spread). The two arrows are the steps man → woman and king → queen. The grey dot, target, is king − man + woman, projected the same way.

Text description of the chart

A 2-D plot. Man is at the lower left and woman above it; an arrow goes from man up to woman. King is at the lower right and queen above it; an arrow goes from king up to queen. The two arrows point in a similar direction: upward, with a small sideways part. Dashed lines join man to king and woman to the target point. The point labelled target, king minus man plus woman, sits between king and queen, nearer to queen. In the full 50 numbers the cosine of the two steps is 0.58, so the parallelogram is only roughly a parallelogram.

Do not trust the picture too much: in the full 50 numbers, the two steps have a cosine of only 0.58. Queen wins because it is still the nearest word to the target. The full table, ranked among all 400,000 words:

Query	Expected	Rank	Top answer
king − man + woman	queen	1	queen 0.852
paris − france + italy	rome	1	rome 0.847
berlin − germany + japan	tokyo	1	tokyo 0.860
walking − walk + swim	swimming	1	swimming 0.807
bigger − big + small	smaller	2	larger 0.924
paris − france + canada	ottawa	12	vancouver 0.802
brother − boy + girl	sister	15	cousin 0.924
refund − money + parcel	delivery	44,574	diphu 0.659
doctor − man + woman	nurse	1	nurse 0.840

Analogies work for famous patterns, such as capitals and verb forms. They fail quietly on others: the answer “vancouver” looks plausible, and nothing warns you that it is wrong. A made-up “Larkfield analogy” such as refund − money + parcel → delivery is nonsense to the vectors: rank 44,574.

Embeddings learn the biases of their text. The last row is not a success. “doctor − man + woman → nurse” at rank 1 is a gender stereotype that the vectors absorbed from news and encyclopedia text. Nobody programmed it; it is in the co-occurrence counts. If such vectors feed a system that ranks people or their texts, the stereotype can turn into unfair decisions. Before you use pretrained vectors, test them for the biases that matter to your task. Measure the outcomes for the groups affected rather than trust a nice nearest-word list, and document what you found.

What came next: one word, many vectors

Every method on this page gives a word one vector, whatever the ticket says around it. “charged” has the same vector in “my card was charged twice” and in “the battery is not charged”. That is the limit that Why word order and context matter showed in practice.

Contextual embeddings fix it. A transformer starts from a table of word vectors like these. Then attention mixes in the other words of the ticket, layer by layer. So the same word gets a different vector in each sentence. The next lesson, The mathematics of attention, shows exactly how. Skip-gram’s idea, predict words from their context without labels, returns at a much larger scale in Training objectives.

Guided practice: train skip-gram in the notebook

1. Open the notebook and run the setup cells. They download the tickets and ticketnet.py.
2. Run “Build the pairs”. Check that ticket T-60140 gives 130 pairs.
3. Run “The model”. Before training, compute the loss of one batch. It must be 6 ln 2 = 4.159.
4. Train for 3 epochs (the lesson used 8). Print the mean loss per epoch.
5. Print nearest("password"), nearest("card") and nearest("late"). For each list, write which neighbours make sense and which look like noise.

Expected output for steps 4 and 5 (your numbers can differ a little on another computer):

epoch 1: 1,357,310 pairs, mean loss 2.638
epoch 2: 1,356,890 pairs, mean loss 2.414
epoch 3: 1,360,876 pairs, mean loss 2.363
password    214x [('reset', 0.95), ('resetting', 0.88), ('link', 0.88), ('forgot', 0.86), ('resend', 0.86)]
card       1323x [('debit', 0.84), ('ending', 0.8), ('visa', 0.8), ('paid', 0.75), ('mastercard', 0.75)]
late         47x [('umbrella', 0.72), ('rescheduled', 0.71), ('later', 0.7), ('excuse', 0.69), ('treat', 0.67)]

Show an example answer

The loss falls fastest in epoch 1, as in the lesson’s chart. The first three epochs give the same losses as the 8-epoch run. After 3 epochs, “password” already sits with “reset”, “link”, “forgot” and “resend”, and “card” with “debit”, “visa” and “mastercard”: frequent, specific words are learned first. “late” gets noisy neighbours (“umbrella”, “treat”): it appears only 47 times, so a few tickets decide its vector. The cosines are higher than after 8 epochs, because the vectors have not yet spread out.

Your turn: change K

Before you run anything, predict: with K = 15 negatives instead of 5, what is the loss before training? And does the Levy and Goldberg optimum move up or down? Then check the first answer in the notebook.

Show the answer

The start loss is (1 + K) ln 2 = 16 × 0.693 = 11.09: each of the 16 terms is ln 2 when U is zero. The optimum moves down: v·u = PMI − ln 15 = PMI − 2.71, instead of PMI − 1.61. More negatives push every pair’s score lower, so only pairs that meet more than 15 times as often as chance get a positive score.

Failure case: the analogy that returns the question

Common mistake

Analogy search without leaving out the query words

Symptom: Tomás writes his own analogy search: scale the vectors to length 1, compute king − man + woman, and return the most similar word. The answer is king (cosine 0.885). Queen comes second, with 0.852. He concludes that GloVe “does not know” the analogy.

Cause: the target a − b + c is built from a, so it usually stays very close to a itself. The nearest word to the target is then one of the query words, most often a. The famous results, and the table above, use the standard rule: leave a, b and c out of the search.

Fix: set the similarity of the three query words to −∞ before you take the top word, as the analogy function above does. Report this rule with your results: without it, analogy scores are not comparable.

Knowledge check

Check your understanding

In skip-gram with negative sampling, what does training do to the centre vector v of password when the context word is reset and the 5 negatives are random words? Answer in words.

Show the answer

It moves v toward u_reset, the context vector of the true neighbour, by the amount the model still misses, 1 − σ(u_reset · v). It moves v away from each negative’s vector u_k, by the probability σ(u_k · v) that the model wrongly gives it. The context vectors move too: u_reset toward v, the negatives away from it. Pull the true context closer, push random words away.

Check your understanding

The skip-gram loss before training is exactly 4.159. Why, and what would it be with K = 2 negatives?

Show the answer

The context table U starts at zero, so every score u · v is 0 and every sigmoid is 0.5. The loss has one term for the true pair and one for each negative, and each term is −ln 0.5 = ln 2. With K = 5 that is 6 ln 2 = 4.159. With K = 2 it is 3 ln 2 = 2.079.

Check your understanding

Pretrained GloVe learned from 6 billion tokens, yet it scores lowest on Larkfield's same-team test (0.320, against 0.585 for skip-gram on the tickets). Omar asks why the bigger model loses. What do you tell him?

Show the answer

The vectors answer the question their text asked. GloVe 6B learned how Wikipedia and news use words. So it puts words near each other by general meaning: late near early, warranty near rental. Larkfield’s teams are a local distinction that this text never saw. The ticket-trained vectors are small and noisy for rare words. But they learned exactly which words appear in which kind of ticket, and that is what routing needs. More data helps only when it is about the right task.

Recap

• Word vectors place each word as a point with 50 latent dimensions: words used in the same contexts end up near each other, directions can carry relations, and a single dimension is usually not readable.
• Skip-gram scores a (centre, context) pair with σ(u_o · v_c) and minimises −ln σ(u_o · v_c) − Σ ln σ(−u_k · v_c), with 5 negatives from unigram^0.75. Its gradient pulls true contexts closer and pushes random words away. On the tickets: 4.159 = 6 ln 2 → 2.293.
• GloVe fits w_i · w̃_j + b_i + b̃_j to ln X_ij by weighted least squares; on our small corpus it is the weakest method.
• Both approximate a factorisation of the PMI matrix (r = 0.852 for skip-gram), and an SVD of shifted PPMI is a third road.
• Ticket-trained skip-gram serves routing best (0.585). Pretrained GloVe knows general English but not Larkfield’s teams (0.320), fails quietly on some analogies, and carries the biases of its text.
• Next: attention gives each word a vector that depends on its context.

Next: The mathematics of attention