From counts to meaning: bag of words, TF-IDF and embeddings

What you will be able to do

Write down exactly how a ticket becomes a vector in three ways: word counts, TF-IDF and a dense embedding. For each one, compute the similarity of two tickets by hand and in code, and see what it fixes and what it still gets wrong.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes tf, df, idf and TF-IDF by hand with NumPy and checks them against scikit-learn to 4 decimals. Then it switches the idf options, trains two routers on the tickets, and measures the synonym pair in counts, TF-IDF and word embeddings. It needs no local setup. About 30 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The vocabulary and the bag of words, from The forward pass, and the embedding table, from Tokens and embeddings.
• The dot product and the length of a vector, from The dot product: a weighted score and The length of a vector in the mathematics course.
• Cosine similarity, from Two ways to say “close”.
• The natural logarithm ln and the summation sign Σ, from Logarithms and The summation sign, read aloud.

What you need on your computer

• Nothing for the notebook route.
• For the local route: the folder ticket-router from Set up PyTorch, with scikit-learn (pip install scikit-learn). The last code block also needs pip install sentence-transformers, which downloads a model of about 90 MB.

Tested with Python 3.12, scikit-learn 1.9.1 and PyTorch 2.14.1 on a CPU.

This lesson is optional: the certificate does not require it.

Two customers, one problem

You saw the idea in The forward pass and in Tokens and embeddings. This page shows the mathematics. There, a ticket became a row of counts, one number for each vocabulary word. Later, each word got a short learned row of numbers instead, and similar words could get similar rows.

Two customers write to Larkfield on the same morning. One writes “my parcel is late”. The other writes “the package has not arrived”. A person sees at once that both mean the same thing. A third customer writes “my refund is late”, which is a payment question. So the question of this page is: when we turn text into a vector, which tickets end up close together, and why?

We answer it three times, with three representations. Each one fixes a problem of the one before it, and each has a price. The diagram puts them side by side.

Three ways to turn a ticket into a vector. TF-IDF keeps the shape of the counts and changes the values; an embedding is short, dense and learned.

Text description of the diagram

The diagram has three boxes with an arrow between each pair. Box 1 is counts, the bag of words. A vector of 2,864 numbers, one per vocabulary word, with about 43 non-zero numbers in an average training ticket: 98.5% zeros. One ticket takes 11,456 bytes as a full vector, or about 352 bytes in a sparse format that stores only the non-zero numbers. Logistic regression on counts routes 97.83% of the validation tickets right. The cosine similarity of “my parcel is late” and “the package has not arrived” is 0. The arrow to box 2 says “weigh each word by its rarity”. Box 2 is TF-IDF. It has the same 2,864 numbers and the same zeros. Each count is multiplied by the word’s idf and the vector is scaled to length 1. It takes the same memory. Validation accuracy 98.83%; the cosine of the two sentences is still 0. The arrow to box 3 says “learn vectors from the words around”. Box 3 is a dense embedding: 64 to 384 numbers, almost none of them 0. They are learned from text, so words used alike get similar vectors. One ticket takes 256 bytes with 64 numbers and 1,536 bytes with 384. A frozen sentence-embedding model routes 94.50% right, and gives the two sentences a cosine of 0.502. A note at the bottom says that memory assumes float32 numbers of 4 bytes, and that a sparse vector stores one value and one position, 4 + 4 bytes, for each non-zero number.

Representation 1: counts

Step 1: a ticket is a point in a space with one axis per word

Fix a vocabulary of V words. Number them from 1 to V. Then a document d, a ticket or a sentence, becomes the vector of its word counts:

$$\mathbf{x}_d = (c_{d,1},\; c_{d,2},\; \dots,\; c_{d,V}) \in \mathbb{R}^V$$

Here c with the indices d and j (written c_d,j) is how often word j appears in document d. ℝ^V means “a list of V real numbers”. Each word is one axis of the space, so a document is one point, or one arrow from the origin.

A tiny example. Take five made-up documents:

Name	Document
d1	my parcel is late
d2	the package has not arrived
d3	refund my card
d4	the parcel arrived broken
d5	my refund is late

Together they use 12 different words, so V = 12. In alphabetical order, d1 has a 1 under “is”, “late”, “my” and “parcel”, and 0 under the other 8 words. Check it in code. CountVectorizer from scikit-learn builds the vocabulary and counts the words; we give it the course’s tokenize, so that it cuts text as the course does. The setup lines load the tickets as in The forward pass.

counts.py

import numpy as np
from ticketnet import load_split, tokenize, Vocab, bag_of_words

train_rows, valid_rows = load_split("data/train.csv"), load_split("data/valid.csv")
vocab = Vocab([r["text"] for r in train_rows])

counts.py

from sklearn.feature_extraction.text import CountVectorizer

docs = ["my parcel is late", "the package has not arrived", "refund my card",
        "the parcel arrived broken", "my refund is late"]
cv = CountVectorizer(tokenizer=tokenize, token_pattern=None, lowercase=False)
C = cv.fit_transform(docs).toarray()          # shape (5 documents, 12 words)
print(cv.get_feature_names_out())
print(C)

The output:

['arrived' 'broken' 'card' 'has' 'is' 'late' 'my' 'not' 'package' 'parcel'
 'refund' 'the']
[[0 0 0 0 1 1 1 0 0 1 0 0]
 [1 0 0 1 0 0 0 1 1 0 0 1]
 [0 0 1 0 0 0 1 0 0 0 1 0]
 [1 1 0 0 0 0 0 0 0 1 0 1]
 [0 0 0 0 1 1 1 0 0 0 1 0]]

Each row is one document and each column is one word. This table of documents and words is the document-term matrix. For the tickets, the same idea gives the matrix of The forward pass, with V = 2,864.

Step 2: almost every number is zero

A ticket uses a few dozen of the 2,864 vocabulary words. So almost every entry of its vector is 0. A vector like this is called sparse. The picture shows the real ticket T-64697 from The forward pass: one small square for each of the 2,864 vocabulary positions, in vocabulary order.

• once
• twice or more
• absent

Ticket T-64697 as a bag of words: 70 of the 2,864 positions are not 0. An average training ticket fills only 43.3 positions.

Text description of the chart

A grid of 2,864 small squares, one for each vocabulary position. They are in vocabulary order, row by row: punctuation first, then words from a to z. Ticket T-64697 has 92 tokens, and 70 different vocabulary words. So 70 squares are coloured and 2,794 squares are grey, 97.6% of them. Most coloured squares stand for a word that appears once. A few stand for words that appear two or more times. “the” appears 7 times; the full stop, the comma and “i” 4 times each; “and” 3 times. The coloured squares are spread over the whole grid, because the ticket’s words come from all parts of the alphabet.

Now count the zeros of all 4,696 training tickets:

counts.py

X = bag_of_words(train_rows, vocab)           # counts, shape (4696, 2864)
nonzero = (X > 0).sum(dim=1).float()          # non-zero entries in each row
print(tuple(X.shape), "| non-zeros per ticket: mean", round(nonzero.mean().item(), 1),
      "median", int(nonzero.median()), "max", int(nonzero.max()))
print("share of zeros:", round(1 - nonzero.mean().item() / X.shape[1], 4))

The output:

(4696, 2864) | non-zeros per ticket: mean 43.3 median 44 max 90
share of zeros: 0.9849

An average ticket has 43.3 non-zero numbers of 2,864: 98.5% zeros. This has a direct cost for Amira, who must store the vectors. A full vector of 2,864 float32 numbers takes 2,864 × 4 = 11,456 bytes. A sparse format stores only the non-zero numbers, each with its position: about 44 × (4 + 4) = 352 bytes. For the whole training set, that is 53.8 MB as full vectors and 2.5 MB as scikit-learn’s sparse matrix. This is why scikit-learn returns sparse matrices, and why we call .toarray() only for tiny examples.

Step 3: similarity is a dot product, divided by the lengths

Two documents are similar when they use the same words. The dot product adds up the products of matching positions:

$$\mathbf{x} \cdot \mathbf{y} = \sum_{j=1}^{V} x_j\, y_j \qquad \lVert \mathbf{x} \rVert = \sqrt{\sum_{j=1}^{V} x_j^2}$$

$$\cos(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x} \cdot \mathbf{y}}{\lVert \mathbf{x} \rVert \, \lVert \mathbf{y} \rVert}$$

In words: x · y multiplies the counts of each word in the two documents and adds the results. A product is non-zero only for a word that both documents use. The length ‖x‖ is the square root of the sum of the squared counts. The cosine similarity divides the dot product by both lengths, so only the direction counts. For counts, it is between 0 and 1.

Check d1 and d5 by hand. They share “is”, “late” and “my”, so d1 · d5 = 1 + 1 + 1 = 3. Each has four words with count 1, so each length is √4 = 2. The cosine is 3 ÷ (2 × 2) = 0.75. Now d1 and d2: they share no word, so every product is 0, and the cosine is 0. In code, we scale every row to length 1 first; then one matrix product gives every cosine at once:

counts.py

def cosine_matrix(M):
    M = M / np.linalg.norm(M, axis=1, keepdims=True)    # every row to length 1
    return (M @ M.T).round(3)                           # all dot products at once

print(cosine_matrix(C.astype(float)))

The output:

[[1.    0.    0.289 0.25  0.75 ]
 [0.    1.    0.    0.447 0.   ]
 [0.289 0.    1.    0.    0.577]
 [0.25  0.447 0.    1.    0.   ]
 [0.75  0.    0.577 0.    1.   ]]

Row 1, column 5 is 0.75, as by hand. Row 1, column 2 is 0: for counts, “my parcel is late” and “the package has not arrived” have nothing in common. “my refund is late”, a payment question, is the closest document to d1.

The problem: common words decide, and length matters

Two things go wrong with raw counts on real tickets.

Common words dominate. “the”, “.”, “i” and “my” appear in almost every ticket, often several times. They add a lot to every dot product, but they say nothing about the team. In T-64697, the five largest counts are all such words: “the” 7, then “.”, “i” and “,” with 4 each, then “and” 3. We measured the effect on the 600 validation tickets. With counts, two tickets for different teams have an average cosine of 0.380. Two tickets for the same team reach only 0.415. Most of the similarity comes from words that every ticket shares.

Length matters for the dot product. Write d1 twice in one ticket: “my parcel is late my parcel is late”. Every count doubles, so its dot product with any other ticket doubles. Long tickets get large dot products with everything. The cosine removes this, because it divides by the length: the doubled ticket has the same direction, so its cosine is unchanged. This is one reason why search and TF-IDF use cosine and rows of length 1.

The second problem has a fix in the formula. The first needs a new idea: weigh each word by how much it tells us.

Representation 2: TF-IDF

Step 1: tf, df and idf

A word that appears in every ticket cannot tell tickets apart. A word that appears in a few tickets can. TF-IDF puts this into three numbers. For a word t, a document d and a collection of N documents:

$$\mathrm{tf}_{t,d} = c_{d,t} \qquad \mathrm{df}_t = \bigl|\{\, d : c_{d,t} > 0 \,\}\bigr|$$

$$\mathrm{idf}_t = \ln \frac{N}{\mathrm{df}_t}$$

In words: the term frequency tf is the count of word t in document d. The document frequency df is the number of documents that contain t at least once; the vertical bars mean “how many”. The inverse document frequency idf is the natural logarithm of N divided by df. The name says what it does: it is large when the document frequency is small. The weight of a word in a document is tf × idf.

The logarithm makes the scale gentle. A word in half of the documents gets ln 2 = 0.69. A word in 1 document of 1,000 gets ln 1,000 = 6.9, ten times more, not 500 times more. A word in every document gets ln 1 = 0: it disappears.

In the five documents, N = 5. “my” is in d1, d3 and d5, so df = 3 and idf = ln(5 ÷ 3) = 0.5108. “parcel” is in d1 and d4: ln(5 ÷ 2) = 0.9163. “package” is only in d2: ln 5 = 1.6094. So “package” counts three times as much as “my”.

Step 2: the formula scikit-learn really uses

This is the textbook formula. scikit-learn’s TfidfVectorizer, the tool behind the course’s TF-IDF baseline, uses a different one by default. If you check its numbers against the textbook, they will not match. Here is exactly what it computes, with its default settings:

$$\mathrm{idf}_t = \ln \frac{1 + N}{1 + \mathrm{df}_t} + 1$$

$$w_{t,d} = \mathrm{tf}_{t,d} \cdot \mathrm{idf}_t \qquad \mathbf{v}_d = \frac{\mathbf{w}_d}{\lVert \mathbf{w}_d \rVert}$$

Three changes, each with a reason:

• The “1 +” in the fraction (smooth_idf=True) acts as if one extra document contained every word once. So no df is ever 0, and the division always works.
• The “+ 1” at the end keeps words that are in every document. With the textbook formula they get weight 0 and vanish; here they get a small weight. With smooth_idf=False, scikit-learn uses ln(N ÷ df) + 1: still not the textbook formula.
• Each row is divided by its length (norm='l2'), so every document vector v has length 1. This is the length fix of the counts section. tf stays the raw count (sublinear_tf=False); with True, it becomes 1 + ln tf.

Now one cell by hand: “parcel” in d1. tf = 1 and df = 2, so idf = ln(6 ÷ 3) + 1 = 1.6931. The other words of d1 get idf 1.6931 (“is”, “late”, in 2 documents) and 1.4055 (“my”, in 3). So d1’s row before scaling is (is 1.6931, late 1.6931, my 1.4055, parcel 1.6931). Its length is √(3 × 1.6931² + 1.4055²) = 3.2520. The weight of “parcel” in d1 is 1.6931 ÷ 3.2520 = 0.5206. Check it, and the whole matrix, against scikit-learn:

tfidf.py

from sklearn.feature_extraction.text import TfidfVectorizer

N = len(docs)
df = (C > 0).sum(axis=0)                          # in how many documents is each word?
idf = np.log((1 + N) / (1 + df)) + 1              # scikit-learn's smoothed idf
W = C * idf                                       # tf x idf, word by word
W = W / np.linalg.norm(W, axis=1, keepdims=True)  # every row to length 1 (l2)

sk = TfidfVectorizer(tokenizer=tokenize, token_pattern=None, lowercase=False)
S = sk.fit_transform(docs).toarray()
print("df: ", df)
print("idf:", idf.round(4))
print("parcel in d1:", W[0, 9].round(4), "| largest difference to scikit-learn:", np.abs(W - S).max())

The output:

df:  [2 1 1 1 2 2 3 1 1 2 2 2]
idf: [1.6931 2.0986 2.0986 2.0986 1.6931 1.6931 1.4055 2.0986 2.0986 1.6931
 1.6931 1.6931]
parcel in d1: 0.5206 | largest difference to scikit-learn: 1.1102230246251565e-16

The largest difference is 1.1 × 10⁻¹⁶, the rounding error of a float64 number: our six lines compute exactly what scikit-learn computes. Try every word in the explorer. Pick a word and follow its weight from the count to the final number.

Explore

TF-IDF, one word at a time

Documents with the word (df)

2 of 5

idf

ln((1 + 5) / (1 + 2)) + 1 = 1.6931

"parcel" is in 2 of the 5 documents, so its idf is 1.6931. Words in 2 documents get a middle idf. Its final weight: d1 0.5206, d4 0.4695.

• final TF-IDF weight (the bar is longer for a larger weight)

"parcel": from count to weight in each document

Document	tf	tf × idf	row length	weight
d1 my parcel is late	1	1.6931	3.2520	0.5206
d2 the package has not arrived	0	0	4.3526	0
d3 refund my card	0	0	3.0407	0
d4 the parcel arrived broken	1	1.6931	3.6061	0.4695
d5 my refund is late	0	0	3.2520	0

weight = tf × idf ÷ row length. The row length is the length of the whole document's tf × idf vector, so every row of the TF-IDF matrix has length 1.

The five made-up documents. For the chosen word: its df and idf, then for each document tf, tf × idf, the length of the document's whole row, and the final weight. All numbers are scikit-learn's or the textbook formula's.

Text description of the explorer

The explorer uses five made-up documents. d1 is “my parcel is late”, d2 “the package has not arrived”, d3 “refund my card”, d4 “the parcel arrived broken” and d5 “my refund is late”. The learner picks one of their 12 words. A second control picks the idf formula: scikit-learn’s ln((1 + N) / (1 + df)) + 1, or the textbook ln(N / df). The readout shows df and idf with the numbers put in. A table has one row for each document. It shows tf, tf × idf, the length of the document’s row, and the final weight tf × idf ÷ row length, with a bar. At the start, the word is “parcel” with scikit-learn’s formula: df 2 of 5, idf ln(6/3) + 1 = 1.6931. In d1, tf 1, tf × idf 1.6931, row length 3.2520, weight 0.5206. In d4 “the parcel arrived broken”, tf 1, row length 3.6061, weight 0.4695. In the other documents, the weight is 0. “my” is in 3 documents and gets the smallest idf, 1.4055, so it has the smallest weight in d1, 0.4322. The words in one document only, such as “package” and “card”, get idf 2.0986. With the textbook formula, “my” gets 0.5108 and its weight in d1 falls to 0.3064, while “parcel” rises to 0.5496.

“my” gets the smallest weight in d1, 0.4322, because it is the only word of d1 that is in three documents. With the textbook formula, the gap is larger: 0.3064 for “my” against 0.5496 for the other words. The picture shows the full count matrix next to its TF-IDF matrix. Rare words light up, and the common “my” and “the” fade.

The five documents as counts and as scikit-learn TF-IDF: words in rows, documents in columns. Darker means a larger number. A dot means 0. The zeros are in the same places.

Text description of the chart

Two tables with the 5 documents in columns and the 12 words in rows: arrived, broken, card, has, is, late, my, not, package, parcel, refund, the. In the counts table, every non-zero cell is 1 and has the same shade. In the TF-IDF table, the same cells are non-zero, with different values. In d1, “is”, “late” and “parcel” have 0.52 and “my” has 0.43. In d2, “has”, “not” and “package” have 0.48, and “arrived” and “the” have 0.39. In d3, “card” has the largest value of the table, 0.69, “refund” 0.56 and “my” 0.46. In d4, “broken” has 0.58, and “arrived”, “parcel” and “the” 0.47. In d5, “is”, “late” and “refund” have 0.52 and “my” 0.43. Words that appear in only one document always get the largest weight in their document.

Step 3: what the weights do on the tickets

Now fit TF-IDF on the 4,696 training tickets. min_df=2 keeps the words that appear in at least two tickets, as the course’s TF-IDF baseline does.

tfidf.py

texts = [r["text"] for r in train_rows]
tfidf = TfidfVectorizer(tokenizer=tokenize, token_pattern=None, lowercase=False, min_df=2).fit(texts)
words, idf_t = tfidf.get_feature_names_out(), tfidf.idf_
order = idf_t.argsort()
print("lowest: ", [(words[i], round(float(idf_t[i]), 2)) for i in order[:5]])
print("highest:", [(words[i], round(float(idf_t[i]), 2)) for i in order[-3:]])
for w in ["refund", "parcel", "password", "late"]:
    print(f"{w:9} idf {idf_t[tfidf.vocabulary_[w]]:.2f}")

The output:

lowest:  [('.', 1.06), ('the', 1.12), (',', 1.14), ('i', 1.19), ('and', 1.27)]
highest: [('spray', 8.36), ('mount', 8.36), ('fiddly', 8.36)]
refund    idf 2.36
parcel    idf 3.66
password  idf 4.15
late      idf 5.58

The full stop is in 4,431 of the 4,696 tickets, so its idf is close to the minimum of 1. A word in only two tickets gets ln(4,697 ÷ 3) + 1 = 8.36, the maximum here. The chart puts selected words on one scale.

The idf of selected words on the 4,696 training tickets, with the number of tickets that contain each word. Grey: words in more than 3,000 tickets.

Text description of the chart

A horizontal bar chart, one bar per word, from the most common to the rarest. The full stop, in 4,431 tickets: idf 1.06. “the”, 4,180 tickets: 1.12. “my”, 3,323: 1.35. “refund”, 1,206: 2.36. “parcel”, 329: 3.66. “broken”, 214: 4.08. “password”, 200: 4.15. “invoice”, 60: 5.34. “warranty”, 53: 5.47. “late”, 47: 5.58. “fiddly”, in 2 tickets: 8.36, the largest idf of all.

Two honest notes. First, idf rewards rare words, not useful words. “fiddly” gets the largest weight, but it says little about the team; “refund” says a lot and gets only 2.36. TF-IDF does not know the teams. Logistic regression learns which words matter for which team. Second, a common word can still win by repeating. In T-64697, “the” appears 7 times, so it stays among the top weights:

tfidf.py

row = next(r for r in valid_rows if r["ticket_id"] == "T-64697")
counts = CountVectorizer(tokenizer=tokenize, token_pattern=None, lowercase=False, min_df=2).fit(texts)
c, t = counts.transform([row["text"]]).toarray()[0], tfidf.transform([row["text"]]).toarray()[0]
print("by count: ", [(words[i], int(c[i])) for i in c.argsort()[::-1][:5]])
print("by TF-IDF:", [(words[i], round(float(t[i]), 2)) for i in t.argsort()[::-1][:5]])

The output:

by count:  [('the', 7), ('.', 4), ('i', 4), (',', 4), ('and', 3)]
by TF-IDF: [('specific', 0.23), ('amount', 0.23), ('the', 0.21), ('recall', 0.2), ('receiving', 0.2)]

The ticket is about a charge: the TF-IDF top now has “amount” and “specific”, but “the” is third. Does the weighting change which tickets look alike? We repeat the measurement of the counts section:

tfidf.py

from sklearn.preprocessing import normalize

v_texts = [r["text"] for r in valid_rows]
teams = np.array([r["team"] for r in valid_rows])
same = teams[:, None] == teams[None, :]               # pairs of tickets with the same team
other = ~np.eye(len(teams), dtype=bool)               # leave out each ticket with itself
for name, vec in [("counts", counts), ("TF-IDF", tfidf)]:
    V = normalize(vec.transform(v_texts))             # rows of length 1
    sims = (V @ V.T).toarray()                        # 600 x 600 cosine similarities
    print(f"{name:7} same team {sims[same & other].mean():.3f}   different teams {sims[~same].mean():.3f}")

The output:

counts  same team 0.415   different teams 0.380
TF-IDF  same team 0.141   different teams 0.109

With counts, the background similarity of unrelated tickets is 0.380, and the same team adds only 9% to it. With TF-IDF, the background falls to 0.109, and the same team stands 29% above it. The shared common words no longer fill the similarity.

What this means for Larkfield’s router

Better vectors should give a better router. We train the same logistic regression on each representation and score the 600 validation tickets. The code is in the notebook; the chart shows the results, with a fifth representation that the next section explains.

Validation accuracy of logistic regression on five representations of the same tickets: counts, 0/1 (word present or not), log of 1 + count, TF-IDF and MiniLM sentence embeddings. 600 validation tickets. One ticket is 0.17 points. The axis starts at 90%.

Text description of the chart

Five bars. Counts: 97.83%. 0/1, a 1 for each word in the ticket and 0 otherwise: 97.83%. Log, ln(1 + count): 98.50%. TF-IDF: 98.83%, the highest. MiniLM, a frozen pretrained sentence-embedding model with 384 numbers per ticket: 94.50%, the lowest. The axis starts at 90%, so the differences look larger than they are: one ticket is 0.17 points.

TF-IDF is the best of the five: 98.83%, 6 more right tickets of 600 than raw counts. Log counts sit between them, because ln(1 + count) also softens the repeated “the”. Differences of 1 to 3 tickets are noise at this size; 6 is a clearer step, but it is still one split and one seed. This is the course’s TF-IDF baseline, and it is still the router that Larkfield should use.

The problem TF-IDF cannot fix: synonyms

Now return to the two customers. TF-IDF changes the values of the vector, never the zeros. A word that is not in a document stays 0. Take d1 “my parcel is late” and d2 “the package has not arrived”. Every product in their dot product is still 0, so the cosine is still 0. The reason is in the space itself: “parcel” and “package” are two different axes, at a right angle to each other. No weighting can turn one axis toward the other.

This is not only a made-up case. “package” appears only 8 times in the training tickets and “parcel” 352 times. Here is one real pair, both for the delivery team. One ticket says “the box I was expecting contained the lid, but it isn’t in the package”. The other is “Re: order LK-905144 - parcel overdue”. They share no token at all, so their TF-IDF cosine is 0.000. Of all 2,817,600 pairs of a validation ticket and a training ticket, 29,920 share no token. For those pairs, counts and TF-IDF cannot say anything.

Representation 3: dense embeddings

Step 1: the company a word keeps

The fix comes from a sentence by the linguist J. R. Firth, in 1957: “You shall know a word by the company it keeps.” This is the distributional hypothesis: words that appear in similar contexts have similar meanings. In English text, “parcel” and “package” both appear next to “arrived”, “late”, “courier” and “delivery”. So a method that learns a vector for each word from its neighbours will give them similar vectors.

Such a vector is a dense embedding: k numbers per word, almost none of them 0. k is much smaller than V, for example 64 or 384 instead of 2,864. The vectors form a table E with one row per word, as in Tokens and embeddings. A document vector can be the average of its words’ rows:

$$\mathbf{e}_t = \mathbf{o}_t^{\top} E \in \mathbb{R}^{k} \qquad \mathbf{s}_d = \frac{1}{n_d} \sum_{i=1}^{n_d} \mathbf{e}_{t_i}$$

In words: o_t is the one-hot vector of word t, all zeros except a 1 at its position. Multiplying it by the table E picks out row t, the word’s embedding e_t. The document vector s_d is the mean of the embeddings of its n_d tokens. A trained sentence model, such as the one below, first mixes each word’s vector with its context and then averages. The output is the same kind of object: one dense vector per text.

A tiny example with made-up 2-number embeddings: parcel = (0.9, 0.2), package = (0.8, 0.3), refund = (0.1, 0.95). The one-hot vectors of parcel and package have a dot product of 0. Their embeddings have a dot product of 0.72 + 0.06 = 0.78, and lengths 0.922 and 0.854. So their cosine is 0.78 ÷ (0.922 × 0.854) = 0.99. parcel and refund: (0.09 + 0.19) ÷ (0.922 × 0.955) = 0.32. Similar words now point the same way.

Step 2: the synonym pair, measured

We use all-MiniLM-L6-v2, a small pretrained sentence-embedding model (Apache-2.0 licence, 22.7 million parameters). It was trained on a large amount of English text, and it turns any text into 384 numbers. We do not train it on the tickets: it is frozen.

embeddings.py

from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
E = encoder.encode(docs, normalize_embeddings=True)   # one row of 384 numbers per document, length 1
print(E.shape)
print((E @ E.T).round(3))

The output:

(5, 384)
[[1.    0.502 0.285 0.585 0.573]
 [0.502 1.    0.189 0.564 0.236]
 [0.285 0.189 1.    0.186 0.632]
 [0.585 0.564 0.186 1.    0.23 ]
 [0.573 0.236 0.632 0.23  1.   ]]

The cosine of d1 and d2 is now 0.502, not 0. The three representations side by side:

The cosine similarity of every pair of the five documents, in counts, TF-IDF and the sentence embedding. The outlined cells are d1 and d2: the two ways to say that a parcel is late.

Text description of the chart

Three tables of 5 by 5 cells, one for each representation, with the documents d1 to d5 in rows and columns. The diagonal is 1 everywhere. Counts: d1 and d2 0, d1 and d3 0.29, d1 and d4 0.25, d1 and d5 0.75. Also d2 and d4 0.45, d3 and d5 0.58; the other pairs 0. TF-IDF: the same zeros; d1 and d5 0.73, d1 and d4 0.24, d1 and d3 0.20, d2 and d4 0.37, d3 and d5 0.49. Sentence embedding: no cell is 0. d1 and d2 0.50, d1 and d4 0.59, d1 and d5 0.57, d2 and d4 0.56, d3 and d5 0.63, and the smallest values, about 0.19, between d3 “refund my card” and d2 or d4.

Read the outlined cells and the d1 row. Counts and TF-IDF put d5 “my refund is late” far above d2: 0.750 and 0.729 against 0. The embedding puts d2 close to d5 (0.502 against 0.573), and d4 “the parcel arrived broken” first (0.585). The chart follows d1 through the three representations:

• d2 "the package has not arrived" (same meaning)
• d5 "my refund is late" (shared words, other team)

The cosine of d1 "my parcel is late" with d2 and with d5, in each representation.

Text description of the chart

Three rows, one per representation, on a scale from 0 to 1. Counts: d2 at 0.000, d5 at 0.750. TF-IDF: d2 at 0.000, d5 at 0.729. Sentence embedding: d2 at 0.502 and d5 at 0.573, close together. Counts and TF-IDF see only the shared words, so the refund document wins by far. The embedding sees that d2 means the same as d1, and the gap almost closes.

On the real pair of delivery tickets, the embedding cosine is 0.255 against TF-IDF’s 0.000. That is lower than for the short sentences, but no longer blind. A plain average of general word vectors does much worse. In the notebook, the average of GloVe word vectors gives d1 and d2 a cosine of 0.905, but d1 and d5 0.939. “my”, “is” and “the” pull every average to the same place, so every pair looks alike.

Step 3: the honest result on routing

So embeddings win? Not on Larkfield’s task. Look again at the accuracy chart: logistic regression on the frozen MiniLM vectors routes 94.50% of the validation tickets right, against 98.83% for TF-IDF. That is 26 more wrong tickets of 600.

The reason is the task. A ticket’s team is decided by exact words: “refund”, “password”, “warranty”, “charged twice”. TF-IDF keeps every one of them on its own axis, and logistic regression learns a weight for each. MiniLM squeezes the whole ticket into 384 numbers that describe its general meaning, and it was never trained on Larkfield’s teams. “My refund is late” and “my parcel is late” are close in meaning, but they belong to different teams. Embeddings fix the synonym problem, not every problem.

What each representation costs, per ticket: counts and TF-IDF, about 352 bytes sparse; a 64-number embedding, 256 bytes; MiniLM, 1,536 bytes. MiniLM also needs a model of 22.7 million parameters to run for every new ticket. Use counts or TF-IDF when the exact words decide, as in routing. Use embeddings when meaning must survive different words, as in search: finding “the package has not arrived” when a person types “parcel late”. Fine-tune a small transformer measures what happens when a pretrained model is trained further on the tickets.

Where the vectors come from

We used a finished model. How do the embedding numbers get their values? The answer takes Firth’s sentence literally: count which words appear near which, then compress those counts into short vectors. word2vec (Mikolov and colleagues, 2013) learns them by predicting a word’s neighbours. GloVe (Pennington, Socher and Manning, 2014) fits them to the logarithms of co-occurrence counts. The next lesson builds both on the tickets and shows that they are close relatives.

Guided practice: one TF-IDF weight by hand

1. Take d3, “refund my card”, with N = 5 documents. Write down the df of “refund”, “my” and “card” from the document table.
2. Compute each idf with scikit-learn’s formula, ln((1 + N) ÷ (1 + df)) + 1.
3. Every tf in d3 is 1, so the row before scaling is the three idf values. Compute its length: the square root of the sum of the squares.
4. Divide the idf of “card” by the length. You should see about 0.690.
5. Check your answer in the explorer: pick “card”. In the notebook, complete tfidf_by_hand() and run its check cell.

Show the answer

“refund” is in d3 and d5 (df 2), “my” in d1, d3 and d5 (df 3), “card” only in d3 (df 1). The idf values are ln(6/3) + 1 = 1.6931, ln(6/4) + 1 = 1.4055 and ln(6/2) + 1 = 2.0986. The length is √(1.6931² + 1.4055² + 2.0986²) = √9.2461 = 3.0407. The weight of “card” is 2.0986 ÷ 3.0407 = 0.6902, the largest weight in the whole matrix. “card” is in one document only, and d3 has only three words.

Your turn: change the options

TfidfVectorizer has two switches that change the formula: smooth_idf=False uses ln(N ÷ df) + 1, and sublinear_tf=True uses 1 + ln tf instead of tf. Before you run them on the five documents, predict: which switch changes the weight of “parcel” in d1, and which one changes nothing at all?

Show the answer

smooth_idf=False changes every weight a little. “parcel” in d1 becomes 0.5255 instead of 0.5206, and the largest change in the matrix is 0.040. sublinear_tf=True changes nothing: in these documents no word appears twice, and 1 + ln 1 = 1, so every tf stays 1. On real tickets it matters, because “the” can appear 7 times: 1 + ln 7 = 2.9 instead of 7.

Common mistake

My TF-IDF does not match scikit-learn

Symptom: Tomás computes TF-IDF by hand from the textbook formula and gets 0.5496 for “parcel” in d1. scikit-learn says 0.5206. The cosine of d1 and d3 is 0.0815 in his version and 0.1998 in scikit-learn’s. He decides that one of the two is wrong.

Cause: both are right, for different formulas. The textbook uses idf = ln(N ÷ df); scikit-learn’s default uses ln((1 + N) ÷ (1 + df)) + 1. The “+ 1” gives common words such as “my” a larger share. So every weight and every cosine changes, although the zeros stay in the same places.

Fix: write the exact formula next to every TF-IDF number, including the library’s defaults: smooth_idf, sublinear_tf and norm. To check a library, copy its formula, as our six lines did: then the largest difference is 1.1 × 10⁻¹⁶.

What it means for a real AI service: “TF-IDF” is a family of formulas, not one. If the search index uses one variant and the code that scores new tickets uses another, the similarities quietly change. Fit the vectorizer once, on the training tickets only, save it, and use the saved one everywhere.

Knowledge check

Check your understanding

Why is the cosine similarity of 'my parcel is late' and 'the package has not arrived' exactly 0 for counts and for TF-IDF? Use the formula in your answer.

Show the answer

The cosine is the dot product divided by the two lengths, and the dot product is the sum of x_j × y_j over all words j. A product is non-zero only for a word that both documents contain. These two documents share no word, so every product is 0, the dot product is 0, and the cosine is 0. TF-IDF multiplies the counts by idf and scales the rows, but a 0 stays 0, so the result does not change. In the vector space, “parcel” and “package” are separate axes at a right angle.

Check your understanding

A word appears in every one of N documents. What is its idf with the textbook formula, and with scikit-learn's default formula? Why does scikit-learn add the + 1?

Show the answer

Textbook: ln(N / N) = ln 1 = 0, so the word gets weight 0 and disappears from every vector. scikit-learn: ln((1 + N) / (1 + N)) + 1 = 0 + 1 = 1, so the word keeps a small weight, the smallest possible. The + 1 at the end keeps such words in the vector instead of removing them completely. The 1 + inside the fraction is the smoothing: it acts as if one extra document contained every word once, so df is never 0.

Check your understanding

Omar reads that embeddings understand meaning and asks why Larkfield's router does not use them. What do you answer, with the numbers?

Show the answer

On the validation tickets, logistic regression on TF-IDF routes 98.83% right, and on the frozen MiniLM embeddings only 94.50%, 26 tickets fewer of 600. The team is decided by exact words such as “refund”, “password” or “warranty”, and TF-IDF keeps each word on its own axis. A general embedding compresses a ticket into 384 numbers about its meaning, and it does not know Larkfield’s teams. “my refund is late” and “my parcel is late” mean similar things but go to different teams. Embeddings help where different words must match, such as search, where they give “parcel” and “package” a cosine of 0.502 instead of 0. They also cost more: a model of 22.7 million parameters must run for every ticket.

Recap

• Counts put a document in a space with one axis per word: 2,864 numbers per ticket, 98.5% of them 0. Store them sparse: about 352 bytes instead of 11,456.
• Cosine similarity = x · y ÷ (‖x‖ ‖y‖). It removes the length, but with raw counts the common words fill most of the similarity.
• TF-IDF weighs each count by idf = ln(N ÷ df). scikit-learn’s default is ln((1 + N) ÷ (1 + df)) + 1, with rows of length 1. Always state which formula you use.
• On the tickets, TF-IDF routes best: 98.83%, against 97.83% for counts and 94.50% for frozen MiniLM embeddings.
• TF-IDF never changes a 0, so synonyms stay at cosine 0. Dense embeddings, learned from the company words keep, give “my parcel is late” and “the package has not arrived” a cosine of 0.502.

Next: how word2vec and GloVe learn those dense vectors from co-occurrence counts.

Next: How embeddings learn meaning: word2vec and GloVe