Why word order and context matter

What you will be able to do

See exactly where counting words fails: tickets whose team depends on the order of the words and on the words around them. This is the problem that attention, in the next lesson, is built to solve.

NotebookSame words, same input: check it in a notebookColab · Kaggle · GitHub

The notebook builds the bag of words of two made-up tickets and checks that they are identical. It counts word pairs and triples, and it trains the bag-of-words network to list the validation tickets that it routes wrongly, by style. About 25 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The bag of words, from The forward pass: a ticket becomes one count for each word of the vocabulary.
• Tokens, embeddings and the average-embedding network, from Tokens and embeddings.
• Grace’s routing policy and its two boundaries (broken on arrival or after use; a refund before or after the return is confirmed), from What deep learning is.

What you need on your computer

• A browser. The notebook opens in Colab or Kaggle with PyTorch already installed.
• For the local route instead: the folder ticket-router from Set up PyTorch: Route 2. The code continues from Tokens and embeddings, so run tokens_and_ids.py first: it makes vocab.

The same words, a different message

Your colleague sends you a message: “Call me, don’t text.” An hour later another one: “Text me, don’t call.” The two messages use exactly the same words. If you only counted the words, you could not tell them apart. But they ask for opposite things.

People read words in order, and each word changes the meaning of the words near it. “Don’t” belongs to “text” in the first message and to “call” in the second. A bag of words keeps the words and throws the order away.

The problem on real tickets

Since Module 2, every router of this course has read a ticket as a bag of words, or as an average of word embeddings. Both forget the order. They still route most tickets right: the bag-of-words network gets 97.3% of the validation tickets after 20 epochs. So where do its mistakes come from?

Every validation ticket has a style, which records how the ticket was written. The style is never an input to a model. It only lets us measure where a model fails. The chart shows the share of tickets that the bag-of-words network routes wrongly, for five styles.

• The words of the request decide
• Order or other words matter

The bag-of-words network on the validation tickets, after 20 epochs, by how each ticket is written. It never fails on plain tickets. It fails on boundaries, negations, two topics and long tickets.

Text description of the chart

Share of validation tickets that the bag-of-words network routes wrongly, by style. Plain tickets: 0 of 145 (0%). On a policy boundary: 3 of 131 (2.3%). With a negation: 2 of 38 (5.3%). Two topics, a solved problem first: 5 of 52 (9.6%). Long tickets: 4 of 21 (19.0%).

The average-embedding network fails on almost the same tickets. It has 19 errors in total, with the same counts for boundaries (3), negations (2), two topics (5) and long tickets (4). Now look at the 46 tickets whose real request comes in the last sentence (“request last”): both networks route all of them right. Position alone is not the problem. The problem starts when the other words point to another team. Long tickets have a second problem: the course cuts every ticket after 120 tokens, so the end of a very long ticket is lost.

Here are four of the real tickets, shortened.

T-64746, a negation. Grace: returns. “I’m not calling about a faulty item, and I’m not asking for a refund. The rattan patio set is simply too large for me to take to a drop-off point. Could you arrange a collection for the return?” Bag of words: payment (57%). Average embedding: payment (56%). The customer says “not … refund”, but a count of “refund” is the same with or without “not”.

T-64813, two topics. Grace: payment. “Earlier this month a return I sent back was refunded quickly … I was charged twice for my pruning shears order … and I’d like one of the payments reversed, please.” Bag of words: returns (98%). Average embedding: returns (98%). The solved return comes first; the real request comes second. Counts cannot tell which topic is finished.

T-65219, on a boundary. Grace: delivery. “The step ladder … has a cracked foot … it feels faulty and unsafe to climb. It was already broken right out of the box when I opened it on the day it came.” Bag of words: warranty (83%). Average embedding: warranty (64%). “cracked” and “faulty” pull toward warranty. Only the words around them, “out of the box … on the day it came”, say “on arrival”.

T-65176, two topics. Grace: account. “Last month my delivery was left at the wrong house, and you sorted a replacement … Still can’t get into my account to check my cushion cover order, though, so could you help?” Bag of words: delivery (87%). Average embedding: delivery (84%). The finished delivery problem has more words than the open account problem.

Be careful with one conclusion. The machine learning course’s linear model also reads a bag of words, and it routes three of these four tickets right. It fails only on T-65176. So not every mistake comes from the lost order: two models can weigh the same counts differently. To see the effect of order alone, you need two tickets whose words are exactly the same.

Same words, same input, same answer

These two tickets are made up for this lesson; no customer wrote them. They use exactly the same 14 tokens, in a different order. Grace’s policy sends them to different teams.

• A: “The lamp was damaged on arrival, not after a month of use.” → delivery
• B: “The lamp was damaged after a month of use, not on arrival.” → warranty

Build their bags of words and compare them. The code uses vocab from Tokens and embeddings, and bag_of_words() from ticketnet.py makes the counts of The forward pass:

same_bag.py

import torch
from ticketnet import bag_of_words, tokenize

a = "The lamp was damaged on arrival, not after a month of use."
b = "The lamp was damaged after a month of use, not on arrival."
xa = bag_of_words([{"text": a}], vocab)
xb = bag_of_words([{"text": b}], vocab)
print(xa.shape, "tokens:", int(xa.sum()), "| the same input:", torch.equal(xa, xb))

The output:

torch.Size([1, 2864]) tokens: 14 | the same input: True

torch.equal checks that two tensors have the same shape and the same numbers. They do: A and B are the same input. A network is a function of its input. The same input always gives the same output, so the trained bag-of-words network must give A and B the same answer. The diagram shows what it says.

Two made-up tickets with the same words become the same input. The network's answer is real: whatever it learns, it must answer A and B alike.

Text description of the diagram

Two made-up tickets. Ticket A says “The lamp was damaged on arrival, not after a month of use.” Grace’s policy sends it to delivery, because it broke on arrival. Ticket B says “The lamp was damaged after a month of use, not on arrival.” Grace’s policy sends it to warranty. Both tickets become the same bag of words of 2,864 counts: the, lamp, was, damaged, on, arrival, not, after, a, month, of, use, the comma and the full stop once each, and every other word 0. The trained bag-of-words network gives both tickets the same answer: warranty 59.4%, delivery 38.0%. That is right for B and wrong for A, and no training can fix it, because the input is the same.

This is not a weakness of this one network. Every model that reads only the counts must answer A and B alike. That includes the average-embedding network and the linear model of the machine learning course. For any pair like this, at least one of the two answers is wrong.

Two more made-up pairs show the same thing. Each pair has the same bag of words, and the trained network gives each pair one answer.

Made-up ticket	Grace’s team	Bag-of-words network
This is not about my account, it is about the late parcel.	delivery	delivery (89.8%)
This is not about the late parcel, it is about my account.	account	delivery (89.8%)
The password problem is solved. The double charge on my card is still open.	payment	returns (66.9%)
The double charge on my card is solved. The password problem is still open.	account	returns (66.9%)

The last pair gets “returns”, which is wrong for both. Made-up sentences like these are shorter than the real tickets, so the network has little to go on. But the main point does not depend on the answer: the two inputs are identical.

Context: a word’s meaning depends on its neighbours

The pairs show a second idea. A word does not have one fixed meaning in a ticket. “damaged” means “on arrival” or “after use” depending on the words next to it. “refund” means “my request” in one sentence and “solved, thank you” in another. “not” changes the word that follows it.

So a good reader of tickets does two things. It keeps the order of the words. And it lets each word take context from the other words that matter for it, near or far. A count gives every word one fixed contribution, whatever surrounds it. An average embedding does the same with one fixed row per word.

A partial fix: count word pairs

A simple idea keeps a little order: count short runs of neighbouring words as well as single words. A run of n neighbouring tokens is an n-gram: the n is the length of the run. Pairs are bigrams (“on arrival”, “not after”), and triples are trigrams (“damaged on arrival”).

Bigrams separate the lamp pair. Ticket A has the pairs “damaged on”, “arrival ,”, “not after” and “use .”; ticket B does not. But they do not separate the password pair:

ngrams.py

from collections import Counter

def ngrams(text, n):
    t = tokenize(text)
    return Counter(tuple(t[i:i + n]) for i in range(len(t) - n + 1))

c = "The password problem is solved. The double charge on my card is still open."
d = "The double charge on my card is solved. The password problem is still open."
for n in (1, 2, 3):
    print(n, "| lamp pair, same:", ngrams(a, n) == ngrams(b, n), "| password pair, same:", ngrams(c, n) == ngrams(d, n))
print(sorted(ngrams(a, 2) - ngrams(b, 2)))

The output:

1 | lamp pair, same: True | password pair, same: True
2 | lamp pair, same: False | password pair, same: True
3 | lamp pair, same: False | password pair, same: False
[('arrival', ','), ('damaged', 'on'), ('not', 'after'), ('use', '.')]

In the password pair, the words that change the meaning, “solved” and “still open”, sit two words away from “problem” and “card”. Every pair of neighbours appears in both tickets. Only triples such as “problem is solved” tell them apart. Whatever window you choose, a sentence can put the important word just outside it.

And every longer window costs a lot. Count only the n-grams that appear at least twice in the training tickets, as the vocabulary does for words:

The vocabulary from the 4,696 training tickets with single words, with word pairs added, and with word triples added. It grows about 7 times, then 17 times.

Text description of the chart

Vocabulary size from the training tickets, counting entries that appear at least twice. Single words: 2,864. Words and word pairs: 19,312. Words, pairs and triples: 47,438.

The costs, in real numbers:

• More weights. The bag-of-words network has 183,685 parameters with words. With pairs it needs 1,236,357, and with triples 3,036,421, for the same 4,696 tickets.
• More unseen inputs. 11.8% of the word pairs in the validation tickets never appeared twice in training, and 33.4% of the triples. A new ticket often uses a pair that the model has never seen, and that pair then counts for nothing.

Does it help? A little. With the machine learning course’s linear model, validation accuracy goes from 98.8% (words) to 99.3% (with pairs) and 99.5% (with triples). The errors on two-topic tickets fall from 6 to 3 to 2. That is 3 or 4 tickets out of 600. And on the three made-up pairs, the n-gram models still give both tickets of each pair the same team. Different inputs make a different answer possible; they do not guarantee it.

A short history: reading word by word

Before 2017, the usual answer was a recurrent network. It reads a ticket one word at a time, like a person reading aloud. It keeps a running summary: a vector that it updates after every word. At the end, the summary goes to the output layer. A recurrent network sees the order, but its summary has a fixed size, so early words fade as the text gets longer. It also must read the words one after another, which is slow on long texts. Improved versions, such as the LSTM, kept early words longer, but the two problems remained. Today, most text models use attention instead.

What a better reader needs

Put the pieces together. To route T-64813, the word “charged” must count more than “refunded”, because “refunded” belongs to a finished story. To route ticket B, “damaged” must read “after a month of use”, and “not” must attach to “on arrival”, four words later. To route T-65176, the network must see that “still can’t” belongs to the account problem. In each case, a word needs information from some other words, chosen by what they say, at any distance.

That is exactly what the next lesson builds: attention. Each word looks at all the other words, gives each one a weight, and takes a weighted average of what they say. The weights come from the words themselves, so they change from ticket to ticket.

Be honest about what to expect. The course’s tiny transformer, built on attention, does read order. On the made-up pairs, it gives the two tickets of each pair different answers. But it is right on only 2 of the 6 tickets. On the real validation tickets it routes 94.3% right, less than the bag-of-words network. Shuffle the words of every validation ticket, and it drops only 1 point, to 93.3%. On these tickets, it barely uses the order. About 4,700 tickets are too few to learn a language from nothing. Module 6 builds that transformer, and Module 7 shows what pretraining adds.

When to use word counts, and what they cost

Word counts are the right tool when the words decide the team. On Larkfield’s tickets they do, most of the time: the linear model routes 98.8% right, and it stays Larkfield’s router. Counts are fast, cheap and easy to explain.

Their cost is a blind spot that is certain, not random. Negations, two topics and tickets on a policy boundary are where the order matters. Measure your router on these styles, not only overall. Add n-grams if a few points matter and you have enough data, and check the size of the vocabulary. For text where order matters a lot, such as long tickets with several topics, you need a model that reads order and context.

Guided practice: prove that the input is the same

1. Open the notebook and run the setup cells.
2. Run the section “Same words, same input”. It builds the bag of words of tickets A and B.
3. Check the output: the same input: True.
4. Run the section “Count pairs and triples”. Find the first n for which each made-up pair differs.
5. Run the section “Where the network fails”. It trains the bag-of-words network for 20 epochs and lists its wrong validation tickets by style.

Expected output of step 4, as in the lesson: the lamp pair differs at n = 2, the password pair at n = 3. Step 5 lists 16 wrong tickets: 5 two-topic, 4 long, 3 boundary, 2 negation, 1 distractor and 1 short ticket.

Show why the answer must be “the same input”

bag_of_words counts each token of the vocabulary. A and B have the same 14 tokens, each once, so all 2,864 counts are equal. The order of the words is never stored anywhere in the tensor. The network computes its output only from this tensor, so it cannot answer A and B differently, whatever its weights are.

Your turn: a pair of your own

Write two short tickets with the same words in a different order, which Grace’s policy sends to different teams. Use only common words, so that no token becomes <unk>. Predict first: is their bag of words the same? Are their word pairs the same? Then check both in the notebook.

Show an example answer

“My card was charged, not my account locked.” (payment) and “My account was locked, not my card charged.” (account). The bag of words is the same (True), and the word pairs differ (False): “card was” is in the first ticket only. Any pair you write with the same words gives True for the bag of words. Whether the pairs differ depends on where you moved the words.

Common mistake

Higher accuracy taken as proof that order is handled

Symptom: Tomás adds word pairs and triples to the linear model. Validation accuracy rises from 98.8% to 99.5%, and he tells Grace that “the order problem is solved”. Then Grace tries the made-up lamp tickets: both still go to warranty.

Cause: a 0.7-point gain is 4 tickets out of 600; it says little about negations or two topics. N-grams see only a short window, and they help only with pairs and triples seen in training. A third of the validation triples were never seen twice in training.

Fix: test order directly. Measure the router by style (negation, two topics, boundary). Keep a small set of made-up pairs with the same words and different teams, and check that the model separates them. Report what it costs too: here, 47,438 vocabulary entries instead of 2,864.

See it end to endFine-tune a small transformerCase study

The fine-tuning case study shows a pretrained model that tells such a same-words pair apart, and where it still fails.

Knowledge check

Check your understanding

Why must every bag-of-words model give the same answer to 'damaged on arrival, not after a month of use' and 'damaged after a month of use, not on arrival'?

Show the answer

The two sentences have exactly the same tokens, each once, so their bags of words are identical: all 2,864 counts are equal. A model computes its output only from its input, and the same input always gives the same output. So any model that reads only the counts, whatever its weights, gives both the same answer. Grace’s policy sends them to different teams (delivery and warranty), so one answer is always wrong.

Check your understanding

The bag-of-words network routes all 46 'request last' tickets right, but 5 of 52 'two topics' tickets wrongly. Both styles put the real request at the end. Why the difference?

Show the answer

The position of the request alone does not matter to a bag of words, because it ignores position. In “request last” tickets, the other words do not point strongly to another team, so the request’s words decide. In “two topics” tickets, a solved problem from another team comes first, with that team’s words. The counts then contain strong words for two teams, and the network cannot tell which topic is finished and which is open.

Check your understanding

Predict: Tomás adds word pairs (bigrams) to fix the password pair, 'The password problem is solved. The double charge on my card is still open.' and its reversed version. Does it work? What does it cost?

Show the answer

It does not work: both tickets have exactly the same word pairs, because “solved” and “still open” sit two words away from “problem” and “card”. Only triples tell them apart. The cost is large. With pairs, the vocabulary grows from 2,864 to 19,312 entries, and the bag-of-words network from 183,685 to 1,236,357 parameters. 11.8% of the validation pairs were never seen twice in training.

Recap

• A bag of words and an average of embeddings both throw the word order away. The bag-of-words network fails most on long tickets (19.0%) and two topics (9.6%), then on negations (5.3%) and boundaries (2.3%), and never on plain tickets.
• Two made-up tickets with the same words get the same input (torch.equal is True). So every count-based model gives them the same answer, and one of the two is wrong.
• A word’s meaning depends on its context: “damaged” needs “on arrival” or “after use”; “refund” can be a request or a finished story.
• N-grams keep a short window of order. They help a little (98.8% → 99.5% for the linear model). But they grow the vocabulary 7 to 17 times, and many new pairs are unseen.
• Recurrent networks read word by word but forget early words and are slow. Attention lets each word choose which other words to read, at any distance.

Next, build attention by hand with three words and two numbers per word. Then look at the real attention weights of the course’s tiny transformer.

Next: Attention step by step