Skip to content
Français

Text as numbers: bag of words and TF-IDF

  • Lesson 10
  • 35 min
  • Module 3: Baselines: rules and classical models

Where we are

Rules read words that we chose. A model needs the text as numbers, and it can choose the words itself. In this lesson, we turn messages into numbers with a bag of words and TF-IDF, with word and character n-grams, and we see why this old method is still a strong baseline.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 10 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceText as numbers: bag of words and TF-IDF

What you will be able to do

Turn each message into a row of numbers that a model can learn from. A rule reads only the words that a person chose. A model can weigh every word of 6,000 messages, but only if the words are numbers first.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It builds a bag of words and TF-IDF for three tiny messages, and turns the 6,000 training messages into numbers. It trains a first model with words and with letters, and sends it a message about a product that it never saw. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this lesson.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.

Go back to the post office. The head clerk wants to learn which letters go to which bag, without writing rules by hand. So for one week, she keeps a tally sheet for every letter. The sheet has one column for each word that she has ever seen on a letter. For each letter, she makes a mark in the column of every word on it.

At the end of the week, she has a big table. It has one row per letter, one column per word, and the bag where each letter went. Now she can look for patterns. Letters with “Paris” and “France” went to the blue bag. Letters with “Paris” and “Texas” did not. She does not need to guess the useful words in advance. The table shows them.

The sheet forgets one thing: the order of the words. “Dog bites man” and “man bites dog” get the same marks. For sorting letters, that is usually fine.

The keyword rules of the last lesson read only the words that Priya and Grace chose: “refund”, “reçu”, “smoke”. They never saw the other words of the message, such as “before I buy” or “has nothing to do with it”. Those words show the real request.

A model can do better, because it can look at every word of thousands of labelled messages and find which words go with which route. But a model computes with numbers, not with text. So before any model, we need a way to turn a message into a row of numbers. The head clerk’s tally sheet is that way.

A bag of words counts each word of a message and forgets the order. The name comes from the picture: you pour the words into a bag, shake it, and count what is inside.

First, we cut each message into tokens: here, the words. Then we make the vocabulary: the list of every token in all the messages. Each message becomes one row, with one column per vocabulary word. The value is how many times the word appears in the message.

Here are three short messages:

  1. where is my parcel
  2. my parcel is late
  3. refund my order

The vocabulary has seven words, in alphabetical order: is, late, my, order, parcel, refund, where. The bag of words is this table of counts:

is late my order parcel refund where
message 1 1 0 1 0 1 0 1
message 2 1 1 1 0 1 0 0
message 3 0 0 1 1 0 1 0

Read the column my. It is 1 in every row. A word that is in every message cannot help to tell messages apart. The columns late, order, refund and where are each in one message only, so they say the most about that message. Plain counts treat all these words the same. We want a number that gives rare words more weight.

TF-IDF does this. The name has two parts:

  • TF, the term frequency: how often the word is in this message. Here, each count is 1.
  • IDF, the inverse document frequency: a weight that is large when few messages have the word, and small when many have it. “Document” is the general name for one text, here one message. “Inverse” means that more messages give a smaller weight.

The TF-IDF value of a word in a message is TF × IDF. Then each row is scaled so that its length is 1. This last step makes a short message and a long message comparable.

Count how many of the three messages have each word. my is in 3 messages, is and parcel are in 2, and the other four words are in 1. scikit-learn computes the IDF from these counts:

Word In how many messages IDF
my 3 1.000
is, parcel 2 1.288
late, order, refund, where 1 1.693

The formula is IDF = 1 + ln((1 + messages) / (1 + messages with the word)), where ln is the natural logarithm. You do not need to remember it. Remember the pattern: the fewer messages have a word, the larger its IDF.

Now take message 1, where is my parcel. Multiply each count by the IDF: where 1.693, is 1.288, my 1.000, parcel 1.288. The length of this row is 2.680: the square root of 1.693² + 1.288² + 1.000² + 1.288². Divide each value by 2.680, and you have the TF-IDF row. Do the same for the other two messages:

is late my order parcel refund where
message 1 0.480 0 0.373 0 0.480 0 0.632
message 2 0.480 0.632 0.373 0 0.480 0 0
message 3 0 0 0.385 0.652 0 0.652 0

In message 1, where now weighs 0.632, and my only 0.373. The rare word counts almost twice as much as the common one. In code, scikit-learn’s TfidfVectorizer does all of this in one line:

tfidf = TfidfVectorizer()
weights = tfidf.fit_transform(tiny)
pd.DataFrame(weights.toarray(), columns=tfidf.get_feature_names_out(), index=["msg 1", "msg 2", "msg 3"]).round(3)
is late my order parcel refund where
msg 1 0.48 0.000 0.373 0.000 0.48 0.000 0.632
msg 2 0.48 0.632 0.373 0.000 0.48 0.000 0.000
msg 3 0.00 0.000 0.385 0.652 0.00 0.652 0.000

The fit part learns the vocabulary and the IDF from the messages. The transform part turns messages into rows. As with every step that learns from data, we fit it on the training messages only.

A bag of words loses the order of words, and sometimes the order matters. “Charged twice” says more than “charged” and “twice” apart. So we often add word pairs to the vocabulary: every two words that follow each other. The general name is n-gram: a group of n tokens. One word is a 1-gram, two words a 2-gram. In scikit-learn, ngram_range=(1, 2) keeps words and word pairs.

We can also cut words into character n-grams: groups of n letters inside a word. The word parcel has the 3-letter groups par, arc, rce and cel. Letter groups help in three cases:

  • Typos. delivred shares most of its letter groups with delivered.
  • Word endings. French remboursé and remboursement share rembours.
  • Short messages. A chat message of four words has few words, but many letter groups.

The cost is size: letter groups make many more columns than words.

Now the same on Larkfield’s 6,000 training messages. The project’s TF-IDF settings are in larkroute/tfidf.py, and the notebook uses the same ones:

words = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, strip_accents="unicode")
X_train = words.fit_transform(train["text"])
print(X_train.shape[0], "messages,", X_train.shape[1], "columns (words and word pairs)")
6000 messages, 14076 columns (words and word pairs)

Each setting has a reason. ngram_range=(1, 2) keeps words and word pairs. min_df=2 keeps only the tokens that appear in at least 2 messages, so a token seen once, such as one order number, makes no column. sublinear_tf=True counts a word that appears five times as a little more than once, not five times more. strip_accents="unicode" removes accents, so livré and livre count as one token. The letter groups of 2 to 5 letters add 24,539 more columns.

So one message is a row of 14,076 numbers. Almost all of them are 0, because one message has only a few of the 14,076 words and pairs. Here is the first test message, a French chat message: “Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?” (“Could the courier leave my parcel LK-596824 with my neighbour?”).

A horizontal bar chart of the 24 TF-IDF weights of message M-100001 that are not zero. The largest are the word pairs pourrait laisser 0.286 and livreur pourrait 0.277, then pourrait 0.257. Common words have small weights: est 0.118, le 0.103, and lk 0.061 at the bottom. A note says that the other 14,052 columns are 0.
One message as a TF-IDF vector. Only 24 of the 14,076 columns are not zero. Rare words and word pairs get large weights; common words such as le and lk get small ones.
Text description of the diagram

The chart lists the 24 columns that are not zero for the message M-100001, from the largest weight to the smallest. The largest weights are word pairs: “pourrait laisser” 0.286 and “livreur pourrait” 0.277. Then come “pourrait” 0.257, “laisser mon” 0.254, “le livreur” 0.252, “colis lk” 0.244 and “chez ma” 0.243. “Livreur”, “ma voisine” and “voisine” each have 0.242. Common words are at the bottom: “mon” 0.122, “est” 0.118, “le” 0.103 and “lk” 0.061. Almost every message has an order number, so “lk” weighs the least. A note under the chart says that the other 14,052 columns are 0.

A row of numbers like this one is called a vector: a list of numbers in a fixed order. A vector or a table where almost every value is 0 is called sparse. scikit-learn stores only the values that are not zero, so 6,000 rows of 14,076 columns fit easily in memory.

Look at the bottom of the chart. lk is in almost every message, because every order number starts with LK, so its weight is small. The order number itself, 596824, makes no column: it appears in only one message, and min_df=2 dropped it. TF-IDF did, by itself, what a person would do: it ignored the common words and kept the telling ones.

The numbers are ready. Now a classifier can learn from them. The next lesson explains how three classifiers work. Here, we use one of them, logistic regression, as a tool, to see whether the numbers are useful. A pipeline joins the vectorizer and the classifier, so that one fit learns both from the training messages:

word_model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=10, max_iter=3000))
word_model.fit(train["text"], train["route"])
rows = [rt.score("TF-IDF words + logistic regression", test, word_model.predict(test["text"]))]

Then Priya adds the letter groups of 2 to 5 letters to the words, with the same classifier. Before you look, predict: do the letters help? French has long word endings, and chat messages are short.

On this inbox, the words alone are better: accuracy 0.968 and US$195 per 1,000 messages, against 0.962 and US$296 with words and letters.
Text description of the chart

A column chart with two columns, the cost of the mistakes in US dollars per 1,000 test messages, for TF-IDF and logistic regression. With words and word pairs only, the cost is 195. With words and letter groups, it is 296. The accuracy is 0.968 with words and 0.962 with words and letters. On French messages, it is 0.956 against 0.952.

The letters did not help here. The words alone reach 0.968 on the test messages and cost US$195 per 1,000. With letters, the model reaches 0.962 and costs US$296. Even the French messages are a little better with words alone (0.956 against 0.952).

This is a real surprise, and it says something about this inbox. As Meet the inbox explains, one language model wrote all the Larkfield messages, so they have few typos and very regular words. A real inbox, with typing errors, slang and very short chat messages, is harder, and there letter groups often help. The only way to know is to test both on your own messages, as here. The case study Route banking questions shows a classical model on real customer questions.

The project’s TF-IDF + logistic regression uses words and letters. Priya chose these settings on the validation messages, where the two were almost equal (0.958 with letters, 0.959 without). The test set then showed a larger difference. The course keeps the project’s choice and reports both rows. A choice made on the test set would make the test score too good.

TF-IDF is old: the idea of IDF comes from the 1970s. It is still the first model to try for text classification, for four reasons:

  • It is cheap. Training on 6,000 messages takes about 3 seconds on a laptop, and routing one message takes about 0.1 milliseconds. No graphics card, no download, no key.
  • It is strong when the words carry the meaning. Support messages say “refund”, “tracking” and “password”. The words are the signal.
  • It is easy to explain. Each column is a word, so you can read which words a model uses. The next lesson does this.
  • It sets the bar. Every newer method in this course must beat it on the same table. On this inbox, the bar is high.

It also has limits:

  • It does not know meaning. “Parcel”, “package” and colis are three unrelated columns. It learns that each one goes with delivery only if the training messages have it.
  • It loses order beyond word pairs. “Not a refund” and “a refund” share most of their columns.
  • It knows only the words of its training messages. A new word gets no column at all, as the failure case below shows.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, predict which column is 1 in all three rows. Then run the cell. The column my is 1 in every row.
  4. Run the TF-IDF cell. In message 1, where has the weight 0.632 and my 0.373.
  5. In section 2, run the cell. You see 6000 messages, 14076 columns, and the weights of the first test message.
  6. In section 3, predict: better on English or on French messages? Then run the cell. You see accuracy 0.968, English 0.973 and French 0.956.
  7. In section 4, run the cell with letter groups. The accuracy is 0.962.

Your turn: the words that mean “billing”

Section titled “Your turn: the words that mean “billing””

The classifier keeps one weight for each column and each route. A large weight for billing means: “this word pushes a message toward billing”. In section 5 of the notebook, find the 5 words or word pairs with the largest weight for billing in word_model. Before you run it, guess two of them.

Show the answer
vec, clf = word_model[0], word_model[-1]
row = list(clf.classes_).index("billing")
names = vec.get_feature_names_out()
billing_words = [str(names[i]) for i in np.argsort(-clf.coef_[row])[:5]]
print(billing_words)
['card', 'charge', 'my card', 'vat receipt', 'vat']

Nobody wrote these words in a rule. The model found them in the training messages. “Vat receipt” is a sign of this inbox. One of its invoice templates asks for a VAT receipt, so the pair is in 56 billing messages of the training set. The next lesson reads these lists for every route.

Common mistake

A word that the model never saw

Symptom: in the launch month, Larkfield sells a new product, the Grow Hub. A customer writes: “My Grow Hub won’t pair with the app, the sensor light keeps blinking”. The model routes it with no error:

new = "My Grow Hub won't pair with the app, the sensor light keeps blinking"
v = word_model[0].transform([new])
print("known words and pairs in the message:", [word_model[0].get_feature_names_out()[i] for i in v.indices])
print(dict(zip(word_model[-1].classes_, word_model.predict_proba([new])[0].round(2))))
known words and pairs in the message: ['app', 'grow', 'keeps', 'light', 'my', 'my grow', 'the', 'the app', 'with', 'with the', 'won']
{'account': 0.03, 'billing': 0.06, 'delivery': 0.46, 'needs_person': 0.07, 'orders': 0.01, 'product': 0.18, 'returns_refunds': 0.19}

The model chooses delivery, the largest probability (0.46). The message is about setting up a device, not about a parcel.

Cause: TF-IDF knows only the words of its training messages. “Hub”, “pair”, “sensor” and “blinking” make no column, so the model sees only common words such as “my”, “the” and “keeps”. It must still choose a route, and it chooses with almost no evidence.

Fix: look at the probability, not only at the route. No route gets even half of the probability here, which is a sign that the model is unsure. A later module sends such messages to a person, and the production module watches for new words and trains the model again with labelled examples.

Consequence in AI work: a TF-IDF model fails quietly on anything new: a new product, a new kind of complaint, a new language. Its accuracy on the old test set stays high while the new messages go to the wrong team.

Check your understanding

In the three tiny messages, “my” is in all three and “where” is in one. Without computing, which gets the larger TF-IDF weight in message 1, and why is that useful for routing?

Show the answer

where gets the larger weight (0.632 against 0.373). The IDF part is large when few messages have a word, and my is in every message, so its IDF is the smallest possible (1.000). A word that is in every message cannot tell messages apart, so it should weigh little. A rare word says more about what this message is about. For Larkfield, lk is in almost every message and gets a small weight; words such as livreur or cancel get large ones.

Check your understanding

Predict: you add letter groups of 2 to 5 letters to the word features of a model for a real chat inbox with many typing errors. Will they help? What did they do on the Larkfield inbox, and why might the two answers differ?

Show the answer

On a real chat inbox with typing errors, letter groups often help. delivred and delivered share most of their letter groups, and short messages get more columns. On the Larkfield inbox, they did not help. Words only reach 0.968 and US$195 per 1,000 test messages; words and letters reach 0.962 and US$296. One language model wrote the Larkfield messages from 27 drafts, so they have few typos and regular words. A result on one inbox is not a rule for every inbox. Test both on your own messages, and choose on the validation set.

Check your understanding

A message says “My Grow Hub won’t pair with the app”. The model was trained before the Grow Hub existed. Explain what TF-IDF does with “Hub” and “pair”, and what you should look at before you trust the route.

Show the answer

“Hub” and “pair” were not in the training messages, so the vectorizer has no column for them. Their meaning is lost, and the model sees only known words such as “my”, “the” and “app”. The model still gives a route (here delivery), but its largest probability is only 0.46. No route gets even half. Look at the probability before you trust the route. A low largest probability is a sign to send the message to a person. Then label new examples and train again.

Recap

  • A bag of words turns a message into counts, one column per word of the vocabulary, and forgets the order.
  • TF-IDF multiplies each count by the IDF, which is large for rare words, and scales each row to length 1. In the tiny example, where weighs 0.632 and my only 0.373.
  • N-grams keep word pairs (“charged twice”); character n-grams keep letter groups, for typos and word endings.
  • On Larkfield, one message is a sparse vector of 14,076 columns, most of them 0. On this inbox, words alone (0.968, US$195 per 1,000) beat words and letters (0.962, US$296). Its messages are unusually regular.
  • TF-IDF is cheap, strong and easy to read. It does not know meaning, and it knows nothing about words that are not in its training messages.

You can now turn messages into numbers. But a row of numbers is not a decision yet. Next, we put three classifiers on top of the same numbers and compare what they cost.

Next: Naive Bayes, logistic regression and linear SVM