Text as numbers: bag of words and TF-IDF
- Lesson 10
- 35 min
- Module 3: Baselines: rules and classical models
Where we are
Rules read words that we chose. A model needs the text as numbers, and it can choose the words itself. In this lesson, we turn messages into numbers with a bag of words and TF-IDF, with word and character n-grams, and we see why this old method is still a strong baseline.
Lesson 10 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceText as numbers: bag of words and TF-IDF
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Turn each message into a row of numbers that a model can learn from. A rule reads only the words that a person chose. A model can weigh every word of 6,000 messages, but only if the words are numbers first.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It builds a bag of words and TF-IDF for three tiny messages, and turns the 6,000 training messages into numbers. It trains a first model with words and with letters, and sends it a message about a product that it never saw. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Keyword rules, and why a word is not a request, from Keyword rules and regular expressions.
- The comparison table and its cost column, from Meet the inbox.
- Logistic regression and scikit-learn pipelines help, from Classification intuition and Preprocess with a pipeline in the machine learning course. This lesson uses them only as a tool; the next lesson explains the classifiers.
What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this lesson.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.
A tally sheet for every letter
Section titled “A tally sheet for every letter”Go back to the post office. The head clerk wants to learn which letters go to which bag, without writing rules by hand. So for one week, she keeps a tally sheet for every letter. The sheet has one column for each word that she has ever seen on a letter. For each letter, she makes a mark in the column of every word on it.
At the end of the week, she has a big table. It has one row per letter, one column per word, and the bag where each letter went. Now she can look for patterns. Letters with “Paris” and “France” went to the blue bag. Letters with “Paris” and “Texas” did not. She does not need to guess the useful words in advance. The table shows them.
The sheet forgets one thing: the order of the words. “Dog bites man” and “man bites dog” get the same marks. For sorting letters, that is usually fine.
The problem: a model needs numbers
Section titled “The problem: a model needs numbers”The keyword rules of the last lesson read only the words that Priya and Grace chose: “refund”, “reçu”, “smoke”. They never saw the other words of the message, such as “before I buy” or “has nothing to do with it”. Those words show the real request.
A model can do better, because it can look at every word of thousands of labelled messages and find which words go with which route. But a model computes with numbers, not with text. So before any model, we need a way to turn a message into a row of numbers. The head clerk’s tally sheet is that way.
The idea: a bag of words
Section titled “The idea: a bag of words”A bag of words counts each word of a message and forgets the order. The name comes from the picture: you pour the words into a bag, shake it, and count what is inside.
First, we cut each message into tokens: here, the words. Then we make the vocabulary: the list of every token in all the messages. Each message becomes one row, with one column per vocabulary word. The value is how many times the word appears in the message.
A tiny example: three messages
Section titled “A tiny example: three messages”Here are three short messages:
where is my parcelmy parcel is laterefund my order
The vocabulary has seven words, in alphabetical order: is, late, my, order, parcel, refund, where. The bag of words is this table of counts:
| is | late | my | order | parcel | refund | where | |
|---|---|---|---|---|---|---|---|
| message 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 |
| message 2 | 1 | 1 | 1 | 0 | 1 | 0 | 0 |
| message 3 | 0 | 0 | 1 | 1 | 0 | 1 | 0 |
Read the column my. It is 1 in every row. A word that is in every message cannot help to tell messages apart. The columns late, order, refund and where are each in one message only, so they say the most about that message. Plain counts treat all these words the same. We want a number that gives rare words more weight.
TF-IDF: rare words weigh more
Section titled “TF-IDF: rare words weigh more”TF-IDF does this. The name has two parts:
- TF, the term frequency: how often the word is in this message. Here, each count is 1.
- IDF, the inverse document frequency: a weight that is large when few messages have the word, and small when many have it. “Document” is the general name for one text, here one message. “Inverse” means that more messages give a smaller weight.
The TF-IDF value of a word in a message is TF × IDF. Then each row is scaled so that its length is 1. This last step makes a short message and a long message comparable.
Every value of the tiny example
Section titled “Every value of the tiny example”Count how many of the three messages have each word. my is in 3 messages, is and parcel are in 2, and the other four words are in 1. scikit-learn computes the IDF from these counts:
| Word | In how many messages | IDF |
|---|---|---|
my |
3 | 1.000 |
is, parcel |
2 | 1.288 |
late, order, refund, where |
1 | 1.693 |
The formula is IDF = 1 + ln((1 + messages) / (1 + messages with the word)), where ln is the natural logarithm. You do not need to remember it. Remember the pattern: the fewer messages have a word, the larger its IDF.
Now take message 1, where is my parcel. Multiply each count by the IDF: where 1.693, is 1.288, my 1.000, parcel 1.288. The length of this row is 2.680: the square root of 1.693² + 1.288² + 1.000² + 1.288². Divide each value by 2.680, and you have the TF-IDF row. Do the same for the other two messages:
| is | late | my | order | parcel | refund | where | |
|---|---|---|---|---|---|---|---|
| message 1 | 0.480 | 0 | 0.373 | 0 | 0.480 | 0 | 0.632 |
| message 2 | 0.480 | 0.632 | 0.373 | 0 | 0.480 | 0 | 0 |
| message 3 | 0 | 0 | 0.385 | 0.652 | 0 | 0.652 | 0 |
In message 1, where now weighs 0.632, and my only 0.373. The rare word counts almost twice as much as the common one. In code, scikit-learn’s TfidfVectorizer does all of this in one line:
tfidf = TfidfVectorizer()weights = tfidf.fit_transform(tiny)pd.DataFrame(weights.toarray(), columns=tfidf.get_feature_names_out(), index=["msg 1", "msg 2", "msg 3"]).round(3) is late my order parcel refund wheremsg 1 0.48 0.000 0.373 0.000 0.48 0.000 0.632msg 2 0.48 0.632 0.373 0.000 0.48 0.000 0.000msg 3 0.00 0.000 0.385 0.652 0.00 0.652 0.000The fit part learns the vocabulary and the IDF from the messages. The transform part turns messages into rows. As with every step that learns from data, we fit it on the training messages only.
Word pairs and letter groups
Section titled “Word pairs and letter groups”A bag of words loses the order of words, and sometimes the order matters. “Charged twice” says more than “charged” and “twice” apart. So we often add word pairs to the vocabulary: every two words that follow each other. The general name is n-gram: a group of n tokens. One word is a 1-gram, two words a 2-gram. In scikit-learn, ngram_range=(1, 2) keeps words and word pairs.
We can also cut words into character n-grams: groups of n letters inside a word. The word parcel has the 3-letter groups par, arc, rce and cel. Letter groups help in three cases:
- Typos.
delivredshares most of its letter groups withdelivered. - Word endings. French
rembourséandremboursementsharerembours. - Short messages. A chat message of four words has few words, but many letter groups.
The cost is size: letter groups make many more columns than words.
The real vocabulary
Section titled “The real vocabulary”Now the same on Larkfield’s 6,000 training messages. The project’s TF-IDF settings are in larkroute/tfidf.py, and the notebook uses the same ones:
words = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, strip_accents="unicode")X_train = words.fit_transform(train["text"])print(X_train.shape[0], "messages,", X_train.shape[1], "columns (words and word pairs)")6000 messages, 14076 columns (words and word pairs)Each setting has a reason. ngram_range=(1, 2) keeps words and word pairs. min_df=2 keeps only the tokens that appear in at least 2 messages, so a token seen once, such as one order number, makes no column. sublinear_tf=True counts a word that appears five times as a little more than once, not five times more. strip_accents="unicode" removes accents, so livré and livre count as one token. The letter groups of 2 to 5 letters add 24,539 more columns.
So one message is a row of 14,076 numbers. Almost all of them are 0, because one message has only a few of the 14,076 words and pairs. Here is the first test message, a French chat message: “Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?” (“Could the courier leave my parcel LK-596824 with my neighbour?”).
Text description of the diagram
The chart lists the 24 columns that are not zero for the message M-100001, from the largest weight to the smallest. The largest weights are word pairs: “pourrait laisser” 0.286 and “livreur pourrait” 0.277. Then come “pourrait” 0.257, “laisser mon” 0.254, “le livreur” 0.252, “colis lk” 0.244 and “chez ma” 0.243. “Livreur”, “ma voisine” and “voisine” each have 0.242. Common words are at the bottom: “mon” 0.122, “est” 0.118, “le” 0.103 and “lk” 0.061. Almost every message has an order number, so “lk” weighs the least. A note under the chart says that the other 14,052 columns are 0.
A row of numbers like this one is called a vector: a list of numbers in a fixed order. A vector or a table where almost every value is 0 is called sparse. scikit-learn stores only the values that are not zero, so 6,000 rows of 14,076 columns fit easily in memory.
Look at the bottom of the chart. lk is in almost every message, because every order number starts with LK, so its weight is small. The order number itself, 596824, makes no column: it appears in only one message, and min_df=2 dropped it. TF-IDF did, by itself, what a person would do: it ignored the common words and kept the telling ones.
A first model on the numbers
Section titled “A first model on the numbers”The numbers are ready. Now a classifier can learn from them. The next lesson explains how three classifiers work. Here, we use one of them, logistic regression, as a tool, to see whether the numbers are useful. A pipeline joins the vectorizer and the classifier, so that one fit learns both from the training messages:
word_model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, strip_accents="unicode"), LogisticRegression(C=10, max_iter=3000))word_model.fit(train["text"], train["route"])rows = [rt.score("TF-IDF words + logistic regression", test, word_model.predict(test["text"]))]Then Priya adds the letter groups of 2 to 5 letters to the words, with the same classifier. Before you look, predict: do the letters help? French has long word endings, and chat messages are short.
Text description of the chart
A column chart with two columns, the cost of the mistakes in US dollars per 1,000 test messages, for TF-IDF and logistic regression. With words and word pairs only, the cost is 195. With words and letter groups, it is 296. The accuracy is 0.968 with words and 0.962 with words and letters. On French messages, it is 0.956 against 0.952.
The letters did not help here. The words alone reach 0.968 on the test messages and cost US$195 per 1,000. With letters, the model reaches 0.962 and costs US$296. Even the French messages are a little better with words alone (0.956 against 0.952).
This is a real surprise, and it says something about this inbox. As Meet the inbox explains, one language model wrote all the Larkfield messages, so they have few typos and very regular words. A real inbox, with typing errors, slang and very short chat messages, is harder, and there letter groups often help. The only way to know is to test both on your own messages, as here. The case study Route banking questions shows a classical model on real customer questions.
The project’s TF-IDF + logistic regression uses words and letters. Priya chose these settings on the validation messages, where the two were almost equal (0.958 with letters, 0.959 without). The test set then showed a larger difference. The course keeps the project’s choice and reports both rows. A choice made on the test set would make the test score too good.
When to use TF-IDF
Section titled “When to use TF-IDF”TF-IDF is old: the idea of IDF comes from the 1970s. It is still the first model to try for text classification, for four reasons:
- It is cheap. Training on 6,000 messages takes about 3 seconds on a laptop, and routing one message takes about 0.1 milliseconds. No graphics card, no download, no key.
- It is strong when the words carry the meaning. Support messages say “refund”, “tracking” and “password”. The words are the signal.
- It is easy to explain. Each column is a word, so you can read which words a model uses. The next lesson does this.
- It sets the bar. Every newer method in this course must beat it on the same table. On this inbox, the bar is high.
It also has limits:
- It does not know meaning. “Parcel”, “package” and colis are three unrelated columns. It learns that each one goes with
deliveryonly if the training messages have it. - It loses order beyond word pairs. “Not a refund” and “a refund” share most of their columns.
- It knows only the words of its training messages. A new word gets no column at all, as the failure case below shows.
Guided practice: TF-IDF in the notebook
Section titled “Guided practice: TF-IDF in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, predict which column is 1 in all three rows. Then run the cell. The column
myis 1 in every row. - Run the TF-IDF cell. In message 1,
wherehas the weight0.632andmy0.373. - In section 2, run the cell. You see
6000 messages, 14076 columns, and the weights of the first test message. - In section 3, predict: better on English or on French messages? Then run the cell. You see accuracy
0.968, English0.973and French0.956. - In section 4, run the cell with letter groups. The accuracy is
0.962.
Your turn: the words that mean “billing”
Section titled “Your turn: the words that mean “billing””The classifier keeps one weight for each column and each route. A large weight for billing means: “this word pushes a message toward billing”. In section 5 of the notebook, find the 5 words or word pairs with the largest weight for billing in word_model. Before you run it, guess two of them.
Show the answer
vec, clf = word_model[0], word_model[-1]row = list(clf.classes_).index("billing")names = vec.get_feature_names_out()billing_words = [str(names[i]) for i in np.argsort(-clf.coef_[row])[:5]]print(billing_words)['card', 'charge', 'my card', 'vat receipt', 'vat']Nobody wrote these words in a rule. The model found them in the training messages. “Vat receipt” is a sign of this inbox. One of its invoice templates asks for a VAT receipt, so the pair is in 56 billing messages of the training set. The next lesson reads these lists for every route.
Common mistake
A word that the model never saw
Symptom: in the launch month, Larkfield sells a new product, the Grow Hub. A customer writes: “My Grow Hub won’t pair with the app, the sensor light keeps blinking”. The model routes it with no error:
new = "My Grow Hub won't pair with the app, the sensor light keeps blinking"v = word_model[0].transform([new])print("known words and pairs in the message:", [word_model[0].get_feature_names_out()[i] for i in v.indices])print(dict(zip(word_model[-1].classes_, word_model.predict_proba([new])[0].round(2))))known words and pairs in the message: ['app', 'grow', 'keeps', 'light', 'my', 'my grow', 'the', 'the app', 'with', 'with the', 'won']{'account': 0.03, 'billing': 0.06, 'delivery': 0.46, 'needs_person': 0.07, 'orders': 0.01, 'product': 0.18, 'returns_refunds': 0.19}The model chooses delivery, the largest probability (0.46). The message is about setting up a device, not about a parcel.
Cause: TF-IDF knows only the words of its training messages. “Hub”, “pair”, “sensor” and “blinking” make no column, so the model sees only common words such as “my”, “the” and “keeps”. It must still choose a route, and it chooses with almost no evidence.
Fix: look at the probability, not only at the route. No route gets even half of the probability here, which is a sign that the model is unsure. A later module sends such messages to a person, and the production module watches for new words and trains the model again with labelled examples.
Consequence in AI work: a TF-IDF model fails quietly on anything new: a new product, a new kind of complaint, a new language. Its accuracy on the old test set stays high while the new messages go to the wrong team.
Knowledge check
Section titled “Knowledge check”Check your understanding
In the three tiny messages, “my” is in all three and “where” is in one. Without computing, which gets the larger TF-IDF weight in message 1, and why is that useful for routing?
Show the answer
where gets the larger weight (0.632 against 0.373). The IDF part is large when few messages have a word, and my is in every message, so its IDF is the smallest possible (1.000). A word that is in every message cannot tell messages apart, so it should weigh little. A rare word says more about what this message is about. For Larkfield, lk is in almost every message and gets a small weight; words such as livreur or cancel get large ones.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you add letter groups of 2 to 5 letters to the word features of a model for a real chat inbox with many typing errors. Will they help? What did they do on the Larkfield inbox, and why might the two answers differ?
Show the answer
On a real chat inbox with typing errors, letter groups often help. delivred and delivered share most of their letter groups, and short messages get more columns. On the Larkfield inbox, they did not help. Words only reach 0.968 and US$195 per 1,000 test messages; words and letters reach 0.962 and US$296. One language model wrote the Larkfield messages from 27 drafts, so they have few typos and regular words. A result on one inbox is not a rule for every inbox. Test both on your own messages, and choose on the validation set.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A message says “My Grow Hub won’t pair with the app”. The model was trained before the Grow Hub existed. Explain what TF-IDF does with “Hub” and “pair”, and what you should look at before you trust the route.
Show the answer
“Hub” and “pair” were not in the training messages, so the vectorizer has no column for them. Their meaning is lost, and the model sees only known words such as “my”, “the” and “app”. The model still gives a route (here delivery), but its largest probability is only 0.46. No route gets even half. Look at the probability before you trust the route. A low largest probability is a sign to send the message to a person. Then label new examples and train again.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A bag of words turns a message into counts, one column per word of the vocabulary, and forgets the order.
- TF-IDF multiplies each count by the IDF, which is large for rare words, and scales each row to length 1. In the tiny example,
whereweighs 0.632 andmyonly 0.373. - N-grams keep word pairs (“charged twice”); character n-grams keep letter groups, for typos and word endings.
- On Larkfield, one message is a sparse vector of 14,076 columns, most of them 0. On this inbox, words alone (0.968, US$195 per 1,000) beat words and letters (0.962, US$296). Its messages are unusually regular.
- TF-IDF is cheap, strong and easy to read. It does not know meaning, and it knows nothing about words that are not in its training messages.
You can now turn messages into numbers. But a row of numbers is not a decision yet. Next, we put three classifiers on top of the same numbers and compare what they cost.