Skip to content
Français

Naive Bayes, logistic regression and linear SVM

  • Lesson 11
  • 40 min
  • Module 3: Baselines: rules and classical models

Where we are

Our messages are now TF-IDF vectors. A vector is not a decision yet: we need a classifier on top. In this lesson, we train Naive Bayes, logistic regression and a linear SVM in one scikit-learn pipeline, and we compare their accuracy, their speed and the words each one uses.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 11 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceNaive Bayes, logistic regression and linear SVM

What you will be able to do

Put a classifier on top of the TF-IDF numbers, so that each message gets a route and a probability. Three classic classifiers do this in seconds. You choose one with the cost table on the validation messages, and you read the words that it uses.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It works a tiny Naive Bayes example and trains the project’s three models. It chooses on validation, scores the test messages once, and reads the top words of each route. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project step)

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.

At the post office, the head clerk now has a week of tally sheets. Each sheet has one row per letter, one column per word, and the bag where each letter went. She trains three new clerks, and each one learns from the sheets in his own way.

The first clerk counts. For each bag, he counts how often each word appeared on its letters. For a new letter, he asks: “In which bag are these words most common?” The second clerk weighs. He gives each word a weight for each bag, plus or minus. For a new letter, he adds up the weights of its words, and the bag with the largest total wins. The third clerk draws a line. He looks for the border between two bags that keeps the letters of each bag as far as possible from the border.

All three use the same tally sheets. They differ in how they learn from them. This lesson trains the three on Larkfield’s messages and compares them on the same cost table.

After the last lesson, every message is a row of 14,076 TF-IDF numbers. That row does not say “delivery”. We need a classifier: a model that learns, from the labelled training rows, a way to turn a new row into a route.

A good classifier for routing gives more than a route. It gives a probability for each of the seven routes, so that we can see when it is unsure. The Grow Hub message of the last lesson got 0.46 for its top route: a clear sign to send it to a person. All three classifiers below give probabilities.

A flow of five boxes. A message, Where is my parcel? Order LK-123456., goes to TF-IDF, one weight for each word. Then to a classifier, one score for each route. Then to 7 probabilities: delivery 0.53, orders 0.29 and so on. Then to the route: the largest, delivery.
A pipeline: TF-IDF turns the message into numbers, the classifier turns the numbers into one probability per route, and the route is the largest. The probabilities are from the project's TF-IDF + logistic regression.
Text description of the diagram

The diagram is a chain of five boxes. The first box is a message: “Where is my parcel? Order LK-123456.” An arrow leads to TF-IDF, which gives one weight for each word. The next arrow leads to a classifier, which gives one score for each route. The next box shows the 7 probabilities: delivery 0.53, orders 0.29, and the other routes smaller. The last box is the route: the largest probability, delivery. The short message gets only 0.53 for its route, so the model is not very sure.

In scikit-learn, a pipeline joins the steps. One fit learns the vocabulary, the IDF and the classifier from the training messages. One predict runs them all on new messages. You met pipelines in Preprocess with a pipeline. Here, each classifier gets its own pipeline with the same TF-IDF step in front.

Naive Bayes: count the words of each route

Section titled “Naive Bayes: count the words of each route”

Naive Bayes is the counting clerk. It is the oldest and simplest of the three, so we start with it.

Four training messages, two routes:

Message Route
parcel late delivery
where parcel delivery
refund order returns_refunds
refund late returns_refunds

Count the words of each route. Each route has 4 words in all. Then add 1 to every count, so that a word never seen in a route does not give a share of 0. This is called smoothing. The vocabulary has 5 words, so each route’s total becomes 4 + 5 = 9:

Word Delivery count Delivery share Returns count Returns share
late 1 (1+1)/9 = 2/9 1 (1+1)/9 = 2/9
order 0 1/9 1 2/9
parcel 2 3/9 0 1/9
refund 0 1/9 2 3/9
where 1 2/9 0 1/9

A new message arrives: late parcel. For each route, multiply the share of the route in the training messages (here 1/2 each) by the shares of the message’s words:

  • delivery: 1/2 × 2/9 (late) × 3/9 (parcel) = 6/162
  • returns_refunds: 1/2 × 2/9 (late) × 1/9 (parcel) = 2/162

Delivery is three times larger. As probabilities that add up to 1: delivery 6/8 = 0.75, returns_refunds 2/8 = 0.25. The notebook checks this with scikit-learn’s MultinomialNB, and gets 0.75 and 0.25.

The method is called naive because it treats each word as if it had nothing to do with the other words of the message. “Charged” and “twice” each count alone. That is rarely true, and the method works well anyway. “Bayes” is Thomas Bayes, whose rule turns “how common is this word in a route” into “how likely is this route for this message”.

Naive Bayes is very fast: it only counts. It trains on 6,000 messages in 0.2 seconds. Its weakness is the naive part: when many words repeat the same idea, it counts the idea many times and becomes too sure.

Logistic regression: a weight for each word

Section titled “Logistic regression: a weight for each word”

Logistic regression is the weighing clerk. For each route, it learns one weight for each of the 14,076 columns. A positive weight pushes a message toward the route, a negative weight pushes it away. For a new message, the model multiplies each TF-IDF value by its weight and adds up the products: one score per route. Then it turns the seven scores into seven probabilities that add up to 1. The largest score gets the largest probability.

Training finds the weights that make the probabilities of the right routes as large as possible on the training messages. The name comes from the logistic function, the S-shaped curve that turns a score into a probability. Classification intuition in the machine learning course shows it step by step.

The weights make the model easy to explain. The project’s command route explain shows the products for one message:

M-100001 (fr, chat): Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?
Label: delivery. Model: delivery (1.00).
Largest weights for delivery: 'voisine' +0.36, 'ma voisine' +0.36, 'colis lk' +0.34, 'chez ma' +0.34, 'colis' +0.33, 'mon colis' +0.28, 'livreur' +0.27, 'le livreur' +0.24

“Voisine” (neighbour), “colis” (parcel) and “livreur” (courier) push the message toward delivery. Together they give a probability of 1.00. A person who reads the message would point at the same words.

A linear SVM (support vector machine) is the clerk who draws a line. Like logistic regression, it learns one weight per column and per route, and it adds up the products into a score. It trains in a different way. It looks for the border between routes that leaves the widest margin. The margin is the gap between the border and the nearest training messages of each route. Only the messages near the border, the support vectors, decide where it goes. “Linear” means that the border is straight: a weighted sum, as in logistic regression.

An SVM gives scores, not probabilities. So the project wraps it in CalibratedClassifierCV, which learns how to turn the scores into probabilities on parts of the training messages. Calibration, which a later module explains in full, means: the probabilities match how often the model is right.

The notebook builds the project’s three models with the settings of larkroute/tfidf.py and larkroute/methods/classical.py:

models = {
"TF-IDF + Naive Bayes": make_pipeline(features(chars=False), ComplementNB(alpha=0.3)),
"TF-IDF + logistic regression": make_pipeline(features(), LogisticRegression(C=10, max_iter=3000)),
"TF-IDF + linear SVM": make_pipeline(features(), CalibratedClassifierCV(LinearSVC(C=0.5), cv=3)),
}

ComplementNB is a version of Naive Bayes that works better when the routes have different sizes. Here, the history has 2,318 delivery messages and only 491 needs_person. features() gives words and word pairs, and with letters, the letter groups too. Naive Bayes uses words only. C sets how strongly a model may follow the training messages. A larger C lets it fit them more closely.

Before you look, predict: which of the three is the cheapest? Priya scores the three on the validation messages first, because that is where we choose:

method accuracy ... mistakes_per_1000 total_per_1000
0 TF-IDF + linear SVM 0.964 ... 404.0 404.0
1 TF-IDF + logistic regression 0.958 ... 472.0 472.0
2 TF-IDF + Naive Bayes 0.941 ... 510.0 510.0

The linear SVM is the cheapest and the most accurate on validation. Logistic regression is second, and Naive Bayes is third. All three are far below a person who reads everything (US$1,500).

Then she scores the test messages once, for the comparison table:

method accuracy macro_f1 takeover_recall ... total_per_1000 ms_per_message
0 TF-IDF + linear SVM 0.966 0.969 1.000 ... 175.0 0.15
1 TF-IDF + logistic regression 0.962 0.965 0.917 ... 296.0 0.14
2 TF-IDF + Naive Bayes 0.943 0.945 0.917 ... 441.0 0.02

The order is the same as on validation, which is a good sign: the choice was not luck. On this inbox, TF-IDF with a linear SVM reaches 0.966 and costs US$175.0 per 1,000 messages. TF-IDF with logistic regression reaches 0.962 and costs US$296.0. Remember that this inbox is easier than a real one. On real customer messages, the same methods score lower. The case study Route banking questions shows them on real data.

Now the comparison table has its first useful rows. Here they are as one picture, with every method that you have met so far:

A scatter chart of cost per 1,000 test messages against accuracy, from 0.5 to 1.0. A dashed red line at 1,500 marks everything to a person, which decides nothing itself. Rules or a person is an orange point above the line at accuracy 0.555 and cost 1,757.8. Four blue points sit together at the bottom right: TF-IDF + Naive Bayes 0.943 and 441, TF-IDF + logistic regression 0.962 and 296, TF-IDF words only + logistic regression 0.968 and 195, and TF-IDF + linear SVM 0.966 and 175.
Cost against accuracy on the 2,000 test messages, for every method so far. Lower and further right is better. The four TF-IDF models cost a fraction of a person; the rules cost more than a person.
Text description of the diagram

The horizontal axis is the accuracy on the test messages, from 0.5 to 1.0. The vertical axis is the cost per 1,000 test messages in US dollars, from 0 to 1,800. A dashed red line at 1,500 marks everything to a person, which decides nothing itself. The rules or a person method is an orange point above this line, at accuracy 0.555 and cost 1,757.8. Four blue points sit close together at the bottom right. TF-IDF with Naive Bayes is at 0.943 and 441. TF-IDF with logistic regression is at 0.962 and 296. TF-IDF with logistic regression on words only is at 0.968 and 195. TF-IDF with a linear SVM is at 0.966 and 175.

The chart shows two things. First, every trained model is far cheaper than the rules and than a person. Second, among the trained models, a small step in accuracy can be a large step in cost. From 0.962 to 0.966 is 8 messages in 2,000, and the cost falls from 296 to 175. The failure case below shows why.

A linear model keeps one weight per column and per route, so you can read what it learned. The notebook trains logistic regression on words only, so that each column is a readable word, and lists the 5 largest weights of each route:

Seven small bar charts, one per route, each with the 5 words of largest weight. orders: cancel 5.7, lk 5.6, delivery address 4.2, please cancel 4.1, one 4.1. delivery: lk 6.1, days 6.1, delivered 4.5, courier 4.4, deliver 4.0. returns_refunds: return 5.8, ago 5.4, received 4.5, weeks 4.5, est arrive 4.5. billing: card 6.0, charge 5.0, my card 4.3, vat receipt 3.9, vat 3.9. product: dimensions 4.4, stock 3.7, if 3.6, only 3.5, red 3.4. account: account 6.2, my account 5.5, mon compte 5.1, compte 5.0, mon 4.6. needs_person: speak 3.5, template 3.2, real 3.1, person 3.1, speak with 3.0.
The 5 words and word pairs with the largest weight for each route, in TF-IDF + logistic regression on words. Most are what a person would choose; some, such as template and red, are habits of this inbox.
Text description of the diagram

There is one small bar chart for each of the seven routes, with the 5 words or word pairs of largest weight and their weights. orders: cancel 5.7, lk 5.6, delivery address 4.2, please cancel 4.1, one 4.1. delivery: lk 6.1, days 6.1, delivered 4.5, courier 4.4, deliver 4.0. returns_refunds: return 5.8, ago 5.4, received 4.5, weeks 4.5, est arrive 4.5. billing: card 6.0, charge 5.0, my card 4.3, vat receipt 3.9, vat 3.9. product: dimensions 4.4, stock 3.7, if 3.6, only 3.5, red 3.4. account: account 6.2, my account 5.5, mon compte 5.1, compte 5.0, mon 4.6. needs_person: speak 3.5, template 3.2, real 3.1, person 3.1, speak with 3.0.

Most words are what Chloé would choose: “cancel” for orders, “courier” for delivery, “card” and “charge” for billing, “mon compte” for account. The model also found words that no rule had. “Ago” and “weeks” go with returns, as in the guidelines’ “I sent the chair back two weeks ago”. “If” goes with product questions, as in “can I return it if…”. French and English words sit side by side, because both are in the training messages.

Some words are habits of this inbox, not of customers. “Template” is in 34 training messages, all needs_person, from “a real person, not another template answer”. “Red” is in 79 training messages, all product, from “is it available in red?”. A real inbox would not have these exact habits; it would have others. Reading the top words is how you find them, before they surprise you on new messages.

All three models are cheap to run:

Model Seconds to train (6,000 messages) Milliseconds per message Running cost per 1,000
TF-IDF + Naive Bayes 0.2 0.02 US$0.00
TF-IDF + logistic regression 3.0 0.14 US$0.00
TF-IDF + linear SVM 1.9 0.15 US$0.00

These times are from the notebook on a laptop; yours will differ. At the course’s price of US$0.05 per CPU-hour, 1,000 messages cost far less than a cent. The cost that matters is the cost of the mistakes.

  • Naive Bayes for a first look in seconds, for very little training data, or for a very large vocabulary. Here it is the weakest of the three (0.943, US$441).
  • Logistic regression as the default: good probabilities, weights that you can read, and route explain shows why it chose a route. It is the project’s main model in the next modules.
  • A linear SVM when its accuracy is clearly better on validation, as here. Its probabilities need the extra calibration step.

None of the three knows the meaning of a word that it never saw, and none reads the order of words beyond word pairs. Later modules add models that do.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, work out the tiny example on paper. Then run the cell. You see delivery 0.75 and returns_refunds 0.25.
  4. In section 2, run the cell that trains the three models. It prints the seconds to train each one.
  5. In section 3, write your prediction. Then run the validation cell. The SVM costs 404.0, logistic regression 472.0, Naive Bayes 510.0.
  6. In section 4, run the test cell. The SVM costs 175.0 per 1,000 test messages.
  7. In section 5, read the top words. Find template in the needs_person row.
  8. In section 6, run the cell. The first words are voisine and ma voisine.

Now add route explain to your project. The command is one file, larkroute/commands/explain.py. The project finds every command by its file, so no other file changes.

  1. In your project folder larkroute, with .venv active, copy the command, its test and the new README from the snapshot end-of-m03:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m03
    cp "$S/larkroute/commands/explain.py" larkroute/commands/
    cp "$S/tests/test_explain.py" tests/
    cp "$S/README.md" .
  2. Run the tests. The last line is 16 passed:

    Terminal window
    python -m pytest -q
  3. Explain the first test message:

    Terminal window
    route explain M-100001

    The last line starts with Largest weights for delivery: 'voisine' +0.36. The command trains the model first, which takes a few seconds.

Your turn: where the two best models disagree

Section titled “Your turn: where the two best models disagree”

Logistic regression and the SVM have almost the same accuracy. Do they make the same mistakes? Predict: on how many of the 2,000 test messages do they give different routes? In section 7 of the notebook, compute disagree, then lr_right and svm_right: how many of those messages each model gets right.

Show the answer
differ = lr_pred != svm_pred
truth = test["route"].to_numpy()
disagree = int(differ.sum())
lr_right = int((lr_pred[differ] == truth[differ]).sum())
svm_right = int((svm_pred[differ] == truth[differ]).sum())
print(disagree, lr_right, svm_right)
24 6 14

The two models agree on 1,976 of 2,000 messages. On the 24 others, the SVM is right 14 times and logistic regression 6 times; on 4, both are wrong. So the whole difference between the two models is in about two dozen messages. Next, see which of them matter most.

Common mistake

One message makes most of the difference

Symptom: the SVM costs US$121 less per 1,000 test messages than logistic regression (175.0 against 296.0). Tomás says: “The SVM is much better. Let’s tell everyone that the SVM halves the cost of mistakes.”

Cause: look at the messages behind the number. On the 2,000 test messages, the difference is US$242, and US$200 of it is one message: the account takeover M-108268. Logistic regression sends it to billing. route explain M-108268 shows why:

Label: account. Model: billing (0.46).
Largest weights for billing: 'card' +0.24, 'my card' +0.18, 'total' +0.17, 'was charged' +0.11, 'order total' +0.10, 'charged' +0.08, 'total was' +0.07, 'rg' +0.07
Largest weights for account: 'account' +0.18, 'the email' +0.17, 'my account' +0.16, 'email on' +0.15, 'account and' +0.14, 'an order' +0.14, 'help' +0.10, 'email' +0.09

The customer wrote about a changed email and a charge on the card, and the billing words won, with a probability of only 0.46. The test set has only 12 account takeovers, so one of them moves the cost by US$100 per 1,000 messages.

Fix: read the messages behind a difference before you believe it. Compare on the validation messages too: there, the SVM is also cheaper (404 against 472), so the choice holds, but the gap is smaller. Report the difference with its cause: “the SVM found one more account takeover”, not “the SVM halves the cost”.

Consequence in AI work: a small test set with a few very expensive messages makes cost differences jump. A team that reports “41% cheaper” from one message promises a saving that the next month will not show. Later in this course, the rare route gets its own lesson.

Check your understanding

In the tiny Naive Bayes example, why does the method add 1 to every count? What would happen to the message “late parcel” without it?

Show the answer

Adding 1 (smoothing) makes sure that a word never seen in a route does not give a share of 0. Without it, parcel has a count of 0 in returns_refunds, so its share is 0. Then the whole product for returns_refunds is 0, and the model is 100% sure of delivery from one word. With smoothing, returns_refunds keeps a small share (1/9), and the probabilities are 0.75 and 0.25. On real data, one unseen word would otherwise rule out every route that does not happen to have it.

Check your understanding

Logistic regression gives “needs_person” a large weight for the word “template”. Is that a good word for routing? What should you do about it?

Show the answer

It works on this inbox: “template” is in 34 training messages, all needs_person, from “a real person, not another template answer”. But it is a habit of how these messages were written, not a sign that customers in general use. A real complaint that asks for a manager may never say “template”, and a normal message might. Read the top words of each route, note such words, and check them on the validation messages and on new messages. Do not trust the model because its top words look clever; trust the scores on messages that it did not learn from.

Check your understanding

The SVM costs US$175.0 per 1,000 test messages and logistic regression US$296.0. Explain where most of the difference comes from, and how you would report it to Grace.

Show the answer

Most of it comes from one account takeover (M-108268) that logistic regression sends to billing. A missed takeover costs US$200, which is US$100 per 1,000 on the 2,000 test messages. The two models disagree on only 24 messages. The SVM was also cheaper on the validation messages (404 against 472), so choosing it is sound. Report it with the validation numbers, as “the SVM is a little more accurate, and it found the one account takeover that logistic regression missed”. Do not report it as “the SVM halves the cost of mistakes”.

Recap

  • A classifier turns a TF-IDF vector into one probability per route. A pipeline joins the vectorizer and the classifier, so both learn from the training messages only.
  • Naive Bayes counts words per route and multiplies their shares, with smoothing for unseen words. Logistic regression adds one weight per word into a score per route. A linear SVM finds the border with the widest margin, and needs calibration for probabilities.
  • On this inbox, chosen on validation and scored once on test: linear SVM 0.966 (US$175.0 per 1,000). Logistic regression: 0.962 (US$296.0). Naive Bayes: 0.943 (US$441.0). All cost a fraction of a person (US$1,500).
  • The top words show what a model learned, including habits of this inbox such as “template” and “red”. route explain shows the words behind one decision.
  • A difference in cost can come from one expensive message. Read the messages behind a difference before you report it.

You can now train and compare classical models. But the rules of the first lesson still know things, such as the safety words, that a person wants guaranteed. Next, we let rules and a model work together, and we measure whether they help or fight.

Next: Rules and models together