Naive Bayes, logistic regression and linear SVM
- Lesson 11
- 40 min
- Module 3: Baselines: rules and classical models
Where we are
Our messages are now TF-IDF vectors. A vector is not a decision yet: we need a classifier on top. In this lesson, we train Naive Bayes, logistic regression and a linear SVM in one scikit-learn pipeline, and we compare their accuracy, their speed and the words each one uses.
Lesson 11 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceNaive Bayes, logistic regression and linear SVM
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Put a classifier on top of the TF-IDF numbers, so that each message gets a route and a probability. Three classic classifiers do this in seconds. You choose one with the cost table on the validation messages, and you read the words that it uses.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It works a tiny Naive Bayes example and trains the project’s three models. It chooses on validation, scores the test messages once, and reads the top words of each route. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- TF-IDF, word pairs and letter groups, from Text as numbers: bag of words and TF-IDF.
- Why we choose on the validation set and score the test set once, from Train, validation and test sets in the machine learning course.
- Probabilities as a model’s output help, from Classification intuition in the machine learning course. This lesson explains what it needs in a few lines.
What you need on your computer (only for the project step)
- Your project folder
larkroutefrom Keyword rules and regular expressions, with.venvactive:python -m pytest -qprints15 passed.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.
Three new clerks
Section titled “Three new clerks”At the post office, the head clerk now has a week of tally sheets. Each sheet has one row per letter, one column per word, and the bag where each letter went. She trains three new clerks, and each one learns from the sheets in his own way.
The first clerk counts. For each bag, he counts how often each word appeared on its letters. For a new letter, he asks: “In which bag are these words most common?” The second clerk weighs. He gives each word a weight for each bag, plus or minus. For a new letter, he adds up the weights of its words, and the bag with the largest total wins. The third clerk draws a line. He looks for the border between two bags that keeps the letters of each bag as far as possible from the border.
All three use the same tally sheets. They differ in how they learn from them. This lesson trains the three on Larkfield’s messages and compares them on the same cost table.
The problem: a vector is not a decision
Section titled “The problem: a vector is not a decision”After the last lesson, every message is a row of 14,076 TF-IDF numbers. That row does not say “delivery”. We need a classifier: a model that learns, from the labelled training rows, a way to turn a new row into a route.
A good classifier for routing gives more than a route. It gives a probability for each of the seven routes, so that we can see when it is unsure. The Grow Hub message of the last lesson got 0.46 for its top route: a clear sign to send it to a person. All three classifiers below give probabilities.
Text description of the diagram
The diagram is a chain of five boxes. The first box is a message: “Where is my parcel? Order LK-123456.” An arrow leads to TF-IDF, which gives one weight for each word. The next arrow leads to a classifier, which gives one score for each route. The next box shows the 7 probabilities: delivery 0.53, orders 0.29, and the other routes smaller. The last box is the route: the largest probability, delivery. The short message gets only 0.53 for its route, so the model is not very sure.
In scikit-learn, a pipeline joins the steps. One fit learns the vocabulary, the IDF and the classifier from the training messages. One predict runs them all on new messages. You met pipelines in Preprocess with a pipeline. Here, each classifier gets its own pipeline with the same TF-IDF step in front.
Naive Bayes: count the words of each route
Section titled “Naive Bayes: count the words of each route”Naive Bayes is the counting clerk. It is the oldest and simplest of the three, so we start with it.
A tiny example with every value
Section titled “A tiny example with every value”Four training messages, two routes:
| Message | Route |
|---|---|
parcel late |
delivery |
where parcel |
delivery |
refund order |
returns_refunds |
refund late |
returns_refunds |
Count the words of each route. Each route has 4 words in all. Then add 1 to every count, so that a word never seen in a route does not give a share of 0. This is called smoothing. The vocabulary has 5 words, so each route’s total becomes 4 + 5 = 9:
| Word | Delivery count | Delivery share | Returns count | Returns share |
|---|---|---|---|---|
late |
1 | (1+1)/9 = 2/9 | 1 | (1+1)/9 = 2/9 |
order |
0 | 1/9 | 1 | 2/9 |
parcel |
2 | 3/9 | 0 | 1/9 |
refund |
0 | 1/9 | 2 | 3/9 |
where |
1 | 2/9 | 0 | 1/9 |
A new message arrives: late parcel. For each route, multiply the share of the route in the training messages (here 1/2 each) by the shares of the message’s words:
- delivery: 1/2 × 2/9 (
late) × 3/9 (parcel) = 6/162 - returns_refunds: 1/2 × 2/9 (
late) × 1/9 (parcel) = 2/162
Delivery is three times larger. As probabilities that add up to 1: delivery 6/8 = 0.75, returns_refunds 2/8 = 0.25. The notebook checks this with scikit-learn’s MultinomialNB, and gets 0.75 and 0.25.
The method is called naive because it treats each word as if it had nothing to do with the other words of the message. “Charged” and “twice” each count alone. That is rarely true, and the method works well anyway. “Bayes” is Thomas Bayes, whose rule turns “how common is this word in a route” into “how likely is this route for this message”.
Naive Bayes is very fast: it only counts. It trains on 6,000 messages in 0.2 seconds. Its weakness is the naive part: when many words repeat the same idea, it counts the idea many times and becomes too sure.
Logistic regression: a weight for each word
Section titled “Logistic regression: a weight for each word”Logistic regression is the weighing clerk. For each route, it learns one weight for each of the 14,076 columns. A positive weight pushes a message toward the route, a negative weight pushes it away. For a new message, the model multiplies each TF-IDF value by its weight and adds up the products: one score per route. Then it turns the seven scores into seven probabilities that add up to 1. The largest score gets the largest probability.
Training finds the weights that make the probabilities of the right routes as large as possible on the training messages. The name comes from the logistic function, the S-shaped curve that turns a score into a probability. Classification intuition in the machine learning course shows it step by step.
The weights make the model easy to explain. The project’s command route explain shows the products for one message:
M-100001 (fr, chat): Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?Label: delivery. Model: delivery (1.00).Largest weights for delivery: 'voisine' +0.36, 'ma voisine' +0.36, 'colis lk' +0.34, 'chez ma' +0.34, 'colis' +0.33, 'mon colis' +0.28, 'livreur' +0.27, 'le livreur' +0.24“Voisine” (neighbour), “colis” (parcel) and “livreur” (courier) push the message toward delivery. Together they give a probability of 1.00. A person who reads the message would point at the same words.
Linear SVM: the widest gap
Section titled “Linear SVM: the widest gap”A linear SVM (support vector machine) is the clerk who draws a line. Like logistic regression, it learns one weight per column and per route, and it adds up the products into a score. It trains in a different way. It looks for the border between routes that leaves the widest margin. The margin is the gap between the border and the nearest training messages of each route. Only the messages near the border, the support vectors, decide where it goes. “Linear” means that the border is straight: a weighted sum, as in logistic regression.
An SVM gives scores, not probabilities. So the project wraps it in CalibratedClassifierCV, which learns how to turn the scores into probabilities on parts of the training messages. Calibration, which a later module explains in full, means: the probabilities match how often the model is right.
One pipeline each, on the same messages
Section titled “One pipeline each, on the same messages”The notebook builds the project’s three models with the settings of larkroute/tfidf.py and larkroute/methods/classical.py:
models = { "TF-IDF + Naive Bayes": make_pipeline(features(chars=False), ComplementNB(alpha=0.3)), "TF-IDF + logistic regression": make_pipeline(features(), LogisticRegression(C=10, max_iter=3000)), "TF-IDF + linear SVM": make_pipeline(features(), CalibratedClassifierCV(LinearSVC(C=0.5), cv=3)),}ComplementNB is a version of Naive Bayes that works better when the routes have different sizes. Here, the history has 2,318 delivery messages and only 491 needs_person. features() gives words and word pairs, and with letters, the letter groups too. Naive Bayes uses words only. C sets how strongly a model may follow the training messages. A larger C lets it fit them more closely.
Choose on the validation messages
Section titled “Choose on the validation messages”Before you look, predict: which of the three is the cheapest? Priya scores the three on the validation messages first, because that is where we choose:
method accuracy ... mistakes_per_1000 total_per_10000 TF-IDF + linear SVM 0.964 ... 404.0 404.01 TF-IDF + logistic regression 0.958 ... 472.0 472.02 TF-IDF + Naive Bayes 0.941 ... 510.0 510.0The linear SVM is the cheapest and the most accurate on validation. Logistic regression is second, and Naive Bayes is third. All three are far below a person who reads everything (US$1,500).
Score the test messages once
Section titled “Score the test messages once”Then she scores the test messages once, for the comparison table:
method accuracy macro_f1 takeover_recall ... total_per_1000 ms_per_message0 TF-IDF + linear SVM 0.966 0.969 1.000 ... 175.0 0.151 TF-IDF + logistic regression 0.962 0.965 0.917 ... 296.0 0.142 TF-IDF + Naive Bayes 0.943 0.945 0.917 ... 441.0 0.02The order is the same as on validation, which is a good sign: the choice was not luck. On this inbox, TF-IDF with a linear SVM reaches 0.966 and costs US$175.0 per 1,000 messages. TF-IDF with logistic regression reaches 0.962 and costs US$296.0. Remember that this inbox is easier than a real one. On real customer messages, the same methods score lower. The case study Route banking questions shows them on real data.
Now the comparison table has its first useful rows. Here they are as one picture, with every method that you have met so far:
Text description of the diagram
The horizontal axis is the accuracy on the test messages, from 0.5 to 1.0. The vertical axis is the cost per 1,000 test messages in US dollars, from 0 to 1,800. A dashed red line at 1,500 marks everything to a person, which decides nothing itself. The rules or a person method is an orange point above this line, at accuracy 0.555 and cost 1,757.8. Four blue points sit close together at the bottom right. TF-IDF with Naive Bayes is at 0.943 and 441. TF-IDF with logistic regression is at 0.962 and 296. TF-IDF with logistic regression on words only is at 0.968 and 195. TF-IDF with a linear SVM is at 0.966 and 175.
The chart shows two things. First, every trained model is far cheaper than the rules and than a person. Second, among the trained models, a small step in accuracy can be a large step in cost. From 0.962 to 0.966 is 8 messages in 2,000, and the cost falls from 296 to 175. The failure case below shows why.
The words each model relies on
Section titled “The words each model relies on”A linear model keeps one weight per column and per route, so you can read what it learned. The notebook trains logistic regression on words only, so that each column is a readable word, and lists the 5 largest weights of each route:
Text description of the diagram
There is one small bar chart for each of the seven routes, with the 5 words or word pairs of largest weight and their weights. orders: cancel 5.7, lk 5.6, delivery address 4.2, please cancel 4.1, one 4.1. delivery: lk 6.1, days 6.1, delivered 4.5, courier 4.4, deliver 4.0. returns_refunds: return 5.8, ago 5.4, received 4.5, weeks 4.5, est arrive 4.5. billing: card 6.0, charge 5.0, my card 4.3, vat receipt 3.9, vat 3.9. product: dimensions 4.4, stock 3.7, if 3.6, only 3.5, red 3.4. account: account 6.2, my account 5.5, mon compte 5.1, compte 5.0, mon 4.6. needs_person: speak 3.5, template 3.2, real 3.1, person 3.1, speak with 3.0.
Most words are what Chloé would choose: “cancel” for orders, “courier” for delivery, “card” and “charge” for billing, “mon compte” for account. The model also found words that no rule had. “Ago” and “weeks” go with returns, as in the guidelines’ “I sent the chair back two weeks ago”. “If” goes with product questions, as in “can I return it if…”. French and English words sit side by side, because both are in the training messages.
Some words are habits of this inbox, not of customers. “Template” is in 34 training messages, all needs_person, from “a real person, not another template answer”. “Red” is in 79 training messages, all product, from “is it available in red?”. A real inbox would not have these exact habits; it would have others. Reading the top words is how you find them, before they surprise you on new messages.
Speed and cost
Section titled “Speed and cost”All three models are cheap to run:
| Model | Seconds to train (6,000 messages) | Milliseconds per message | Running cost per 1,000 |
|---|---|---|---|
| TF-IDF + Naive Bayes | 0.2 | 0.02 | US$0.00 |
| TF-IDF + logistic regression | 3.0 | 0.14 | US$0.00 |
| TF-IDF + linear SVM | 1.9 | 0.15 | US$0.00 |
These times are from the notebook on a laptop; yours will differ. At the course’s price of US$0.05 per CPU-hour, 1,000 messages cost far less than a cent. The cost that matters is the cost of the mistakes.
When to use which
Section titled “When to use which”- Naive Bayes for a first look in seconds, for very little training data, or for a very large vocabulary. Here it is the weakest of the three (0.943, US$441).
- Logistic regression as the default: good probabilities, weights that you can read, and
route explainshows why it chose a route. It is the project’s main model in the next modules. - A linear SVM when its accuracy is clearly better on validation, as here. Its probabilities need the extra calibration step.
None of the three knows the meaning of a word that it never saw, and none reads the order of words beyond word pairs. Later modules add models that do.
Guided practice: three classifiers
Section titled “Guided practice: three classifiers”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, work out the tiny example on paper. Then run the cell. You see
delivery0.75andreturns_refunds0.25. - In section 2, run the cell that trains the three models. It prints the seconds to train each one.
- In section 3, write your prediction. Then run the validation cell. The SVM costs
404.0, logistic regression472.0, Naive Bayes510.0. - In section 4, run the test cell. The SVM costs
175.0per 1,000 test messages. - In section 5, read the top words. Find
templatein theneeds_personrow. - In section 6, run the cell. The first words are
voisineandma voisine.
Now add route explain to your project. The command is one file, larkroute/commands/explain.py. The project finds every command by its file, so no other file changes.
-
In your project folder
larkroute, with.venvactive, copy the command, its test and the new README from the snapshotend-of-m03:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m03cp "$S/larkroute/commands/explain.py" larkroute/commands/cp "$S/tests/test_explain.py" tests/cp "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m03"Copy-Item "$S\larkroute\commands\explain.py" larkroute\commands\Copy-Item "$S\tests\test_explain.py" tests\Copy-Item "$S\README.md" . -
Run the tests. The last line is
16 passed:Terminal window python -m pytest -q -
Explain the first test message:
Terminal window route explain M-100001The last line starts with
Largest weights for delivery: 'voisine' +0.36. The command trains the model first, which takes a few seconds.
Your turn: where the two best models disagree
Section titled “Your turn: where the two best models disagree”Logistic regression and the SVM have almost the same accuracy. Do they make the same mistakes? Predict: on how many of the 2,000 test messages do they give different routes? In section 7 of the notebook, compute disagree, then lr_right and svm_right: how many of those messages each model gets right.
Show the answer
differ = lr_pred != svm_predtruth = test["route"].to_numpy()disagree = int(differ.sum())lr_right = int((lr_pred[differ] == truth[differ]).sum())svm_right = int((svm_pred[differ] == truth[differ]).sum())print(disagree, lr_right, svm_right)24 6 14The two models agree on 1,976 of 2,000 messages. On the 24 others, the SVM is right 14 times and logistic regression 6 times; on 4, both are wrong. So the whole difference between the two models is in about two dozen messages. Next, see which of them matter most.
Common mistake
One message makes most of the difference
Symptom: the SVM costs US$121 less per 1,000 test messages than logistic regression (175.0 against 296.0). Tomás says: “The SVM is much better. Let’s tell everyone that the SVM halves the cost of mistakes.”
Cause: look at the messages behind the number. On the 2,000 test messages, the difference is US$242, and US$200 of it is one message: the account takeover M-108268. Logistic regression sends it to billing. route explain M-108268 shows why:
Label: account. Model: billing (0.46).Largest weights for billing: 'card' +0.24, 'my card' +0.18, 'total' +0.17, 'was charged' +0.11, 'order total' +0.10, 'charged' +0.08, 'total was' +0.07, 'rg' +0.07Largest weights for account: 'account' +0.18, 'the email' +0.17, 'my account' +0.16, 'email on' +0.15, 'account and' +0.14, 'an order' +0.14, 'help' +0.10, 'email' +0.09The customer wrote about a changed email and a charge on the card, and the billing words won, with a probability of only 0.46. The test set has only 12 account takeovers, so one of them moves the cost by US$100 per 1,000 messages.
Fix: read the messages behind a difference before you believe it. Compare on the validation messages too: there, the SVM is also cheaper (404 against 472), so the choice holds, but the gap is smaller. Report the difference with its cause: “the SVM found one more account takeover”, not “the SVM halves the cost”.
Consequence in AI work: a small test set with a few very expensive messages makes cost differences jump. A team that reports “41% cheaper” from one message promises a saving that the next month will not show. Later in this course, the rare route gets its own lesson.
Knowledge check
Section titled “Knowledge check”Check your understanding
In the tiny Naive Bayes example, why does the method add 1 to every count? What would happen to the message “late parcel” without it?
Show the answer
Adding 1 (smoothing) makes sure that a word never seen in a route does not give a share of 0. Without it, parcel has a count of 0 in returns_refunds, so its share is 0. Then the whole product for returns_refunds is 0, and the model is 100% sure of delivery from one word. With smoothing, returns_refunds keeps a small share (1/9), and the probabilities are 0.75 and 0.25. On real data, one unseen word would otherwise rule out every route that does not happen to have it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Logistic regression gives “needs_person” a large weight for the word “template”. Is that a good word for routing? What should you do about it?
Show the answer
It works on this inbox: “template” is in 34 training messages, all needs_person, from “a real person, not another template answer”. But it is a habit of how these messages were written, not a sign that customers in general use. A real complaint that asks for a manager may never say “template”, and a normal message might. Read the top words of each route, note such words, and check them on the validation messages and on new messages. Do not trust the model because its top words look clever; trust the scores on messages that it did not learn from.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The SVM costs US$175.0 per 1,000 test messages and logistic regression US$296.0. Explain where most of the difference comes from, and how you would report it to Grace.
Show the answer
Most of it comes from one account takeover (M-108268) that logistic regression sends to billing. A missed takeover costs US$200, which is US$100 per 1,000 on the 2,000 test messages. The two models disagree on only 24 messages. The SVM was also cheaper on the validation messages (404 against 472), so choosing it is sound. Report it with the validation numbers, as “the SVM is a little more accurate, and it found the one account takeover that logistic regression missed”. Do not report it as “the SVM halves the cost of mistakes”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A classifier turns a TF-IDF vector into one probability per route. A pipeline joins the vectorizer and the classifier, so both learn from the training messages only.
- Naive Bayes counts words per route and multiplies their shares, with smoothing for unseen words. Logistic regression adds one weight per word into a score per route. A linear SVM finds the border with the widest margin, and needs calibration for probabilities.
- On this inbox, chosen on validation and scored once on test: linear SVM 0.966 (US$175.0 per 1,000). Logistic regression: 0.962 (US$296.0). Naive Bayes: 0.943 (US$441.0). All cost a fraction of a person (US$1,500).
- The top words show what a model learned, including habits of this inbox such as “template” and “red”.
route explainshows the words behind one decision. - A difference in cost can come from one expensive message. Read the messages behind a difference before you report it.
You can now train and compare classical models. But the rules of the first lesson still know things, such as the safety words, that a person wants guaranteed. Next, we let rules and a model work together, and we measure whether they help or fight.