Rules and models together
- Lesson 12
- 35 min
- Module 3: Baselines: rules and classical models
Where we are
We have rules and three classical models in the comparison table. Each one is right where the other is wrong. In this lesson, we let the rules decide the certain cases and the model decide the rest, and we stop the two from giving different answers for the same message.
Lesson 12 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceRules and models together
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Use rules and a model together without letting a rule overrule a model that is right more often. Measure each rule against the model, and let a rule go first only where its mistakes are cheap and its catches are expensive.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It puts Grace’s four strong rules and TF-IDF + logistic regression together in three ways. It measures each rule against the model, scores the ways on validation and then once on test, and asks you to build a fourth way. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The rule book, the strong rules and the cost of “rules or a person”, from Keyword rules and regular expressions.
- TF-IDF + logistic regression, the validation set, and the one-message difference, from Naive Bayes, logistic regression and linear SVM.
What you need on your computer (only for the project step)
- Your project folder
larkroutefrom the last lesson, with.venvactive:python -m pytest -qprints16 passed.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.
The “fragile” stamp
Section titled “The “fragile” stamp”At the post office, the trained clerks now sort most letters well. They learned from a year of tally sheets. But the head clerk still has a few rules that she wants kept. One of them: “Every parcel with a fragile stamp goes on the careful shelf.” She does not care what the clerk thinks the parcel is. A plate that goes on the careful shelf by mistake costs a minute. A broken vase costs a customer.
Another of her old rules says: “Every letter with the word Paris goes to the France bag.” The trained clerks are better at this than the rule, because they read the whole address. If the rule goes first, it sends the Texas letters to France again.
So the question is not “rules or a model”. It is: which rule may overrule the model, and when?
The problem: rules that make the model worse
Section titled “The problem: rules that make the model worse”Grace marked four rules as strong: legal or press threats, safety problems, an account that someone else used, and a double charge. “These are the cases that I trust more than a model,” she says. Priya builds what Grace asks for. If a strong rule matches, its route wins; otherwise TF-IDF + logistic regression decides. The project calls this method strong rules + logistic regression.
The result, on the 2,000 test messages:
| Method | Accuracy | Cost per 1,000 |
|---|---|---|
| TF-IDF + logistic regression | 0.962 | US$296.0 |
| strong rules + logistic regression | 0.932 | US$443.0 |
The rules that Grace trusts most make the model worse: 60 more messages wrong, and US$147 more per 1,000 messages. How can four careful rules do that? Look at two messages where a strong rule changed the model’s right answer:
- M-100053: “I would like to return order LK-195050, as the colour is not what I expected. … Separately, I was charged twice for order LK-806352…” The label is
returns_refunds, the first request. The model saysreturns_refunds. The double-charge rule matches “charged twice” and sends it tobilling. - M-100516: “…the smoker box can’t arrive that day. Please arrange delivery for another day…” The label is
delivery, and the model saysdelivery. The safety rule matches “smoker” and sends it toneeds_person.
The model read the whole message. The rule read one phrase. When the rule went first, it threw away the better answer.
The idea: decide who wins, then measure it
Section titled “The idea: decide who wins, then measure it”When a rule and a model can both decide, there are several ways to put them together. Each one answers the question “who wins when they disagree?” differently:
Text description of the diagram
The flow chart starts with a new message and the question: does a strong rule match? If no, the message gets the model’s route. If yes, a second question: do the rule and the model agree? If yes, the message gets that route. If no, there are two branches. With “rules first”, the message gets the rule’s route. With “a disagreement goes to a person”, a person reads the message.
- The model alone. The rules play no part in routing.
- Strong rules first. The rule wins every disagreement. This is the project’s method.
- A disagreement goes to a person. When a strong rule and the model disagree, neither wins: a person reads the message for US$1.50.
- Only one rule first. One chosen rule wins its disagreements; the model decides the rest.
There is no right answer in general. It depends on how often each side is right and on what each kind of mistake costs. So we measure, on the validation messages, as always.
Each strong rule against the model
Section titled “Each strong rule against the model”The fairest test of a rule is this: on the messages that the rule matches, who is right more often, the rule or the model? Priya counts it on the 2,000 validation messages:
- The rule is right
- The model is right
Text description of the chart
A column chart with two columns for each of Grace’s four strong rules, on the validation messages that the rule matches. Legal or press threat: 36 messages; the rule is right on 16 (44%), the model on 34 (94%). Safety problem: 43 messages; the rule 30 (70%), the model 41 (95%). Someone else used the account (takeover): 7 messages; the rule 4 (57%), the model 7 (100%). Double charge: 75 messages; the rule 43 (57%), the model 71 (95%).
The model wins on every rule. The legal-threat rule is right on 16 of its 36 messages; on the same messages, the model is right on 34. Many customers mention a lawyer or a consumer protection office only as a possible next step, after their real request. The labels of this inbox follow the request. The model has learned this from the training messages. The rule cannot.
On all four rules together, the strong rules match 159 validation messages, and on 70 of them the rule and the model disagree. On those 70, the model is right 64 times, and the rule 6 times. So “strong rules first” changes 70 decisions, and 64 of the changes are mistakes.
Choose on the validation messages
Section titled “Choose on the validation messages”Now the four ways, scored on the validation messages. Before you look, predict: which is the cheapest?
Text description of the chart
A column chart of the cost per 1,000 validation messages for four ways. The model alone: 472.0. Strong rules first: 589.0. A disagreement goes to a person: 432.5. Only the safety rule first: 445.0.
On validation, “strong rules first” is the most expensive (US$589.0), as the per-rule counts predicted. “A disagreement goes to a person” is the cheapest (US$432.5). It sends the 70 disagreements to a review, which costs US$52.5 per 1,000 messages. In return, it catches a few expensive mistakes of the model. “Only the safety rule first” is also cheaper than the model alone. The next section explains why.
The test messages, once
Section titled “The test messages, once”Priya chose “a disagreement goes to a person” on validation. Now she scores all four ways on the test messages, once:
| Way | Validation: accuracy, cost per 1,000 | Test: accuracy, cost per 1,000 |
|---|---|---|
| The model alone | 0.958, US$472.0 | 0.962, US$296.0 |
| Strong rules first | 0.929, US$589.0 | 0.932, US$443.0 |
| A disagreement goes to a person | 0.926, US$432.5 | 0.932, US$339.8 |
| Only the safety rule first | 0.953, US$445.0 | 0.955, US$310.0 |
On test, the choice does not hold. The model alone is the cheapest (US$296.0). “A disagreement goes to a person” costs US$339.8: 61 reviews, and almost nothing to catch. On the test set, the model is right on 60 of the 61 disagreements. The validation set happened to have a few expensive mistakes that the reviews caught; the test set did not.
This is an honest result, and an uncomfortable one. Two things are true. “Strong rules first” is clearly worse on both sets: the strong rules should not overrule this model. And the gain of “a disagreement goes to a person” is too small to trust: it depends on a handful of expensive messages. Priya writes both results in the comparison table and keeps the model alone as the main router.
When a rule may go first
Section titled “When a rule may go first”Look again at the last row: “only the safety rule first”. It is cheaper than the model on validation (US$445.0 against US$472.0). It costs a little more on test (US$310.0 against US$296.0). Why does this one rule do so much better than the four together?
The answer is in the cost table. When the safety rule is wrong, a team message goes to needs_person, and a senior person reads it and forwards it: US$2. When the model misses a real safety message, a team answers it automatically: US$40. So the rule may be wrong 20 times for every safety message that it catches, and still break even.
On validation, the rule changed 15 of the model’s routes. Two of them were real needs_person messages that the model had missed (US$80 saved). The other 13 were not (US$26 lost). On test, the model had already found all 99 needs_person messages, so the rule changed 14 routes and caught nothing. The guarantee cost US$14 per 1,000 messages there: 14 messages read by a senior person.
So a rule may go first when two things are true:
- Its mistakes are cheap compared with what it catches. A safety word sent to a person costs US$2; a missed safety problem costs US$40, and much more in real life.
- A person wants a guarantee, not an average. Grace can tell a customer: “Every message that says hurt or smoke is read by a senior person.” A model cannot promise that.
The double-charge rule fails the first test. When it is wrong, a return or a delivery message goes to billing: a wrong queue, US$4, or US$20 if urgent. When it catches something, it saves the same US$4. So it needs to be right more often than the model, and it is not (57% against 95%).
How to stop the two from fighting
Section titled “How to stop the two from fighting”Here is what Priya and Grace agree, as a short checklist for any rule that sits next to a model:
- Measure each rule against the model on the validation messages that the rule matches. If the model is right more often, the rule does not decide.
- Let a rule go first only for a guarantee with cheap mistakes, such as the safety words. Write down why, and check its cost at each new model version.
- Use the other rules as flags or tests, not as routes. “Charged twice” can add a note for the billing team. The rule tests check that the rule book still does what it says.
- Give every rule an owner and a reason. The
whyfield inconfig/rules.jsonis the start. A rule that nobody can explain is a rule that nobody can safely remove.
With rules in their place, Larkfield’s best classical router is TF-IDF + a linear model. On this inbox, it costs US$175.0 to US$296.0 per 1,000 test messages, against US$1,500 for a person who reads everything.
Guided practice: rules and the model
Section titled “Guided practice: rules and the model”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see Grace’s four strong rules.
- In section 2, write your prediction. Then run the cell. On validation, strong rules first costs
589.0, and a disagreement to a person432.5. - In section 3, run the cell. For every strong rule,
model rightis larger thanrule right. - In section 4, run the cell. You see
159matches and70disagreements, with the model right on64. - In section 5, run the test cell once. The model alone costs
296.0.
Now finish the project stage of this module. The last file, larkroute/methods/classical.py, adds three methods to route compare: Naive Bayes, the linear SVM and strong rules + logistic regression. The project finds every method by its file, so no other file changes.
-
In your project folder
larkroute, with.venvactive, copy the methods and their test from the snapshotend-of-m03:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m03cp "$S/larkroute/methods/classical.py" larkroute/methods/cp "$S/tests/test_classical.py" tests/Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m03"Copy-Item "$S\larkroute\methods\classical.py" larkroute\methods\Copy-Item "$S\tests\test_classical.py" tests\ -
Run the tests. The last line is
18 passed. Your project is now the same as the snapshotend-of-m03.Terminal window python -m pytest -q -
Score every method of this module on the test messages. It takes about 15 seconds:
Terminal window route compare --methods "everything to a person,rules or a person,TF-IDF + Naive Bayes,TF-IDF + logistic regression,TF-IDF + linear SVM,strong rules + logistic regression"method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msgTF-IDF + linear SVM 0.966 0.969 1.000 1.000 0.971 0.954 175.0 0.0 175.0 0.14TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.14TF-IDF + Naive Bayes 0.943 0.945 0.917 1.000 0.952 0.922 441.0 0.0 441.0 0.02strong rules + logistic regression 0.932 0.923 0.917 0.970 0.928 0.944 443.0 0.0 443.0 0.16everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.03The table is sorted by total cost, cheapest first. These are the rows of the comparison table for this module. The
ms/msgcolumn changes a little from run to run. -
Compare the model and the strong rules on the validation messages, where you choose:
Terminal window route compare --split validation --methods "TF-IDF + logistic regression,strong rules + logistic regression"The last two lines show
472.0for the model and589.0for the strong rules first.
Your turn: only the safety rule first
Section titled “Your turn: only the safety rule first”Grace says: “Safety is not negotiable. I want every message with a safety word read by a senior person, even when the model is usually right.” In section 6 of the notebook, write safety_first(part): the route is needs_person when the rule a safety problem matches, and the model’s route otherwise. Then compute safety_cost, its cost per 1,000 validation messages. Before you run it, predict: more or less than the model alone (US$472.0)?
Show the answer
def safety_first(part): model_pred = model.predict(part["text"]) return ["needs_person" if safety["regex"].search(t) else m for t, m in zip(part["text"], model_pred)]
safety_cost = round(rt.cost_per_1000(validation, safety_first(validation)), 1)print(safety_cost)445.0Less than the model alone. The rule caught 2 safety messages that the model sent to a team (US$40 each), and its 13 wrong routes cost only US$2 each. On the test messages, the same rule costs US$310.0, against US$296.0 for the model alone. There, the model had found every needs_person message, so the rule only added 14 senior reads. Either way, the guarantee is cheap, because the rule’s mistakes are cheap.
Common mistake
Changing the choice after the test
Symptom: Priya chose “a disagreement goes to a person” on the validation messages (US$432.5 against US$472.0 for the model alone). On the test messages, the model alone wins (US$296.0 against US$339.8). Tomás says: “Easy. The test set says the model alone is best, so let’s choose that.”
Cause: the test set is there to give one honest number for the choice that you made, not to make the choice. If you change the choice because of the test result, the test set has become a second validation set. Its score is then too good in the same way, and nothing honest is left. Here, the two sets also disagree because the difference is small: it comes from a few expensive messages among 2,000.
Fix: report the test result for the choice as it is: the reviews did not pay on the test messages. Then make the next decision with more evidence, not with the test set. Use more labelled messages, or cross-validation on the training messages, as in Train, validation and test sets. Priya keeps the model alone as the main router, for a reason that does not depend on the test set. The gain on validation came from a handful of messages, and simpler is safer.
Consequence in AI work: a team that picks the winner on the test set reports a cost that the next month will not show. When two options are this close, the right report is “about the same, and here is why”, not a winner.
Knowledge check
Section titled “Knowledge check”Check your understanding
Grace's four strong rules are “the cases that she trusts more than a model”. Yet strong rules + logistic regression costs US$443.0 per 1,000 test messages, against US$296.0 for the model alone. Explain why, with the numbers of the validation messages.
Show the answer
When a strong rule goes first, it wins every disagreement with the model. On the validation messages, the strong rules match 159 messages and disagree with the model on 70. The model is right on 64 of the 70, and the rule on only 6. So the rules change 64 right answers into wrong ones. On each rule’s own messages, the model is right more often than the rule. For example, for legal threats, the rule is right on 16 of 36 and the model on 34. A rule reads one phrase. The model reads the whole message, and it has learned that a threat after the real request does not change the route.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The safety rule may go first, but the double-charge rule may not. Use the cost table to explain the difference.
Show the answer
When the safety rule is wrong, a team message goes to needs_person, and a senior person forwards it: US$2. When it catches a safety message that the model missed, it saves US$40. So its mistakes are cheap and its catches are expensive. It can be wrong 20 times per catch and still break even, and it gives Grace a guarantee. When the double-charge rule is wrong, a message goes to the wrong queue (US$4, or US$20 if urgent). A catch saves about the same. So it must be right more often than the model to help, and it is not: 57% against 95% on its validation messages.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: your project now has the end-of-m03 stage. Which command shows the six methods of this module in one table, and which method is the cheapest on the test messages?
Show the answer
route compare --methods "everything to a person,rules or a person,TF-IDF + Naive Bayes,TF-IDF + logistic regression,TF-IDF + linear SVM,strong rules + logistic regression". After python -m pytest -q prints 18 passed, the table lists the six rows, sorted by total cost. The cheapest is TF-IDF + linear SVM: accuracy 0.966 and US$175.0 per 1,000 test messages on this inbox. The most expensive is rules or a person, at US$1,757.8.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Strong rules first makes TF-IDF + logistic regression worse on this inbox. It gives 0.932 and US$443.0 per 1,000 test messages, against 0.962 and US$296.0. On the 70 validation disagreements, the model is right 64 times.
- Measure each rule against the model on the messages that it matches. Here, the model wins for all four strong rules.
- A rule may go first when its mistakes are cheap and its catches are expensive, and a person wants a guarantee. The safety rule is one: US$2 per mistake, US$40 per catch.
- A disagreement goes to a person looked best on validation (US$432.5) and did not hold on test (US$339.8). Report the test result as it is; do not change the choice because of it.
- Your project is at the stage
end-of-m03: 18 tests,route explain, and six methods inroute compare. On this inbox, the best classical router is TF-IDF + a linear SVM: 0.966 and US$175.0 per 1,000 messages.
You can now build, compare and combine classical routers. But TF-IDF still treats “parcel” and colis as two unrelated words, and it knows nothing about a word that it never saw. In Module 4, we use models that give similar sentences similar numbers, in English and in French.