Skip to content
Français

Rules and models together

  • Lesson 12
  • 35 min
  • Module 3: Baselines: rules and classical models

Where we are

We have rules and three classical models in the comparison table. Each one is right where the other is wrong. In this lesson, we let the rules decide the certain cases and the model decide the rest, and we stop the two from giving different answers for the same message.

Lesson 12 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceRules and models together

What you will be able to do

Use rules and a model together without letting a rule overrule a model that is right more often. Measure each rule against the model, and let a rule go first only where its mistakes are cheap and its catches are expensive.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It puts Grace’s four strong rules and TF-IDF + logistic regression together in three ways. It measures each rule against the model, scores the ways on validation and then once on test, and asks you to build a fourth way. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project step)

  • Your project folder larkroute from the last lesson, with .venv active: python -m pytest -q prints 16 passed.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.

At the post office, the trained clerks now sort most letters well. They learned from a year of tally sheets. But the head clerk still has a few rules that she wants kept. One of them: “Every parcel with a fragile stamp goes on the careful shelf.” She does not care what the clerk thinks the parcel is. A plate that goes on the careful shelf by mistake costs a minute. A broken vase costs a customer.

Another of her old rules says: “Every letter with the word Paris goes to the France bag.” The trained clerks are better at this than the rule, because they read the whole address. If the rule goes first, it sends the Texas letters to France again.

So the question is not “rules or a model”. It is: which rule may overrule the model, and when?

The problem: rules that make the model worse

Section titled “The problem: rules that make the model worse”

Grace marked four rules as strong: legal or press threats, safety problems, an account that someone else used, and a double charge. “These are the cases that I trust more than a model,” she says. Priya builds what Grace asks for. If a strong rule matches, its route wins; otherwise TF-IDF + logistic regression decides. The project calls this method strong rules + logistic regression.

The result, on the 2,000 test messages:

Method Accuracy Cost per 1,000
TF-IDF + logistic regression 0.962 US$296.0
strong rules + logistic regression 0.932 US$443.0

The rules that Grace trusts most make the model worse: 60 more messages wrong, and US$147 more per 1,000 messages. How can four careful rules do that? Look at two messages where a strong rule changed the model’s right answer:

  • M-100053: “I would like to return order LK-195050, as the colour is not what I expected. … Separately, I was charged twice for order LK-806352…” The label is returns_refunds, the first request. The model says returns_refunds. The double-charge rule matches “charged twice” and sends it to billing.
  • M-100516: “…the smoker box can’t arrive that day. Please arrange delivery for another day…” The label is delivery, and the model says delivery. The safety rule matches “smoker” and sends it to needs_person.

The model read the whole message. The rule read one phrase. When the rule went first, it threw away the better answer.

The idea: decide who wins, then measure it

Section titled “The idea: decide who wins, then measure it”

When a rule and a model can both decide, there are several ways to put them together. Each one answers the question “who wins when they disagree?” differently:

A flow chart. A new message goes to the question: does a strong rule match? If no, the model's route. If yes, the question: do the rule and the model agree? If yes, that route. If no, there are two branches: rules first leads to the rule's route; a disagreement goes to a person leads to a person reads it.
Rules and a model together. When a strong rule matches and the model agrees, there is nothing to decide. When they disagree, you must choose who wins: the rule, or a person.
Text description of the diagram

The flow chart starts with a new message and the question: does a strong rule match? If no, the message gets the model’s route. If yes, a second question: do the rule and the model agree? If yes, the message gets that route. If no, there are two branches. With “rules first”, the message gets the rule’s route. With “a disagreement goes to a person”, a person reads the message.

  1. The model alone. The rules play no part in routing.
  2. Strong rules first. The rule wins every disagreement. This is the project’s method.
  3. A disagreement goes to a person. When a strong rule and the model disagree, neither wins: a person reads the message for US$1.50.
  4. Only one rule first. One chosen rule wins its disagreements; the model decides the rest.

There is no right answer in general. It depends on how often each side is right and on what each kind of mistake costs. So we measure, on the validation messages, as always.

The fairest test of a rule is this: on the messages that the rule matches, who is right more often, the rule or the model? Priya counts it on the 2,000 validation messages:

  • The rule is right
  • The model is right
On the validation messages that each strong rule matches (legal threat, safety, takeover, double charge), the model is right more often than the rule, for all four rules.
Text description of the chart

A column chart with two columns for each of Grace’s four strong rules, on the validation messages that the rule matches. Legal or press threat: 36 messages; the rule is right on 16 (44%), the model on 34 (94%). Safety problem: 43 messages; the rule 30 (70%), the model 41 (95%). Someone else used the account (takeover): 7 messages; the rule 4 (57%), the model 7 (100%). Double charge: 75 messages; the rule 43 (57%), the model 71 (95%).

The model wins on every rule. The legal-threat rule is right on 16 of its 36 messages; on the same messages, the model is right on 34. Many customers mention a lawyer or a consumer protection office only as a possible next step, after their real request. The labels of this inbox follow the request. The model has learned this from the training messages. The rule cannot.

On all four rules together, the strong rules match 159 validation messages, and on 70 of them the rule and the model disagree. On those 70, the model is right 64 times, and the rule 6 times. So “strong rules first” changes 70 decisions, and 64 of the changes are mistakes.

Now the four ways, scored on the validation messages. Before you look, predict: which is the cheapest?

The four ways on the validation messages. Rules first is the most expensive; sending the disagreements to a person is the cheapest.
Text description of the chart

A column chart of the cost per 1,000 validation messages for four ways. The model alone: 472.0. Strong rules first: 589.0. A disagreement goes to a person: 432.5. Only the safety rule first: 445.0.

On validation, “strong rules first” is the most expensive (US$589.0), as the per-rule counts predicted. “A disagreement goes to a person” is the cheapest (US$432.5). It sends the 70 disagreements to a review, which costs US$52.5 per 1,000 messages. In return, it catches a few expensive mistakes of the model. “Only the safety rule first” is also cheaper than the model alone. The next section explains why.

Priya chose “a disagreement goes to a person” on validation. Now she scores all four ways on the test messages, once:

Way Validation: accuracy, cost per 1,000 Test: accuracy, cost per 1,000
The model alone 0.958, US$472.0 0.962, US$296.0
Strong rules first 0.929, US$589.0 0.932, US$443.0
A disagreement goes to a person 0.926, US$432.5 0.932, US$339.8
Only the safety rule first 0.953, US$445.0 0.955, US$310.0

On test, the choice does not hold. The model alone is the cheapest (US$296.0). “A disagreement goes to a person” costs US$339.8: 61 reviews, and almost nothing to catch. On the test set, the model is right on 60 of the 61 disagreements. The validation set happened to have a few expensive mistakes that the reviews caught; the test set did not.

This is an honest result, and an uncomfortable one. Two things are true. “Strong rules first” is clearly worse on both sets: the strong rules should not overrule this model. And the gain of “a disagreement goes to a person” is too small to trust: it depends on a handful of expensive messages. Priya writes both results in the comparison table and keeps the model alone as the main router.

Look again at the last row: “only the safety rule first”. It is cheaper than the model on validation (US$445.0 against US$472.0). It costs a little more on test (US$310.0 against US$296.0). Why does this one rule do so much better than the four together?

The answer is in the cost table. When the safety rule is wrong, a team message goes to needs_person, and a senior person reads it and forwards it: US$2. When the model misses a real safety message, a team answers it automatically: US$40. So the rule may be wrong 20 times for every safety message that it catches, and still break even.

On validation, the rule changed 15 of the model’s routes. Two of them were real needs_person messages that the model had missed (US$80 saved). The other 13 were not (US$26 lost). On test, the model had already found all 99 needs_person messages, so the rule changed 14 routes and caught nothing. The guarantee cost US$14 per 1,000 messages there: 14 messages read by a senior person.

So a rule may go first when two things are true:

  • Its mistakes are cheap compared with what it catches. A safety word sent to a person costs US$2; a missed safety problem costs US$40, and much more in real life.
  • A person wants a guarantee, not an average. Grace can tell a customer: “Every message that says hurt or smoke is read by a senior person.” A model cannot promise that.

The double-charge rule fails the first test. When it is wrong, a return or a delivery message goes to billing: a wrong queue, US$4, or US$20 if urgent. When it catches something, it saves the same US$4. So it needs to be right more often than the model, and it is not (57% against 95%).

Here is what Priya and Grace agree, as a short checklist for any rule that sits next to a model:

  1. Measure each rule against the model on the validation messages that the rule matches. If the model is right more often, the rule does not decide.
  2. Let a rule go first only for a guarantee with cheap mistakes, such as the safety words. Write down why, and check its cost at each new model version.
  3. Use the other rules as flags or tests, not as routes. “Charged twice” can add a note for the billing team. The rule tests check that the rule book still does what it says.
  4. Give every rule an owner and a reason. The why field in config/rules.json is the start. A rule that nobody can explain is a rule that nobody can safely remove.

With rules in their place, Larkfield’s best classical router is TF-IDF + a linear model. On this inbox, it costs US$175.0 to US$296.0 per 1,000 test messages, against US$1,500 for a person who reads everything.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, run the cell. You see Grace’s four strong rules.
  4. In section 2, write your prediction. Then run the cell. On validation, strong rules first costs 589.0, and a disagreement to a person 432.5.
  5. In section 3, run the cell. For every strong rule, model right is larger than rule right.
  6. In section 4, run the cell. You see 159 matches and 70 disagreements, with the model right on 64.
  7. In section 5, run the test cell once. The model alone costs 296.0.

Now finish the project stage of this module. The last file, larkroute/methods/classical.py, adds three methods to route compare: Naive Bayes, the linear SVM and strong rules + logistic regression. The project finds every method by its file, so no other file changes.

  1. In your project folder larkroute, with .venv active, copy the methods and their test from the snapshot end-of-m03:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m03
    cp "$S/larkroute/methods/classical.py" larkroute/methods/
    cp "$S/tests/test_classical.py" tests/
  2. Run the tests. The last line is 18 passed. Your project is now the same as the snapshot end-of-m03.

    Terminal window
    python -m pytest -q
  3. Score every method of this module on the test messages. It takes about 15 seconds:

    Terminal window
    route compare --methods "everything to a person,rules or a person,TF-IDF + Naive Bayes,TF-IDF + logistic regression,TF-IDF + linear SVM,strong rules + logistic regression"
    method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msg
    TF-IDF + linear SVM 0.966 0.969 1.000 1.000 0.971 0.954 175.0 0.0 175.0 0.14
    TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.14
    TF-IDF + Naive Bayes 0.943 0.945 0.917 1.000 0.952 0.922 441.0 0.0 441.0 0.02
    strong rules + logistic regression 0.932 0.923 0.917 0.970 0.928 0.944 443.0 0.0 443.0 0.16
    everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00
    rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.03

    The table is sorted by total cost, cheapest first. These are the rows of the comparison table for this module. The ms/msg column changes a little from run to run.

  4. Compare the model and the strong rules on the validation messages, where you choose:

    Terminal window
    route compare --split validation --methods "TF-IDF + logistic regression,strong rules + logistic regression"

    The last two lines show 472.0 for the model and 589.0 for the strong rules first.

Grace says: “Safety is not negotiable. I want every message with a safety word read by a senior person, even when the model is usually right.” In section 6 of the notebook, write safety_first(part): the route is needs_person when the rule a safety problem matches, and the model’s route otherwise. Then compute safety_cost, its cost per 1,000 validation messages. Before you run it, predict: more or less than the model alone (US$472.0)?

Show the answer
def safety_first(part):
model_pred = model.predict(part["text"])
return ["needs_person" if safety["regex"].search(t) else m for t, m in zip(part["text"], model_pred)]
safety_cost = round(rt.cost_per_1000(validation, safety_first(validation)), 1)
print(safety_cost)
445.0

Less than the model alone. The rule caught 2 safety messages that the model sent to a team (US$40 each), and its 13 wrong routes cost only US$2 each. On the test messages, the same rule costs US$310.0, against US$296.0 for the model alone. There, the model had found every needs_person message, so the rule only added 14 senior reads. Either way, the guarantee is cheap, because the rule’s mistakes are cheap.

Common mistake

Changing the choice after the test

Symptom: Priya chose “a disagreement goes to a person” on the validation messages (US$432.5 against US$472.0 for the model alone). On the test messages, the model alone wins (US$296.0 against US$339.8). Tomás says: “Easy. The test set says the model alone is best, so let’s choose that.”

Cause: the test set is there to give one honest number for the choice that you made, not to make the choice. If you change the choice because of the test result, the test set has become a second validation set. Its score is then too good in the same way, and nothing honest is left. Here, the two sets also disagree because the difference is small: it comes from a few expensive messages among 2,000.

Fix: report the test result for the choice as it is: the reviews did not pay on the test messages. Then make the next decision with more evidence, not with the test set. Use more labelled messages, or cross-validation on the training messages, as in Train, validation and test sets. Priya keeps the model alone as the main router, for a reason that does not depend on the test set. The gain on validation came from a handful of messages, and simpler is safer.

Consequence in AI work: a team that picks the winner on the test set reports a cost that the next month will not show. When two options are this close, the right report is “about the same, and here is why”, not a winner.

Check your understanding

Grace's four strong rules are “the cases that she trusts more than a model”. Yet strong rules + logistic regression costs US$443.0 per 1,000 test messages, against US$296.0 for the model alone. Explain why, with the numbers of the validation messages.

Show the answer

When a strong rule goes first, it wins every disagreement with the model. On the validation messages, the strong rules match 159 messages and disagree with the model on 70. The model is right on 64 of the 70, and the rule on only 6. So the rules change 64 right answers into wrong ones. On each rule’s own messages, the model is right more often than the rule. For example, for legal threats, the rule is right on 16 of 36 and the model on 34. A rule reads one phrase. The model reads the whole message, and it has learned that a threat after the real request does not change the route.

Check your understanding

The safety rule may go first, but the double-charge rule may not. Use the cost table to explain the difference.

Show the answer

When the safety rule is wrong, a team message goes to needs_person, and a senior person forwards it: US$2. When it catches a safety message that the model missed, it saves US$40. So its mistakes are cheap and its catches are expensive. It can be wrong 20 times per catch and still break even, and it gives Grace a guarantee. When the double-charge rule is wrong, a message goes to the wrong queue (US$4, or US$20 if urgent). A catch saves about the same. So it must be right more often than the model to help, and it is not: 57% against 95% on its validation messages.

Check your understanding

Practical check: your project now has the end-of-m03 stage. Which command shows the six methods of this module in one table, and which method is the cheapest on the test messages?

Show the answer

route compare --methods "everything to a person,rules or a person,TF-IDF + Naive Bayes,TF-IDF + logistic regression,TF-IDF + linear SVM,strong rules + logistic regression". After python -m pytest -q prints 18 passed, the table lists the six rows, sorted by total cost. The cheapest is TF-IDF + linear SVM: accuracy 0.966 and US$175.0 per 1,000 test messages on this inbox. The most expensive is rules or a person, at US$1,757.8.

Recap

  • Strong rules first makes TF-IDF + logistic regression worse on this inbox. It gives 0.932 and US$443.0 per 1,000 test messages, against 0.962 and US$296.0. On the 70 validation disagreements, the model is right 64 times.
  • Measure each rule against the model on the messages that it matches. Here, the model wins for all four strong rules.
  • A rule may go first when its mistakes are cheap and its catches are expensive, and a person wants a guarantee. The safety rule is one: US$2 per mistake, US$40 per catch.
  • A disagreement goes to a person looked best on validation (US$432.5) and did not hold on test (US$339.8). Report the test result as it is; do not change the choice because of it.
  • Your project is at the stage end-of-m03: 18 tests, route explain, and six methods in route compare. On this inbox, the best classical router is TF-IDF + a linear SVM: 0.966 and US$175.0 per 1,000 messages.

You can now build, compare and combine classical routers. But TF-IDF still treats “parcel” and colis as two unrelated words, and it knows nothing about a word that it never saw. In Module 4, we use models that give similar sentences similar numbers, in English and in French.

Next: Sentence embeddings for classification