Keyword rules and regular expressions
- Lesson 9
- 35 min
- Module 3: Baselines: rules and classical models
Where we are
Our labels are now good enough to learn from. Before we train a model, we try the simplest thing that could work: rules that a person writes. In this lesson, we write keyword rules and regular expressions, see where they are the best tool, and see how they break.
Lesson 9 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceKeyword rules and regular expressions
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Write keyword rules that people can read and change, and measure what they cost before you trust them. A rule book that catches three messages in four can still cost more than a person who reads every message.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It writes two regular expressions and runs Larkfield’s 11 rules on the 2,000 test messages. It splits their cost by kind of mistake, and asks you to add one rule. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The seven routes and the labelling guidelines, from Labelling guidelines.
- The cost table, from The cost of each mistake. A review costs US$1.50 and a wrong queue US$4. An urgent message in a wrong queue costs US$20, and a missed account takeover US$200.
- The test set: 2,000 messages that no method learns from, from Meet the inbox.
What you need on your computer (only for the project step; the notebook needs none of it)
- Your project folder
larkrouteat the stage of the last module, with its.venvactive.python -m pytest -qprints12 passed. - The labs repository
nextia-learning-labsnext to your project folder, as in The project on your computer.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.
A sorting card at the post office
Section titled “A sorting card at the post office”Think of a new clerk in a post office. On her first day, she gets a small card with sorting rules. “A letter with an express stamp goes in the red bag. A letter to France goes in the blue bag. A parcel over 2 kg goes on the shelf.” She reads each letter, finds the first rule that fits, and puts the letter in that bag. A letter that fits no rule goes to her manager.
The card is a good start. Anyone can read it, and anyone can change it in one minute. Nobody needs a model or a week of training.
But one morning a letter arrives for “Paris, Texas”. The card says “Paris goes to France”, so the letter goes in the blue bag and travels to the wrong country. The rule saw the word. It did not understand the address.
The problem: a word is not a request
Section titled “The problem: a word is not a request”Larkfield’s support inbox has the same problem. A customer writes:
“Can I return the storage box if I don’t like the colour? Just checking.”
The word “return” is there. But the customer has not bought anything yet. By the labelling guidelines, a question about the return policy before buying goes to the product team. You met this trap in A message in the wrong queue, with Tomás’s quick rule. This lesson measures it on the whole test set.
A person reads the whole message and sees the request. A rule sees only the words in its pattern. So before we build any model, we must know how far simple rules go, and what they cost when they are wrong.
The idea: keyword rules and regular expressions
Section titled “The idea: keyword rules and regular expressions”A keyword rule says: “if the message contains these words, send it to this route”. It is the clerk’s card, written for a computer. Each rule has a route, a list of words, and a reason that a person can read.
To write the list of words, we use a regular expression, or regex for short. A regex is a small pattern language for text. The name comes from mathematics, where “regular” describes this kind of simple pattern. Python’s re module finds a regex in a string. Three pieces are enough for this lesson:
| Piece | Means | Example | Matches |
|---|---|---|---|
\b |
a word edge: the start or the end of a word | \bcard\b |
“card”, not “discard” |
\d{6} |
six digits | LK-\d{6} |
“LK-123456” |
(a|b) and \w* |
a or b; then any letters |
(refund\w*|return\w*) |
“refund”, “refunded”, “returning” |
The flag re.IGNORECASE makes the pattern ignore upper and lower case, so “Refund” matches too.
A tiny example: three messages, two patterns
Section titled “A tiny example: three messages, two patterns”Here are two patterns. The first finds a Larkfield order number: \bLK-\d{6}\b. The second is a refund rule: \b(refund\w*|return\w*)\b. We try them on three short messages:
import re
samples = ["Where is order LK-123456? Tracking has not moved.", "I sent the chair back. When is my refund coming?", "Can I return the bench if I don't like the colour?"]for text in samples: order = re.search(r"\bLK-\d{6}\b", text) refund = re.search(r"\b(refund\w*|return\w*)\b", text, re.IGNORECASE) print(order.group(0) if order else "-", "|", refund.group(0) if refund else "-", "|", text)The output:
LK-123456 | - | Where is order LK-123456? Tracking has not moved.- | refund | I sent the chair back. When is my refund coming?- | return | Can I return the bench if I don't like the colour?The order number pattern is perfect: it finds the number, and nothing else. The refund pattern is right for the second message. For the third, it finds “return”, and the route would be wrong. The pattern did its job exactly. The job was the wrong one.
This is the difference that the whole lesson is about. A pattern is excellent at finding a form: an order number, a product code, an exact phrase. It is weak at knowing a request, because the same words appear in many requests.
Larkfield’s rule book
Section titled “Larkfield’s rule book”Priya and Grace wrote 11 rules in config/rules.json, the file that the project has had since the start. Each rule has a route, a pattern, a reason (why) and a mark strong. Grace marked as strong the four rules that she trusts most: legal threats, safety problems, an account that someone else used, and a double charge. The last lesson of this module uses that mark.
The rules run in order. The first rule that matches decides the route. A message that no rule matches goes to a person, who reads it and sends it on. The project calls this method rules or a person.
Text description of the diagram
The diagram is a ladder from top to bottom. A new message arrives at rule 1: does it contain lawyer, journalist or consumer protection? If yes, the route is needs_person. If no, the message goes down to rule 2: hurt, smoke or on fire? If yes, the route is needs_person. If no, it goes down through rules 3 to 10. They look for words such as manager, hacked, charged twice, invoice, refund, tracking, cancel and dimensions, and the first one that matches gives its route. If none matches, the message reaches rule 11: password, log in or my account. If yes, the route is account. If no rule matched at all, a person reads the message.
The order matters. Take a message that says “I was charged twice, and I want to return the chair”. It matches the double-charge rule first, so it goes to billing, even if the return is the main request. Moving one rule up or down changes the route of many messages.
Run the rules on the test messages
Section titled “Run the rules on the test messages”Priya runs the rule book on the 2,000 test messages. In the notebook, first_match() returns the first rule that matches, or None:
hits = [first_match(t, rules) for t in test["text"]]rule_pred = [h["route"] if h else "review" for h in hits]matched = np.array([h is not None for h in hits])right = np.array(rule_pred) == test["route"].to_numpy()print("matched by a rule:", matched.sum(), "of", len(test), "- share", np.round(matched.mean(), 3))print("right when a rule matched:", right[matched].sum(), "- share", np.round(right[matched].mean(), 3))print("sent to a person:", (~matched).sum())The output:
matched by a rule: 1527 of 2000 - share 0.764right when a rule matched: 1110 - share 0.727sent to a person: 473Two words name these numbers. Coverage is the share of messages that a rule decides: 76.4%. Precision is the share of those decisions that are right: 72.7%. So the rules decide three messages in four, and about three of those four decisions are right.
Tomás is pleased. “Three quarters of the inbox, with no model and no training. Let’s put it live.” Grace asks the question of Module 1: “What does it cost?”
What the rules cost
Section titled “What the rules cost”rt.score() scores a method with the same columns and the same cost table as route compare. Priya scores “everything to a person”, which is Larkfield today, and the rules:
method accuracy macro_f1 ... total_per_10000 everything to a person 0.000 0.000 ... 1500.01 rules or a person 0.555 0.625 ... 1757.8The rules cost US$1,757.8 per 1,000 messages. A person who reads every message costs US$1,500. The rule book that “handles three quarters of the inbox” costs more than no automation at all.
To see why, Priya puts each test message in one group of the cost table and adds up the dollars per group:
Text description of the diagram
The chart has two columns, in US dollars per 1,000 test messages. The left column, everything to a person, is one block of 1,500: a person reads all 2,000 messages. The right column, rules or a person, is made of six blocks and reaches 1,757.8, above a dashed red line at 1,500. From the bottom, the blocks are:
- 354.75 dollars for the 473 messages that no rule matched and a person reads;
- 634 for 317 messages in a wrong queue;
- 460 for 46 urgent messages in a wrong queue;
- 200 for 2 missed account takeovers;
- 60 for 3
needs_personmessages sent to a team; - 49 for 49 team messages sent to
needs_person.
The reviews cost US$354.75 per 1,000. That part is cheap, because a review costs only US$1.50. The expensive part is the 417 messages that a rule sends to a wrong route (20.9% of the test set). A wrong queue costs US$4, an urgent one US$20, and a missed account takeover US$200. Even the cheapest mistake, a team message sent to needs_person, costs US$2: more than a review. Together, the wrong decisions cost US$1,403 per 1,000: almost as much as reading everything.
So coverage was the wrong number to look at. A rule that decides a message wrongly is worse than no rule, because a person would have read that message for US$1.50.
Read the messages that the rules got wrong
Section titled “Read the messages that the rules got wrong”The numbers say how much. Only the messages say why. Here are five real test messages, the rule that caught each one, and the word that it matched:
| Message | Label | Rule (route) | Matched | What went wrong |
|---|---|---|---|---|
| M-100017: “I think someone got into my account… a new order, which definitely wasn’t me. I’m also looking to return an item…” | account (account takeover) |
returns, refunds (returns_refunds) |
“return” | The takeover rule looks for “hacked” or “didn’t order”, and this customer wrote “wasn’t me”. The return rule came next. Cost: US$200. |
| M-100056: “J’ai reçu le mauvais article…” (“I received the wrong item”) | returns_refunds |
invoices and receipts (billing) |
“reçu” | In French, reçu means “a receipt” and also “received”. |
| M-100318: “…j’ai pas trouvé ses dimensions. Vous pouvez me les donner? Le remboursement, ça n’a rien à voir” (“I did not find its dimensions… the refund has nothing to do with it”) | product |
returns, refunds (returns_refunds) |
“remboursement” | The customer said “not a refund”. A rule cannot read “not”. |
| M-100516: “…the smoker box can’t arrive that day. Please arrange delivery for another day…” | delivery |
a safety problem (needs_person) |
“smoker” | A smoker box is a Larkfield product. The pattern for smoke catches it. |
| M-100053: “I would like to return order LK-195050… Separately, I was charged twice…” | returns_refunds |
a double charge (billing) |
“charged twice” | The main request comes first; the double-charge rule is higher in the rule book. |
Each rule matched a real word, and each route is wrong. The word reçu alone decides 60 test messages, and only 8 of them are billing messages. Each time, a person would see the request at once.
A rule that almost never fires
Section titled “A rule that almost never fires”Rules also break in a quieter way: they miss. The takeover rule looks for didn'?t (make|place|order), with a straight apostrophe ('). Phones and word processors often write a curly one (’). In the test set, 1,396 messages use the curly apostrophe, and only 78 use the straight one. So “an order that I didn’t make” does not match, and the takeover rule decides only 3 test messages.
Priya tries the obvious fix, didn['’]?t, which accepts both apostrophes. The rule now matches 18 test messages and finds 6 of the 12 account takeovers. But 12 of the 18 have another main request. For example: “How do I return the sun lounger? I changed my mind. Order LK-121497. Also, I noticed an order I didn’t make on my account.” The customer mentions the account at the end, and the label follows the first request, a return. Each fix opens a new door. No error message warned about the apostrophe; only reading the messages showed it.
Common mistake
Rules or a person costs more than a person
Symptom: the rules match 76.4% of the test messages, and a quick look at a few matches looks fine. Tomás wants to send every matched message straight to its team.
Cause: coverage counts decisions, not good decisions. The 417 wrong matches go to a wrong route with no warning, and each one costs more than a review.
Fix: score every method with the cost table before you use it, not with coverage. Use route compare or rt.score(), which put the review cost and the cost of each kind of mistake in one number. Keep rules for the narrow jobs below, where they are almost never wrong.
Consequence in AI work: a team ships a rule book because “it handles most of the inbox”. The wrong queues appear later as slow answers, and nobody connects them to the rules. Here, the rules cost US$257.8 more per 1,000 messages than no automation.
Where rules are the best tool
Section titled “Where rules are the best tool”Rules failed as a router here. They are still the best tool for some jobs, because they are exact, fast, free to run and easy to read.
- Finding a form. An order number (
LK-\d{6}), a product code, an email address or a date has a fixed shape. A regex finds it with no mistakes. The team that gets the message can use the order number at once. - Exact phrases with a clear meaning. “Charged twice” almost always means a double charge. A rule for it is a good flag for a person or a model, even when it is not a good final route.
- A first version on day one. A new product has no labelled messages yet. A rule on its name can send its messages somewhere sensible until there are labels to learn from.
- A test of known behaviour. A rule book is easy to test: “this message must match this rule”. The project’s tests do this, as you see below.
Rules are a bad router when the same words appear in many requests, as “return”, “refund” and “account” do here. Every new rule also adds work: someone must read the mistakes and check that a fix for one message does not break others. With 300 rules written by five people, nobody knows why a message went where it went. So the cost of rules is people’s time. The computer time is nothing: about 0.03 milliseconds per message.
Guided practice: run the rule book
Section titled “Guided practice: run the rule book”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, predict which message has an order number. Then run the cell. Only the first message has
LK-123456. - In section 2, run the cell. You see the 11 rules of the rule book.
- In section 3, write your prediction of the coverage. Then run the cell. You see
1527 of 2000and a share of0.764. - Run the scoring cell. The rules cost
1757.8; everything to a person costs1500.0. - In section 4, run the cell. The largest line is
wrong queue, with634.00per 1,000. - In section 5, read the five messages and the word that each rule matched.
- In section 6, run the cell. You see
60messages sent to billing because of reçu.
Now add the tests of the rule book to your project. They check that clear messages get the right rule, and they also record one known weakness: a question before buying goes to returns_refunds. A test that records a weakness is useful. If someone changes the rules, the test shows whether the weakness changed too.
-
In your project folder
larkroute, with.venvactive, copy the test file from the snapshotend-of-m03:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m03cp "$S/tests/test_rules.py" tests/Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m03"Copy-Item "$S\tests\test_rules.py" tests\ -
Open
tests/test_rules.pyand read the three tests. The second one is calledtest_a_refund_word_is_not_always_a_refund_request. -
Run the tests:
Terminal window python -m pytest -qThe last line is
15 passed: the 12 tests from before and the 3 new ones. -
Run the first comparison again, and find the rules row:
Terminal window route compare --quickmethod accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msgTF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.12everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.03The rules are the last row: the most expensive way to route here. The
ms/msgcolumn (milliseconds per message) changes a little from run to run.
Your turn: a rule for questions before buying
Section titled “Your turn: a rule for questions before buying”The refund rule sends questions before buying to returns_refunds. In the test set, 36 of the 144 product questions mention a return or a refund, and the rules send 32 of them to returns_refunds. Priya writes a rule for these questions:
policy = {"route": "product", "strong": False, "why": "a question before buying", "pattern": r"\b(can i return|may i return|if i (buy|get|order)|before i (buy|order|decide)|je peux le retourner|si je (l'achète|commande))\b"}Where must it go in the rule book, and why there? Before you run anything, predict: will the cost go below US$1,500? In section 7 of the notebook, make fixed_rules: a copy of rules with policy inserted just before the first returns_refunds rule. The next cell scores both rule books on the validation messages.
Show the answer
policy["regex"] = re.compile(policy["pattern"], re.IGNORECASE)at = next(i for i, r in enumerate(rules) if r["route"] == "returns_refunds")fixed_rules = rules[:at] + [policy] + rules[at:]before validation cost per 1,000: 1677.0after validation cost per 1,000: 1660.2The rule must come before the refund rule. The first rule that matches decides, so after the refund rule it would never run. It saves US$16.8 per 1,000 validation messages, and the rule book still costs more than a person who reads everything. One rule fixes one kind of message. The 417 wrong decisions have many different causes.
Why validation, and not test? You wrote this rule after you read test messages. If you also judge it on the test messages, you judge it on the messages that it was written for. Then the score is too good. The validation set has messages that played no part in writing the rule. Train, validation and test sets in the machine learning course explains this rule in detail.
Knowledge check
Section titled “Knowledge check”Check your understanding
The rules decide 76.4% of the test messages, and 72.7% of those decisions are right. A person who reads everything costs US$1,500 per 1,000 messages. Explain how the rules can cost more (US$1,757.8).
Show the answer
The reviews are cheap: the 473 messages that no rule matched cost US$1.50 each, US$354.75 per 1,000. The expensive part is the wrong decisions. 417 messages go to a wrong route, and each one costs more than a review. Most cost US$4, an urgent one US$20, and each of the two missed account takeovers US$200. The wrong decisions add about US$1,403 per 1,000. A wrong automatic decision is worse than a review, so a high coverage with a precision of 72.7% loses money. Judge a method by its cost, not by its coverage.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: the refund rule meets the French message “je regardais votre corner sofa… j’ai pas trouvé ses dimensions… Le remboursement, ça n’a rien à voir” (“I did not find its dimensions… the refund has nothing to do with it”). Which route does the rule give, and why would a person give another one?
Show the answer
The rule gives returns_refunds, because the pattern rembours\w* matches “remboursement”. A pattern sees words, not their meaning, so it cannot read “has nothing to do with it”. A person reads the request: the customer asks for the dimensions of a product, so the route is product. This is message M-100318 of the test set, and it is the guidelines’ first rule: label the request, not the words.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name two jobs where a regular expression is the right tool for Larkfield, and one job where it is the wrong tool. Give a reason for each.
Show the answer
Right tool: finding the order number with \bLK-\d{6}\b, because it has a fixed form and a regex finds it with no mistakes. Right tool: flagging an exact phrase with a clear meaning, such as “charged twice”, for a person or a model to use. A rule for a new product’s name is also good on the first days, before there are labels. Wrong tool: choosing the route from words such as “return”, “refund” or “account”, because the same words appear in many different requests. On this inbox, the rules send 417 test messages to a wrong team.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A keyword rule sends a message to a route when its pattern matches. A regular expression writes the pattern:
\bfor a word edge,\d{6}for six digits,(a|b)for a choice. - Larkfield’s 11 rules decide 76.4% of the test messages (coverage), and 72.7% of those decisions are right (precision). The first rule that matches decides, so the order of the rules matters.
- Rules or a person costs US$1,757.8 per 1,000 messages, more than a person who reads everything (US$1,500). The 417 wrong decisions cost more than all the reviews.
- A word is not a request: “return” before buying, reçu that means “received”, “not a refund”, a smoker box. A pattern cannot read meaning.
- Rules are the best tool for finding a form (order numbers, codes), for flags, and for a first version before labels exist. Judge every method by its cost.
You can now write rules and measure what they cost. But a rule reads only the words that a person chose for it. Next, we turn each message into numbers, so that a model can choose the useful words itself.