Reasoning models for the hard cases
- Lesson 22
- 35 min
- Module 5: LLMs as classifiers
Where we are
Some messages are hard for every method: two requests in one, sarcasm, a policy edge case. A reasoning model thinks before it answers. In this lesson, we send only the hard messages to a reasoning model, and we see why it is almost never worth it for the whole inbox.
Lesson 22 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceReasoning models for the hard cases
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Find the messages that are hard for every method, and test whether a slower, more expensive model that “thinks” first does better on them. Measure where it helps, where it cannot help, and what each answer costs in time and money.
NotebookDo this lesson in a notebook
The notebook rebuilds the hard set from five methods’ predictions and replays the reasoning model’s recorded answers on it. It lists the messages where only the reasoning model is right. It also shows the 12 messages that every method gets wrong, and the time and cost of each answer. You need no account and no key. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The LLM with descriptions and examples, from Few-shot examples and label descriptions.
- The hard kinds of messages (vague, two requests, policy edge cases, sarcasm), from This inbox is easier than a real one and the labelling guidelines.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
The senior expert’s desk
Section titled “The senior expert’s desk”At the post office, a few letters defeat every clerk. The address is half missing, or one letter asks for two things. The manager keeps a senior expert for those. She is slow and she is expensive, so she never sees the easy letters.
She solves many of the hard letters. But some letters say only “see my last letter”. No expert can guess what the last letter said. Those letters need a phone call to the customer.
The problem: the last mistakes are the hardest
Section titled “The problem: the last mistakes are the hardest”The LLM with descriptions and examples gets 0.970 of the test messages right on this inbox, and the fine-tuned encoder 0.980. The remaining mistakes are not random. A careful person also finds most of them hard: a vague follow-up, two requests in one message, a case at the border of two rules.
Tomás has an idea: “Use the best model there is.” The best hosted models are reasoning models. They are also the slowest and the most expensive. Before we pay for one, we must know which messages it would get right that cheaper models get wrong.
The idea: a reasoning model, only for the hard messages
Section titled “The idea: a reasoning model, only for the hard messages”A reasoning model is an LLM that writes hidden working notes before its answer. These notes are reasoning tokens: you pay for them as output tokens, but you do not see them. A setting called effort (low, medium, high) controls how much it may think. The course used the Azure deployment chat-strong (model gpt-6.1-sol) at effort “medium”. It had the same few-shot prompt: the route descriptions and the 8 most similar training messages.
We do not send it every message. We first find the hard set: the test messages that at least two of five methods get wrong. The five are TF-IDF with logistic regression, the two embedding models with logistic regression, the fine-tuned encoder, and the LLM with examples.
A tiny worked example: is this message hard?
Section titled “A tiny worked example: is this message hard?”Count how many of the five methods get each message wrong. Here are three test messages:
| Message | Label | Methods wrong | Hard? |
|---|---|---|---|
| M-105011 “Can I return the rug if I don’t like the colour?” | product | fewer than 2 | no |
| M-109403 “stop the emails and close my account pls. Also, can you cancel an order …” | account | 2 or more | yes |
| M-105667 “could you follow up on my earlier message about order LK-969836? Thanks!” | billing | all 5 | yes, and the hardest |
Text description of the diagram
The first box: the 2,000 test messages. Five methods route every message: TF-IDF with logistic regression, the two embedding models, the fine-tuned encoder and the LLM with examples. The second box sorts the messages by how many methods get them wrong. All five are right on 1,696 messages, and one is wrong on 174. Two or more are wrong on 130 messages, 6.5% of the test set: this is the hard set. The third box: the reasoning model on the 130 hard messages gets 97 right, a share of 0.746. The LLM with examples gets 90. The reasoning model costs US$1.94 per 1,000 messages and takes 2.0 seconds per message. The fourth box: all five methods are wrong on 12 messages. The reasoning model gets none of them right. On 11 of them, it gives the same wrong route as the LLM with examples.
Larkfield: the reasoning model on the 130 hard messages
Section titled “Larkfield: the reasoning model on the 130 hard messages”The reasoning model gets 97 of the 130 hard messages right. The LLM with examples, on the same 130, gets 90. That is a gain of 7 messages, out of 2,000 test messages. Count the movements one by one, as in the few-shot lesson:
| On the 130 hard messages | LLM with examples right | LLM with examples wrong |
|---|---|---|
| Reasoning model right | 87 | 10 (only the reasoning model) |
| Reasoning model wrong | 3 (only the LLM with examples) | 30 |
Where the extra accuracy comes from
Section titled “Where the extra accuracy comes from”The course’s answer key marks each hard message with its kind. The chart shows both models by kind:
Text description of the diagram
For each hard kind, two bars: the reasoning model, then the LLM with examples. Two requests, 48 messages: the reasoning model gets 45 right, the LLM with examples 38. Vague, 58 messages: 29 and 29. No hard kind, 17 messages: 16 and 16. Policy edge, 7 messages: 7 and 7.
The whole gain is on messages with two requests: 45 of 48 against 38. The guidelines say: route a message with two requests by its main request, usually the first one. Here is one of them:
@Larkfield, stop the emails and close my account pls. Also, can you cancel an order before it ships? Tracking for LK-136362 hasn’t changed in 16 days—is my herb kit lost?Label:
account. Reasoning model:account, “Main request is stopping emails and closing the account.” LLM with examples:orders, “The first main request is cancelling an order before it ships.”
The reasoning model follows the “first request” rule more often. But look at its reasoning tokens: for this message, 0. It wrote no working notes at all. For 7 of the 10 messages that only it gets right, it wrote none. At effort “medium”, it wrote no reasoning tokens for 89 of the 130 messages, and 13 on average. So here, most of the gain comes from a stronger model, not from long thinking.
On vague messages, both models get 29 of 58. The reasoning model thinks most on the messages it gets wrong: 35.7 reasoning tokens on average, against 5.3 on the ones it gets right. More thinking is a sign of a hard message, not a promise of a right answer.
Where thinking cannot help
Section titled “Where thinking cannot help”Twelve test messages are wrong for all five methods. The reasoning model gets 0 of the 12. On 11 of them, it gives the same wrong route as the LLM with examples. Read two:
@Larkfield, could you follow up on my earlier message about order LK-969836? Thanks!Label:
billing. Reasoning model:returns_refunds, written confidence 0.46.
@Larkfield Hi, about the large one I mentioned—see my last message. Could you let me know how that works? Thanks!Label:
delivery. Reasoning model:product, written confidence 0.88.
The first message names an order, and nothing else. The second asks “how that works” about “the large one”. The label comes from the earlier message, which no method can see. No amount of thinking can find information that is not in the message. The guidelines say: label the best guess and mark the message for review. A person who can open the customer’s history is the right route.
Time and money
Section titled “Time and money”| On the 130 hard messages | Right | Median time | Slowest 10% | Cost per 1,000 messages |
|---|---|---|---|---|
LLM with examples (chat-small) |
90 | 1.75 s | about US$0.09 | |
Reasoning model (chat-strong, medium) |
97 | 2.04 s | 3.2 s | US$1.94 |
The reasoning model costs about 20 times more per message. Its prices are higher (US$2.00 per million input tokens and US$10.00 per million output tokens, against US$0.10 and US$0.50), and the reasoning tokens are billed as output. All 130 answers together cost about US$0.25. Sending all 2,000 test messages would cost about US$3.88. Prices change: check the provider’s current price page.
Why it is almost never worth it for the whole inbox
Section titled “Why it is almost never worth it for the whole inbox”The money is small at Larkfield’s volume. The reasons not to send every message to a reasoning model are elsewhere:
- The gain is small and narrow. On this inbox, 7 more right answers out of 2,000, all from one kind of message. We do not even know how it does on the other 1,870 messages: the course did not record them.
- You do not know in advance which messages are hard. The hard set was found with the labels: we counted the mistakes of five methods. In production, there are no labels. You need a signal that a message is unsure, such as a checked confidence, before you send it on. That is a cascade, and Cascades builds one.
- The hardest messages need a person, not a model. The 12 that everyone gets wrong lack information. A reasoning model answers them anyway, with written confidences from 0.43 to 0.91.
- Time and limits. Each call takes about 2 seconds, and the slowest 10% over 3 seconds. Each model has its own rate limit, so check yours. The price per token is 20 times higher here: at a large shop’s volume, both matter.
Use a reasoning model for a small, well-defined slice where you measured a gain. Here, that slice is “two requests in one message”. On this inbox, even that slice could go to a person for US$1.50 a message. Or the guidelines’ rule could become a better example in the prompt.
Guided practice: rebuild the hard set
Section titled “Guided practice: rebuild the hard set”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, train TF-IDF and load the other four methods. The table shows
130messages wrong for two or more methods, and12for all five. - In section 2, replay the reasoning model. You see
reasoning model: 97 of 130 right; LLM with examples: 90 of 130. - In section 4, run the cell. The reasoning model gets
0 of 12. Read the texts. - In section 5, run the cell. You see the reasoning tokens, the cost and the time.
Your turn: the messages only the reasoning model gets right
Section titled “Your turn: the messages only the reasoning model gets right”In section 3 of the notebook, complete only_reasoning. It holds the sorted IDs of the hard messages where only the reasoning model is right. Predict first: how many will there be, and which hard kind will most of them be?
Show the answer
only_reasoning = sorted(hard["message_id"][r_ok & ~f_ok])Check passed: only_reasoning is right.10 messages; the LLM with examples alone is right on 3Ten messages. Seven of them have two routes in the column routes: two requests in one message. The reasoning model routes them by the first request, as the guidelines say. The other three are vague follow-ups about a broken item, where it chose returns_refunds and the LLM with examples chose product.
Common mistake
A hard set chosen with the labels
Symptom: a team reports “the reasoning model fixes 10 of our hardest messages”, and plans to send every message to it. In production, the gain disappears into the cost.
Cause: the hard set was found by counting mistakes against the labels. In production there are no labels, so nobody knows which messages are hard. Sending everything to the reasoning model pays its price on the 93.5% of messages that cheaper methods already get right. And the 12 hardest messages stay wrong, because they lack information.
Fix: choose the messages for the expensive step with a signal that exists without labels. Examples are a checked confidence, or a disagreement between two cheap methods. Measure the whole chain on the test set, as a cascade, with the cost of every step. Send messages that lack information to a person.
Consequence in AI work: an evaluation on a slice chosen with the answers looks better than any real system can be. Always ask: could the system have chosen this slice by itself?
Knowledge check
Section titled “Knowledge check”Check your understanding
The reasoning model gets 97 of the 130 hard messages right, and the LLM with examples 90. Where does the gain come from, and how much of it is 'thinking'?
Show the answer
The whole gain is on messages with two requests: 45 of 48 against 38. On vague messages, both get 29 of 58, and on the policy edge cases both get 7 of 7. The reasoning model applies the rule “route by the first request” more often. But it wrote no reasoning tokens for 7 of the 10 messages that only it gets right, and for 89 of the 130 hard messages. So most of the gain comes from a stronger model, not from long thinking.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the reasoning model get none of the 12 messages that all five methods get wrong? What should happen to such messages?
Show the answer
They lack the information. Most say “see my last email” or “the large one I mentioned”, and the label depends on that earlier message, which no model can read. Thinking cannot add missing facts. On 11 of the 12, the reasoning model gives the same wrong route as the LLM with examples. Such messages should go to a person, who can open the customer’s history, as the guidelines say for vague messages.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás wants to send every message to the reasoning model, because 'it is the best model'. Give three reasons against it on this inbox, with numbers.
Show the answer
First, the gain is narrow: 7 more right answers out of 2,000 test messages, all on messages with two requests. Second, it costs about 20 times more per message: US$1.94 per 1,000 against about US$0.09. It also takes 2.04 s against 1.75 s. Third, the hard set was chosen with the labels. In production, you do not know which messages are hard, so you would pay for the 93.5% that cheaper methods already get right. And the 12 hardest messages stay wrong anyway.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The hard set: the 130 test messages (6.5%) that at least two of five methods get wrong.
- A reasoning model writes hidden reasoning tokens before it answers. On the hard set, it gets 97 of 130 (0.746), against 90 for the LLM with examples.
- The gain is all on two requests in one message (45 of 48 against 38). It came mostly without reasoning tokens: a stronger model, not long thinking.
- On the 12 messages that every method gets wrong, it gets 0: the information is not in the message. A person must read them.
- It costs about US$1.94 per 1,000 messages, about 20 times the LLM with examples, and takes about 2 s.
- The hard set was chosen with the labels. In production, you need a signal without labels: a cascade.
Next, we put every cost of every model in one place: tokens, prices, seconds, rate limits and privacy, against the cost of the mistakes.
Next: Cost per 1,000 messages