Model routing

Lesson 11
60 min
Module 4: Optimize with measured trade-offs

Lesson 11 of 19 · AI System Design and Cost EngineeringModel routing

Course overview

1. Start with requirements

2. Design the end-to-end architecture

3. Estimate resources and cost

4. Optimize with measured trade-offs

5. Design for failure and growth

6. Communicate the proposal

7. Case studies

What you will be able to do

Decide with a measurement, not a hope, whether a stronger model is worth its price. You compare three designs on the same 67 questions: the small model only, the strong model only, and a routed design. The routed design asks the strong model only when a rule fires. Then you write the result in the proposal, even when the result is “routing does not pay here”.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

The cost model and its sensitivity table, from Unit economics: Read the sensitivity table. It said that “chat-strong for every answer” is the biggest risk in the table.
The quality and latency targets Q4 and Q1, from Quality attributes: Larkfield’s quality attributes, and the first estimate of the slow answers, from A first estimate for Q1.
Grace’s 67 questions and how the RAG course scores the assistant’s answers, from Answer measures in RAG: Building AI That Uses Your Data.

What you need on your computer

Your folder design-pack with its active virtual environment, and python -m pytest -q printing 35 passed: the snapshot end-of-m03.
The labs folder nextia-learning-labs next to it.

You do not call a model in this lesson. The course team ran the 268 model calls; the pack holds the measured results. Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A help line with two levels

Think of a phone help line at a bank. A first-level agent takes every call. Most questions are simple: a card is blocked, a payment is late. When the agent cannot help, the call goes to a specialist. The specialist costs much more per hour, and the customer waits longer, because two people now spend time on the call.

The bank does this for a reason: it does not want to pay specialists for simple calls. But the plan works only if two things are true. The specialist must solve calls that the first agent could not solve. And the agent must know when to pass a call on. If the specialist fails on the same calls, the bank pays twice and the customer waits twice, for nothing.

The problem: a better model for every answer?

Ines, the owner of Bramble Books, wants better answers. Tomás has a simple plan: “Use chat-strong for everything. It is the better model, so the answers will be better.”

The pack already knows what this costs. In the last module’s sensitivity table, chat-strong for every answer added more than 200% to the monthly cost. This module’s design file starts from US$51.90 a month, not Module 3’s US$52.67, because it assumes that the answer cache answers 10% of the questions. The next lesson explains that number. With chat-strong for every answer, the month goes from US$51.90 to US$173.99: +235%. One question costs US$0.003197 instead of US$0.000171, about 19 times more.

There is a second cost: time. Module 1 estimated the slow answers by adding the p95 of each step. With chat-strong, that sum is 0.48 + 0.32 + 6.56 = 7.36 seconds. The target Q1 is 6 seconds for 95% of the answers. So chat-strong for everything fails a target before anyone asks if its answers are better.

Omar asks the question that matters: “Can we pay for the strong model only on the questions where it helps?”

The idea: model routing

Model routing means that a rule chooses which model answers each request. The common pattern is the help line: the cheap model answers first, and a rule sends some requests to the strong model. This is escalation, and the rule is the escalation rule. The name comes from networks, where a router sends each packet on one of several paths.

A good escalation rule uses a signal that is cheap to check and that appears when the small model is likely to be wrong. The design pack tests three signals:

The small model declines. It says that the passages do not answer the question.
A citation check fails. The answer cites a passage that it did not get, or it quotes a number that is not in any cited passage.
The search looks weak. The best passage’s reranker score is below a threshold T. (A reranker scores how well a passage matches the question; higher is better.)

A flow from left to right. A question with its top 5 passages goes to chat-small, which answers for US$0.000171 in 1.52 s. An escalation rule asks: did chat-small decline, does the citation check fail, is the best score below T? If no rule fires, the system sends chat-small's answer. If a rule fires, chat-strong answers again for US$0.003197 in 2.48 s, added to the first call, and the system sends chat-strong's answer. — A routed design: chat-small answers every question first; chat-strong answers again only when the escalation rule fires. An escalated question pays for both calls and waits for both.

Look at the right side of the picture. An escalated question is never cheaper or faster than a small-only question. It costs both calls, and the person waits for both. Routing saves money only compared with “strong for everything”. It adds quality only if the strong model fixes the escalated questions.

A tiny example: five questions

Here are five of Grace’s 67 questions from the course team’s run, first pass, with every value visible. The rule here is “escalate when chat-small declines or a citation check fails”.

Question	chat-small	Rule fires?	chat-strong	Result
Q01 “A customer was charged twice for the same order. How quickly do we refund the extra charge?”	right, US$0.000125, 1.44 s	no	not asked	right
Q38 “Which pump kit fits the PW-2200?”	right, US$0.000136, 1.13 s	no	not asked	right
Q45 “How long does a refund take to reach the customer’s card?”	wrong, US$0.000303, 3.31 s	no (good citations)	not asked	wrong
Q14 “Do we offer a student discount?”	declines (right: no document says), US$0.000104, 1.21 s	yes	declines, US$0.001914, 1.64 s	right
Q13 “Does Larkfield sell extended warranties?”	declines (wrong: the answer exists), US$0.000112, 1.26 s	yes	declines, US$0.002028, 1.49 s	wrong

Add up the three designs on these five questions:

chat-small only: 0.000125 + 0.000136 + 0.000303 + 0.000104 + 0.000112 = US$0.000780, 3 of 5 right.
Routed: the same five small calls + 0.001914 + 0.002028 = US$0.004722, 3 of 5 right. Q14 and Q13 each waited for two calls: 2.85 s and 2.75 s.
chat-strong only: 0.003146 + 0.002704 + 0.005398 + 0.001914 + 0.002028 = US$0.015190, 3 of 5 right. It also got Q45 wrong.

The routed design cost six times the small design and fixed nothing. Look at why. Q14 was a correct “I don’t know”, so the second call only confirmed it. Q13 was a search miss. Its answer is in a French document about the Quebec legal warranty, and that passage was not in the top 5. The strong model got the same five passages. A stronger writer cannot cite a page that it never received.

Larkfield: three designs, one workload

Five questions prove nothing; 67 questions in two passes are a measurement. The course team ran all three designs on Grace’s 67 questions. Each run replayed the requests that the RAG course recorded: the same passages, the same prompt, the same answer format. Only the model changed. The RAG course’s automatic check scored each answer. Then the pack’s calculator priced each design on this module’s base workload.

Guided practice: compare the three designs

In your folder design-pack, with the environment active, ask for the new command:

python -m costmodel options

This command arrives in Module 4. Download the end-of-module-4 snapshot or keep going.

Copy the code of this module’s options and the module’s design file from the snapshot end-of-m04.
- macOS / Linux
- Windows (PowerShell)
Terminal window
S=../nextia-learning-labs/C26/snapshots/end-of-m04 cp "$S/costmodel/options.py" costmodel/ cp "$S/design.toml" .
Terminal window
$S = "..\nextia-learning-labs\C26\snapshots\end-of-m04" Copy-Item "$S\costmodel\options.py" costmodel\ Copy-Item "$S\design.toml" .
Open design.toml. Two things are new. cache_hit_share is now [0.0, 0.1, 0.3]: the design assumes that 10% of the questions are answered from the cache in the base case. The next lesson explains this number. At the end is a section [labour], for the last lesson of the module. The line model = "small" in [answers] has not changed.

Compare the three designs:

python -m costmodel options

Model routing: the same 67 questions, the same search results (measured), base workload (cost model)
  design                    correct  US$/question   p50 s   p95 s  US$/month
  chat-small only             86.6%      0.000171    1.52    3.79      51.90
  chat-strong only            85.8%      0.003197    2.48    6.56     173.99
  routed (small first)        86.6%      0.000815    1.76    4.02      77.89
Cheapest design with the best measured quality: chat-small only

The command prints two more tables, about caching and batching. The next lesson reads them. The times are the model call only, from the course team’s computer to the provider.

Run the tests:
Terminal window
```
python -m pytest -q
```
```
...................................                                      [100%]
35 passed in 0.06s
```
The count has not changed. The tests of options.py arrive in the last lesson of the module, together with the code that they also test.

Read the option matrix

Kwame puts the three designs into one table with the targets. This is the module’s main picture: an option matrix of quality, latency and cost, measured on one workload.

Design	Correct (Q4: at least 85%)	Slow answers (Q1: p95 of the whole answer at most 6 s)	US$ per question	US$ per month (base)
chat-small only	86.6%: meets	0.48 + 0.32 + 3.79 = 4.59 s: meets	0.000171	51.90
chat-strong only	85.8%: meets, just	0.48 + 0.32 + 6.56 = 7.36 s: fails	0.003197 (19×)	173.99 (+235%)
routed (small first)	86.6%: meets	0.48 + 0.32 + 4.02 = 4.82 s: meets	0.000815 (4.8×)	77.89 (+50%)

The slow-answer column adds the p95 of each step, as Module 1 did: the question’s embedding (0.48 s), the reranker (0.32 s) and the model. It is an upper estimate. The same three designs as a chart:

The three designs on the same 67 questions: chat-small only (small), routed, and chat-strong only (strong). The quality stays at about 86%; the cost per question changes 19 times.

Text description of the chart

A scatter chart with three labelled points. The x axis is the cost per 1,000 questions, from 0 to 4 US dollars. The y axis is the share of correct answers, from 0 to 100%. chat-small only: US$0.17 per 1,000 questions, 86.6% correct. Routed: US$0.82, 86.6%. chat-strong only: US$3.20, 85.8%. The three points are at the same height; only their position from left to right changes.

All three points are at the same height. The strong model was not better on these questions, and routing did not change a single result. The cheapest design with the best measured quality is chat-small only, so the course-end design keeps it.

Two more numbers need a word. The calculator prices the routed design as chat-small for every question, plus an average chat-strong answer for 20.15% of them. That gives 0.000171 + 0.2015 × 0.003197 = US$0.000815. In the real run, the routed design cost a little less, US$0.00065 per question. The escalated questions were mostly short “I don’t know” answers. The calculator’s number is the careful one, and the decision is the same with both.

Why routing did not help

On average, 13.5 of the 67 questions were escalated per pass. Kwame looked at each escalated question, and at each question that chat-small got wrong. That review explains the result better than the totals.

What happened to the escalated questions in pass 1. None of the 14 changed its result; the three wrong answers that were not escalated were wrong for chat-strong too.

Text description of the diagram

In pass 1, chat-small declined 14 questions, so the rule sent them to chat-strong. Eight were questions that the documents cannot answer, like the student discount. chat-strong also declined them: the answer was correct twice and paid twice. Five were search misses, where the right passage was not in the top 5: Q13, Q26, Q28, Q36 and Q60. chat-strong also declined them. One was an injection bait, Q56, a question near a document with a planted instruction. chat-strong declined it too, and it stayed a miss. Three other wrong answers were never escalated. These were the contradictions Q45 and Q46 and the injection bait Q57. chat-small answered them with good citations, so no rule fired. chat-strong gets all three wrong as well.

Every remaining miss has a cause that a stronger model cannot fix:

A search miss needs better search: the right passage must reach the top 5. The RAG course shows how to find these in Find where it fails.
A contradiction between two documents needs a fix in the documents, or a prompt rule about which one wins.
An injection bait needs the defences of AI Security, Privacy and Responsible Design, not a stronger model.

This is the lesson’s main point: measure before you route. A stronger model writes better from the same passages. When the passages are wrong, it writes the same wrong answer, or declines, at 19 times the price.

Try a third rule: the search score

The decline rule fires on the wrong questions. Perhaps a different signal fires on the right ones? The explorer adds the third rule: escalate also when the best passage scores below T. Moving T to the right sends more questions to chat-strong. Predict first.

Explore

Does sending more questions to the strong model buy quality?

Sent to chat-strong too (pass 1 · pass 2): 14 · 13
Correct of 67 (pass 1 · pass 2): 58 · 58
US$ per question (× chat-small alone): US$0.000651 (×3.8)
p95 of the model calls (pass 1 · pass 2): 3.93 · 4.10 s

With the threshold off, only the two fixed rules escalate: chat-small declined or failed the citation check on 14 questions in pass 1 and 13 in pass 2. Correct: 58 and 58 of 67, exactly as chat-small alone: the extra calls did not change one result. Each question costs US$0.000651 on average, 3.8 times chat-small alone. The p95 of the model calls is 4.10 s.

pass 1
pass 2

The course team's measured run: Grace's 67 questions, the same passages, two passes. Escalate when chat-small declines, when a citation check fails, or when the best passage's reranker score is below T. Costs include both calls when a question is escalated.

Text description of the explorer

The explorer has one control, a slider for the threshold T on the best passage’s reranker score. It runs from off to above the highest score, in 83 steps; each step passes one question’s score. Three panels share the threshold axis. The first shows the correct answers of 67 for pass 1 and pass 2, with a dashed line at the quality target of 57 (85%). The second shows the US$ per question, with dashed lines for chat-small only (US$0.000171) and chat-strong only (US$0.003197). The third shows the p95 seconds of the model calls, with a dashed line at the 6-second target. With T off, 14 and 13 questions are escalated; 58 and 58 are correct; the cost is US$0.00065; the p95 is 3.9 and 4.1 s. At T = 0, about 21 to 23 questions are escalated; still 58 and 58 correct; the cost is about US$0.0011. At T = 2, 37 and 35 are escalated; 57 and 58 are correct; US$0.0018; p95 7.1 and 6.7 s. At the far right, all 67 are escalated; 57 and 58 are correct; US$0.0034 per question; p95 9.8 and 10.0 s. The correct count stays between 57 and 59 at every threshold; the cost and the p95 rise.

The quality line stays flat, between 57 and 59, at every threshold. The cost rises from US$0.00065 to US$0.0034 per question. The p95 of the model calls rises from about 4 to 10 seconds, far above the target. The score rule fails for the same reason: when the search is weak, both models get the same weak passages.

When to use routing

Use routing when you have measured two things on your own questions. First, the strong model is better on some kind of question. Second, a cheap signal finds that kind of question before the answer goes out. Good examples are a language that the small model handles badly, or long questions with many steps, if your evaluation set shows the gap.

Do not route because a model is “stronger” on a public leaderboard. Do not route to fix a search problem. And always compare with the right baseline: the small model alone, not “strong for everything”.

Routing costs more than the second call. It doubles the evaluation work, because Priya must test both models before every change. It adds code and a rule to keep up to date. Each escalation adds latency to the slowest answers. So the proposal keeps the measurement and a trigger to measure again. Triggers are a new model version, a new kind of question, or a new tenant’s documents.

Common mistakes

Common mistake

Routing is called a saving

Symptom: the proposal says “routing saves 55% compared with the strong model”. The cloud bill then grows by 50% after the change.

Cause: the comparison used the wrong baseline. The design before the change was chat-small only, at US$51.90 a month. Routing costs US$77.89.

Fix: compare every option with the current design, on the same questions and the same workload. Write the baseline in the option matrix, in the first row.

Common mistake

Two models compared on different questions

Symptom: a test shows chat-strong at 95% correct and chat-small at 86%. On Grace’s set, the two models are equal.

Cause: the strong model was tested on 20 easy questions from a demo, with a different search. The small model was tested on the 67 questions with the real search.

Fix: run every design on the same questions, with the same passages and the same prompt. The course team replayed the recorded requests, so only the model changed. Run more than one pass: chat-small changed its result on 2 of 67 questions between its two passes.

Reset. If your pack breaks, copy the snapshot end-of-m04 as the snapshots README shows, and delete buildbuy.py and tests/test_options.py until the last lesson of the module.

Your turn: put the routed design in the cost model

The design file chooses which model answers, in the section [answers]. Change model = "small" to model = "routed". Before you run anything, predict two things: the monthly cost, and the dominant driver.

Then run python -m costmodel costs, and change the value back to "small".

Show the answer

The option matrix already gave the total: US$77.89. The answer tokens grow from US$6.76 to about US$32, so they become the dominant driver:

Monthly cost, scenario base, month 0 (2 tenants, 43,950 questions)
  Driver                                US$ per month   share
  Model tokens: answers                         32.25   41.4%
  Compute: API                                  27.43   35.2%
  Database server                               13.50   17.3%
  Database storage                               4.05    5.2%
  Retries                                        0.65    0.8%
  Model tokens: question embeddings              0.01    0.0%
  Model tokens: summaries                        0.00    0.0%
  Compute: ingestion worker                      0.00    0.0%
  File storage                                   0.00    0.0%
  Model tokens: ingestion                        0.00    0.0%
  Database backups                               0.00    0.0%
  Requests                                       0.00    0.0%
  Logs                                           0.00    0.0%
  Network out                                    0.00    0.0%
  Total                                         77.89
Biggest driver: Model tokens: answers (41% of the total)

The retries also grow, from US$0.14 to US$0.65, because a retry of a strong call is a strong call. One line in the design file moved the biggest driver from the always-on API to the tokens. Change the value back to "small"; costs prints US$51.90 again.

Knowledge check

Check your understanding

Predict: the rule 'escalate when chat-small declines' fires on a question that the documents cannot answer, such as 'Do we offer a student discount?'. What does the escalation add to the result, the cost and the time?

Show the answer

Nothing to the result: chat-small declined correctly, and chat-strong declines too. The cost rises from about US$0.0001 to about US$0.002 for that question, because it pays for both calls. The person also waits for both: 1.21 + 1.64 = 2.85 s instead of 1.21 s in the measured run. In pass 1, 8 of the 14 escalated questions were of this kind.

Check your understanding

Tomás writes: 'The routed design costs US$0.000815 per question, a quarter of chat-strong's US$0.003197, so routing saves money.' What is wrong with the sentence, and how would you write it?

Show the answer

The baseline is wrong. The current design is chat-small only, at US$0.000171 per question and US$51.90 a month. Compared with it, routing costs 4.8 times more per question and +50% a month (US$77.89), for the same 86.6% correct. A true sentence: “On Grace’s 67 questions, routing costs 50% more a month than chat-small alone, for the same quality. We keep chat-small.”

Check your understanding

Ines asks: 'chat-strong got 85.8% and chat-small 86.6%. So is the strong model worse?' Answer her with the measurement, and name one change that would really improve the answers.

Show the answer

No, the measurement shows no difference: 57 and 58 correct for chat-strong, 58 and 58 for chat-small, in two passes of 67 questions. The gap is one answer. chat-small itself changed its result on 2 questions between its two passes. The strong model is not better here, and it is slower (p95 6.56 s against 3.79 s) and 19 times more expensive. A real improvement is better search: five of the nine misses are search misses, where the right passage was not in the top 5. Fixing the two contradictions in the documents would help too.

Recap

Model routing: a rule chooses which model answers. The usual pattern is small first, then escalation to a strong model when a cheap signal fires.
An escalated question pays for both calls and waits for both. Routing saves money only against “strong for everything”; it adds quality only if the strong model fixes the escalated questions.
The measured result on Grace’s 67 questions: chat-small 86.6%, US$0.000171, p95 3.79 s · chat-strong 85.8%, US$0.003197, p95 6.56 s · routed 86.6%, US$0.000815, p95 4.02 s. Monthly: US$51.90 / 173.99 / 77.89.
No escalation changed a result. The misses are search misses, contradictions and injection baits: a stronger model does not fix a missing passage.
A score threshold as a third rule keeps the quality at 57 to 59 correct. It raises the cost up to US$0.0034 and the p95 up to 10 s.
The design keeps chat-small only, and records when to measure again.

Next: Caching and batching