A routing service
- Lesson 35
- 45 min
- Module 9: Run it in production
Where we are
We can choose a model, its thresholds and its zones in a notebook. A notebook cannot answer the inbox all day. In this lesson, Amira runs the router as a small service that loads the model once, checks its input and returns the label, the confidence and the model version.
Lesson 35 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceA routing service
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Run the router as a small service that the inbox can call all day. It loads the model once, refuses bad input, and answers with the route, the zone, the confidence and the versions. It logs every decision without the message text, and it puts the unsure messages in Chloé’s review queue.
Before you startWhat to know and to have ready
What you need to know
- The three zones (decide, review, person) and the threshold 0.58 that Grace’s review limit chose, from Decide, review or send to a person.
- TF-IDF with logistic regression, the model that the service uses, from Naive Bayes, logistic regression and linear SVM.
- What a request, a response and a status code are. HTTP fundamentals in the API course explains them, if they are new to you.
What you need on your computer
- The project folder
larkroutefrom Module 8, with its.venvvirtual environment and itsdata/folder. The labs foldernextia-learning-labsis next to it. Rungit -C nextia-learning-labs pullin the parent folder to get the latest files. - If you do not have the Module 8 project, copy the snapshot
nextia-learning-labs/C36/snapshots/end-of-m08to a new folderlarkroute. Then follow itsREADME.mdto make the environment and get the data. - Two terminal windows: one runs the service, and the other sends it messages.
Tested on macOS with Python 3.12.13, FastAPI 0.162.0, uvicorn 0.54.0 and scikit-learn 1.9.1. The Windows (PowerShell) commands are written out, but we did not run them.
The sorting counter at the post office
Section titled “The sorting counter at the post office”Think of the sorting counter in a post office. In the morning, the sorter opens the rulebook and learns the rules for the day. She does not read the whole rulebook again for each letter. When a letter arrives, she first checks it: is there an address? A letter with no address goes back, with a note that says why.
Then she sorts the letter into a bin. If she is sure, it goes straight into a street’s bin. If she is not sure, it goes into a tray for the supervisor. At the end of the day, the supervisor goes through the tray.
She also keeps a logbook. For each letter, she writes its number, the bin and the edition of the rulebook she used. She never copies what the letter says: that is private. If someone asks later why a letter went to the wrong street, the logbook answers.
Larkfield needs the same counter for its inbox, and it must be open all day.
The problem: a notebook cannot answer the inbox
Section titled “The problem: a notebook cannot answer the inbox”Until now, Priya ran the router in a notebook or with one route command. Each run reads the data, trains or loads a model, scores the messages and stops. Larkfield’s inbox works differently. Messages arrive one at a time, by email, chat and social media, all day. The help desk software needs an answer for each one in a few milliseconds.
Amira, the developer, lists what goes wrong if she only wraps Priya’s code in a loop:
- Loading the model for each message is slow. On the test laptop, loading the saved model took 26.7 ms each time. Routing one message took 0.49 ms. So most of the time goes into loading, not into routing.
- Bad input gets a confident-looking answer. We gave the model an empty text, with no check before it. It sent the empty text to
product, with confidence 0.3036. An empty message would wait in the review queue for a product agent. - Nobody can say which model decided. Priya retrains the model often. If the answers do not name the model, a wrong route next month cannot be traced back.
- A log with the message text is a privacy risk. Messages contain names, order numbers and addresses. People copy log files, share them and keep them for a long time.
Text description of the chart
A column chart with two columns. The first column is the time for one message when the service loads the model for each message: 27.2 ms. The second column is the time when the model is already in memory: 0.49 ms. The second column is so short that only its value label shows.
The first load of the day is slower again: 673 ms, because Python must also load scikit-learn. A service pays this cost once, when it starts.
The idea: a small service
Section titled “The idea: a small service”A service is a program that runs all the time and answers requests from other programs. Here, the help desk software sends a message, and the service answers with a route. The two programs talk over HTTP, the same protocol that a web browser uses. The service listens on a port, a number that names one door on the computer. Ours is 8036.
The service has one endpoint for routing: an address and a method, POST /route. A POST request carries data in its body. Here, the body is JSON: text that holds names and values, such as {"message_id": "M-999001", "text": "…"}. The answer is JSON too.
Each step of the sorting counter becomes one rule of the service:
Text description of the diagram
The diagram has four groups, from top to bottom. The first group happens once, when the service starts. The file models/current names the version, v1. The service loads model.joblib and card.json from models/v1/ and keeps them in memory. The second group checks the input of each message. A message arrives at POST /route, with a message ID, a text and a channel. The service asks: is the input right? The ID must look like M-999001, the text must not be empty, and the channel must be one that Larkfield uses. If not, the answer is HTTP 422 with the reason, and no decision is made. If the input is right, the third group routes the message into a zone, with the model in memory. The model gives a confidence for each route. If the best route is needs_person, the zone is person. If the confidence is 0.58 or more, the zone is decide. Below 0.58, the zone is review. The fourth group is what comes out. The answer holds the route, the zone, the confidence, the model version and the label version. Every decision goes into queue/decisions.jsonl, without the text. Review decisions also go into queue/review_queue.jsonl, for Chloé.
Load once, check the input
Section titled “Load once, check the input”The service loads the model when it starts, and keeps it in memory. This is the sorter who learns the rulebook in the morning. In the code, FastAPI calls a function named lifespan once at start-up, and that function loads the model.
Then the service checks every request before the model sees it. The service uses FastAPI, a Python library for building services. FastAPI checks requests with Pydantic models. A Pydantic model is a class that lists the fields of the request and the rule for each field. The Message class in larkroute/service.py says:
message_idmust look likeM-999001: one to three capital letters, a hyphen, and digits.textmust have between 1 and 5,000 characters, and it must not be only spaces.channelmust beemail,chat,socialormarketplace. If it is missing, it isemail.
If a field breaks its rule, FastAPI does not call our code. It answers with the status code 422 (“Unprocessable Content”), and a body that names the field and the problem. A status code is the number at the start of every HTTP answer: 200 means “OK”, and the 400s mean “your request is wrong”.
Route into a zone, and name the versions
Section titled “Route into a zone, and name the versions”For a good request, the model gives a confidence for each route. The service takes the best route and chooses the zone with the thresholds from the model’s card:
| Message (constructed) | Best route | Confidence | Rule | Zone | Route in the answer |
|---|---|---|---|---|---|
| M-999001: parcel delivered, nothing at my door | delivery | 0.9937 | 0.9937 ≥ 0.58 | decide | delivery |
| M-999005: a question about my order and the invoice | billing | 0.394 | 0.394 < 0.58 | review | billing |
| M-999006: the hedge trimmer caught fire, my hand is burned | needs_person | 0.8519 | best route is needs_person | person | needs_person |
These are messages we wrote for the lesson, and the values are the service’s real answers. Notice the third row. A person reads every message that the model sends to needs_person, even when it is sure. On validation, the cheapest choice had no low-confidence cut for the person zone: its threshold is 0.00. So in this model, the person zone holds only the needs_person messages.
Every answer also names two versions:
- The model version, such as
v1, says which saved model decided. Each version lives in its own folder,models/v1/, and the project never writes over a version. - The label version, such as
labels-4bad2e4ef7, is a fingerprint of the labels that the model learned from. The project computes it from the message IDs and their routes with SHA-256, a function that turns any text into a fixed code. The same labels always give the same code, and one changed label gives a new code.
Log the decision, not the text
Section titled “Log the decision, not the text”The service writes one line of JSON for each decision in queue/decisions.jsonl. The format JSON lines has one JSON object per line, so other programs can read it line by line. The line has the message ID, the route, the zone, the confidence, the two versions, the channel and the time. It never has the text. The message ID is enough: an authorised person can look up the message in the help desk, where its access rules already apply.
Review decisions also go to queue/review_queue.jsonl. This is Chloé’s tray: the message ID, the suggested route and the confidence. In a real help desk, the service would put the message into a queue screen instead of a file. The idea is the same.
Set up the service
Section titled “Set up the service”Now you add the service to your project. The project is the snapshot end-of-m08 from Module 8. You copy this module’s files from the snapshot end-of-m09. A snapshot is a copy of the project at one point of the course. This module has one snapshot for its three lessons, so you also copy the files of the next two lessons now.
-
Open a terminal in the parent folder of
larkroute. Go into the project and activate its environment:Terminal window cd larkroutesource .venv/bin/activatepython -m pytest -qTerminal window cd larkroute.venv\Scripts\Activate.ps1python -m pytest -qThe last line says
28 passed. -
Copy the eight new code files, the four new test files,
requirements.txtandREADME.mdfrom the snapshotend-of-m09:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m09for f in model_store service drift release; do cp "$S/larkroute/$f.py" larkroute/; donefor f in train serve drift release; do cp "$S/larkroute/commands/$f.py" larkroute/commands/; donefor t in service drift release model_store; do cp "$S/tests/test_$t.py" tests/; donecp "$S/requirements.txt" "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m09"foreach ($f in "model_store","service","drift","release") { Copy-Item "$S\larkroute\$f.py" larkroute\ }foreach ($f in "train","serve","drift","release") { Copy-Item "$S\larkroute\commands\$f.py" larkroute\commands\ }foreach ($t in "service","drift","release","model_store") { Copy-Item "$S\tests\test_$t.py" tests\ }Copy-Item "$S\requirements.txt", "$S\README.md" .The commands are found by their file names, so
routenow has four new commands:train,serve,driftandrelease. -
Install the new packages.
requirements.txtnow pins FastAPI, uvicorn and httpx:Terminal window python -m pip install -r requirements.txtAmong the packages that pip installs, you see
fastapi-0.162.0,uvicorn-0.54.0,httpx-0.28.1andstarlette-1.10.0. uvicorn is the server program that runs a FastAPI service. httpx lets the tests call the service without a network. -
Run the tests and the style check:
Terminal window python -m pytest -qruff check .You see
42 passedandAll checks passed!. Nine of the 14 new tests check the service. One sends a known message and checks that the answer names both versions. Six send bad input and expect 422. One checks that the log has no message text, and one checks that a version is never written over. -
Train the model and save it as version
v1.--make-currenttells the service to load this version:Terminal window route train --version v1 --make-currentSaved models/v1: 6000 messages, 7 routes, labels-4bad2e4ef7; validation accuracy 0.958; decide at 0.58, person below 0.00. Now the current model. -
Open
models/v1/card.jsonin your editor. The model card is the model’s identity sheet: its version, its routes, its label version, its thresholds and its validation scores. Its validation numbers are the same as in Module 8. On validation, 90.2% of messages are decided, 5.4% go to review and 4.4% go to a person. The cost is US$305.8 per 1,000 messages. The filemodels/currentcontains one word:v1.
Run the service and send it messages
Section titled “Run the service and send it messages”-
In the first terminal, start the service:
Terminal window route serveINFO: Started server process [2493]INFO: Waiting for application startup.INFO: Application startup complete.INFO: Uvicorn running on http://127.0.0.1:8036 (Press CTRL+C to quit)Your process number will differ. The address
127.0.0.1means “this computer only”, so no other computer can reach the service. -
Open a second terminal in the
larkroutefolder. Ask the service which model it runs:Terminal window curl -s http://127.0.0.1:8036/healthTerminal window Invoke-RestMethod http://127.0.0.1:8036/health{"status":"ok","model_version":"v1","label_version":"labels-4bad2e4ef7","routes":["account","billing","delivery","needs_person","orders","product","returns_refunds"]}PowerShell shows the same fields as a list, not as one line of JSON.
-
Send the first message of the table above:
Terminal window curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \-d '{"message_id": "M-999001", "text": "Tracking says my parcel was delivered yesterday, but nothing is at my door. Order LK-123456.", "channel": "chat"}'Terminal window Invoke-RestMethod -Method Post -Uri http://127.0.0.1:8036/route -ContentType 'application/json' `-Body '{"message_id": "M-999001", "text": "Tracking says my parcel was delivered yesterday, but nothing is at my door. Order LK-123456.", "channel": "chat"}'{"message_id":"M-999001","route":"delivery","zone":"decide","confidence":0.9937,"model_version":"v1","label_version":"labels-4bad2e4ef7"} -
Send two more messages in the same way, with these bodies. Change only the
-dpart (or-Bodyin PowerShell):{"message_id": "M-999006", "text": "The hedge trimmer caught fire while I was using it. My hand is burned. I will contact my lawyer.", "channel": "email"}{"message_id": "M-999007", "text": "My new Grow Hub keeps blinking orange and will not connect to the Wi-Fi.", "channel": "social"}The answers:
{"message_id":"M-999006","route":"needs_person","zone":"person","confidence":0.8519,"model_version":"v1","label_version":"labels-4bad2e4ef7"}{"message_id":"M-999007","route":"product","zone":"review","confidence":0.4671,"model_version":"v1","label_version":"labels-4bad2e4ef7"}The Grow Hub is Omar’s new product. The model has never seen a message about it, so it guesses
productwith low confidence, and the message goes to review. Keep this in mind for the next lesson. -
Send a message with an empty text:
Terminal window curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' -d '{"message_id": "M-999002", "text": ""}'{"detail":[{"type":"string_too_short","loc":["body","text"],"msg":"String should have at least 1 character","input":"","ctx":{"min_length":1}}]}The status code is 422: add
-w "\nHTTP %{http_code}\n"to thecurlcommand to see it. In PowerShell,Invoke-RestMethodshows the 422 as an error, with the same body. The body says which field failed (loc) and why (msg). -
In the first terminal, stop the service with Ctrl+C. The last line is
INFO: Finished server process [2493]. -
Open
queue/decisions.jsonlandqueue/review_queue.jsonlin your editor.
queue/decisions.jsonl has one line for each routed message. Here are the lines of our run for the three messages:
{"message_id": "M-999001", "route": "delivery", "zone": "decide", "confidence": 0.9937, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "chat", "time": 1794062680.778}{"message_id": "M-999006", "route": "needs_person", "zone": "person", "confidence": 0.8519, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "email", "time": 1794062680.796}{"message_id": "M-999007", "route": "product", "zone": "review", "confidence": 0.4671, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "social", "time": 1794062680.805}Your time values will differ. The time is the number of seconds since 1 January 1970, the usual way computers store a moment. The empty message has no line, because the service refused it before any decision. Look for the words of a message in the file: “Grow Hub” is not there. queue/review_queue.jsonl has one line for M-999007, with "suggested_route": "product".
The terminal of the service printed one line for each request too, for example "POST /route HTTP/1.1" 422 Unprocessable Entity. That is uvicorn’s access log: the address, the method and the status code, never the body.
Your turn: one message in each zone
Section titled “Your turn: one message in each zone”Write three messages of your own. Try to write one that lands in each zone: decide, review and person. Before you send each one, predict its route, its zone and roughly its confidence. Send them, then find each one in queue/decisions.jsonl, and the review one in queue/review_queue.jsonl.
Show an example
We sent the three messages of the table, and one more: “Hi, I have a question about my order and the invoice. Can someone call me?” Two topics in one message (orders and billing) give the model two good answers, so neither is sure. The answer was billing, zone review, confidence 0.394, and the message went to the review queue.
A message that names one problem with its usual words lands in decide, like M-999001 (0.9937). A safety problem or a threat lands in person, like M-999006. A message about something the model never saw lands in review, like the Grow Hub message (0.4671). If all your messages land in decide, write a vaguer one: “Hello, I wrote to you last week. Any news?”
When to use a service, and what it costs
Section titled “When to use a service, and what it costs”Use a service when another program needs an answer for each message, at the moment it arrives. That is the case for an inbox that routes messages as they come in. The service is cheap here: one process, about 260 MB of disk for the environment, and half a millisecond of computer time per message.
Do not build a service when a daily batch is enough. If Grace’s team only needs yesterday’s messages sorted each morning, a scheduled route command is simpler. No process must keep running, and nothing listens on a port.
A service has its own costs. Someone must start it again after a crash, and watch it. Logs, metrics and traces in the MLOps course shows how a team does that. A service that other computers can reach also needs sign-in and limits, which ours does not have: it listens only on 127.0.0.1. And the model file is a pickle, a Python format that can run code when it is loaded. Load only model files that you made yourself.
Tomás asks: “Why not call the hosted LLM from the service, for every message?” You can, but each call takes 1.7 seconds instead of half a millisecond. It also needs a network, and it sends the customer’s text to another company. Cascades showed a better mix: the cheap model first, and the LLM only for the unsure messages.
Common mistake
Two different models, both called v1
Symptom: Priya fixes a few labels and trains again with the same name, v1. Next week, Grace asks why a refund request went to product. The log says "model_version": "v1", but two different models had that name. Nobody can tell which one decided.
In this project, the second training stops with an error instead:
$ route train --version v1models/v1 exists: a version is never overwritten. Choose a new version name.Cause: a version name is only useful if it always means the same model. If you write over a version, its name stops pointing to one model, and the log loses its meaning.
Fix: give every trained model a new version name, such as v2. Keep the old folder. The label version adds a second check: new labels give a new fingerprint, even if someone reuses a name in another system.
Consequence in AI work: without stable versions, you cannot explain a past decision, compare two models fairly, or go back to the model that worked. The next two lessons need all three.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name the four things that the routing service does that a notebook call does not, and the problem that each one solves.
Show the answer
- It loads the model once, at start-up. On the test laptop, loading took about 27 ms each time. Routing one message took about 0.5 ms. Loading for each message would make every answer much slower.
- It checks the input. A bad request gets HTTP 422 with the reason, and no decision. Without the check, an empty text went to
productwith confidence 0.3036. - It names the model version and the label version in every answer. A wrong route can be traced back to the model and the labels that made it.
- It logs decisions without the text, and sends review decisions to Chloé’s queue. The log answers “what happened?” without copying private messages.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: the model sends a message to needs_person with confidence 0.99. Which zone does the service choose, and why? What would happen if the threshold person_below were 0.40 and a delivery message had confidence 0.35?
Show the answer
The zone is person, whatever the confidence. A person reads every message that the model sends to needs_person, because those messages are safety problems, legal threats or escalated complaints. The confidence does not change that.
With person_below at 0.40, a delivery message with confidence 0.35 would also go to the person zone, with the route needs_person. The model is too unsure to suggest a team. With the current card (person_below 0.00), the same message goes to review with the suggested route delivery.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: start the service, send a message whose text has only spaces, and say what you get back and what the service writes to its two files.
Show the answer
The answer is HTTP 422 with a body like {"detail":[{"type":"value_error","loc":["body","text"],"msg":"Value error, text is empty","input":" ",…}]}. The Message class refuses a text that is only spaces. FastAPI never calls the routing code, so the service writes nothing to queue/decisions.jsonl and nothing to queue/review_queue.jsonl. Only uvicorn’s access log in the terminal shows the request, with status 422.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A service runs all the time and answers each message over HTTP. Ours has
POST /routeandGET /health, on port 8036 of this computer only. - It loads the model once. On the test laptop, a load took about 27 ms and one message took 0.49 ms.
- It checks the input with FastAPI and Pydantic: a bad ID, an empty text or an unknown channel gets HTTP 422 and no decision.
- Every answer has the route, the zone, the confidence, the model version and the label version. A version is never written over.
- It logs every decision without the text in
queue/decisions.jsonl, and sends review decisions toqueue/review_queue.jsonlfor Chloé.
The service now answers all day. But the inbox does not stay the same: the Grow Hub message already went to review. Next, you replay Omar’s launch month through the service’s model and see what changes.
Next: Drift and new topics