Skip to content
Français

A routing service

  • Lesson 35
  • 45 min
  • Module 9: Run it in production

Where we are

We can choose a model, its thresholds and its zones in a notebook. A notebook cannot answer the inbox all day. In this lesson, Amira runs the router as a small service that loads the model once, checks its input and returns the label, the confidence and the model version.

Lesson 35 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceA routing service

What you will be able to do

Run the router as a small service that the inbox can call all day. It loads the model once, refuses bad input, and answers with the route, the zone, the confidence and the versions. It logs every decision without the message text, and it puts the unsure messages in Chloé’s review queue.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • The project folder larkroute from Module 8, with its .venv virtual environment and its data/ folder. The labs folder nextia-learning-labs is next to it. Run git -C nextia-learning-labs pull in the parent folder to get the latest files.
  • If you do not have the Module 8 project, copy the snapshot nextia-learning-labs/C36/snapshots/end-of-m08 to a new folder larkroute. Then follow its README.md to make the environment and get the data.
  • Two terminal windows: one runs the service, and the other sends it messages.

Tested on macOS with Python 3.12.13, FastAPI 0.162.0, uvicorn 0.54.0 and scikit-learn 1.9.1. The Windows (PowerShell) commands are written out, but we did not run them.

Think of the sorting counter in a post office. In the morning, the sorter opens the rulebook and learns the rules for the day. She does not read the whole rulebook again for each letter. When a letter arrives, she first checks it: is there an address? A letter with no address goes back, with a note that says why.

Then she sorts the letter into a bin. If she is sure, it goes straight into a street’s bin. If she is not sure, it goes into a tray for the supervisor. At the end of the day, the supervisor goes through the tray.

She also keeps a logbook. For each letter, she writes its number, the bin and the edition of the rulebook she used. She never copies what the letter says: that is private. If someone asks later why a letter went to the wrong street, the logbook answers.

Larkfield needs the same counter for its inbox, and it must be open all day.

The problem: a notebook cannot answer the inbox

Section titled “The problem: a notebook cannot answer the inbox”

Until now, Priya ran the router in a notebook or with one route command. Each run reads the data, trains or loads a model, scores the messages and stops. Larkfield’s inbox works differently. Messages arrive one at a time, by email, chat and social media, all day. The help desk software needs an answer for each one in a few milliseconds.

Amira, the developer, lists what goes wrong if she only wraps Priya’s code in a loop:

  1. Loading the model for each message is slow. On the test laptop, loading the saved model took 26.7 ms each time. Routing one message took 0.49 ms. So most of the time goes into loading, not into routing.
  2. Bad input gets a confident-looking answer. We gave the model an empty text, with no check before it. It sent the empty text to product, with confidence 0.3036. An empty message would wait in the review queue for a product agent.
  3. Nobody can say which model decided. Priya retrains the model often. If the answers do not name the model, a wrong route next month cannot be traced back.
  4. A log with the message text is a privacy risk. Messages contain names, order numbers and addresses. People copy log files, share them and keep them for a long time.
Loading the model for every message adds about 27 ms. Once the model is in memory, one message takes 0.49 ms. Measured on the test laptop; your times will differ.
Text description of the chart

A column chart with two columns. The first column is the time for one message when the service loads the model for each message: 27.2 ms. The second column is the time when the model is already in memory: 0.49 ms. The second column is so short that only its value label shows.

The first load of the day is slower again: 673 ms, because Python must also load scikit-learn. A service pays this cost once, when it starts.

A service is a program that runs all the time and answers requests from other programs. Here, the help desk software sends a message, and the service answers with a route. The two programs talk over HTTP, the same protocol that a web browser uses. The service listens on a port, a number that names one door on the computer. Ours is 8036.

The service has one endpoint for routing: an address and a method, POST /route. A POST request carries data in its body. Here, the body is JSON: text that holds names and values, such as {"message_id": "M-999001", "text": "…"}. The answer is JSON too.

Each step of the sorting counter becomes one rule of the service:

Four groups from top to bottom. Once, when the service starts: models/current names the version v1, and the service loads model.joblib and card.json and keeps them in memory. Then, for each message, check the input: a message arrives at POST /route with message_id, text and channel. A question: is the input right? An ID like M-999001, a text that is not empty, a known channel. If no: HTTP 422 with the reason, and no decision. If yes, with the model in memory: route it into a zone. The model gives a confidence for each route. A second question: which zone? If the best route is needs_person: person. If the confidence is 0.58 or more: decide. If below 0.58: review. What comes out: the answer with route, zone, confidence, model_version and label_version; queue/decisions.jsonl with every decision and never the text; queue/review_queue.jsonl with the review decisions, for Chloé.
The routing service. The model is loaded once. Each message is checked, routed into a zone and logged without its text.
Text description of the diagram

The diagram has four groups, from top to bottom. The first group happens once, when the service starts. The file models/current names the version, v1. The service loads model.joblib and card.json from models/v1/ and keeps them in memory. The second group checks the input of each message. A message arrives at POST /route, with a message ID, a text and a channel. The service asks: is the input right? The ID must look like M-999001, the text must not be empty, and the channel must be one that Larkfield uses. If not, the answer is HTTP 422 with the reason, and no decision is made. If the input is right, the third group routes the message into a zone, with the model in memory. The model gives a confidence for each route. If the best route is needs_person, the zone is person. If the confidence is 0.58 or more, the zone is decide. Below 0.58, the zone is review. The fourth group is what comes out. The answer holds the route, the zone, the confidence, the model version and the label version. Every decision goes into queue/decisions.jsonl, without the text. Review decisions also go into queue/review_queue.jsonl, for Chloé.

The service loads the model when it starts, and keeps it in memory. This is the sorter who learns the rulebook in the morning. In the code, FastAPI calls a function named lifespan once at start-up, and that function loads the model.

Then the service checks every request before the model sees it. The service uses FastAPI, a Python library for building services. FastAPI checks requests with Pydantic models. A Pydantic model is a class that lists the fields of the request and the rule for each field. The Message class in larkroute/service.py says:

  • message_id must look like M-999001: one to three capital letters, a hyphen, and digits.
  • text must have between 1 and 5,000 characters, and it must not be only spaces.
  • channel must be email, chat, social or marketplace. If it is missing, it is email.

If a field breaks its rule, FastAPI does not call our code. It answers with the status code 422 (“Unprocessable Content”), and a body that names the field and the problem. A status code is the number at the start of every HTTP answer: 200 means “OK”, and the 400s mean “your request is wrong”.

For a good request, the model gives a confidence for each route. The service takes the best route and chooses the zone with the thresholds from the model’s card:

Message (constructed) Best route Confidence Rule Zone Route in the answer
M-999001: parcel delivered, nothing at my door delivery 0.9937 0.9937 ≥ 0.58 decide delivery
M-999005: a question about my order and the invoice billing 0.394 0.394 < 0.58 review billing
M-999006: the hedge trimmer caught fire, my hand is burned needs_person 0.8519 best route is needs_person person needs_person

These are messages we wrote for the lesson, and the values are the service’s real answers. Notice the third row. A person reads every message that the model sends to needs_person, even when it is sure. On validation, the cheapest choice had no low-confidence cut for the person zone: its threshold is 0.00. So in this model, the person zone holds only the needs_person messages.

Every answer also names two versions:

  • The model version, such as v1, says which saved model decided. Each version lives in its own folder, models/v1/, and the project never writes over a version.
  • The label version, such as labels-4bad2e4ef7, is a fingerprint of the labels that the model learned from. The project computes it from the message IDs and their routes with SHA-256, a function that turns any text into a fixed code. The same labels always give the same code, and one changed label gives a new code.

The service writes one line of JSON for each decision in queue/decisions.jsonl. The format JSON lines has one JSON object per line, so other programs can read it line by line. The line has the message ID, the route, the zone, the confidence, the two versions, the channel and the time. It never has the text. The message ID is enough: an authorised person can look up the message in the help desk, where its access rules already apply.

Review decisions also go to queue/review_queue.jsonl. This is Chloé’s tray: the message ID, the suggested route and the confidence. In a real help desk, the service would put the message into a queue screen instead of a file. The idea is the same.

Now you add the service to your project. The project is the snapshot end-of-m08 from Module 8. You copy this module’s files from the snapshot end-of-m09. A snapshot is a copy of the project at one point of the course. This module has one snapshot for its three lessons, so you also copy the files of the next two lessons now.

  1. Open a terminal in the parent folder of larkroute. Go into the project and activate its environment:

    Terminal window
    cd larkroute
    source .venv/bin/activate
    python -m pytest -q

    The last line says 28 passed.

  2. Copy the eight new code files, the four new test files, requirements.txt and README.md from the snapshot end-of-m09:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m09
    for f in model_store service drift release; do cp "$S/larkroute/$f.py" larkroute/; done
    for f in train serve drift release; do cp "$S/larkroute/commands/$f.py" larkroute/commands/; done
    for t in service drift release model_store; do cp "$S/tests/test_$t.py" tests/; done
    cp "$S/requirements.txt" "$S/README.md" .

    The commands are found by their file names, so route now has four new commands: train, serve, drift and release.

  3. Install the new packages. requirements.txt now pins FastAPI, uvicorn and httpx:

    Terminal window
    python -m pip install -r requirements.txt

    Among the packages that pip installs, you see fastapi-0.162.0, uvicorn-0.54.0, httpx-0.28.1 and starlette-1.10.0. uvicorn is the server program that runs a FastAPI service. httpx lets the tests call the service without a network.

  4. Run the tests and the style check:

    Terminal window
    python -m pytest -q
    ruff check .

    You see 42 passed and All checks passed!. Nine of the 14 new tests check the service. One sends a known message and checks that the answer names both versions. Six send bad input and expect 422. One checks that the log has no message text, and one checks that a version is never written over.

  5. Train the model and save it as version v1. --make-current tells the service to load this version:

    Terminal window
    route train --version v1 --make-current
    Saved models/v1: 6000 messages, 7 routes, labels-4bad2e4ef7; validation accuracy 0.958; decide at 0.58, person below 0.00. Now the current model.
  6. Open models/v1/card.json in your editor. The model card is the model’s identity sheet: its version, its routes, its label version, its thresholds and its validation scores. Its validation numbers are the same as in Module 8. On validation, 90.2% of messages are decided, 5.4% go to review and 4.4% go to a person. The cost is US$305.8 per 1,000 messages. The file models/current contains one word: v1.

  1. In the first terminal, start the service:

    Terminal window
    route serve
    INFO: Started server process [2493]
    INFO: Waiting for application startup.
    INFO: Application startup complete.
    INFO: Uvicorn running on http://127.0.0.1:8036 (Press CTRL+C to quit)

    Your process number will differ. The address 127.0.0.1 means “this computer only”, so no other computer can reach the service.

  2. Open a second terminal in the larkroute folder. Ask the service which model it runs:

    Terminal window
    curl -s http://127.0.0.1:8036/health
    {"status":"ok","model_version":"v1","label_version":"labels-4bad2e4ef7","routes":["account","billing","delivery","needs_person","orders","product","returns_refunds"]}

    PowerShell shows the same fields as a list, not as one line of JSON.

  3. Send the first message of the table above:

    Terminal window
    curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \
    -d '{"message_id": "M-999001", "text": "Tracking says my parcel was delivered yesterday, but nothing is at my door. Order LK-123456.", "channel": "chat"}'
    {"message_id":"M-999001","route":"delivery","zone":"decide","confidence":0.9937,"model_version":"v1","label_version":"labels-4bad2e4ef7"}
  4. Send two more messages in the same way, with these bodies. Change only the -d part (or -Body in PowerShell):

    {"message_id": "M-999006", "text": "The hedge trimmer caught fire while I was using it. My hand is burned. I will contact my lawyer.", "channel": "email"}
    {"message_id": "M-999007", "text": "My new Grow Hub keeps blinking orange and will not connect to the Wi-Fi.", "channel": "social"}

    The answers:

    {"message_id":"M-999006","route":"needs_person","zone":"person","confidence":0.8519,"model_version":"v1","label_version":"labels-4bad2e4ef7"}
    {"message_id":"M-999007","route":"product","zone":"review","confidence":0.4671,"model_version":"v1","label_version":"labels-4bad2e4ef7"}

    The Grow Hub is Omar’s new product. The model has never seen a message about it, so it guesses product with low confidence, and the message goes to review. Keep this in mind for the next lesson.

  5. Send a message with an empty text:

    Terminal window
    curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' -d '{"message_id": "M-999002", "text": ""}'
    {"detail":[{"type":"string_too_short","loc":["body","text"],"msg":"String should have at least 1 character","input":"","ctx":{"min_length":1}}]}

    The status code is 422: add -w "\nHTTP %{http_code}\n" to the curl command to see it. In PowerShell, Invoke-RestMethod shows the 422 as an error, with the same body. The body says which field failed (loc) and why (msg).

  6. In the first terminal, stop the service with Ctrl+C. The last line is INFO: Finished server process [2493].

  7. Open queue/decisions.jsonl and queue/review_queue.jsonl in your editor.

queue/decisions.jsonl has one line for each routed message. Here are the lines of our run for the three messages:

{"message_id": "M-999001", "route": "delivery", "zone": "decide", "confidence": 0.9937, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "chat", "time": 1794062680.778}
{"message_id": "M-999006", "route": "needs_person", "zone": "person", "confidence": 0.8519, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "email", "time": 1794062680.796}
{"message_id": "M-999007", "route": "product", "zone": "review", "confidence": 0.4671, "model_version": "v1", "label_version": "labels-4bad2e4ef7", "channel": "social", "time": 1794062680.805}

Your time values will differ. The time is the number of seconds since 1 January 1970, the usual way computers store a moment. The empty message has no line, because the service refused it before any decision. Look for the words of a message in the file: “Grow Hub” is not there. queue/review_queue.jsonl has one line for M-999007, with "suggested_route": "product".

The terminal of the service printed one line for each request too, for example "POST /route HTTP/1.1" 422 Unprocessable Entity. That is uvicorn’s access log: the address, the method and the status code, never the body.

Write three messages of your own. Try to write one that lands in each zone: decide, review and person. Before you send each one, predict its route, its zone and roughly its confidence. Send them, then find each one in queue/decisions.jsonl, and the review one in queue/review_queue.jsonl.

Show an example

We sent the three messages of the table, and one more: “Hi, I have a question about my order and the invoice. Can someone call me?” Two topics in one message (orders and billing) give the model two good answers, so neither is sure. The answer was billing, zone review, confidence 0.394, and the message went to the review queue.

A message that names one problem with its usual words lands in decide, like M-999001 (0.9937). A safety problem or a threat lands in person, like M-999006. A message about something the model never saw lands in review, like the Grow Hub message (0.4671). If all your messages land in decide, write a vaguer one: “Hello, I wrote to you last week. Any news?”

Use a service when another program needs an answer for each message, at the moment it arrives. That is the case for an inbox that routes messages as they come in. The service is cheap here: one process, about 260 MB of disk for the environment, and half a millisecond of computer time per message.

Do not build a service when a daily batch is enough. If Grace’s team only needs yesterday’s messages sorted each morning, a scheduled route command is simpler. No process must keep running, and nothing listens on a port.

A service has its own costs. Someone must start it again after a crash, and watch it. Logs, metrics and traces in the MLOps course shows how a team does that. A service that other computers can reach also needs sign-in and limits, which ours does not have: it listens only on 127.0.0.1. And the model file is a pickle, a Python format that can run code when it is loaded. Load only model files that you made yourself.

Tomás asks: “Why not call the hosted LLM from the service, for every message?” You can, but each call takes 1.7 seconds instead of half a millisecond. It also needs a network, and it sends the customer’s text to another company. Cascades showed a better mix: the cheap model first, and the LLM only for the unsure messages.

Common mistake

Two different models, both called v1

Symptom: Priya fixes a few labels and trains again with the same name, v1. Next week, Grace asks why a refund request went to product. The log says "model_version": "v1", but two different models had that name. Nobody can tell which one decided.

In this project, the second training stops with an error instead:

$ route train --version v1
models/v1 exists: a version is never overwritten. Choose a new version name.

Cause: a version name is only useful if it always means the same model. If you write over a version, its name stops pointing to one model, and the log loses its meaning.

Fix: give every trained model a new version name, such as v2. Keep the old folder. The label version adds a second check: new labels give a new fingerprint, even if someone reuses a name in another system.

Consequence in AI work: without stable versions, you cannot explain a past decision, compare two models fairly, or go back to the model that worked. The next two lessons need all three.

Check your understanding

Name the four things that the routing service does that a notebook call does not, and the problem that each one solves.

Show the answer
  • It loads the model once, at start-up. On the test laptop, loading took about 27 ms each time. Routing one message took about 0.5 ms. Loading for each message would make every answer much slower.
  • It checks the input. A bad request gets HTTP 422 with the reason, and no decision. Without the check, an empty text went to product with confidence 0.3036.
  • It names the model version and the label version in every answer. A wrong route can be traced back to the model and the labels that made it.
  • It logs decisions without the text, and sends review decisions to Chloé’s queue. The log answers “what happened?” without copying private messages.

Check your understanding

Predict: the model sends a message to needs_person with confidence 0.99. Which zone does the service choose, and why? What would happen if the threshold person_below were 0.40 and a delivery message had confidence 0.35?

Show the answer

The zone is person, whatever the confidence. A person reads every message that the model sends to needs_person, because those messages are safety problems, legal threats or escalated complaints. The confidence does not change that.

With person_below at 0.40, a delivery message with confidence 0.35 would also go to the person zone, with the route needs_person. The model is too unsure to suggest a team. With the current card (person_below 0.00), the same message goes to review with the suggested route delivery.

Check your understanding

Practical check: start the service, send a message whose text has only spaces, and say what you get back and what the service writes to its two files.

Show the answer

The answer is HTTP 422 with a body like {"detail":[{"type":"value_error","loc":["body","text"],"msg":"Value error, text is empty","input":" ",…}]}. The Message class refuses a text that is only spaces. FastAPI never calls the routing code, so the service writes nothing to queue/decisions.jsonl and nothing to queue/review_queue.jsonl. Only uvicorn’s access log in the terminal shows the request, with status 422.

Recap

  • A service runs all the time and answers each message over HTTP. Ours has POST /route and GET /health, on port 8036 of this computer only.
  • It loads the model once. On the test laptop, a load took about 27 ms and one message took 0.49 ms.
  • It checks the input with FastAPI and Pydantic: a bad ID, an empty text or an unknown channel gets HTTP 422 and no decision.
  • Every answer has the route, the zone, the confidence, the model version and the label version. A version is never written over.
  • It logs every decision without the text in queue/decisions.jsonl, and sends review decisions to queue/review_queue.jsonl for Chloé.

The service now answers all day. But the inbox does not stay the same: the Grow Hub message already went to review. Next, you replay Omar’s launch month through the service’s model and see what changes.

Next: Drift and new topics