Online inference

What you will be able to do

Give the agent the escalation score at the moment the ticket opens, not the next morning. Agree how long the help desk may wait, and decide what happens when the score does not come in time.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• The batch job, score_tickets and the morning delay, from Batch inference.
• Request validation and stable error responses, from Validation and error responses in Build and Deploy Your First API.
• A batch endpoint with a body limit, from Serve a house-price model: Part 5 in the API course. This lesson builds on it.

What you need on your computer

• escalation/settings.py, scoring.py and batch.py, and data/july_tickets.csv, from Batch inference: Guided practice.
• A terminal in ~/projects/escalation-service with .venv active and MODEL_SHA256 set, as in step 3 of that practice.
• If your project is missing or broken, copy the snapshot end-of-m01 and do the guided practice of Batch inference. Use a snapshot has the steps.

This lesson changes no project file. You measure the scoring code in a Python session.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

The card terminal at the till

You pay with a card in a shop. The terminal sends the payment to the bank, and you wait at the till until it says “approved”. A few seconds are normal. After half a minute, the queue behind you gets angry. And if the bank does not answer at all, the shop has a plan: it asks for another card, or for cash.

That is the other way to use a model. The question comes with a person waiting, the answer must come back in that one exchange, and there must be a plan for “no answer”.

The problem: a flag 23 hours late

The batch job from the last lesson flags a ticket the morning after it arrives. In July, tickets arrived at every hour of the day, most of them from 07:00 to 22:00. A ticket that arrives at 09:05 waits until 08:00 the next day: about 23 hours. Its customer can escalate within 72 hours, so a third of that time is gone before anyone looks.

Grace asks for something else: “Show the flag when the agent opens the ticket. Then a senior agent can take it before the customer writes again.” Now the score must be ready while a person waits. Every millisecond of the scoring is in the agent’s wait.

The idea: online serving

Online serving means that the model answers one request at a time, at the moment the request arrives, while the caller waits. The call is synchronous: the help desk sends the ticket, waits, and continues only when the answer arrives. The time from the request to the answer is the latency.

Online: one ticket, one request, one answer, while the agent waits. The flag reaches the senior team at 09:05, not the next morning.

Text description of the diagram

The same three actors as in the batch diagram: the help desk system, escalation-service (online), and the senior team. At 09:05, an agent opens the new ticket T-120001. The help desk system sends POST /v1/score with that one ticket. The service checks the contract, then runs its three steps: frame, prepare and model. It answers 200 with the score 0.35 and the flag true, in about 3 ms. At 09:05, the ticket joins the senior team’s review queue. A note at the bottom: if no answer comes in 50 ms, the page opens without a score, and the nightly batch scores the ticket.

Compare it with the batch picture. The actors are the same, and the model and the contract are the same. Only the moment changes: the score arrives in milliseconds, so the flag reaches the senior team the same minute. The HTTP route POST /v1/score comes in Module 3. In this lesson you measure the scoring work behind it.

The latency budget

Omar asks the question that decides the design: “How long does the help desk wait?”

Grace answers for her agents: “The ticket page must open in 300 ms. Slower than that, and agents notice.” But the page does more than ask for a score. It loads the ticket and the customer’s history, and it asks ticket-api for the category. The help desk team needs most of the 300 ms for that work.

So they agree on a latency budget: the part of the total time that one step may use. escalation-service gets 50 ms of the page’s 300 ms. The help desk waits at most 50 ms for the score. If no answer comes, the page opens without a score, and the nightly batch job scores the ticket the next morning. The page is never slow because of the model.

The budget is a promise about the slow requests, not the usual one. So it is written as a p95: 95 of every 100 scores must arrive within 50 ms, as the help desk sees them, network included. (Benchmark the service teaches percentiles in full.) This course keeps this budget: p95 under 50 ms. Module 4 measures the service against it.

How much of the budget does the model use? Predict first, then try the explorer. The first value is measured; the others are simulated, as if the model were slower, for example a large text model.

Explore

One ticket against the help desk's 50 ms

Time per ticket

1.76 ms (measured)

Tickets in the request

1 ticket

One request takes

1.76 ms

Result

Online fits

1 ticket × 1.76 ms per ticket (measured) = 1.76 ms. An agent opens one ticket, and the score comes well inside the help desk's 50 ms. Over HTTP, the whole request took about 3 to 4 ms (Module 4 measures it). Online serving fits.

• Fits the help desk's 50 ms
• Over 50 ms, under 30 s
• Over 30 s: use a job
• Light, dashed: simulated time

One request with one ticket. Move the time per ticket from the measured 1.76 ms to simulated slower models, and compare the request with the 50 ms budget.

Text description of the explorer

The explorer shows one request with one ticket on a time axis from 1 ms to 1 hour. Each step of the axis is 10 times the one before. Two dashed lines mark the limits: the help desk waits 50 ms, and the gateway stops any request at 30 s. One control sets the time per ticket: 1.76 ms (measured), then 5, 10, 20, 25, 40, 100 and 400 ms (simulated). Simulated times are drawn with a light fill and a dashed outline.

At the measured 1.76 ms, the request takes 1.76 ms, far inside 50 ms. Over HTTP, the whole request took about 3 to 4 ms. Up to 25 ms per ticket, the request fits easily. At 40 ms it still fits, but it uses most of the budget. At 100 ms and 400 ms, one ticket takes longer than 50 ms, so the page opens without a score every time. The answer to the question: about 28 times slower (50 ÷ 1.76), between the 40 ms and 100 ms steps.

The model has room: about 28 times. That room is not free to spend. The 50 ms must also hold the network, and the slow moments when several agents open tickets at once. The failure case at the end of this lesson shows such a moment.

Where the time of one request goes

Tomás has an idea: “If each request costs time, send all the day’s tickets in one request.” Before you answer him, measure what one call of score_tickets costs. Start a Python session in the project folder (python), and type:

Python session, in ~/projects/escalation-service

import time
import pandas as pd
from escalation.settings import load_settings
from escalation.bundle import load_bundle
from escalation.batch import COLUMNS, check_rows
from escalation.scoring import score_tickets
settings = load_settings()
bundle = load_bundle(settings.model_bundle, settings.model_sha256)
rows = pd.read_csv("data/july_tickets.csv", usecols=COLUMNS, dtype=str, keep_default_na=False, na_values=[""])
tickets, bad = check_rows(rows, first_row=2)
len(tickets), len(bad)

The session loads the bundle once, with every check, and reuses the batch job’s check_rows to turn the 1,400 July rows into checked tickets. The output:

(1400, 0)

Now score the tickets in two ways. First, one call per ticket, as an online service does. Then one call for all of them, as the batch job does. The first line is a warm-up call, so that the first measured call is not slower than the rest:

Python session (continued)

results, timings = score_tickets(bundle, tickets[:1])
start = time.perf_counter()
for ticket in tickets:
    results, timings = score_tickets(bundle, [ticket])

one_by_one = time.perf_counter() - start
timings.header()
start = time.perf_counter()
results, timings = score_tickets(bundle, tickets)
all_at_once = time.perf_counter() - start
timings.header()
print(f"one by one: {one_by_one:.2f} s, all at once: {all_at_once * 1000:.1f} ms, {one_by_one / all_at_once:.0f} times faster")

Press Enter on the empty line after the for loop, so that the loop runs. The outputs, in order (the test computer, without a container):

'frame;dur=0.46, prepare;dur=1.09, model;dur=0.04'
'frame;dur=2.98, prepare;dur=1.77, model;dur=0.07'
one by one: 2.44 s, all at once: 5.7 ms, 429 times faster

The first line is the last single ticket: 1.59 ms of work, and the model is only 0.04 ms of it. Building the table (frame) and the preprocessing (prepare) cost 39 times more than the model. The second line is all 1,400 tickets in one call: only about three times the work of one ticket. Most of the cost of a call is a fixed cost that does not grow with the number of tickets. timings.header() is the Server-Timing header that the service sends in Module 3.

Amira measured the same in a container with 1 CPU, the size that Module 4 uses:

The same 1,400 tickets, the same model. One call per ticket took 2.46 s (1.76 ms per ticket). One call for all took 7 ms: about 350 times less. The second bar is too short to see.

Text description of the chart

A column chart of two measurements in a container with 1 CPU, scoring the 1,400 July tickets with score_tickets. One call per ticket: 2,460 ms, which is 1.76 ms per ticket. One call for all the tickets: 7 ms, which is 0.005 ms per ticket. The second column is so short that only its label, 7 ms, shows above the axis. The ratio is about 350. The steps of one call for one ticket were frame 0.64, prepare 1.23 and model 0.05 ms. For all 1,400 tickets, they were frame 2.80, prepare 1.70 and model 0.07 ms.

On your computer the numbers are different, and they change from run to run. The shape does not change: one call per ticket repeats the fixed cost 1,400 times.

So Tomás is half right. One call for many tickets has a much higher throughput, and that is exactly why the batch job scores 500 tickets per call. But online serving has one ticket per request, because the agent opens one ticket. Online pays the fixed cost on every request, and it buys a short wait with it. At 1.76 ms, that price fits easily in 50 ms.

Availability and request limits

A batch job must work once a night. An online service must answer whenever a ticket arrives, and in July tickets arrived at every hour of the day. The share of time in which a service answers correctly is its availability. For example, 99.9% availability allows about 43 minutes without answers in a 30-day month.

Two things keep this affordable at Larkfield. First, the help desk has a plan for “no answer”: the page opens without a score, and the nightly batch catches up. So an hour without the service delays some flags, but it never blocks an agent. Second, the service must say clearly when it is not ready, so the platform sends no requests to it. That is the /ready endpoint of Module 3.

Online serving also needs request limits: rules about what one request may ask for. They protect the latency of everyone else. If one caller sends 50,000 tickets in a request, every agent behind it waits.

• One ticket per request for POST /v1/score. The body is one TicketIn, so the work of one request is always small.
• A body size limit. The service refuses a body larger than 2,000,000 bytes, before it reads it. The API course’s house-price API used the same idea (Part 5).
• Timeouts on both sides. The help desk waits at most 50 ms. Larkfield’s gateway, the proxy in front of every service, stops any request after 30 s.

The 30 s limit matters for large work, such as a whole month of tickets. The next lesson starts there.

When online is the right choice

Online serving fits when a person or a program waits for one answer now. Examples: a flag while the ticket opens, a fraud check while a payment waits, a price on a page.

It does not fit when nobody waits, or when the work of one request is too large for the caller’s wait. Then use a batch job or a queued job.

It costs more than batch. A service runs all day, and someone watches it. It needs a latency budget, limits, a health check and a plan for “no answer”. And it pays the fixed cost of a call for every single ticket.

Guided practice: measure one call

1. Open a terminal in ~/projects/escalation-service, and activate .venv.

2. Set MODEL_SHA256 to your bundle’s digest, as in Batch inference: Guided practice, step 3.

3. Start Python with python. The prompt changes to >>>.

4. Type the first block of Where the time of one request goes. The last line shows (1400, 0).

5. Type the second block. Press Enter on the empty line after the for loop.

6. Compare your two header() lines. Check that model is the smallest of the three steps in both.

7. Read the last line. One call per ticket is a few hundred times slower than one call for all.

8. Leave Python with exit().

Show an example of the output

On the test computer, three runs of the same session gave 2.43 s, 2.84 s and 2.54 s one by one. All at once took 5.7 to 5.8 ms: 417, 499 and 435 times faster. Other programs were running on the computer, so the times moved. Your numbers can be larger or smaller. If one by one is less than 100 times slower, check that the loop calls score_tickets with [ticket], a list of one.

Your turn: 100 tickets

Before you measure, predict the time for the first 100 July tickets, scored one by one and in one call. Use the numbers of this lesson: 1.76 ms per ticket one by one, and about 7 ms for 1,400 tickets in one call. Then change both tickets in the second block to tickets[:100], and run it.

Show the answer

The prediction: one by one, 100 × 1.76 ms ≈ 180 ms. In one call, less than the 7 ms of 1,400 tickets, because most of a call is the fixed cost: about 2 ms.

On the test computer, four runs gave 199, 321, 239 and 286 ms one by one. One call took 2.2 to 5.6 ms. The one-by-one time is higher than predicted because other work was running on the computer. One call for 100 tickets costs about as much as one call for one ticket: the fixed cost again.

Common mistake

Fast alone, too slow together

Symptom: Tomás sends one request at a time. Each answer takes about 4 ms, far inside 50 ms. “It always fits,” he says. Then the benchmark of Module 4 sends 8 requests at once to 1 worker on 1 CPU. Every answer is 200, and the log shows no error:

1000 requests, concurrency 8: p50 31.6 ms, p95 72.5 ms, p99 105.7 ms, max 136.1 ms, 216.5 requests/s, 0 errors.

The p95 is 72.5 ms, above the 50 ms budget. If 8 agents opened tickets at the same moment, more than one ticket in 20 would open without a score.

Cause: one worker scores one request at a time. When 8 requests arrive together, each one waits for the others in front of it. The usual request (p50) took 31.6 ms instead of 4 ms, and the slow ones took longer.

Fix: compare the p95 with the budget at the load that the service really gets. Larkfield gets about 45 tickets a day, so 8 at once almost never happens. With one request at a time, the p95 was 6.1 ms. If the load grows, give the service more CPUs: 2 workers on 2 CPUs had a p95 of 37.3 ms with 8 at once. Benchmark the service shows how to measure it.

For a real AI service: one fast request proves little. Agree the budget as a percentile, and measure it at the real load and at a busy moment. Keep the plan for “no answer”, because one day it will happen.

Check your understanding

Grace's page has 300 ms in total. Why does escalation-service get only 50 ms of it, and what happens to a ticket whose score does not come within 50 ms?

Show the answer

The page also loads the ticket, the customer’s history and the category from ticket-api. The help desk team needs most of the 300 ms for that. So the score gets a budget of 50 ms. The help desk stops waiting after 50 ms and opens the page without a score. The nightly batch job then scores that ticket the next morning, so the flag is late, but the agent is never blocked.

Check your understanding

One call of score_tickets for one ticket took about 1.8 ms, and one call for 1,400 tickets took about 7 ms. Explain why, and what it means for online and batch scoring.

Show the answer

Most of the cost of a call is fixed. Building the table and running the preprocessing take about the same time for one ticket or for many. The model itself is almost free (0.05 ms). One call per ticket pays that fixed cost 1,400 times. So batch scoring, with many tickets per call, has a much higher throughput. Online serving pays the fixed cost on every request to give one quick answer, which is fine as long as it fits the latency budget.

Check your understanding

Tomás wants POST /v1/score to accept up to 50,000 tickets in one request. Give two reasons against it.

Show the answer

First, a request with many tickets takes much longer. The agents’ requests wait behind it, so their scores miss the 50 ms budget. Second, a large request can pass the gateway’s 30 s limit and fail completely. And a body of 50,000 tickets is far beyond the 2,000,000-byte limit. One ticket per request keeps every request small. Large work belongs in the batch job or in a queued job (next lesson).

Recap

• Online serving answers one request at a time while the caller waits: a synchronous call. Its key number is the latency, the time from the request to the answer.
• Larkfield’s latency budget: escalation-service gets 50 ms of the help desk’s 300 ms page. Without an answer in 50 ms, the page opens without a score, and the nightly batch catches up.
• One call costs about 1.76 ms per ticket, and the model is only 0.05 ms of it. 1,400 tickets in one call took 7 ms: the fixed cost of a call dominates.
• An online service must be available whenever tickets arrive. Limits keep each request small: one ticket, a body limit, and timeouts.
• A budget is a promise about the slow requests (p95 under 50 ms). 8 requests at once gave a p95 of 72.5 ms on 1 CPU; Larkfield’s real load is about 45 tickets a day.

Next, you handle work that is too large for one request: a whole month of tickets, or a much slower model.

Next: Asynchronous inference