Skip to content

Connect the API

  • Lesson 8
  • 40 min
  • Module 3: Build the prediction service

What you will be able to do

Put the model behind four HTTP routes. They check every ticket against the contract, answer errors in one stable shape, and name the model and threshold in every answer.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

At a pharmacy, you hand in a prescription form. If a field is missing, the pharmacist gives it back and points to the field. If the form is complete, you get the medicine with a printed label: the name, the dose, the date and the pharmacy. Months later, anybody can read the label and know what you got, and from whom.

A prediction API needs the same two things. It rejects a ticket that breaks the contract, and it names the field. And every answer carries its label: which model, which threshold, which request.

Suppose the API answered only {"score": 0.35}. The help desk must decide: flag the ticket or not? With version 1.0.0, the threshold is 0.19, so 0.35 is flagged. In Module 5, version 1.1.0 uses a threshold of 0.20. Tomás’s candidate 1.1.0-rc1 moves every score up. With the old threshold 0.19, it flags 86% of July’s tickets instead of 20%. The same number means different things for different models.

A bare score has a second problem. When Grace asks in August “why was ticket T-120001 flagged?”, nobody can say which model scored it. And a client that receives three error shapes needs three pieces of code to read them, as you saw in the API course.

So the API follows three rules:

  1. Check before scoring. A ticket that breaks the contract gets 422 and the names of the wrong fields. The model never sees it.
  2. One error shape. Every error, from 401 to 500, has the same body: code, message, request_id and fields. A client reads error.code, never the text.
  3. Every score names its source. The answer has the score, the flag, the threshold and the model_version, plus warnings for values the model never saw. The service also writes one log line per score, with the request ID.

The diagram follows one request through these rules:

POST /v1/score gets a request ID. Is the API key right? No: 401 unauthorized. Yes: does the ticket match the contract? No: 422 invalid_ticket. Yes: score_tickets with the bundle in memory, which gives a 200 answer with score, flag, threshold, model_version and warnings, and writes one prediction log line. If scoring raises an error: 500 internal_error. The 401, 422 and 500 answers all have one error shape: code, message, request_id, fields.
One request through the service. Every path ends in the 200 answer or in the one error shape.
Text description of the diagram

A request to POST /v1/score gets a request ID first. The API key check comes next: a missing or wrong key ends in 401 unauthorized. Then the contract check: a ticket that does not match ends in 422 invalid_ticket. A ticket that passes goes to score_tickets(), which uses the bundle in memory. It gives the 200 answer with score, flag, threshold, model_version and warnings, and writes one prediction log line. An error during scoring ends in 500 internal_error. The three error answers have one shape: code, message, request_id and fields.

These are the API course’s patterns from the ticket-api, applied to a model. The new parts are the threshold and the model version in each answer, the warnings and the prediction log.

Make escalation/errors.py. It has the shape and three handlers, as in the API course:

escalation/errors.py
"""One error shape for every failure, as in the ticket API:
{"error": {"code": ..., "message": ..., "request_id": ..., "fields": [...]}}"""
import logging
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from starlette.exceptions import HTTPException
logger = logging.getLogger("escalation")
CODES = {401: "unauthorized", 404: "not_found", 405: "method_not_allowed", 429: "too_many_jobs"}
class ErrorDetail(BaseModel):
code: str = Field(description="A stable code that a program can check.")
message: str = Field(description="A short explanation for a person.")
request_id: str | None = None
fields: list[str] = []
class ErrorResponse(BaseModel):
error: ErrorDetail
def field_name(error: dict) -> str:
if error["type"] == "json_invalid":
return "body"
return ".".join(str(part) for part in error["loc"][1:]) or "body"
def error_response(request: Request, status: int, code: str, message: str, fields=()) -> JSONResponse:
detail = ErrorDetail(
code=code,
message=message,
request_id=getattr(request.state, "request_id", None),
fields=list(fields),
)
return JSONResponse(status_code=status, content=ErrorResponse(error=detail).model_dump())
def add_error_handlers(app: FastAPI) -> None:
@app.exception_handler(RequestValidationError)
async def invalid_request(request: Request, exc: RequestValidationError):
fields = sorted({field_name(e) for e in exc.errors()})
return error_response(
request, 422, "invalid_ticket", "The ticket does not match the model's contract.", fields
)
@app.exception_handler(HTTPException)
async def http_error(request: Request, exc: HTTPException):
code = CODES.get(exc.status_code, "http_error")
return error_response(request, exc.status_code, code, str(exc.detail))
@app.exception_handler(Exception)
async def unexpected(request: Request, exc: Exception):
logger.exception("unexpected error id=%s", getattr(request.state, "request_id", None))
return error_response(request, 500, "internal_error", "Something went wrong on the server.")

invalid_request turns every contract error into 422 invalid_ticket, with the sorted names of the wrong fields. A body that is not JSON at all gives the field body. unexpected logs the full error for you, but sends the client only a short message and the request ID: the inside of the server stays inside.

Make escalation/security.py. It is the API course’s key check:

escalation/security.py
import secrets
from fastapi import Request
from fastapi.security import APIKeyHeader
from starlette.exceptions import HTTPException
api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
def require_api_key(request: Request) -> None:
"""Allow the request only with the right X-API-Key, if a key is set."""
expected = request.app.state.settings.api_key
if expected is None:
return
given = request.headers.get("X-API-Key") or ""
if not secrets.compare_digest(given.encode(), expected.encode()):
raise HTTPException(401, "A valid X-API-Key header is required.")

Without API_KEY, every request passes, which is convenient on your computer. With API_KEY set, a request needs the header X-API-Key with that value. secrets.compare_digest compares in constant time, so the response time does not tell an attacker how many characters were right. Set REQUIRE_API_KEY=true in production: then settings.py refuses to start without a key.

Make escalation/routes.py. The first part has the router and the response models:

escalation/routes.py
import json
import logging
from typing import Annotated
from fastapi import APIRouter, Body, Depends, Request, Response
from pydantic import BaseModel, Field
from escalation.contract import TicketIn
from escalation.errors import ErrorResponse, error_response
from escalation.jobs import QueueFull
from escalation.scoring import score_tickets
from escalation.security import require_api_key
predictions = logging.getLogger("escalation.predictions")
router = APIRouter(prefix="/v1", dependencies=[Depends(require_api_key)])
ERRORS = {
401: {"model": ErrorResponse, "description": "The API key is missing or wrong."},
422: {"model": ErrorResponse, "description": "The ticket does not match the contract."},
}
class ScoreOut(BaseModel):
ticket_id: str
score: float = Field(ge=0, le=1, description="0 to 1. Higher means a higher risk of escalation.")
flag: bool = Field(description="True: send the ticket to the senior team.")
threshold: float = Field(description="The score at and above which a ticket is flagged.")
model_version: str
warnings: list[str] = Field(description="Values the model never saw. The score is less certain.")
class ModelInfo(BaseModel):
model_version: str
bundle_sha256: str
threshold: float
trained_at: str
features: list[str]
class JobIn(BaseModel):
tickets: list[TicketIn] = Field(min_length=1)
class JobStatus(BaseModel):
job_id: str
status: str = Field(description="queued, running, done or failed")
tickets: int
created_at: str
finished_at: str | None = None
seconds: float | None = None
model_version: str | None = None
results: list[dict] | None = None
error: str | None = None

dependencies=[Depends(require_api_key)] puts the key check on every route under /v1. /health and /ready are in main.py, outside the router, so a platform can call them without a key. ScoreOut is the label from the pharmacy: each field has a description, which becomes the API documentation.

The second part is the two model routes:

escalation/routes.py (continued)
@router.post("/score", tags=["scoring"], summary="Score one new ticket", responses=ERRORS)
def score(ticket: TicketIn, request: Request, response: Response) -> ScoreOut:
state = request.app.state
[result], timings = score_tickets(state.bundle, [ticket])
response.headers["Server-Timing"] = timings.header()
record = {
"request_id": request.state.request_id,
"ticket_id": ticket.ticket_id,
"model_version": state.bundle.version,
"score": result.score,
"flag": result.flag,
}
predictions.info(json.dumps(record))
return ScoreOut(
ticket_id=ticket.ticket_id,
score=result.score,
flag=result.flag,
threshold=state.bundle.threshold,
model_version=state.bundle.version,
warnings=result.warnings,
)
@router.get("/model", tags=["model"], summary="Which model answers, and how to check it")
def model(request: Request) -> ModelInfo:
bundle = request.app.state.bundle
return ModelInfo(
model_version=bundle.version,
bundle_sha256=bundle.digest,
threshold=bundle.threshold,
trained_at=bundle.metadata["trained_at"],
features=bundle.contract["features"],
)

The parameter ticket: TicketIn does the contract check: FastAPI validates the body with the class from Module 1 before the function runs. The route then calls the same score_tickets() as the batch job. So the API and the batch job always score a ticket in the same way. The Server-Timing header carries the three timed steps, for Module 4’s measurements.

The route writes one log line per score, as JSON, to the logger escalation.predictions. Why? A score is a prediction about the next 72 hours. Only later does Larkfield know if the ticket was escalated. In Module 5, Monitor model health joins these lines with the real outcomes by ticket_id, to measure precision and recall on live tickets. Without the log, the evidence is gone. The line holds no customer data: only IDs, the version and the result.

The third part wires the job runner of Module 2 to two routes:

escalation/routes.py (continued)
def status_of(job) -> JobStatus:
return JobStatus(
job_id=job.id,
status=job.status,
tickets=len(job.results) if job.results is not None else len(job.tickets),
created_at=job.created_at,
finished_at=job.finished_at,
seconds=job.seconds,
model_version=job.model_version,
results=job.results if job.status == "done" else None,
error=job.error,
)
@router.post(
"/jobs",
tags=["jobs"],
status_code=202,
summary="Score many tickets later",
responses={**ERRORS, 429: {"model": ErrorResponse, "description": "Too many jobs are waiting."}},
)
def submit_job(job_in: Annotated[JobIn, Body()], request: Request, response: Response) -> JobStatus:
limit = request.app.state.settings.max_job_tickets
if len(job_in.tickets) > limit:
return error_response(request, 413, "job_too_large", f"A job can have at most {limit} tickets.")
try:
job = request.app.state.jobs.submit(job_in.tickets)
except QueueFull:
return error_response(request, 429, "too_many_jobs", "Too many jobs are waiting. Try again later.")
response.headers["Location"] = f"/v1/jobs/{job.id}"
response.headers["Retry-After"] = "2"
return status_of(job)
@router.get("/jobs/{job_id}", tags=["jobs"], summary="The status of a job, and its scores when done")
def get_job(job_id: str, request: Request) -> JobStatus:
job = request.app.state.jobs.jobs.get(job_id)
if job is None:
return error_response(request, 404, "job_not_found", "There is no job with this ID here.")
return status_of(job)

Each ticket in a job passes the same contract check, because JobIn holds a list of TicketIn. The rest is Module 2’s design: 202 Accepted, the Location of the status, Retry-After, and 429 when four jobs already wait.

Start the service as in Load and initialize, without API_KEY. In a second terminal, score the example ticket from contract.py. The option -i shows the headers too:

Terminal 2
curl -i http://127.0.0.1:8000/v1/score -H "Content-Type: application/json" -d '{"ticket_id":"T-120001","channel":"email","team":"payment","segment":"home","region":"west","priority":1,"order_value":89.5,"word_count":112,"customer_tenure_days":400,"prior_tickets_90d":2,"created_hour":10,"prior_escalations_90d":1}'

The output:

HTTP/1.1 200 OK
date: Thu, 08 Oct 2026 14:35:05 GMT
server: uvicorn
content-length: 119
content-type: application/json
server-timing: frame;dur=1.07, prepare;dur=1.81, model;dur=0.07
x-request-id: 53ffd3d9d351
{"ticket_id":"T-120001","score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[]}

The score 0.351 is above the threshold 0.19, so flag is true. In the service’s terminal, the prediction log line has the same request ID:

INFO escalation.predictions: {"request_id": "53ffd3d9d351", "ticket_id": "T-120001", "model_version": "1.0.0", "score": 0.35065699401024064, "flag": true}
INFO escalation: POST /v1/score 200 4.9ms id=53ffd3d9d351

Now break the contract twice in one ticket: priority "high" and created_hour 24. Change those two values in the command and send it again. The output:

HTTP/1.1 422 Unprocessable Content
...
x-request-id: e53bd384c649
{"error":{"code":"invalid_ticket","message":"The ticket does not match the model's contract.","request_id":"e53bd384c649","fields":["created_hour","priority"]}}

Both wrong fields are named, and the model never ran. Next, two cases from Module 1. Remove the key "order_value":89.5, completely. Then put it back as "order_value":null and set "channel":"social". The two outputs:

{"error":{"code":"invalid_ticket","message":"The ticket does not match the model's contract.","request_id":"179de6f59408","fields":["order_value"]}}
{"ticket_id":"T-120001","score":0.4310802648610386,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":["channel: 'social' is a value the model never saw"]}

A missing key is an error; a null value is a known missing value, and the pipeline’s imputer fills it. The new channel is scored, with a warning in the answer.

Ask the service which model answers:

Terminal 2
curl http://127.0.0.1:8000/v1/model

The output:

{"model_version":"1.0.0","bundle_sha256":"8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74","threshold":0.19,"trained_at":"2026-10-08 13:53:54 UTC","features":["channel","team","segment","region","priority","order_value","word_count","customer_tenure_days","prior_tickets_90d","created_hour","prior_escalations_90d"]}

bundle_sha256 is the full digest, so anybody can compare it with the release record. Your digest and trained_at are your own.

Last, a job with the same ticket three times. Send {"tickets":[…]} with the ticket three times to /v1/jobs, then call the URL in Location. The outputs (headers shortened):

HTTP/1.1 202 Accepted
location: /v1/jobs/b28f86e15848
retry-after: 2
...
{"job_id":"b28f86e15848","status":"queued","tickets":3,"created_at":"2026-10-08T14:35:05+00:00","finished_at":null,"seconds":null,"model_version":null,"results":null,"error":null}
{"job_id":"b28f86e15848","status":"done","tickets":3,"created_at":"2026-10-08T14:35:05+00:00","finished_at":"2026-10-08T14:35:05+00:00","seconds":0.003,"model_version":"1.0.0","results":[{"ticket_id":"T-120001","score":0.35065699401024064,"flag":true},...],"error":null}

The job’s scores equal the online score, 0.35065699401024064, because both paths call score_tickets() with the same bundle.

Stop the service. Start it again with a key, then send the example ticket without and with the header:

Terminal
API_KEY=change-me-1234 fastapi run
Terminal 2
curl -i http://127.0.0.1:8000/v1/score -H "Content-Type: application/json" -d '…the example ticket…'
curl http://127.0.0.1:8000/v1/score -H "Content-Type: application/json" -H "X-API-Key: change-me-1234" -d '…the example ticket…'

On Windows, set $env:API_KEY = "change-me-1234" before fastapi run, and add -Headers @{"X-API-Key"="change-me-1234"} to Invoke-WebRequest. The output (headers shortened):

HTTP/1.1 401 Unauthorized
...
{"error":{"code":"unauthorized","message":"A valid X-API-Key header is required.","request_id":"68782a77b2f0","fields":[]}}
{"ticket_id":"T-120001","score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[]}

/health still answers {"status":"ok"} without a key. In the Docker course, the key came from a secret file with API_KEY_FILE, and settings.py reads that too. Stop the service, and start it again without API_KEY for the next part.

FastAPI documents the routes for you at /docs. Open http://127.0.0.1:8000/docs:

Swagger UI for Escalation Service, version 1.0.0, OAS 3.1, with the description Scores new support tickets for the risk of escalation within 72 hours. Four sections: scoring with POST /v1/score Score one new ticket; model with GET /v1/model Which model answers, and how to check it; jobs with POST /v1/jobs Score many tickets later and GET /v1/jobs/{job_id} The status of a job, and its scores when done; health with GET /health and GET /ready.
The service's routes in Swagger UI, grouped by their tags. The summaries come from the route decorators.

Select POST /v1/score to see the example ticket from contract.py, the ScoreOut fields with their descriptions, and the 401 and 422 answers with ErrorResponse. Select Try it out and Execute to score the example. This page has no Authorize button: require_api_key reads the header itself, so the OpenAPI document does not describe the key. Use curl when a key is set. In production, SHOW_DOCS=false turns the page off.

The API tests need a client: a test client that sends requests to the app in memory, with no server and no port. Add it to tests/conftest.py. First the import, after import pytest:

tests/conftest.py
from fastapi.testclient import TestClient

Then the fixture, after the other fixtures at the end of the file:

tests/conftest.py (continued)
@pytest.fixture
def client():
from escalation.main import create_app
with TestClient(create_app()) as test_client:
yield test_client

create_app is imported inside the fixture, so it runs after the fixture settings_env (from Batch inference) has set MODEL_SHA256 for the test bundle. The with block runs the lifespan, so each test gets the real bundle, loaded and checked as in the server.

Now make tests/test_api.py. Each test proves one rule of this lesson, with the client and ticket fixtures:

tests/test_api.py
import time
def test_score_one_ticket(client, ticket):
response = client.post("/v1/score", json=ticket)
assert response.status_code == 200
body = response.json()
assert body["model_version"] == "1.0.0"
assert 0 <= body["score"] <= 1
assert body["flag"] == (body["score"] >= body["threshold"])
assert "model;dur=" in response.headers["Server-Timing"]
def test_a_contract_error_has_the_error_shape(client, ticket):
response = client.post("/v1/score", json={**ticket, "priority": "high", "created_hour": 24})
assert response.status_code == 422
error = response.json()["error"]
assert error["code"] == "invalid_ticket"
assert error["fields"] == ["created_hour", "priority"]
def test_an_unknown_category_is_scored_with_a_warning(client, ticket):
body = client.post("/v1/score", json={**ticket, "channel": "social"}).json()
assert body["warnings"] == ["channel: 'social' is a value the model never saw"]
def test_the_model_endpoint_names_the_bundle(client):
body = client.get("/v1/model").json()
assert body["model_version"] == "1.0.0"
assert len(body["bundle_sha256"]) == 64
def test_ready_names_the_model(client):
assert client.get("/ready").json() == {"status": "ready", "model_version": "1.0.0"}
def test_a_job_is_accepted_then_done(client, ticket):
response = client.post("/v1/jobs", json={"tickets": [ticket] * 3})
assert response.status_code == 202
location = response.headers["Location"]
for _ in range(50):
body = client.get(location).json()
if body["status"] == "done":
break
time.sleep(0.02)
assert body["status"] == "done"
assert len(body["results"]) == 3
def test_an_unknown_job_is_404(client):
assert client.get("/v1/jobs/nothere").json()["error"]["code"] == "job_not_found"
def test_a_key_is_required_when_set(monkeypatch, ticket):
from fastapi.testclient import TestClient
from escalation.main import create_app
monkeypatch.setenv("API_KEY", "test-key")
with TestClient(create_app()) as client:
assert client.post("/v1/score", json=ticket).status_code == 401
assert client.post("/v1/score", json=ticket, headers={"X-API-Key": "test-key"}).status_code == 200

Run them:

Terminal
python -m pytest tests/test_api.py -v

The output (shortened):

tests/test_api.py::test_score_one_ticket PASSED [ 12%]
tests/test_api.py::test_a_contract_error_has_the_error_shape PASSED [ 25%]
...
tests/test_api.py::test_a_key_is_required_when_set PASSED [100%]
============================== 8 passed in 0.95s ===============================

python -m pytest runs the whole suite: 28 passed.

Guided practice: wrap the classifier in the API

Section titled “Guided practice: wrap the classifier in the API”

This is the outline’s practice for the module: put the classifier behind the API with the same patterns as the API course’s ticket-api.

  1. Write escalation/errors.py, escalation/security.py and escalation/routes.py as above. If you copied them from the snapshot in the last lesson, compare them with this page.
  2. Add the client fixture to tests/conftest.py, and write tests/test_api.py.
  3. Run python -m pytest. The last line says 28 passed.
  4. Start the service with fastapi run.
  5. Score the example ticket with curl -i. Check the four fields score, flag, threshold and model_version, and the header x-request-id.
  6. Find the prediction log line with the same request ID in the service’s terminal.
  7. Call /v1/model. Check that bundle_sha256 starts with the 12 characters of the start-up log line.
Show the expected result of steps 5 and 6

A 200 answer with "score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[], and a log line INFO escalation.predictions: {"request_id": "…", "ticket_id": "T-120001", …} with the same ID as the x-request-id header. The ID itself is random.

The contract has extra="forbid". Send the example ticket with one more field, "colour":"red". Predict first: status code, code and fields?

Show the answer
{"error":{"code":"invalid_ticket","message":"The ticket does not match the model's contract.","request_id":"5c935dcc4129","fields":["colour"]}}

The status is 422. An extra field is rejected and named, because a client that sends colour may believe that the model uses it. The model does not, so the API says so at once.

Common mistake

Every request gets 422 with the field body

Symptom: a script on Windows sends tickets, and every answer is:

{"error":{"code":"invalid_ticket","message":"The ticket does not match the model's contract.","request_id":"c2c4f45d10bf","fields":["body"]}}

The service’s log shows POST /v1/score 422 for each request, and no prediction line.

Cause: the body is not valid JSON, so the contract check never reads a single field. Here, the shell changed the quotes: the body arrived as {'ticket_id':'T-120001'}, with single quotes, which JSON does not allow. field_name() reports a JSON error as the field body.

Fix: send the body exactly as JSON. In PowerShell, keep the JSON in a variable with single quotes outside and double quotes inside, as in the worked example, or let ConvertTo-Json make it. Check one request with -i before you send many.

For a real AI service: a 422 is cheap and safe: no model ran, and the client learns what to fix. Count 422 answers by fields. A sudden rise of body errors means a broken client; a rise of one field means that a client changed what it sends. Module 5 counts these signals.

Check your understanding

Why does every answer of POST /v1/score include the threshold and the model version, not only the score?

Show the answer

A score means nothing without the model and threshold that go with it. Version 1.0.0 flags at 0.19 and version 1.1.0 at 0.20. The faulty candidate rc1 moves all scores up, so 0.19 flags 86% of tickets. With the version and threshold in the answer and the log, a client decides correctly. Later, anybody can say which model scored a ticket.

Check your understanding

The service writes one JSON log line per score with the request ID, ticket ID, model version, score and flag. Why, and what must it not contain?

Show the answer

The real outcome (escalated or not) arrives days later. The log lines let the team join each prediction with its outcome by ticket ID. Then they can measure precision and recall on live tickets, per model version (Module 5). The request ID links the line to the client’s answer. The line must not contain customer data, such as the text of the ticket or a name: only IDs and the result.

Check your understanding

A client sends a ticket without the key order_value, and another sends order_value: null. What does each get, and why the difference?

Show the answer

Without the key: 422 invalid_ticket with fields: ["order_value"], because the contract requires the key. With null: 200 and a score. Null is the agreed way to say “there is no order”, and the pipeline’s imputer fills it as in training. A missing key is more likely a broken client than a real missing value.

Recap

  • POST /v1/score checks the ticket with the contract before the model runs, and answers with score, flag, threshold, model_version and warnings.
  • GET /v1/model names the version, the full digest, the threshold and the features. The job routes wire Module 2’s JobRunner.
  • Every error has one shape: code, message, request_id, fields. Every response has X-Request-ID.
  • One JSON log line per score keeps the evidence for monitoring when the outcomes arrive.
  • The API key protects /v1; /health and /ready stay open for the platform.
  • tests/test_api.py proves each rule: 28 passed.

Next, you prove that the served scores are the scores that training gave: parity.

Next: Check parity