Asynchronous inference

What you will be able to do

Score work that is too large for one request, such as a month of tickets with a slow model. The caller gets an answer at once and collects the scores later, and the service stays fast for the agents.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The batch job and its outputs, from Batch inference.
• The latency budget (p95 under 50 ms for one score), the 30 s gateway limit and the fixed cost of a call, from Online inference.

What you need on your computer

• escalation/settings.py, scoring.py and batch.py, and data/july_tickets.csv, from Batch inference: Guided practice.
• A terminal in ~/projects/escalation-service with .venv active and MODEL_SHA256 set, as in step 3 of that practice.
• If your project is missing or broken, copy the snapshot end-of-m01 and do the guided practice of Batch inference. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

A ticket at the dry cleaner

You bring a coat to the dry cleaner. Nobody cleans it while you wait at the counter. You get a paper ticket with a number, and you come back tomorrow. You show the number, and you get your coat. If the rail is full, the shop tells you at once: “Not today, come back tomorrow.” You do not wait at the counter, and the shop cleans the coats at its own speed.

The problem: work larger than the wait

Larkfield’s quality team checks one month of tickets at a time. Their tool sends the month to escalation-service and wants the scores back. For July, that is 1,400 tickets.

With today’s model, that is no problem. One call for 1,400 tickets took 7 ms. But Priya is testing a text model that reads the words of each ticket, and such models are much slower. Suppose it needs 40 ms per ticket. Then the month takes 1,400 × 40 ms = 56 s. Larkfield’s gateway stops every request after 30 s, so the request can never finish. The quality team gets an error, and the service did 30 s of work for nothing.

Where is the limit? Predict first, then try it. The first time per ticket is measured; the larger ones are simulated.

Explore

1,400 tickets in one request against the 30 s gateway

Time per ticket

1.76 ms (measured)

Tickets in the request

1,400 tickets

One request takes

2.46 s

Result

One request, a long wait

1,400 tickets × 1.76 ms per ticket (measured) = 2.46 s. One request still finishes before the 30 s gateway limit, but the caller waits with the connection open. (One call for all of them is much faster: 1,400 tickets in one call took 7 ms.)

• Fits the help desk's 50 ms
• Over 50 ms, under 30 s
• Over 30 s: use a job
• Light, dashed: simulated time

One request with the 1,400 July tickets. Move the time per ticket from the measured 1.76 ms to simulated slower models, and find where the request no longer fits in 30 s.

Text description of the explorer

The explorer shows one request with 1,400 tickets on a time axis from 1 ms to 1 hour. Each step of the axis is 10 times the one before. Dashed lines mark the help desk’s 50 ms and the gateway’s 30 s. One control sets the time per ticket: 1.76 ms (measured), then 5, 10, 20, 25, 40, 100 and 400 ms (simulated). Simulated times have a light fill and a dashed outline.

The explorer assumes that each ticket costs its full time, as a slow model’s work does. At the measured 1.76 ms, that gives 2.46 s, the measured time for 1,400 tickets one by one; one call for all 1,400 took only 7 ms. At 5 ms it takes 7 s, and at 10 ms 14 s. At 20 ms it takes 28 s: still under 30 s, but close to it. At 25 ms it takes 35 s, at 40 ms 56 s, at 100 ms 2.3 minutes and at 400 ms 9.3 minutes. From 25 ms, the gateway stops the request, and the explorer says: use a job. The exact limit is 30,000 ms ÷ 1,400 ≈ 21.4 ms per ticket.

The limit is about 21 ms per ticket: 30,000 ms ÷ 1,400. And 20 ms is already dangerous, because 28 s leaves no room for a busy moment or a slightly larger month.

The explorer multiplies the tickets by the time per ticket. For today’s model, that is too pessimistic. Most of the 1.76 ms is the fixed cost of a call, and one call for all 1,400 tickets took 7 ms. A slow text model spends its time on each ticket, so for the simulated times the multiplication is right.

Tomás asks: “Then make the gateway wait 10 minutes.” That moves the problem. The quality tool holds a connection open for 10 minutes, and a network hiccup in minute 9 loses all the work. Meanwhile, the service spends its CPU on one huge request while agents wait for their 50 ms scores.

The idea: asynchronous inference

Asynchronous inference separates the request from the work. The caller sends the work and gets a receipt at once. The service does the work later, in the background. The caller comes back to collect the result. Asynchronous means “not at the same time”: the answer does not come in the same exchange as the question.

The work that the caller hands over is a job. The jobs that wait for their turn form a queue, a line in which the first job in is the first job out. Three HTTP details carry the dry cleaner’s ticket:

• 202 Accepted: “I have your work, but it is not done.” The body has the job ID, the number on the paper ticket.
• Location: /v1/jobs/<id>: where to ask about this job.
• Retry-After: 2: how many seconds to wait before asking again. Asking again and again until the job is done is called polling.

Asynchronous: the answer to the request is a receipt, not the scores. The caller polls until the job is done. A full queue says no at once, with 429.

Text description of the diagram

The same three actors: the help desk system, escalation-service (jobs), and the senior team. The help desk system sends POST /v1/jobs with 1,400 tickets. The service answers at once: 202 Accepted, with a job ID, the status queued, a Location header and Retry-After: 2. The job waits in the queue, then the job worker scores it. In a loop, every Retry-After seconds, the help desk system sends GET /v1/jobs/{id}, and the service answers queued or running. Then a GET answers done, with the 1,400 scores, and the help desk system gives the review list to the senior team. A note at the bottom: if the queue is full, the service answers 429, try again later.

The job runner: jobs.py

Make escalation/jobs.py. It holds the jobs, the queue and one worker thread that scores the jobs one after another:

escalation/jobs.py

"""Asynchronous scoring: accept a large job now, score it later, keep the result.

    POST /v1/jobs            202 Accepted, with the job's ID and status URL
    GET  /v1/jobs/{job_id}   queued → running → done (or failed), and the scores

One worker thread takes jobs from a queue, one at a time. The queue has a
limit: when it is full, a new job gets 429, so a burst cannot use all memory.

The jobs live in this process's memory. That is enough to learn the pattern,
but a restart loses every job, and a second worker process cannot see them.
A real service keeps jobs in a queue service and results in storage (see the course *Scaling APIs and AI Workloads*).
"""

import logging
import queue
import threading
import time
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone

from escalation.contract import TicketIn
from escalation.scoring import score_tickets

logger = logging.getLogger("escalation")


class QueueFull(Exception):
    pass


@dataclass
class Job:
    id: str
    tickets: list[TicketIn]
    status: str = "queued"
    created_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat(timespec="seconds"))
    finished_at: str | None = None
    seconds: float | None = None
    model_version: str | None = None
    results: list[dict] | None = None
    error: str | None = None


class JobRunner:
    def __init__(self, get_bundle, max_queued: int, extra_ms_per_ticket: float = 0.0):
        self.get_bundle = get_bundle
        self.extra_ms = extra_ms_per_ticket
        self.queue: queue.Queue[Job] = queue.Queue(maxsize=max_queued)
        self.jobs: dict[str, Job] = {}
        self.thread = threading.Thread(target=self.work, daemon=True, name="job-worker")
        self.thread.start()

    def submit(self, tickets: list[TicketIn]) -> Job:
        job = Job(id=uuid.uuid4().hex[:12], tickets=tickets)
        try:
            self.queue.put_nowait(job)
        except queue.Full:
            raise QueueFull from None
        self.jobs[job.id] = job
        return job

    def work(self) -> None:
        while True:
            job = self.queue.get()
            job.status = "running"
            started = time.perf_counter()
            try:
                bundle = self.get_bundle()
                results, _ = score_tickets(bundle, job.tickets)
                if self.extra_ms:  # a stand-in for a slow model, such as a large text model
                    time.sleep(self.extra_ms * len(job.tickets) / 1000)
                job.results = [
                    {"ticket_id": t.ticket_id, "score": r.score, "flag": r.flag}
                    for t, r in zip(job.tickets, results)
                ]
                job.model_version = bundle.version
                job.status = "done"
            except Exception as error:  # a failed job must not stop the worker
                logger.exception("job %s failed", job.id)
                job.status, job.error = "failed", str(error)
            job.seconds = round(time.perf_counter() - started, 3)
            job.finished_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
            job.tickets = []  # the input is not needed any more
            self.queue.task_done()

Read it in three parts. submit gives the job an ID and puts it in the queue without waiting. If the queue is full, put_nowait fails at once, and submit raises QueueFull. work runs forever in its own thread. It takes the next job, scores all its tickets in one call of score_tickets, and stores the results in the job. If anything goes wrong, the job gets the status failed with the error, and the worker continues with the next job.

extra_ms_per_ticket is only for learning. It adds a sleep of that many milliseconds per ticket, to simulate a slow model. Its setting, EXTRA_MS_PER_TICKET, is 0 unless you set it.

The HTTP routes POST /v1/jobs and GET /v1/jobs/{id} come in Connect the API. Here you try the runner directly, in a Python session.

Worked example: two jobs in a Python session

Amira starts Python in the project folder and loads the bundle and the July tickets, as in the last lesson:

Python session, in ~/projects/escalation-service

import time
import pandas as pd
from escalation.settings import load_settings
from escalation.bundle import load_bundle
from escalation.batch import COLUMNS, check_rows
from escalation.jobs import JobRunner, QueueFull
settings = load_settings()
bundle = load_bundle(settings.model_bundle, settings.model_sha256)
rows = pd.read_csv("data/july_tickets.csv", usecols=COLUMNS, dtype=str, keep_default_na=False, na_values=[""])
tickets, bad = check_rows(rows, first_row=2)

A small job. She makes a runner with a queue of 4, the default of MAX_QUEUED_JOBS, and submits 3 tickets:

Python session (continued)

runner = JobRunner(lambda: bundle, max_queued=4)
job = runner.submit(tickets[:3])
job.id, job.status
time.sleep(1)
job.status, job.seconds, job.model_version
job.results[0]

The outputs:

('68e0ec7a36af', 'running')
('done', 0.004, '1.0.0')
{'ticket_id': 'T-119233', 'score': 0.037624397312637836, 'flag': False}

submit returned at once with an ID. The worker thread had already taken the job, so its status was running, not queued. One second later the job was done. It took 4 ms, and its first result is the score of T-119233 from the batch job. The same model and the same contract give the same score in every pattern.

A slow job. Now Amira simulates Priya’s slow text model with 40 ms per ticket (simulated), and submits the whole month. Then, while the big job runs, she submits five small jobs of 10 tickets:

Python session (continued)

slow = JobRunner(lambda: bundle, max_queued=4, extra_ms_per_ticket=40)
start = time.perf_counter()
big = slow.submit(tickets)
print(f"accepted in {(time.perf_counter() - start) * 1000:.1f} ms: job {big.id}, {big.status}")
time.sleep(0.5)
for n in range(1, 6):
    try:
        small = slow.submit(tickets[:10])
        print(f"job {n} of 10 tickets: accepted")
    except QueueFull:
        print(f"job {n} of 10 tickets: refused, the queue is full")

while big.status in ("queued", "running"):
    print(f"after {time.perf_counter() - start:2.0f} s: {big.status}")
    time.sleep(10)

big.status, big.seconds, len(big.results)

The outputs:

accepted in 2.9 ms: job 19733f90f765, running
job 1 of 10 tickets: accepted
job 2 of 10 tickets: accepted
job 3 of 10 tickets: accepted
job 4 of 10 tickets: accepted
job 5 of 10 tickets: refused, the queue is full
after  1 s: running
after 11 s: running
after 21 s: running
after 31 s: running
after 41 s: running
after 51 s: running
('done', 56.014, 1400)

The month was accepted in 2.9 ms, although the work took 56 s. As a single request, it would have failed at 30 s. The half-second sleep let the worker take the big job, so the queue was empty. Four small jobs then filled it, and the fifth was refused at once. The loop is polling: it asks every 10 seconds, as a client does with Retry-After. After 56 s, the job was done with 1,400 results: 1,400 × 40 ms = 56 s, as predicted.

The four small jobs waited behind the big one. Their wait is the price of one worker. Each job is safe from the gateway, but a job can wait for minutes in a busy queue.

The life of a job

Each job moves through a few states:

The states of a job. A full queue refuses new work at once. Scores and errors stay with the job, but only in the memory of one process.

Text description of the diagram

A job starts in the state queued, after POST /v1/jobs answers 202. If the queue is full, the request is refused with 429, and no job exists. The job worker takes a queued job, and it becomes running. A running job ends as done, with its scores kept in the job, or as failed, with its error kept in the job. A note on running: the jobs are in the memory of this process, so a restart loses every job.

Three limits keep the queue safe. The queue holds at most MAX_QUEUED_JOBS (4) waiting jobs, so a burst of jobs cannot use all the memory: the fifth gets 429 too_many_jobs. A job holds at most MAX_JOB_TICKETS (5,000) tickets, so one job cannot take the worker for hours: more gets 413 job_too_large. And the results stay in the memory of one process until it restarts. That is the weak point, as the failure case at the end of this lesson shows. A real service keeps results for a fixed time, for example one day, and says so in its documentation.

Choose a serving pattern

You now know three patterns. The same model and the same score_tickets sit behind each of them. What changes is who starts the work, who waits, and how the results arrive. Three questions decide it:

1. Who waits for the result, and how long can they wait? Nobody, until a known time: batch. A person, for less than a second: online. A program, for minutes: asynchronous.
2. How large is one piece of work? One case: online. Many cases that arrive together on a schedule: batch. Many cases that someone sends on demand: asynchronous.
3. What happens when it fails? The answer gives the recovery plan that the pattern needs.

The table summarises the module.

	Batch	Online	Asynchronous
Who starts the work	A scheduler, at a fixed time	The caller, one request per case	The caller, one job for many cases
How long the caller waits	Nobody waits; results are ready by a known time	The whole latency: 50 ms budget at Larkfield	Milliseconds for the receipt (202), then minutes for the result
How results arrive	Files: scores.csv, summary.json	In the answer: score, flag, model version	Polling GET /v1/jobs/{id} until done
Key number	Throughput: 1,400 tickets in 0.75 s	Latency: about 3 to 4 ms per request	Time in the queue plus the work
Limits	5% bad rows; columns of the contract	One ticket, body size, timeouts	Queue of 4 (429), 5,000 tickets (413)
Failure recovery	Run again into the same folder: same input and bundle, same output	The caller’s plan for no answer: open the page without a score; the batch catches up	Submit again after 429, failed or a lost job; keep jobs in a queue service and storage
Larkfield example	Mei’s nightly file for the morning review	The flag when an agent opens a ticket	A month of tickets for the quality team, with a slow model

When asynchronous is the right choice

Use asynchronous inference when a caller asks for work on demand, and the work is longer than the caller can wait in one request. Typical cases are large files, slow models, and work whose time you cannot predict.

Do not use it when the work fits in the latency budget, or when it only happens on a schedule. Online and batch are simpler.

It costs more than both. The service must store jobs and results, limit the queue, and delete old results, and every caller needs a polling loop. A real system also needs a queue service and storage, which are more things to run and to pay for.

Guided practice: run the jobs

1. Make escalation/jobs.py, as in The job runner: jobs.py.

2. Open a terminal in ~/projects/escalation-service, activate .venv, and set MODEL_SHA256.

3. Start Python with python, and type the first block of Worked example: two jobs in a Python session.

4. Type the small-job block. Check that the status becomes done and that job.results[0] has the score 0.037624397312637836 for T-119233.

5. Type the slow-job block. Expect the loop to run for about a minute.

6. Check the last line: 'done', about 56 seconds, and 1,400 results. Check that exactly one of the five small jobs was refused.

7. Leave Python with exit().

8. Run python -m pytest. It still says 20 passed: jobs.py gets its tests with the HTTP routes in Module 3.

Show what can be different on your computer

Your job IDs are different: they are random. The status right after submit can be queued or running, depending on whether the worker thread has already taken the job. Two small jobs can be refused instead of one. Then the worker had not taken the big job after half a second, so the big job also used a place in the queue. Your seconds for the slow job is 56 plus a little. The sleep of 56 s is simulated, and the real scoring adds a few milliseconds.

Your turn: the largest request that fits

Priya’s text model needs 40 ms per ticket (simulated). What is the largest number of tickets that one request can hold before it passes the 30 s gateway limit? Write down your answer, then check it with the explorer. Its second control sets the number of tickets.

Explore

Time per ticket and tickets per request

Time per ticket

40 ms (simulated)

Tickets in the request

1,400 tickets

One request takes

56 s

Result

Use a job

1,400 tickets × 40 ms per ticket (simulated) = 56 s. The gateway stops every request after 30 s, so this request can never finish. Send it as a job: the service answers 202 at once, and the scores are ready after about 56 s.

• Fits the help desk's 50 ms
• Over 50 ms, under 30 s
• Over 30 s: use a job
• Light, dashed: simulated time

Two controls: the time per ticket (measured 1.76 ms, or simulated) and the number of tickets in one request. The readout says whether the request fits in 50 ms, in 30 s, or needs a job.

Text description of the explorer

The explorer starts at 40 ms per ticket (simulated) with 1,400 tickets. That is 56 s, over the 30 s limit, so it says: use a job. A second control sets the tickets in one request: 1, 20, 200, 1,400 or 5,000.

At 40 ms, one ticket fits the help desk’s 50 ms. With 20 tickets, a request takes 800 ms. With 200 tickets, it takes 8 s: one request, but a long wait. With 1,400 tickets it takes 56 s, and with 5,000 tickets 3.3 minutes: both need a job. At the measured 1.76 ms, even 5,000 tickets one by one take only 8.8 s.

Show the answer

30,000 ms ÷ 40 ms = 750 tickets. In the explorer, 200 tickets take 8 s and fit, and 1,400 take 56 s and do not. But 750 tickets would take the whole 30 s, with no room for a busy moment. A safe request is much smaller, for example 200 tickets. For a month of tickets, use a job.

Common mistake

GET for a job answers 404 after a restart

Symptom: the quality team submits the July job and gets 202 with a job ID. A few minutes later, polling suddenly answers 404, although the ID is correct. Amira tried it with the finished service of Module 3. She submitted 3 tickets and got the job ID e6cd444d099c. She saw it done, restarted the service, and asked again:

HTTP/1.1 404 Not Found
...
{"error":{"code":"job_not_found","message":"There is no job with this ID here.","request_id":"ec357afe6eb3","fields":[]}}

Cause: the jobs live in the memory of one process. A restart, for example a new release or a crash, starts a new process with an empty jobs dictionary. Every queued, running and finished job is gone. With two worker processes, each process has its own jobs, so a status request can also reach the process that never had the job.

Fix: for this course, a caller that gets 404 for a job that it submitted must submit the job again. In a real system, keep the jobs in a queue service and the results in storage outside the process. Then any process can answer, and a restart loses nothing. Messaging and queue services are the topic of the course Scaling APIs and AI Workloads.

For a real AI service: an accepted job is a promise. Before you accept work that takes minutes, decide where the job lives when the process stops. Tell callers how long the results are kept.

Check your understanding

Why does POST /v1/jobs answer 202 Accepted and not 200 OK with the scores?

Show the answer

202 Accepted means “the work is accepted, but it is not done yet”. The scores do not exist when the service answers; the job is only in the queue. The answer gives the job ID and the Location to ask about the job, and Retry-After says when to ask again. The caller gets the scores later with GET /v1/jobs/{id}, when the status is done.

Check your understanding

The queue limit is 4. One big job runs, and the quality team submits 6 jobs at once. Predict what happens to each of the 6, and say why a limit is better than an unlimited queue.

Show the answer

The running job is not in the queue any more. The first 4 new jobs are accepted and wait in the queue (202). Jobs 5 and 6 are refused at once with 429 too_many_jobs, and the team must try them again later. Without a limit, a burst of jobs could fill the memory of the service, and every job, including the agents’ requests, would suffer. A limit says no early and clearly, while the service stays healthy.

See it end to endScore hospital readmissions asynchronouslyCase study

The case study submits a large discharge file as a queued job and polls for the result. It also shows what a restart does to an in-memory job.

Module practice: choose a pattern for three workloads

Worked example first. Amira’s answers for Larkfield’s three needs are the columns of the table in Choose a serving pattern. Read each column from the top: who starts the work, who waits and how long, how the results arrive, and the failure recovery.

Your task. Choose a pattern for each of the three fictional workloads below. For each one, write the answers to the three questions, the pattern, how users receive the results, and what failure recovery the pattern needs.

1. The churn list. A small online shop’s marketing team sends a discount email every Monday morning to customers who are likely to stop buying. A churn model scores all 80,000 customers. The emails go out at 09:00.
2. The card payment. A payment company checks each card payment for fraud while the customer waits at the till. The shop’s terminal waits at most 2 seconds for the whole payment. The fraud model needs about 15 ms per payment.
3. The policy re-score. An insurance company re-scores all 2 million home insurance policies once a month, when it updates its risk model. The model is slow: about 200 ms per policy. The actuaries ask for a re-score from their own tool when the new model is ready, and they need the results within two days.

Show a model answer

1. The churn list: batch. Nobody waits for one score; the marketing team needs the list by 09:00 on Monday. All 80,000 customers are scored together. Run the job on Sunday night, write the list and a summary into a dated folder, and send the emails from the list. Users receive results as a file. Recovery: set bad customer rows aside with a failure limit, and write the file atomically. If the job fails, alert someone and run it again before 09:00. A missing list means no emails this week, not wrong emails.

2. The card payment: online. A customer waits, and one payment is one case. The fraud check gets a latency budget, a part of the 2 seconds, for example 100 ms; 15 ms fits it. Users receive the result in the answer: approve, refuse or check by hand. Recovery: the terminal needs a plan for no answer within the budget. The company must decide it in advance, for example “approve small payments, check large ones by hand”. The service needs high availability, because payments arrive all day.

3. The policy re-score: asynchronous (or batch, if it runs on a fixed date). 2 million × 200 ms = 400,000 s, about 4.6 days with one worker, so no request can hold it. The actuaries’ tool submits a job and gets a job ID at once. The service splits the work into many smaller parts and runs them on several workers, to finish within two days. Users receive results by polling the job, then downloading a results file from storage. Recovery: a run of several days will see a restart. So the jobs and their progress live in a queue service and storage, not in one process’s memory. A failed part is retried, not the whole job.

Module check

Check your understanding

Name the three serving patterns of this module, and for each one, say how the users receive the results.

Show the answer

Batch: the users read the results from files that a scheduled run wrote, such as scores.csv with summary.json. Online: the caller receives the score in the answer to its own request, while it waits. Asynchronous: the caller receives a job ID at once (202), then polls GET /v1/jobs/{id} until the status is done, and gets the scores from that answer.

Check your understanding

Larkfield's model becomes 30 times slower per ticket. Predict what happens to each of the three Larkfield needs, using the numbers of this module.

Show the answer

30 × 1.76 ms ≈ 53 ms per ticket. Online: one ticket takes more than the 50 ms budget, so the page opens without a score most of the time; online no longer fits. Batch: about 45 tickets a night take about 2.4 s, and even 1,400 take about 74 s. That is long before 08:00, so batch still works. Asynchronous: a month of 1,400 tickets takes about 74 s, more than the 30 s gateway. So it must be a job, and a job still works. The slower model breaks only the pattern with a person waiting.

Check your understanding

Completion check: for batch, online and asynchronous serving, explain how users receive the results and what failure recovery each pattern needs.

Show the answer

Batch: users read a file, such as scores.csv, at a known time. Recovery: bad rows go to rejected.csv. Too many bad rows, or a missing column, fail the run with no output. The job writes its files atomically. summary.json records the input and bundle hashes, so the job can run again before anyone needs the results.

Online: the caller receives the score in the answer, within the latency budget (50 ms at Larkfield). Recovery is on the caller’s side: a timeout and a plan for no answer. At Larkfield, the page opens without a score, and the nightly batch catches up. The service needs availability and a clear “not ready” signal.

Asynchronous: the caller receives a job ID (202) and polls until the job is done, then reads the scores. Recovery: 429 means try again later; a failed job keeps its error and is submitted again; a lost job (after a restart) is submitted again. In production, jobs and results live in a queue service and storage, so a restart loses nothing.

Recap

• Asynchronous inference accepts a job at once (202 Accepted, a job ID, Location, Retry-After), scores it in the background, and the caller polls for the result. A queue holds the waiting jobs.
• A request with 1,400 tickets stops fitting in a 30 s gateway at about 21 ms per ticket. At 40 ms per ticket (simulated), the month took 56 s as a job, and it was accepted in 2.9 ms.
• Limits keep the service healthy: 4 waiting jobs (429), 5,000 tickets per job (413). In-memory jobs are lost on a restart; real systems use a queue service and storage (see Scaling APIs and AI Workloads).
• Module 2: choose the pattern by asking who waits, how large one piece of work is, and how it recovers from failure. Batch for scheduled lists, online for one case while someone waits, asynchronous for large work on demand. The same score_tickets serves all three.

Reset to a known state. Module 2 made no container and no image. If your project is broken, copy the snapshot end-of-m02 into ~/projects/escalation-service, as in Use a snapshot. Then run python -m pytest: 20 passed. You can delete the out/ folder; it is in .gitignore.

Next, you put the scoring behind an HTTP service. The service loads the bundle once, at start-up, so each request does only the fast work.

Next: Load and initialize