Job lifecycle

What you will be able to do

Accept a new ticket in milliseconds, even when the AI provider is busy. Save the AI work as a job in a queue, let a separate worker do it, and let the caller ask for the result later.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The async design of I/O versus CPU work: the API calls the provider three times while the customer waits.
• The provider’s quota from Find the bottleneck: 100 chat requests per minute, then HTTP 429.
• A transaction, and SELECT … FOR UPDATE, from Transactions in the databases course.

What you need on your computer

• The project after Failure and stampede behaviour: python -m pytest prints 148 passed, 1 deselected.
• PostgreSQL, Azurite and Valkey running (docker compose up -d db azurite cache), with the small data loaded.
• The course’s labs folder next to your project, as in Run the AI workload on your computer.

After this lesson, your project matches the snapshot end-of-m04. The next two lessons use the same files and add no code. The AI provider is the course’s simulated provider: its times and its quota come from real recorded calls. Every number labelled “measured” comes from an Apple M4 computer (10 cores, 16 GB) with nothing else busy. Your numbers will differ, but the shapes stay the same. Tested on macOS (Apple silicon); the Windows commands are written but not tested.

A ticket at the dry cleaner

You bring a shirt to a dry cleaner. The person at the counter does not clean it while you stand there. They write a ticket with a number, give you half of it, and put the shirt on a rail. You leave after one minute. Behind the counter, the cleaners take shirts from the rail, in order, as fast as they can. Tomorrow, you come back with your number and collect the shirt.

Three things make this work. The counter only takes work, so it is always quick. The rail keeps the work, so nothing is lost when the cleaners are busy. And your number lets you ask about your shirt later.

The problem: the customer waits for the AI

Until now, POST /v1/tickets did the AI work during the request. The API asked the provider three things, one after the other: the team and priority, a draft reply, and an embedding. That takes about 2.2 seconds per ticket. The customer’s app waited the whole time.

Waiting is not the worst part. The provider allows 100 chat requests per minute, and each ticket needs two. So the provider can serve about 50 tickets per minute. Omar’s Black Friday numbers say that the busiest minute can bring 90 tickets. In that minute, the provider must refuse some calls.

The course measured what customers see. Tickets arrived at 10 per second for 30 seconds, with the recorded quota. With the async design of Module 2, only about 55 of 300 tickets were saved. About 245 customers got 503 ai_busy, and their tickets were not saved. A customer wrote a message, and the help desk never saw it, because a draft reply was not ready.

Sam asks the obvious question: “Saving a ticket is one row in the database. Why does it depend on the AI at all?” It does not have to.

The idea: a queue and a worker

Split the request into two parts, as the dry cleaner does. The API saves the ticket and a note that says “this ticket needs its AI work”. It answers at once. A separate program reads the notes, one after the other, and does the AI work.

• A job is the note: one piece of work to do later, with its own number. Here, one job is the AI work of one ticket.
• A queue is the rail: the jobs that wait, in order. The first job in is the first job out.
• A worker is the cleaner: a program that takes a job from the queue, does it, and saves the result. It runs as its own process, next to the API. This is not the same as the worker processes of Horizontal scaling: those are copies of the API that answer requests (fastapi run --workers 4). A queue worker answers no request; it only takes jobs.

The API now answers 202 Accepted. In HTTP, 202 means “I have your request, and the work is not done yet”. The answer has the job’s number and a Location header: the address where the caller can ask about the job.

A tiny example with every number visible. Six tickets arrive in the same second. One worker runs 4 jobs at the same time, and each job takes 2.2 seconds. Each customer gets an answer in about 15 milliseconds:

Ticket	Answer to the customer	Job starts	Job is done
1 to 4	202, at 0.015 s	0 s	2.2 s
5 and 6	202, at 0.015 s	2.2 s (a place is free)	4.4 s

Jobs 5 and 6 wait 2.2 seconds in the queue. Nobody watches them wait: the customers were told “accepted” at once. When a burst is larger than the workers can handle, the queue gets longer, and the jobs wait longer. The work is late, but it is not lost.

One ticket with a queue: the API only saves; the worker does the three AI calls later; the caller reads the result with the job's address.

Text description of the diagram

The diagram has three parts. The top part is the request: the customer’s app talks only to the API, and the API talks only to the database. The answer comes back in about 15 milliseconds, before any AI call. The middle part is the worker. It talks to the database and to the provider, and no customer waits for it. Each of the three results goes to the database as soon as it is ready, with the job’s number; the next lesson explains why. The last part is the customer’s app asking for the result with GET /v1/jobs/1.

Where to keep the queue

A queue needs a place that all API processes and all workers can reach, and that keeps the jobs when a program stops. The course team measured two candidates on the same synthetic jobs that do nothing. Four workers took 2,000 empty jobs as fast as they could.

Empty jobs per second through each queue (4 workers, median of 3 passes), and the pace that the provider's quota allows.

Text description of the chart

A column chart with three columns. A Valkey list moved 7,171.6 empty jobs per second. A table in PostgreSQL moved 2,585.9 jobs per second. The provider’s quota allows 0.83 tickets per second. That column is too small to see: it is about 3,000 times lower than the PostgreSQL column.

What this tells us. Valkey is almost three times faster, and both are far too fast to matter. The AI work, not the queue, sets the pace. So the choice depends on other things, and PostgreSQL wins on two of them:

1. The ticket and its job are saved in one transaction. Either both exist, or neither does. With a queue in another system, the API saves the ticket, then sends the job. If the API stops between the two steps, the ticket never gets its AI work, and nobody notices. Keeping the job in the same database as the ticket is called the transactional outbox pattern.
2. A job is a row that SQL can read. “How many jobs wait, and how old is the oldest?” is one query. Mei can answer it with the tools that she already uses.

Other good queues exist: Redis or Valkey streams, RabbitMQ, and cloud queues such as Azure Service Bus or Amazon SQS. Choose one of them when you need thousands of jobs per second, or when many services share the queue. Then plan how the ticket and the job stay together.

Larkfield: the jobs table and the worker

Amira’s migration 014_jobs.sql adds the table. These are the columns that matter in this lesson:

migrations/014_jobs.sql (part)

CREATE TABLE jobs (
    job_id          bigint      GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    ticket_id       text        NOT NULL REFERENCES tickets (ticket_id) ON DELETE CASCADE,
    actor           text        NOT NULL,
    state           text        NOT NULL DEFAULT 'queued',
    attempts        integer     NOT NULL DEFAULT 0,
    max_attempts    integer     NOT NULL DEFAULT 5,
    run_after       timestamptz NOT NULL DEFAULT now(),
    locked_by       text,
    locked_until    timestamptz,
    ...

The API saves the ticket and the job in the same with db.connection() block, so they are in one transaction:

ticket_api/intake.py (accept, part)

ticket = repository.insert_ticket(conn, new.customer_id, new.subject, new.body, new.channel)
job = jobs.enqueue(conn, ticket["ticket_id"], actor, idempotency_key=key, ...)

The column actor says who asked for the work: customer:C-0022. The worker trusts this row, never a message from outside. The authentication course builds on this idea in Job identity and context.

Two workers never take the same job

Mei asks the database question: “If two workers look at the queue at the same moment, do they both take job 1?” Not with SKIP LOCKED. Each worker claims jobs with one statement:

ticket_api/jobs.py (claim, simplified)

UPDATE jobs SET state = 'running', attempts = attempts + 1, locked_by = 'worker-1',
       locked_until = now() + interval '30 seconds'
WHERE job_id IN (
    SELECT job_id FROM jobs
    WHERE state = 'queued' AND run_after <= now()
    ORDER BY run_after, job_id LIMIT 4
    FOR UPDATE SKIP LOCKED)
RETURNING ...

FOR UPDATE locks the rows that the worker picks, as in the databases course. SKIP LOCKED tells a second worker to step over those rows and take the next ones, instead of waiting. So each job goes to exactly one worker.

The worker also sets a lease: locked_until, 30 seconds from now. A lease is a loan with an end time. A job takes about 2.2 seconds, so the lease is long enough. If the worker stops in the middle, the lease ends, and another worker may take the job again. The next lesson is about exactly that case.

A job that takes minutes, such as a long report, needs more. Its worker must extend the lease while it works, with a regular “still working” update called a heartbeat. Without one, a slow job and a stopped worker look the same.

The life of a job

A job moves through six states. Each arrow is one SQL statement in ticket_api/jobs.py, so the state in the table is always the truth:

The states of a job and the event that moves it along each arrow.

Text description of the diagram

Read it from the top. Every job starts as queued. The only way to running is a worker’s claim. From running, there are four ways out. succeeded is the goal. failed is an error that a retry cannot fix, such as an answer that is not valid. A job can also go back to queued for a later try, or become a dead_letter after the last attempt. A person can send a dead letter back to the queue. cancelled is possible only before a worker takes the job, because a running job may already have called the provider. The next lesson explains retries, the 429 arrow and dead letters.

The caller follows the job with three addresses:

• GET /v1/jobs/{id}: the state, the attempts, the last error, and the result when the job has succeeded.
• POST /v1/jobs/{id}/cancel: cancel a queued job. Any other state answers 409 conflict.
• GET /v1/queue: the queue’s numbers: queued, ready, running, dead letters, finished in the last minute, and the oldest job’s age.

Guided practice: move the AI work to the queue

1. In ~/projects/ticket-api, with the virtual environment active, copy the files of Module 4 from the snapshot end-of-m04:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m04
   for f in README.md compose.yaml contract/openapi.json migrations/014_jobs.sql \
     scripts/duplicate_demo.py scripts/load.py scripts/queue_watch.py scripts/send_tickets.py \
     tests/test_cache.py tests/test_intake.py tests/test_jobs.py \
     ticket_api/cache.py ticket_api/config.py ticket_api/errors.py ticket_api/intake.py \
     ticket_api/job_routes.py ticket_api/jobs.py ticket_api/main.py ticket_api/models.py \
     ticket_api/ratelimit.py ticket_api/repository.py ticket_api/worker.py; do cp "$S/$f" "$f"; done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m04"
   $files = 'README.md', 'compose.yaml', 'contract\openapi.json', 'migrations\014_jobs.sql',
     'scripts\duplicate_demo.py', 'scripts\load.py', 'scripts\queue_watch.py', 'scripts\send_tickets.py',
     'tests\test_cache.py', 'tests\test_intake.py', 'tests\test_jobs.py',
     'ticket_api\cache.py', 'ticket_api\config.py', 'ticket_api\errors.py', 'ticket_api\intake.py',
     'ticket_api\job_routes.py', 'ticket_api\jobs.py', 'ticket_api\main.py', 'ticket_api\models.py',
     'ticket_api\ratelimit.py', 'ticket_api\repository.py', 'ticket_api\worker.py'
   foreach ($f in $files) { Copy-Item "$S\$f" $f }

2. If .env has a line INTAKE_MODE=…, delete it. The default is now queue.

3. Apply the new migration: python -m ticket_api.migrate. It prints applied 014_jobs.

4. Run the tests: python -m pytest. Expect 163 passed, 1 deselected.

5. Load the small data again, so that new tickets start at T-500001: python -m scripts.load --reset.

6. Start the simulated provider in one terminal (python -m simulator) and the API in a second one (fastapi dev). Do not start a worker yet.

7. In a third terminal, send a ticket. Then ask for its job:

   curl -i -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" \
     -d '{"customer_id": "C-0022", "subject": "Charged twice", "body": "Two payments for one order."}'
   curl http://127.0.0.1:8000/v1/jobs/1

   In PowerShell, write curl.exe instead of curl, on one line, and put \" around each name and value inside the JSON.

8. Start the worker in the third terminal. With --once, it stops when the queue is empty: python -m ticket_api.worker --once --name worker-1.

9. Ask for the job again: curl http://127.0.0.1:8000/v1/jobs/1.

10. Send a second ticket, and cancel its job at once: curl -X POST http://127.0.0.1:8000/v1/jobs/2/cancel. Then cancel it a second time.

Show the expected result

Step 7, the author’s run (your times will differ):

HTTP/1.1 202 Accepted
location: /v1/jobs/1
server-timing: rate_limit;dur=6.6, db;dur=17.4

{"ticket_id":"T-500001","job_id":1,"state":"queued","status_url":"/v1/jobs/1","queued_ahead":0,"estimated_wait_seconds":null}
{"job_id":1,"ticket_id":"T-500001","state":"queued","attempts":0,"max_attempts":5,"created_at":"…","started_at":null,"finished_at":null,"last_error":null,"result":null}

The ticket is saved, the answer took milliseconds, and no AI call happened yet. Step 8 prints three HTTP Request: POST http://127.0.0.1:8300/v1/… lines, then:

INFO ticket_api.worker: job 1: succeeded (attempt 1)
Worker worker-1: up to 4 jobs at a time. Takes what is queued, then stops.
Done in 2.3 s: 1 taken, 1 succeeded, 0 waiting for the quota, 0 to retry later, 0 failed, 0 dead letters.

Step 9 shows the result (shortened):

{"job_id":1,"ticket_id":"T-500001","state":"succeeded","attempts":1,…,"last_error":null,
 "result":{"team":"billing","priority":1,"draft_reply":"Thank you for your message about \"Charged twice\". …","embedding_version":"embed-small-384-v1"}}

Step 10: the first cancel answers 200 with "state":"cancelled". The second answers 409 with {"error":{"code":"conflict","message":"Job 2 is cancelled: only a queued job can be cancelled.",…}}.

Your turn: two workers on one queue

Predict first. Twenty jobs wait. You start two workers at the same moment, each with --once. How many jobs does each one take? Can one job appear in both logs?

1. Send 20 tickets without a worker running: python -m scripts.send_tickets --rate 5 --seconds 4.
2. In two terminals, at the same time, start python -m ticket_api.worker --once --name worker-1 and python -m ticket_api.worker --once --name worker-2.
3. Compare the job numbers in the two logs.

Show the answer

In the author’s run, worker-1 took 9 jobs and worker-2 took 11:

worker-1: jobs 58, 59, 60, 61, 63, 64, 68, 69, 71         Done in 6.1 s: 9 taken, 9 succeeded …
worker-2: jobs 54, 55, 56, 57, 62, 65, 66, 67, 70, 72, 73  Done in 6.7 s: 11 taken, 11 succeeded …

Your split will differ, because the worker that asks first gets the next jobs. No job is in both logs: SKIP LOCKED gave each job to one worker. Two workers finished 20 jobs in under 7 seconds; one worker needs about 11. The quota still limits them both, as the lesson Backpressure shows.

Jobs that never start

Common mistake

The API accepts tickets, but no worker runs

Symptom: every POST /v1/tickets answers 202 in milliseconds, so everything looks fine. But no ticket gets a team, a priority or a draft reply, and GET /v1/jobs/{id} says queued for every job.

Cause: the worker is not running. Maybe it never started after a deployment, or it stopped with an error. The API cannot see that: it only saves jobs. A queue hides a missing worker, because the customers still get fast answers.

Diagnose: look at the queue’s numbers. python -m scripts.queue_watch --every 2 prints one line every 2 seconds. With no worker, queued and oldest s grow, and done/min stays at 0. In the author’s run, 31 jobs waited; then one worker started at time 0:

  time  queued  ready  running  oldest s  done/min  dead
     0      31     31        0      64.1         0     0
     2      27     27        4       7.6         0     0
     4      23     23        4       9.5         4     0
     6      20     20        4      11.5         7     0
     8      16     16        4      13.5        11     0
    10      12     12        4      15.4        15     0
    12       9      9        4      17.4        18     0
    14       5      5        4      19.3        22     0
    16       2      2        3      21.3        26     0
    18       0      0        3       0.0        28     0
    20       0      0        0       0.0        31     0

At time 0, the oldest job had waited 64 seconds. Then queued fell by about 2 jobs per second, and the queue was empty after about 18 seconds.

Fix:

1. Start the worker: python -m ticket_api.worker. In Docker Compose, the worker service runs it.
2. Watch the oldest job’s age, not only the errors. An age that grows with no end means “no worker” or “workers too slow”. Scaling signals turns this number into a signal to add workers.

When to use a queue, and what it costs

Use a queue when the work is slow or unreliable, and the caller does not need the result in the same moment. The AI work of a new ticket is a good fit. The customer needs to know that the help desk has the ticket, not the draft reply. A person reads the draft reply later anyway.

Do not use a queue when the caller needs the answer now. POST /v1/answers from Module 3 answers a customer’s question on the screen. A queue would only make the customer ask again and again. For that endpoint, the cache and the async design are the right tools.

A queue has costs, and Kwame wants them written down:

• One more program to run and watch. The worker can stop while the API looks healthy, as the common mistake above shows.
• The result comes later. The caller must ask for it, with GET /v1/jobs/{id}, or be told when it is ready.
• More states to handle. A job can wait, fail, be cancelled or need a person. Each state needs an answer for the customer and for the help desk.
• A job can run twice. A lease that ends gives the same job to a second worker. The next lesson makes that safe.

Knowledge check

Check your understanding

With the async design, about 245 of 300 tickets were not saved during the burst. Why does a queue in the same database save every one of them, even when the provider refuses calls?

Show the answer

With a queue, saving the ticket no longer depends on the provider. The API saves the ticket and its job in one short transaction and answers 202 in about 15 ms; it makes no AI call. The provider’s limit now only decides how fast the worker finishes the jobs. A refusal makes a job wait in the queue; it does not lose the ticket. The cost is time: the AI results arrive later, and in a long burst much later.

Check your understanding

A Valkey list moved 7,172 empty jobs per second and the PostgreSQL table 2,586. Why did Larkfield choose the slower one?

Show the answer

Both are thousands of times faster than the need. The quota allows about 0.83 tickets per second, so the queue’s own speed does not matter here. PostgreSQL gives two things that do matter. The ticket and its job are saved in one transaction, so there is never a ticket without its job. And jobs are rows that SQL can count and read: how many wait, how old the oldest is, which ones failed. With another queue, the API would need extra work to keep the ticket and the job together.

Check your understanding

Predict: a customer sends POST /v1/jobs/7/cancel while worker-1 runs job 7. What happens, and why does the API not stop the work?

Show the answer

The API answers 409 conflict: only a queued job can be cancelled. A running job may already have called the provider and saved some results, and the worker does not ask the API before each step. To stop a running job, the worker would need to check for a cancel request between steps and clean up what it saved. The project keeps it simple: cancel is possible until a worker takes the job.

Recap

• A queue keeps jobs until a worker does them. The API saves the ticket and its job in one transaction and answers 202 Accepted with the job’s address.
• With the async design, about 245 of 300 tickets in a burst were not saved. With a queue, saving a ticket does not depend on the provider.
• PostgreSQL (2,586 jobs/s) and a Valkey list (7,172/s) are both far faster than the AI work needs. PostgreSQL keeps the ticket and the job together and lets SQL read the queue.
• FOR UPDATE SKIP LOCKED gives each job to one worker. A 30-second lease gives a job back if its worker stops.
• A job is queued, running, succeeded, failed, cancelled or dead_letter. GET /v1/jobs/{id} returns the state and the result; cancel works only while queued.
• Watch the oldest job’s age: a queue hides a stopped worker.

Next: Delivery and retries