Delivery and retries

What you will be able to do

Make the AI work of a ticket safe to receive twice. Repeat what may work next time, a limited number of times, and set aside what still fails, so that a person can look at it.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Jobs, the worker, the lease and the job states, from Job lifecycle.
• A unique constraint, from Integrity constraints in the databases course.
• HTTP 429 and the Retry-After hint, from Find the bottleneck.

What you need on your computer

• The project at end-of-m04, from the previous lesson: python -m pytest prints 163 passed, 1 deselected.
• PostgreSQL and Valkey running; the simulated provider (python -m simulator) and the API (fastapi dev) in two terminals.

This lesson adds no code. The crash and the “naive worker” are constructed cases: the course makes them on purpose to show what can go wrong. The provider is simulated. Tested on macOS (Apple silicon); the Windows commands are not tested.

A pizza order, made twice

You order a pizza by phone. The pizzeria takes your order, and then the line drops before you hear “OK”. Did they get it? You do not know, so you call again.

A careless pizzeria makes a second pizza, and you pay for two. A careful one asks for your phone number, finds your order on the list, and says “We have it already; it is in the oven.” You can call ten times and still get one pizza.

The careful pizzeria does not stop you from calling again. It makes your second call harmless.

The problem: the same work arrives twice

The ticket API meets the same situation in two places.

The caller repeats a request. A customer’s app sends a new ticket, and the network drops the answer. The app does not know whether the ticket arrived, so it sends it again. Without protection, the help desk gets two tickets and the provider does the AI work twice.

The queue delivers a job twice. A worker takes job 1, does the work, and stops before it marks the job succeeded. The process crashed, or the computer was restarted. The job is still running, but nobody holds it. Its 30-second lease ends, and another worker takes it: the same job, a second time. This is a duplicate delivery.

The second case is not a bug in the queue; it is the price of never losing a job. A queue that gives a job back when its worker disappears cannot know how far the worker got. So it must sometimes deliver a job twice. This promise is called at-least-once delivery: every job runs at least once, and sometimes more.

What happens then depends on the worker. The course built a naive worker on purpose: it calls the provider for every step, and it saves results without the job’s number. Delivered twice after a crash, it left 6 AI results for one ticket and made 6 provider calls. That is two teams, two draft replies, two embeddings, and a double bill.

The idea: make repeated work harmless

An operation is idempotent when doing it twice has the same effect as doing it once. A lift button is idempotent: pressing it five times calls one lift. “Add 10 euros to the account” is not; “set the balance to 110 euros” is. The word means “the same power” in Latin.

You cannot stop the repeats. You make each repeat harmless instead, with two tools:

1. An idempotency key for a repeated request. The caller sends a name for its request in the header Idempotency-Key. The API keeps the name with the job. When the same name comes again, it returns the first answer and saves nothing new.
2. The job’s number on every result, for a repeated delivery. The worker saves each step’s result with the job’s ID, and the database allows one result per job and step. Before a step, the worker checks which results the job has. It skips the steps that are done.

The agents course uses the same idea for actions in the outside world. Its operation IDs give each intended change one name, so a repeated refund pays once.

A tiny example with the course’s demo. Worker-1 does all three steps of job 1 and crashes before it marks the job succeeded. The lease ends, and worker-2 takes the job:

	Provider calls	Results saved	Job state
Worker-1, attempt 1	classify, draft_reply, embed (3)	3, each with job 1	running (crash)
Worker-2, attempt 2: the project’s worker	none: all 3 steps are done	0 new	succeeded
Worker-2, attempt 2: the naive worker (constructed)	3 again	3 more, without the job’s ID	—
Total	project: 3; naive: 6	project: 3; naive: 6

A duplicate delivery: the same job reaches a second worker after a crash. The project's worker finds the finished steps; a naive worker repeats them.

Text description of the diagram

The top half happens in both cases: a crash after the work and before the last update. The lease turns that crash into a second delivery. The box at the bottom compares two workers. The project’s worker reads the job’s results first, so the second delivery only marks the job as done. The naive worker trusts nothing it can read, so it repeats every call and doubles every result.

What “one intended effect” means here

Be exact about the promise, because Kwame will ask. After any number of deliveries, a ticket has one job, one team and priority, one draft reply and one embedding: one result per step. That is the intended effect.

The number of provider calls is not always 3. The demo crashed after the last save, so the second delivery made no call. If a worker crashes between a provider call and its save, the next delivery repeats that one call. The result is still saved once, but the provider bills that call twice. At the course’s prices, one chat step costs about US$0.00003. For a draft reply, that is acceptable. For a step with an outside effect, such as an email to the customer, it is not. Such a step needs its own key that the other service checks, like the operation IDs above.

Larkfield: one ticket per idempotency key

The API looks for the key before it saves anything. The key belongs to the caller (the job’s actor), so two customers can use the same key without a conflict:

ticket_api/intake.py (accept, part)

if key:
    earlier = jobs.find_by_key(conn, actor, key)
    if earlier is not None:
        if earlier["request_sha256"] != digest:
            raise jobs.IdempotencyConflict(key)          # 422 idempotency_key_reused
        ...
        return earlier["ticket_id"], earlier, ahead, jobs.stats(conn), True   # replayed

digest is a SHA-256 fingerprint of the request’s body. The same key with the same body is a repeat: the API answers 202 with the first ticket and job, and the header Idempotent-Replayed: true. The same key with a different body is a mistake in the client: 422 idempotency_key_reused.

Two copies of a request can also arrive at the same moment, before either one is saved. A unique index jobs_idempotency_key on (actor, idempotency_key) lets only one of them insert. The other one fails with a unique violation, and accept_once looks up the key again. It then returns the first request’s ticket.

Larkfield: one result per job and step

Migration 014_jobs.sql adds the job’s ID to ai_runs, and one rule:

migrations/014_jobs.sql (part)

ALTER TABLE ai_runs ADD COLUMN job_id bigint REFERENCES jobs (job_id) ON DELETE SET NULL;
CREATE UNIQUE INDEX ai_runs_one_per_job_task ON ai_runs (job_id, task) WHERE job_id IS NOT NULL;

The worker uses the rule twice. First, before the steps, it reads the job’s results (repository.job_runs) and skips every step that has one. Second, it saves each result with ON CONFLICT (job_id, task) … DO NOTHING. If another worker saved the same step a moment earlier, the insert adds nothing, and the worker does not update the ticket again. The author tried a second classify row for job 1 by hand:

ERROR:  duplicate key value violates unique constraint "ai_runs_one_per_job_task"
DETAIL:  Key (job_id, task)=(1, classify) already exists.

With the worker’s ON CONFLICT … DO NOTHING, the same insert printed INSERT 0 0: no error, and no second row.

One last guard is in jobs.finish. It marks a job succeeded only if this worker still holds it (locked_by). A slow worker whose lease ended cannot overwrite the state that the newer worker wrote.

Retries: a few, then wait longer, then stop

Not every failure is a crash. Sometimes the provider answers 503 or does not answer in time. A retry is a new attempt of the same job later. Retries are safe now, because the work is idempotent. But they are not free.

Tomás proposes the simple rule: “Just retry until the provider answers.” During an outage of the provider, that rule sends hundreds of calls to a service that cannot answer. When the provider comes back, every job tries again in the same second, and the provider fails again under the load. Retries need three limits:

1. Wait longer after each failure. This is exponential backoff: the longest wait doubles after each attempt. The project starts at 2 seconds (RETRY_BASE_SECONDS) and never waits more than 60 (RETRY_CAP_SECONDS).
2. Pick a random wait below that limit. This is jitter. Jobs that failed in the same second then try again at different moments, not all together.
3. Stop after a few attempts. After 5 attempts (JOB_MAX_ATTEMPTS), the job becomes a dead letter: it leaves the queue and waits for a person. The name comes from the post office, where a letter that cannot be delivered or returned goes to a special office.

The waits of one job, with the project’s settings (jobs.backoff_seconds):

After attempt	Longest wait	Actual wait
1	2 s	random, 0 to 2 s
2	4 s	random, 0 to 4 s
3	8 s	random, 0 to 8 s
4	16 s	random, 0 to 16 s
5	—	no retry: dead_letter

The author set the simulated provider to outage and sent one ticket. The worker’s log:

09:25:55,599 INFO ticket_api.worker: job 2: retried (attempt 1)
09:25:56,147 INFO ticket_api.worker: job 2: retried (attempt 2)
09:25:58,716 INFO ticket_api.worker: job 2: retried (attempt 3)
09:25:59,768 INFO ticket_api.worker: job 2: retried (attempt 4)
09:26:01,827 INFO ticket_api.worker: job 2: dead_letter (attempt 5)

The random waits were short this time: 6.2 seconds from the first attempt to the dead letter. GET /v1/jobs/2 then showed "state":"dead_letter" and "last_error":"ProviderUnavailable: the provider answered 503". After the outage, POST /v1/jobs/2/retry sent it back with 0 attempts, and the worker finished it in 2.0 seconds.

Two endings look alike, but they are not the same. A failed job had an error that a retry cannot fix, such as an answer that is not valid; the worker stops at once. A dead letter had errors that might go away, but they did not go away in 5 attempts. Both keep last_error, so a person can decide.

A retry rule that turned waiting into failure

Common mistake

Counting a 429 as a failed attempt (found in the course's own worker)

Symptom (measured): the first version of the course’s worker had a burst of 180 tickets, 6 per second, with the recorded quota on. Within 50 seconds, 99 of 180 jobs were dead letters. The provider was not down at all. It only asked the worker to slow down.

Cause: that worker treated 429 like a 503: retry later, and spend one attempt. Its waits were seconds long (0 to 2, 0 to 4, 0 to 8 …). But the quota counts calls over a whole minute. So a job used its 5 attempts in about half a minute, while the quota was still full, and became a dead letter.

Diagnose: compare when the dead letters appear with the quota’s window. In this run, the first dead letters came 16 seconds into the burst. All 99 came within 50 seconds: less than the minute that the quota needs to fill again. To see why each job ended, group the dead letters by their last error: SELECT last_error, count(*) FROM jobs WHERE state = 'dead_letter' GROUP BY 1;.

Fix: a 429 is not the job’s fault. The worker now gives the job back without spending an attempt. It waits for the provider’s hint or a growing backoff, whichever is longer:

ticket_api/worker.py (process, part)

except ProviderRateLimited as error:
    # Over the quota is not the job's fault: the job waits and keeps its attempts.
    state = "waiting"
    self.rate_limited += 1
    delay = max(error.retry_after or 0,
                jobs.backoff_seconds(min(self.rate_limited, 6), ...))
    await asyncio.to_thread(self._release, job, delay)

Outages (5xx, timeouts) still spend attempts and still end as dead letters. The same burst after the fix:

• Before the fix: dead letters
• After the fix: dead letters (dashed, 0)
• After the fix: jobs waiting

The same burst (180 tickets, 6 per second, quota on), before and after the fix. One row every 2 seconds.

Text description of the chart

A line chart over 140 seconds. Before the fix, the dead letters start at about 16 seconds and climb to 99 at 50 seconds, then stay there. After the fix, the dead letters stay at 0 for the whole run. Instead, the jobs that the quota refused wait in the queue: up to 116 at about 30 seconds. Then the queue gets shorter, as the quota lets about 50 jobs per minute through. At 138 seconds, 25 jobs still waited, and the oldest had waited about two minutes. The burst came from the local autoscaler run of Module 5, a simulation on this computer.

What this tells us. After the fix, no job was lost, but jobs waited minutes. That is the honest result of a quota: the work is late, not gone. The next lesson decides how much waiting work the queue should accept.

Module practice: one deliberately repeated message

This is the module’s practice and its completion check. scripts/duplicate_demo.py repeats a message in both ways. It sends the same request twice with one idempotency key. Then it delivers the same job twice after a simulated crash. It runs the API and two workers inside one process, on your .env database, with the simulated provider.

1. Load the data again, so the demo’s ticket is T-500001: python -m scripts.load --reset.

2. Make sure the simulated provider runs (python -m simulator). The API does not need to run.

3. Run the demo: python -m scripts.duplicate_demo. It takes about 8 seconds, because it waits for the crashed worker’s lease to end.

4. Look at the job’s AI results in the database:

   docker compose exec db psql -U tickets -d tickets -c "SELECT run_id, job_id, task, left(output, 40) AS output FROM ai_runs WHERE ticket_id = 'T-500001' ORDER BY run_id;"

5. Run the tests that check the same rules: python -m pytest tests/test_jobs.py -k "twice or idempotency" -v.

Show the expected result

Step 3 prints two WARNING ticket_api: API_KEY is not set lines and some INFO and HTTP Request lines first. Then:

Part 1: the same request twice, with the same Idempotency-Key
  first:  202 ticket T-500001 job 1
  second: 202 ticket T-500001 job 1 (Idempotent-Replayed: true)
  in the database: tickets 1, jobs 1, AI results 0
Part 2: the same job delivered twice (a simulated crash)
  after worker-1 (crashed before finishing): job running, attempt 1; tickets 1, jobs 1, AI results 3
  after worker-2: job succeeded, attempt 2; tickets 1, jobs 1, AI results 3
  provider calls for this job: 3 (3 steps: classify, draft_reply, embed)

Step 4, the author’s run (your run_id numbers can differ):

 run_id | job_id |    task     |                  output
--------+--------+-------------+------------------------------------------
    295 |      1 | classify    | {"team": "billing", "priority": 1}
    296 |      1 | draft_reply | Thank you for your message about "Charge
    297 |      1 | embed       | vector embed-small-384-v1
(3 rows)

Step 5 selects 2 tests, test_the_same_idempotency_key_gives_the_same_ticket and test_a_job_delivered_twice_has_one_effect, and ends with 2 passed, 13 deselected.

Completion check. Show three things for one ticket. The repeated request returned the same ticket and job. The second delivery ended with the job succeeded at attempt 2. And the database holds exactly one result per step, with the job’s ID. Then explain in two sentences which line of the project stops each duplicate.

Your turn: the same key from another customer

Predict first. C-0022 sends a ticket with Idempotency-Key: my-first-key-0001. Then C-0003 sends a different ticket with the same key. Is the second answer a replay, a 422, or a new ticket?

With the API running, send both:

for c in C-0022 C-0003; do
  curl -s -i -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" \
    -H "Idempotency-Key: my-first-key-0001" \
    -d "{\"customer_id\": \"$c\", \"subject\": \"Late parcel\", \"body\": \"My parcel is late.\"}"
done

In PowerShell, send the two requests one by one with curl.exe.

Show the answer

A new ticket. In the author’s run:

HTTP/1.1 202 Accepted
{"ticket_id":"T-500082","job_id":82,"state":"queued",…}
HTTP/1.1 202 Accepted
{"ticket_id":"T-500083","job_id":83,"state":"queued",…}

The key belongs to its caller: the unique index is on (actor, idempotency_key), and the two actors are customer:C-0022 and customer:C-0003. Customers choose their keys on their own, so two of them can pick the same one. If keys were global, one customer’s key would block another customer’s ticket, or worse, return another customer’s ticket. Your ticket numbers depend on what you sent before.

When to use these tools, and what they cost

Use an idempotency key on every request that creates something and that a client may repeat: new tickets, payments, messages. Keep keys for as long as a client might repeat a request; this project keeps them with the job. Make every queued job idempotent, always: with at-least-once delivery, a second delivery is a question of time.

Retries with backoff fit errors that can go away: timeouts, 503, a dependency that restarts. Do not retry an error that repeats the same way, such as a request that the provider rejected; mark it failed at once. Do not count a “slow down” (429) as a failure.

The costs are a unique index and one more read before each step, and a few calls that can repeat after a crash. Dead letters also need a person, a screen or a query, and a decision. Five attempts with these waits give up after less than a minute. That is right for a short glitch. For a long outage, Module 5 adds a circuit breaker, so that jobs wait instead of using up their attempts.

See it end to endQueue and cache for an open-data APICase study

The open-data API case study makes each CSV export a job with an idempotency key. The worker names the file by the job’s ID, so a second delivery writes the same file.

Knowledge check

Check your understanding

Why can a queue that never loses a job not promise to deliver each job exactly once?

Show the answer

When a worker disappears, the queue cannot know how far it got. Maybe it did nothing; maybe it did all the work and crashed before the last update. To never lose a job, the queue gives it to another worker when the lease ends. In the second case, that is a second delivery. So the queue promises “at least once”, and the worker makes a repeat harmless. It saves each step’s result with the job’s ID, once, and skips the finished steps.

Check your understanding

The demo's second delivery made no provider call. Describe a crash after which the next delivery makes one call again, and say why the ticket still has one draft reply.

Show the answer

Worker-1 calls the provider for draft_reply, gets the answer, and crashes before it saves it. The job has a classify result but no draft_reply result. The next delivery skips classify, calls the provider again for draft_reply, and saves it: the provider billed that step twice. The ticket has one draft reply, because only one row for (job_id, 'draft_reply') can exist. The effect is once; one call was repeated.

Check your understanding

Before the fix, 99 of 180 jobs became dead letters during a burst, while the provider was not down. Explain why, and what the worker does now with a 429.

Show the answer

The worker counted each 429 as a failed attempt and waited only seconds before the next one. The quota counts calls over a whole minute, so a job used all 5 attempts while the quota was still full. A 429 means “slow down”, not “this job is broken”. Now the worker gives the job back without spending an attempt, and waits for the provider’s hint or a growing backoff, whichever is longer. In the same burst, no job became a dead letter; jobs waited in the queue instead.

Recap

• A queue that never loses a job delivers some jobs twice: at-least-once delivery. Work must be idempotent: twice has the effect of once.
• An Idempotency-Key per caller makes a repeated request return the same ticket and job (Idempotent-Replayed: true); another body with the same key is 422 idempotency_key_reused.
• Each step’s result is saved with the job’s ID, and UNIQUE (job_id, task) allows one. A crashed job delivered twice: 3 results and 3 provider calls; a naive worker (constructed): 6 and 6.
• The intended effect is one result per step. A crash between a call and its save repeats that one call, not the result.
• Retries wait longer after each failure, at random: exponential backoff with jitter. After 5 attempts, the job is a dead letter, for a person. failed is for errors that a retry cannot fix.
• A 429 must not cost an attempt: counting it turned 99 of 180 jobs into dead letters; after the fix, 0.

Next: Backpressure