Final assignment

What you will be able to do

Use every module of the course on a job that you did not build in it. You move the slow AI work of a second Larkfield service into a bounded background queue. Then you prove four things with load tests and recorded runs. You know what the service can take, and why. A repeated message has one effect. Overload ends in a clear answer, not in a pile that grows. And your report says what your tests cannot say about production.

Before you startWhat to know and to have readyAbout 180 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• The service: your docs-assistant from the final assignment of the databases course. If you do not have it, build its small core first: documents with versions and a stored file, and chunks with a vector and an embedding version. You do not need that assignment’s index tuning or restore drill here.
• Or another Larkfield job, if you prefer. It must be new, not the ticket intake of this course. It must call the provider for at least one chat step and one embedding step, and some of its jobs must have many steps. An example is a nightly summary of each customer’s open tickets, with an embedding of each summary. Say in your report which service you chose. The steps below use the document assistant; change the names for your job.
• The course’s project, as a source of code: your ticket-api at the end of Module 6, or the snapshot end-of-m06 (see the snapshots). You may copy code from it: the simulated provider simulator/, ticket_api/jobs.py, worker.py, ratelimit.py, resilience.py, loadtest/shapes.py, and the scripts percentiles, machine, queue_watch, compare and cost. Say in your report what you copied and what you changed. Do not copy the ticket routes and rename them: the reviewer looks for the decisions that this service needs.
• Docker with PostgreSQL 18 and pgvector, Azurite and Valkey, as in the course. The AI provider is the course’s simulated one. No account and no money are needed.
• Every load test in this assignment targets your own services on your own computer. Never point a load test at a real AI provider or at a service that you do not own.
• You can do the assignment in more than one session. Run the tests after each change.

The situation: a policy day

The help desk’s agents ask the document assistant questions while they work on tickets. Its answers come from Larkfield’s policy documents, split into chunks with an embedding each. Making the chunks of a new document version is slow work that waits for the provider. A long document has many chunks, and each chunk needs an embedding.

Omar brings the new workload. On the day before Black Friday, the policy team publishes new versions of many documents at once: returns, delivery times, refunds. A script uploads them all within a few minutes. The assistant may cite a new version only after its chunks and vectors exist. And the policy team wants a short summary of each version, written by the model, for the help desk’s news page.

Kwame asks his usual questions: “What will this cost, and what is the limit?” Mei watches her database connections. Tomás has already said “add more workers” twice.

After this assignment, an upload is saved at once and answers 202 Accepted. A job in a PostgreSQL queue does the AI work later, in a separate worker:

• Split the document version into chunks (no provider call).
• Summarize the version: one chat call to chat-small.
• Embed each chunk: one call to embed-small per chunk, or per batch of chunks if you choose batches.

Your data. Use the 36 policy documents in data/source/documents/ of your ticket-api (Markdown, CC0). Write a seeded script that makes new versions of them, with small changes, for your load tests. All the data is made up: say so in your report.

Your provider. Run the course’s simulated provider. Its chat quota is 100 requests per minute, and its embedding quota is 100,000 tokens per minute, as recorded from the real account. The simulator times your summary like a draft reply, because it does not know your prompt. Say so wherever you show a summary’s time.

Part 1: capacity assumptions

Uses Run the AI workload on your computer, Workload model, Latency and capacity, Find the bottleneck and Dependency limits.

1. Write the workload model in docs/workload.md: what arrives (uploads, and the questions that agents ask at the same time), how often on a normal day, and on the policy day. Give the busiest minute and the multiple that you plan for, with the reason for each number.
2. Measure one job alone, with the simulated provider: the time of each step, the number of chunks, and the tokens of each call. Use at least 10 documents of different lengths. Report the median and the slowest, not only the mean.
3. Compute what each limit allows, in jobs per minute, with every number visible. Do it for the chat quota (one summary per job), the embedding quota (the tokens of a job’s chunks) and the workers (jobs at a time ÷ the time of one job). Add the database connections of every process together. Say which limit comes first, and at which workload.
4. Compute the AI cost of one job and of the policy day from the token counts and the dated prices in prices.py. Say that prices change.
5. Give each assumption a letter, as in Capacity recommendation. Every claim later in your report names the letters that it rests on.

Part 2: a bounded background queue

Uses I/O versus CPU work, Job lifecycle and Backpressure.

1. Add a jobs table with a migration. Save the document version and its job in one transaction: there is never a version without its job, or a job without its version. The upload answers 202 with the job’s address in Location.
2. Add a worker that claims jobs with FOR UPDATE SKIP LOCKED and a lease. Give the jobs the states of the course, or explain each state that you change. Add GET /v1/jobs/{id}, a cancel route for a queued job (409 conflict otherwise), and GET /v1/queue. The queue route shows the queued, ready and running jobs, the dead letters and the oldest job’s age.
3. Until its job succeeds, a new version is not searchable, and the assistant keeps citing the previous published version. Say how a reader can tell that a newer version is waiting.
4. Bound the queue. Choose QUEUE_MAX from the longest wait that you can promise to the policy team, not from the space in the database. Show the arithmetic. When the queue is full, answer 503 queue_full with a Retry-After from the real pace, and save nothing.
5. Add a rate limit for each caller, with 429 rate_limited and Retry-After, so that one script cannot fill the queue for everyone.
6. Give each accepted upload an estimate of its wait. Say how wrong the estimate can be at the start of a burst.

Part 3: duplicate-safe job processing

Uses Delivery and retries, Failure containment and Graceful shutdown.

1. Accept an Idempotency-Key on the upload, for each caller. A repeat with the same body returns the same version and job, with Idempotent-Replayed: true. The same key with another body answers 422. When two copies arrive at the same moment, a unique index decides, not a check in Python.
2. Save each step’s result at once, with the job’s ID, under a unique rule: one summary per job, and one chunk row per job and chunk position. Before a step, the worker skips what is already saved. A second delivery must not add a chunk, a vector or a summary.
3. Retry what may work next time, with exponential backoff, jitter and a limit of attempts, then a dead letter. Mark an error that a retry cannot fix as failed at once. A 429 from the provider makes the job wait without spending an attempt.
4. Set the provider timeout from your measured slowest call, with room. Add a circuit breaker in front of the provider. While it is open, jobs wait without spending attempts.
5. Handle SIGTERM: take no new job, drain for less time than your platform’s grace period, then give the unfinished jobs back without spending an attempt.
6. Measure your longest job. A document with many chunks takes much longer than a ticket. If a job can run longer than its lease, add a heartbeat that extends the lease, or split the job into smaller jobs. Say which, and why.

Part 4: load tests and a capacity recommendation

Uses Horizontal scaling, Scaling signals, Load-test design, Compare configurations and Capacity recommendation.

1. Write the four parts of each test before you run it: the question, the workload from Part 1, the shape, and the environment with its readings. Reset the data, empty the cache and restart the simulated provider before every pass. Record python -m scripts.machine before and after each run.
2. Run a steady baseline at a normal day’s rate, and a burst at the policy day’s busiest minute. Read two kinds of time: the upload’s answer (p50, p95, p99), and the time from submit to result of each job. Read the queue every second: its length and its oldest job.
3. Run an overload test above the bound, as in Backpressure. Count the accepted, refused and lost uploads, the Retry-After values and the queue’s peak.
4. Compare two configurations that differ in one setting, with at least three passes each. Examples are one worker against two, or a batch of 16 chunks per embedding call against one chunk per call. Report the median and the range of every column of the scorecard. Say “no difference shown” when the ranges overlap.
5. Choose a scaling signal for the worker from your runs, with its minimum and maximum. Say what more workers can fix, and what they cannot.
6. Write docs/capacity-report.md. It has the question, the workload, what you measured, and one demonstrated improvement with before and after. It also has the cost, the next bottleneck with what to measure before the next change, and “What this test does not say”.

No production capacity claim from a laptop test. Your report must not say how many uploads production can take, or how fast the real provider answers on the policy day. You measured one computer, the simulated provider, made-up data and short runs. Claims about shapes and comparisons are fine, with their conditions: “with a bound of 50, an overload lost no upload”, or “the chat quota comes first”. An absolute production number is not. If someone needs one, your report says which real run must measure it.

The failure paths you must check

A working happy path is not enough. Show each of these with a test or a recorded run, and say which. Every run goes to your own services on your own computer. Label each constructed case as constructed.

1. The same upload sent twice with one Idempotency-Key gives one version and one job. The second answer has Idempotent-Replayed: true. Another body with the same key gets 422.
2. A worker crashes in the middle of a job, after some chunks are saved (constructed). The job is delivered again after its lease. In the end, every chunk position has exactly one row and one vector, and the summary is saved once. Report the provider calls of the job, and explain any call that was repeated.
3. A full queue answers 503 queue_full with a Retry-After, and saves no version. One caller over its limit gets 429 rate_limited, while other callers still get 202.
4. With the provider’s quota used up (python -m simulator --mode quota makes it 10 times smaller), jobs wait, spend no attempt and become no dead letter.
5. A simulated outage opens the circuit breaker. Count the calls that reached the provider during the outage. After the outage, the waiting jobs succeed. With the breaker off (constructed), show that the attempts run out, and retry one dead letter on purpose.
6. SIGTERM during a long job gives the job back with its attempts unchanged and its saved steps kept. After a SIGKILL (constructed), the job stays running until its lease ends; then another worker finishes it without a duplicate.
7. With no worker running, uploads still get 202. Show the reading that tells you: the oldest job’s age grows, and nothing finishes.
8. The connections of all your API processes and workers together stay below the database’s limit. Show the sum for your largest configuration, with each pool size.

Fresh-environment run

Before you submit, prove that the project works from nothing, on your computer:

1. Copy the project to a new folder, without .venv, .env and your run outputs.
2. Delete the database and the files with docker compose down --volumes. Then start PostgreSQL, Azurite and Valkey again.
3. Make a new virtual environment, install from requirements.txt, and make .env from .env.example.
4. Apply the migrations, load the data, and run every test.
5. Start the simulated provider, the API and one worker. Upload one document, follow its job to succeeded, and run your duplicate check (failure path 2) once.

Paste the commands and their outputs into your report. If a step needed something that is not in the README, fix the README, and run the steps again.

What to submit

Your project folder (without .env, .venv and large run outputs), with your migrations, code, tests, load-test files, scripts, docs/workload.md, docs/capacity-report.md, your filled scorecards and REPORT.md:

# Larkfield document assistant: background ingestion under load
## Workload and assumptions         (the workload model, one job measured alone, each limit in jobs per
                                     minute, the cost, the lettered assumptions)
## The queue                        (the jobs table, one transaction, states and routes, the bound and its
                                     arithmetic, the rate limit, the wait estimate)
## One effect per message           (idempotency keys, saved steps and their unique rules, retries, 429
                                     handling, timeout, breaker, drain, lease or heartbeat)
## Load-test evidence               (each test's four parts, the machine and its load, the baseline, the
                                     burst, the overload, the comparison with medians and ranges)
## Capacity recommendation          (the improvement, the next bottleneck, what this test does not say)
## Failure paths                    (each of the eight, with the test or the run that shows it)
## Fresh-environment run            (the commands and their outputs)
## Limits                           (simulated provider, made-up data, your computer, what you copied,
                                     what you did not test)

How it is marked

A reviewer marks each required item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all four must be demonstrated. You can improve an item and submit it again.

Required item	Demonstrated	Partially demonstrated, for example	Not yet demonstrated, for example
Capacity assumptions	The workload has numbers and reasons. One job is measured alone, with medians and the slowest. Each limit is computed in jobs per minute, and the first one is named. The cost uses token counts and dated prices. Every claim names its assumptions.	The limits are computed, but the embedding quota or the database connections are missing. Or the job’s time comes from one run.	No stated workload: “ten times more” with no numbers. Or the report turns a laptop result into a production number.
Load-test evidence	Each test has its question, workload, shape and environment, and the machine’s load is recorded. The burst reports the time from submit to result, not only the 202. The comparison changes one setting, with at least three passes, medians and ranges. Where the ranges overlap, it says “no difference shown”.	The tests ran, but with one pass each. Or only the upload’s answer time was measured. Or the load average is missing.	No load test, or a load test against a service that you do not own. Or two settings changed between the compared runs.
Duplicate-safe job processing	A repeated request and a repeated delivery each leave one effect, shown by a test and a recorded run, and the database’s unique rules enforce it. A 429 spends no attempt. Retries are bounded, with a dead letter. The breaker, the drain and the lease or heartbeat are in place, and the longest job is measured.	One effect per message, but a 429 spends attempts. Or the longest job can outlive its lease, and nothing handles it.	A second delivery adds chunks, vectors or a second summary. Or the idempotency key is global, so one caller can replay another caller’s upload.
Explicit limits and overload behaviour	The queue has a bound chosen from a promised wait, with the arithmetic. Overload answers 503 queue_full or 429 rate_limited with an honest Retry-After. It saves nothing that it refuses, and loses nothing that it accepts. The worker count has a maximum, and the connections stay below the database’s limit.	The bound works, but Retry-After is a fixed number. Or the bound has no reason, or its race is not mentioned.	The queue has no bound, so a long burst grows the backlog without an end. Or a refused upload is half saved.

Whatever else is strong, the work is not complete while a duplicate delivery produces two effects, or while overload is unbounded. A fast 202 does not make up for a second set of vectors, or for a queue that grows without a limit. And a sentence that turns a laptop test into a production capacity fails the item that it appears in.

The course’s own results can help you judge yours, but they are not targets. In Backpressure, a bound of 100 accepted 172 to 173 of 300 tickets. It refused 127 to 128 with a Retry-After of 88 to 128 seconds, and the queue peaked at 107. In Delivery and retries, a job delivered twice left 3 AI results and made 3 provider calls; the naive worker made 6 of each. In Failure containment, the breaker cut the calls during a 20-second outage from 150 to 8. Your service, its jobs and your computer are different.

AI assistants

You may use an AI assistant to practise, to explain an error, or to suggest a test, a query or a load shape. Check each suggestion, and be ready to explain your final work. For each limit, unique rule and retry setting, say what it does, why you chose its value, and which test or run proves it. Mark in your report the decisions that are your own, and the suggestions that you rejected, with the reason. An assistant may not choose your capacity numbers for you, and it may not produce your evidence: every time, count, percentile and cost in your report comes from a run on your computer. Do not ask it for a production capacity either: it has not measured your service.

Common mistake

A long document is processed twice

Symptom: short documents end at attempt 1, but the longest ones end at attempt 2. The provider’s count shows more embedding calls than chunks, although the chunk table has one row per position.

Cause: the job ran longer than its lease. A ticket’s job in the course took about 2.2 seconds, far below the 30-second lease. A document with many chunks can take much longer, especially when the quota makes some calls wait. When the lease ends, another worker claims the job while the first one still works on it. Your unique rules stopped the duplicate rows, but both workers paid for some calls.

Fix: measure the longest job under load, not alone. Then add a heartbeat that extends locked_until while the worker makes progress, or split the job into batches of chunks with a job each. Check again with the longest document in a burst, and compare the provider’s calls with the chunks.

Recap

In this course, you made a slow, quota-limited AI service behave well under load:

• Measure first: a workload model with numbers, percentiles instead of averages, and the bottleneck found by splitting the time.
• Use the waiting: async calls for I/O work, more processes only where the limits allow, and a sum of every per-process setting.
• Cache with care: keys that hold everything that changes the answer, including who asks; invalidation by generations; and a cache that fails open.
• Queue the slow work: jobs saved with their data in one transaction, one effect per message, bounded retries, dead letters, and a bound with an honest Retry-After.
• Contain failure: timeouts from measurements, a circuit breaker, a scaling signal that more workers can fix, and a drain on every stop.
• Report honestly: fair comparisons with passes and ranges, and a recommendation that says what the test does not say.

If you have not done one yet, choose a case study for the certificate: Capacity plan from a public traffic trace or Queue and cache for an open-data API.