Object-storage patterns

What you will be able to do

Keep the photos and receipts that customers attach in object storage, and their details in PostgreSQL. Let clients upload and download each file through a link that allows one action on one file for a few minutes. Refuse files that are too large.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The four kinds of store, and why a photo is not a row, from Storage categories.
• The attachments table and why its foreign key refuses deletes, from Integrity constraints.
• The connection pool and with db.connection(), from Connection management.

What you need on your computer

• The project ~/projects/ticket-api at the end of Module 4. python -m pytest prints 87 passed.
• PostgreSQL running, and the labs repository in ~/projects. Step 5 loads the small data again, in place of the large data of Module 4. If you cloned it some time ago, run git -C ../nextia-learning-labs pull first.
• Docker, for the object-storage emulator. Without Docker, see the note in step 4.

This lesson adds object storage to Compose, one migration (009_attachment_uploads.sql), two new modules (files.py, attachments.py) and a test file. It also changes six files of the API.

The cloakroom

At a concert, you leave your coat at the cloakroom. The attendant hangs it on hook 214 and gives you a small ticket with the number 214. The coat stays on the hook; the ticket goes in your pocket. The concert hall does not keep coats in the seats.

The ticket does three useful things. It names one hook, so it cannot fetch another coat. It works for one action, “give me my coat”. And the cloakroom closes at midnight. After that, the ticket is worth nothing.

The problem: 43 rows and nowhere for the files

The help desk already describes its files. Open psql on the small data and look at ticket T-30034, the parcel that arrived damaged:

SELECT ticket_id, file_name, content_type, size_bytes, object_key
FROM attachments WHERE ticket_id = 'T-30034';

 ticket_id |  file_name  | content_type | size_bytes |                            object_key
-----------+-------------+--------------+------------+------------------------------------------------------------------
 T-30034   | photo-1.png | image/png    |       3613 | tickets/T-30034/fcf3867b-6929-4624-9e25-c526ce2f3e67/photo-1.png
 T-30034   | photo-2.png | image/png    |       3767 | tickets/T-30034/d38ab247-9828-4dce-a1ee-b0b2e6baa45e/photo-2.png
(2 rows)

The rows name each file and its place, object_key. But that place does not exist yet. The 43 files of the small data are only in a folder of the project, data/small/files/. No program can upload a new photo, and no agent can open one.

Amira lists the first ideas of the team, and what each one costs:

• Put the bytes in a bytea column of PostgreSQL. It works for a few receipts. But every photo then goes through the API, the connection pool and every backup. A database of 9 MB with a year of photos becomes many gigabytes, and each restore takes longer.
• Keep the files on the API server’s disk. The API runs in a container. When it is rebuilt, the disk can go with it, as Persistent data showed. A second API server would not see the first one’s files.
• Send the bytes through the API to some store. The API then spends a worker on every slow upload from a phone.

The idea: object storage, with the details in PostgreSQL

Object storage keeps files, called objects, in a flat space. Each object has a key, a name such as tickets/T-30034/fcf3867b-…/photo-1.png. You put an object, get it, or delete it, always by its key. Objects live in a container (Amazon S3 calls it a bucket). Object storage has no tables, no joins and no constraints, and it is cheap per gigabyte. Azure Blob Storage, Amazon S3 and Google Cloud Storage are object stores.

It is the cloakroom. The hooks hold the coats; the ticket holds the hook number. In the same way, PostgreSQL keeps one row per file: its ticket, name, type, size, checksum and key. Object storage keeps the bytes. The row is the cloakroom ticket, and object_key is the hook number.

What	Where	For T-30034’s first photo
The bytes of the photo	object storage	3,613 bytes of PNG
Which ticket it belongs to	PostgreSQL, attachments.ticket_id	T-30034
Its key in object storage	PostgreSQL, attachments.object_key	tickets/T-30034/fcf3867b-…/photo-1.png
Its size and checksum	PostgreSQL, size_bytes, sha256	3613, 91c6510a03d4…

The key contains the attachment’s UUID. So two photos called photo-1.png never get the same key, and nobody can guess a key from a ticket number alone.

On your computer, the course uses Azurite, Microsoft’s Azure Blob Storage emulator (version 3.37.0, MIT licence). It runs in Docker, answers the same requests as the cloud service, and needs no account and no money. The API talks to it through Microsoft’s azure-storage-blob package (12.31.0).

Signed URLs: a cloakroom ticket for one file

The API should not carry the bytes. But the client cannot have the storage account’s key either. With that key, a phone could read, change or delete every customer’s files.

The answer is a signed URL: a link that allows one action on one object until a time, and nothing else. The API makes it with the account key, and the client never sees the key. Azure calls it a SAS (shared access signature). Amazon S3 calls it a presigned URL. Here is the download URL from the course’s own run, in four parts:

The signature is computed from the other parts with a secret key. So a changed file name, a later time or another action makes it wrong, and the storage answers 403. All five tries are real runs.

Text description of the diagram

A signed URL has four parts. The path names one file: …/attachments/tickets/T-30002/34cf7dbf…/photo-1.png. The parameter se gives the expiry time, 2026-10-10T04:18:10Z. The parameter sp gives the action: r for read, or cw for create and write. The parameter sig is the signature, made from the other parts with the account key. The diagram lists five real tries. The same URL before it expires gives 200 and 3390 bytes. Another file name with the same signature gives 403. The same URL after its expiry time gives 403. A read URL used to upload gives 403. An upload URL used to download gives 403.

Sam asks: “Can somebody who sees the link change photo-1.png to photo-2.png?” They can change the text, but the signature was made for photo-1.png. The storage computes the signature again for the changed link, finds a different value, and refuses with AuthorizationFailure. Without the account key, nobody can make a matching signature.

So the link is safe to hand to a browser, for a short time. It is not safe to log, paste into a ticket or keep for a week. Anybody who has it can use it until it expires. That is why the project gives upload links 10 minutes and download links 5 minutes.

Larkfield’s upload, in four requests

Mei and Amira design the flow so that the API never touches the bytes, and the database never trusts the client:

Four requests. The API decides; the object storage carries the bytes. The row starts as pending and becomes stored only after the API has checked the real file.

Text description of the diagram

Step 1: the client asks the API to start an upload of photo-1.png, image/png, 3390 bytes, for ticket T-30002. The API checks the size limit, inserts a row with status pending, and returns an upload URL that allows create and write for 10 minutes. Step 2: the client sends the file straight to the object storage with that URL and gets 201. The file never passes through the API. Step 3: the client asks the API to complete the upload. The API asks the storage for the real size, 3390 bytes, and reads the file to compute its SHA-256. If the size is as announced and at most 10 MiB, the row becomes stored. If the file is larger than 10 MiB, the API deletes it, the row becomes rejected, and the answer is 413. Step 4: the client asks for a download URL, which allows reading for 5 minutes, and gets the file from the storage. The API holds no database connection while it talks to the object storage.

Three decisions are worth a closer look.

The row comes first, as pending. Migration 009 adds a status column with three values: pending, stored and rejected. A CHECK says that a stored file must have a checksum and an upload time. So the database cannot claim a file that nobody checked. A pending row whose file never arrives is harmless: the ticket page shows stored files only.

Complete checks the real file, not the client’s word. The client announced 3,390 bytes. At complete, the API asks the storage for the size, reads the bytes and computes the SHA-256. Only then does the row change, with one UPDATE … WHERE status = 'pending'. Completing twice gives the same answer and changes nothing.

No connection is held during storage calls. In Connection management, a pool of 10 connections served the API. A storage call can take seconds on a slow network. So attachments.py borrows a connection for a short read and returns it. Then it talks to the storage, and then it borrows a connection again for a short write:

ticket_api/attachments.py (complete_upload, shortened)

with db.connection() as conn:  # 1. a short read
    row = repository.get_attachment(conn, str(attachment_id))
size = files.size(row["object_key"])  # 2. the object storage, with no connection held
...
with db.connection() as conn:  # 3. a short write, only if the row is still pending
    updated = repository.finish_attachment(conn, str(attachment_id), stored=problem is None,
                                           size_bytes=size, sha256=sha)

Behind files is a small contract, FileStore in ticket_api/files.py: upload URL, download URL, size, read, put, delete, list keys. AzureBlobStore speaks to Azurite or Azure. LocalFileStore keeps files in a folder and signs its own links, so the tests need no Docker. Another cloud would need a third class, and nothing else in the API would change.

Size limits: what a signed URL cannot promise

The help desk keeps files up to 10 MiB (10,485,760 bytes). The API refuses a larger announcement at once, with 413 file_too_large. But a client can announce 1,000 bytes and then send something much bigger. The course tried it, with a link made for a 1,000-byte file:

PUT 12,000,000 bytes to the upload URL      -> 201
POST /v1/attachments/<id>/complete          -> 413
{"error":{"code":"file_too_large","message":"The file is larger than 10485760 bytes.","request_id":"21877b981272","fields":[]}}

An Azure SAS has no size condition. The storage accepted 12 MB, because the link allows “create and write this key”, with no limit. So the project checks the size at complete, deletes the file and marks the row rejected. The file stayed in the storage for a few seconds. A client that never calls complete leaves it there. The scan in Recovery and retention finds such old pending uploads.

Amazon S3 has a different tool. A presigned POST carries a policy that S3 checks during the upload, including a content-length-range condition, for example from 1 to 10,485,760 bytes. S3 refuses a larger file before it stores it. This course describes it and does not run it, because it needs an S3 account or another emulator. Either way, keep the last check on your side. Migration 009 also adds CHECK (size_bytes <= 10485760), so no program can record a larger file.

Lifecycle rules: the storage cleans up by itself

Files also have a lifetime. A rejected upload is waste after an hour. Grace’s retention plan, in the last lesson of this module, may say “delete closed tickets’ photos after two years”. The cloud services can do part of this work themselves, with lifecycle rules. A lifecycle rule lives in the storage account. It deletes the objects that match a prefix and an age, or moves them to a cheaper tier.

Azurite does not run lifecycle rules. So the example below follows the shape of the Azure documentation, and nobody ran it in this course. It deletes objects under attachments/uploads-tmp/ 2 days after their last change:

An Azure lifecycle rule (from the documentation's structure; not run)

{
  "rules": [
    {
      "name": "delete-unfinished-uploads",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["attachments/uploads-tmp/"] },
        "actions": { "baseBlob": { "delete": { "daysAfterModificationGreaterThan": 2 } } }
      }
    }
  ]
}

Three facts from the documentation matter for a plan. A new rule can take up to 24 hours before its first run. Conditions are ages in days: after creation, after the last change, or after the last read if access tracking is on. A filter can only include objects by prefix or tag; it cannot exclude any.

A lifecycle rule knows nothing about your database. It sees ages and prefixes, not “this ticket is closed”. So use it for whole groups of objects that the database no longer needs, under their own prefix. Delete a specific customer’s files from your program, as the last lesson of this module does.

Guided practice: add object storage to ticket-api

1. In ~/projects/ticket-api, copy this lesson’s files from the snapshot end-of-m05. On macOS or Linux:

   S=../nextia-learning-labs/C20/snapshots/end-of-m05
   cp $S/compose.yaml $S/.env.example $S/pyproject.toml $S/requirements*.txt .
   cp $S/contract/openapi.json contract/
   cp $S/ticket_api/{config,errors,main,models,tickets,repository,files,attachments}.py ticket_api/
   cp $S/migrations/009_attachment_uploads.sql migrations/
   cp $S/tests/test_files.py tests/

   On Windows, in PowerShell (written, not tested):

   $S = "..\nextia-learning-labs\C20\snapshots\end-of-m05"
   Copy-Item "$S\compose.yaml", "$S\.env.example", "$S\pyproject.toml", "$S\requirements*.txt" .
   Copy-Item "$S\contract\openapi.json" contract\
   foreach ($f in "config","errors","main","models","tickets","repository","files","attachments") { Copy-Item "$S\ticket_api\$f.py" ticket_api\ }
   Copy-Item "$S\migrations\009_attachment_uploads.sql" migrations\
   Copy-Item "$S\tests\test_files.py" tests\

   The copied tickets.py and repository.py also contain the similar-ticket search. It stays unused until Vector storage.

2. Open .env.example. Copy its last three lines, which start with # Object storage, to the end of your .env. The account name devstoreaccount1 and its long key are Azurite’s public test key. Microsoft publishes them, and every Azurite has the same ones, so they are not a secret. A real storage account’s key is a secret: it never goes in a file that you commit.

3. Install the new package (it brings azure-storage-blob 12.31.0):

   python -m pip install -r requirements.txt

4. Start Azurite next to PostgreSQL:

   docker compose up -d db azurite

   Without Docker, Azurite also comes as an npm package (npm install -g azurite, then azurite-blob --skipApiVersionCheck). The course did not test that route.

5. Load the small data again. The loader applies 009 and puts the 43 files into the object storage:

   python -m scripts.load --reset

   applied  009_attachment_uploads
   Database is up to date (9 migrations).
   Loaded the small data in 0.5 s:
     customers                 40
     tickets                  200
     messages                 331
     attachments               43
     documents                 36
     ai_runs                  294
     ticket_embeddings        200
     43 files uploaded to the object storage.
     3 messages have no ticket: they are in orphaned_messages.

6. Run the tests. They use LocalFileStore, so they do not need Azurite:

   python -m pytest

   The last line is 95 passed, 1 deselected. The deselected test talks to Azurite. Run it on its own with python -m pytest -m azurite: 1 passed, 95 deselected.

7. Start the API with fastapi dev. In a second terminal, in the project folder, copy one of the small data’s photos to photo-1.png:

   cp data/small/files/tickets/T-30014/*/photo-2.png photo-1.png

   On Windows: Copy-Item data\small\files\tickets\T-30014\*\photo-2.png photo-1.png. The file has 3,390 bytes. If your .env sets API_KEY, add -H "X-API-Key: <your key>" to every request to port 8000 below.

8. Start an upload for ticket T-30002:

   curl -s -X POST http://127.0.0.1:8000/v1/tickets/T-30002/attachments \
     -H 'Content-Type: application/json' \
     -d '{"file_name": "photo-1.png", "content_type": "image/png", "size_bytes": 3390}'

   {"attachment_id":"34cf7dbf-8b5f-4b2b-ab87-1776f335a986","upload_url":"http://127.0.0.1:10000/devstoreaccount1/attachments/tickets/T-30002/34cf7dbf-8b5f-4b2b-ab87-1776f335a986/photo-1.png?se=2026-10-10T04%3A23%3A10Z&sp=cw&sv=2026-10-06&sr=b&sig=6QSxque36Itx%2B5gphGJKF0DCanEaJ8VtNAXL4MzTi8s%3D","method":"PUT","headers":{"x-ms-blob-type":"BlockBlob","Content-Type":"image/png"},"expires_at":"2026-10-10T04:23:10Z","max_bytes":10485760}

   Your ID, times and signature differ. Copy two values into shell variables, with your own values: ID=34cf7dbf-… and URL='http://127.0.0.1:10000/…'. Keep the single quotes around the URL, because it contains &.

9. Look at the row in psql: SELECT file_name, size_bytes, status, sha256 FROM attachments WHERE ticket_id = 'T-30002';. It shows photo-1.png | 3390 | pending | with no checksum. Ask the API to complete it now, before the upload:

   curl -s -X POST http://127.0.0.1:8000/v1/attachments/$ID/complete

   {"error":{"code":"upload_missing","message":"The file is not in storage. Upload it first.","request_id":"7e0312d6acff","fields":[]}}

10. Upload the file straight to the storage, then complete:

    curl -s -o /dev/null -w '%{http_code}\n' -X PUT -H 'x-ms-blob-type: BlockBlob' \
      -H 'Content-Type: image/png' --data-binary @photo-1.png "$URL"
    curl -s -X POST http://127.0.0.1:8000/v1/attachments/$ID/complete

    201
    {"attachment_id":"34cf7dbf-8b5f-4b2b-ab87-1776f335a986","file_name":"photo-1.png","content_type":"image/png","size_bytes":3390,"status":"stored","uploaded_at":"2026-10-10T04:13:10.151351Z"}

    In PowerShell, the same three requests are shorter, because Invoke-RestMethod reads the JSON for you (written, not tested):

    $r = Invoke-RestMethod -Method Post -Uri http://127.0.0.1:8000/v1/tickets/T-30002/attachments -ContentType 'application/json' -Body '{"file_name": "photo-1.png", "content_type": "image/png", "size_bytes": 3390}'
    Invoke-WebRequest -Method Put -Uri $r.upload_url -Headers @{ 'x-ms-blob-type' = 'BlockBlob' } -ContentType 'image/png' -InFile photo-1.png
    Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/v1/attachments/$($r.attachment_id)/complete"

11. Get a download URL, and use it:

    curl -s http://127.0.0.1:8000/v1/attachments/$ID/download

    The answer has a url with sp=r, valid for 5 minutes. Put it in a variable, DL='…'. Then curl -s -o /dev/null -w '%{http_code} %{size_download} bytes\n' "$DL" prints 200 3390 bytes. Change photo-1.png? to photo-2.png? in the URL and try again: 403.

12. Try the size limit both ways. First announce 20,000,000 bytes: the API answers 413 with file_too_large at once. Then announce 1,000 bytes for a file called small.png. Put its attachment_id in ID and its upload_url in URL, as in step 8. Send 12,000,000 bytes to it:

    head -c 12000000 /dev/zero > big.png
    curl -s -o /dev/null -w '%{http_code}\n' -X PUT -H 'x-ms-blob-type: BlockBlob' \
      -H 'Content-Type: image/png' --data-binary @big.png "$URL"

    The storage answers 201. Complete that upload: 413 and file_too_large. In psql, the row of small.png is now rejected, and the file is gone from the storage. On Windows, make the file with [IO.File]::WriteAllBytes("$PWD\big.png", (New-Object byte[] 12000000)). Delete big.png and photo-1.png when you finish.

The outputs in this module come from one computer, an Apple M4 with macOS 26.6 and 16 GB of memory. Docker Desktop 4.81.0 had 8 GB and 10 CPUs. It ran PostgreSQL 18.6 with pgvector 0.8.7, and Azurite 3.37.0. Your times will differ; the counts will not. The course tested every step on macOS (Apple silicon) only. The Windows and Linux commands are written, not run.

Common mistake

Azurite refuses every request: the API version is not supported

Symptom: python -m scripts.load --reset or the first upload stops with azure.core.exceptions.HttpResponseError and this message:

The API version 2026-10-06 is not supported by Azurite. Please upgrade Azurite to latest version and retry. If you are using Azurite in Visual Studio, please check you have installed latest Visual Studio patch. Azurite command line parameter "--skipApiVersionCheck" or Visual Studio Code configuration "Skip Api Version Check" can skip this error.

Cause: every request of azure-storage-blob names the storage API version that it speaks. Version 12.31.0 of the package speaks 2026-10-06. That version is newer than Azurite 3.37.0. Azurite refuses versions that it does not know, to be safe. The course reproduced it by starting Azurite without the flag (a constructed case).

Diagnose: look at how Azurite was started: the command: line of the azurite service in compose.yaml, or docker compose ps azurite.

Fix: start Azurite with --skipApiVersionCheck, as the project’s compose.yaml does. The flag tells Azurite to answer anyway. This is safe for the calls that this project makes: put, get, size, delete, list and signed URLs. The tests prove that they work. If you wrote your own Compose file, add the flag, and run docker compose up -d azurite again. The other fix is a newer Azurite, when one supports the version.

Independent variation: the wrong size

Start a new upload of photo-1.png for T-30002, but announce "size_bytes": 1000. Upload the real file, 3,390 bytes, and complete. Before you run it, predict the status code and the error code. Then check the row in psql.

Show the answer

The upload itself works (201), because the link has no size condition. complete answers:

{"error":{"code":"size_mismatch","message":"The file has 3390 bytes, not 1000.","request_id":"b6b5cc8b1873","fields":[]}}

The status code is 422. The file is smaller than 10 MiB, so this is not file_too_large. It is not the file that the client announced. The API deletes it, and the row becomes rejected. A client that wants to retry starts a new upload.

When to use object storage, and what it costs

Omar asks the product manager’s question: “What does this cost us?” Object storage is the cheapest place per gigabyte. The bytes never pass through the API or the database, so neither must grow for them. Use it for any file that a person uploads or downloads whole: photos, receipts, PDFs, exports, and the files of a document assistant.

It is not free of work:

• Two places for one attachment. The row and the file can disagree: a row whose file is missing, or a file that no row knows. The database’s transactions do not reach the object storage. Recovery and retention builds the scan that finds both.
• Backups are separate. pg_dump does not copy the files.
• A signed link works for anybody who has it. Keep its lifetime short, and never log it.
• Small, structured data does not belong there. A ticket’s subject, its status or an AI run’s cost need queries, constraints and joins. They stay in PostgreSQL.

A bytea column can still be the right answer for a few small, critical files. Use it when a file must be in the same transaction and backup as its row. A signature image of a few kilobytes is an example.

Knowledge check

Check your understanding

Why does the row start as pending, and what turns it into stored? What would go wrong if the API stored the row as stored at the start?

Show the answer

At the start, the file does not exist yet; the client only announced it. The row is pending until complete. Then the API asks the storage for the real size, reads the bytes and computes the SHA-256. Only if the size is as announced and at most 10 MiB does the row become stored. If the row were stored at the start, the ticket would show files that never arrived, or files of another size. The CHECK that a stored file has a checksum and an upload time stops that.

Check your understanding

Predict: a customer shares a download link in a public forum. What can other people do with it, and for how long?

Show the answer

They can download that one file, and nothing else, until the time in se, at most 5 minutes after the API made the link. They cannot change the file name to get another file, because the signature would not match (403). They cannot upload with it, because sp=r allows reading only (403). After the expiry time, the link gives 403. That is why the project keeps download links short and makes a new one for each request.

Check your understanding

An Azure SAS cannot limit the size of an upload. Name the places where the project enforces the 10 MiB limit, and the S3 feature that checks it during the upload.

Show the answer

The API checks the announced size when the upload starts (413 before any upload). It checks the real size at complete: it deletes a larger file, and the row becomes rejected. The database has a CHECK (size_bytes <= 10485760), so no program can record a larger file. Amazon S3’s presigned POST can carry a content-length-range condition, which S3 checks during the upload. The course describes it but does not run it.

Recap

• Object storage keeps files as objects with a key. PostgreSQL keeps one row per file: ticket, name, type, size, checksum and object_key.
• A signed URL allows one action on one object until a time. Another file name, the wrong action or a late try each gave 403.
• The upload takes four requests: start (a pending row), PUT to the storage, complete (the API checks the real file), download. The API never carries the bytes, and holds no connection during storage calls.
• An Azure SAS has no size limit: a 12 MB upload got 201. So the API checks at complete and deletes the file (413). S3’s presigned POST can check during the upload.
• Lifecycle rules delete or move objects by prefix and age, in the cloud; Azurite does not run them.
• Your project has Azurite, migration 009 and the file endpoints. python -m pytest prints 95 passed, 1 deselected.

Next: Vector storage