Ownership and source of truth

What you will be able to do

Decide which store keeps the original of each piece of data, and which stores keep copies. Then a lost copy costs time, never data. Then make a copy again from its original.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The four kinds of store and the storage map, from Storage categories.
• What a ticket vector is, from Storage categories: the idea.

What you need on your computer

• The project ~/projects/ticket-api with the small data loaded. python -m pytest prints 34 passed.
• The labs repository or its ZIP file in ~/projects, from From SQLite to PostgreSQL. If you cloned it some time ago, run git -C ../nextia-learning-labs pull in the project folder first. With the ZIP file, download and unzip it again.
• Optional: about 2 GB of free disk space for an embedding model. Without it, you can do everything except the rebuild. The lesson shows its output.

This lesson adds three files to your project. After it, your project matches the snapshot end-of-m01.

A recipe and a cake

Your grandmother’s recipe card is in a drawer. On Sunday, you bake her cake from it.

If the dog eats the cake, you are annoyed, but you bake another one. It costs flour, eggs and an hour. Nothing is lost for good, because the recipe is still in the drawer.

Now imagine the opposite. The card is gone, and only the cake is left. You can taste it and guess, but you cannot get the exact recipe back from the cake.

The card is the original. The cake is a copy, made from the original with a method: the oven, the time, the order of the steps. A copy is safe to lose only while its original and its method still exist.

The problem: “keep everything in the vector database”

Tomás has a plan for the similar-ticket feature. “Vector databases store metadata too,” he says. “Let’s put each ticket’s text, status and customer next to its vector. Then everything is in one place, and the search can return it all.”

Mei asks one question: “What happens to the tickets when we lose the vectors?”

Vectors get lost more often than you might think. A new embedding model makes every old vector useless, because two models put the same text in different places. Module 5 measures it. A query vector from one model, among the vectors of another model, found 0.4% of the right neighbours. There was no error. So vectors are regularly deleted and made again. An index is dropped and rebuilt. A migration goes wrong.

Try a loss on your own project. This is a constructed case: you delete every vector on purpose. Open psql (docker compose exec db psql -U tickets -d tickets) and run:

TRUNCATE ticket_embeddings;
SELECT count(*) AS tickets FROM tickets;
SELECT count(*) AS vectors FROM ticket_embeddings;

The answers are 200 tickets and 0 vectors. TRUNCATE deleted every row of ticket_embeddings and nothing else. Now run the similar-ticket query of the setup lesson again:

SELECT ticket_id, subject FROM tickets ORDER BY (SELECT embedding FROM ticket_embeddings WHERE ticket_id = 'T-30002') <=> (SELECT embedding FROM ticket_embeddings e WHERE e.ticket_id = tickets.ticket_id) LIMIT 4;

 ticket_id |              subject
-----------+-----------------------------------
 T-30002   | Charged twice for order LK-182074
 T-30003   | Merge two accounts
 T-30004   | robot vacuum not delivered
 T-30001   | Tracking number does not work
(4 rows)

There is no error. Without vectors, every distance is empty, so PostgreSQL returns tickets in any order. Your rows can differ. The feature is broken, but the help desk has lost no data: the 200 tickets, their messages and their files are all still there.

In Tomás’s design, the same TRUNCATE would have deleted the only copy of 200 tickets. Nobody could make them again, because a vector cannot be turned back into its text. That is the cake without the recipe card.

Quit psql with \q. You make the vectors again later in this lesson.

The idea: one original, many copies

For each fact, one store is the source of truth: the store that keeps the original. If two stores disagree, the source of truth wins. Data in it is authoritative. Another word for the same idea is system of record.

Every other store keeps derived data: a copy made from an original by a known method. A derived copy can always be made again, so losing it costs time, not data. A disposable copy is a derived copy that nobody even rebuilds: a cache entry is filled again by the next request.

Three rules follow:

1. Every fact has exactly one original, in one store, with a backup.
2. Every copy records what it was made from: which original, and which method and version.
3. A copy is never the only place of a fact. If it is, it is not a copy: it is an original in the wrong store.

Ownership goes with it. The store that keeps the original owns the fact. A change goes to the owner first; the copies follow.

A tiny worked example: the recipe of one vector

The vector of T-30002 is derived. Here is everything that went into it. The original is the ticket’s text in PostgreSQL, its subject and body joined by a new line:

Charged twice for order LK-182074
My card was charged two times for order LK-182074, 449.00 dollars each time. Please refund one of the payments.

The method is in scripts/embed.py, which you add to the project below:

scripts/embed.py (part)

MODELS = {  # embedding version: (model, revision, prefix, licence)
    "e5-small-v1": (
        "intfloat/multilingual-e5-small",
        "614241f622f53c4eeff9890bdc4f31cfecc418b3",
        "query: ",
        "MIT",
    ),
    ...
}

The model is intfloat/multilingual-e5-small, the model of the RAG course. The revision fixes one exact version of its files: another revision could give other numbers. The text gets the prefix "query: ", and the result is normalized to length 1. The result is 384 numbers.

Same original, same model, same revision, same prefix: same vector. That is the whole promise of a derived copy.

How the ticket vectors are derived, and how they are made again: the same original through the same recipe.

Text description of the diagram

The original is the ticket text in PostgreSQL. The recipe is the model, its revision, the prefix and the normalization. The copy is the 200 rows of ticket_embeddings, and the HNSW index of Module 5 is a copy of that copy. If the copy is lost, or nobody trusts it, python -m scripts.embed --rebuild --all runs the same original through the same recipe again.

Add the storage map to the project

Mei writes the help desk’s ownership down, so that it does not live only in her head. Add her map and the rebuild script to your project.

1. In the project folder, copy the three files of this lesson from the snapshot end-of-m01. If your labs folder is nextia-learning-labs-main, use that name.

   macOS and Linux

   cp ../nextia-learning-labs/C20/snapshots/end-of-m01/docs/storage-map.md docs/
   cp ../nextia-learning-labs/C20/snapshots/end-of-m01/scripts/embed.py scripts/
   cp ../nextia-learning-labs/C20/snapshots/end-of-m01/requirements-embed.txt .

   Windows

   Copy-Item ..\nextia-learning-labs\C20\snapshots\end-of-m01\docs\storage-map.md docs\
   Copy-Item ..\nextia-learning-labs\C20\snapshots\end-of-m01\scripts\embed.py scripts\
   Copy-Item ..\nextia-learning-labs\C20\snapshots\end-of-m01\requirements-embed.txt .

2. Open docs/storage-map.md in VS Code. Read the table from left to right. Each row gives the data, the store of its original, and the stores of its copies. The last column says what to do when a copy is lost.

   docs/storage-map.md

   | Data | Original in | Copies in | If a copy is lost |
   |---|---|---|---|
   | Customers | PostgreSQL `customers` | none | not a copy: restore it from a backup |
   | Tickets and messages | PostgreSQL `tickets`, `messages` | ticket vectors (made from the text) | not a copy: restore it from a backup |
   | Files that customers attach | object storage (Azurite on your computer), one object per file | none | not a copy: back up the object storage separately |
   | File details (name, size, checksum, owner) | PostgreSQL `attachments` | none | restore it from a backup; check that every file still exists |
   | Policy documents | PostgreSQL `documents` (each version) | none | restore it from a backup |
   | AI runs (model, prompt version, tokens, cost) | PostgreSQL `ai_runs` | none | restore it from a backup |
   | Ticket vectors | none: they are a copy | PostgreSQL `ticket_embeddings` (pgvector) | make them again from the ticket text with the same model and version: `python -m scripts.embed --rebuild` |
   | Similar-ticket index (HNSW) | none: it is a copy | PostgreSQL index on `ticket_embeddings` | build it again: `REINDEX INDEX ...` or drop and create it |
   | Cached answers (if you add a cache) | none: they are a copy | the cache | nothing to do: the next request fills it again |

   Only the table is shown. The file starts with one sentence about what it is. It ends with two rules. Every copy says what it was made from. A copy is never the only place where a fact is kept.

3. Run the tests:

   python -m pytest

   The last line is still 34 passed. The new files add no test: the map is a document, and the script is optional.

Your project now matches the snapshot end-of-m01. Two rows of the map describe things that do not exist yet: the HNSW index arrives in Module 5, and the project has no cache. A map can show the plan as well as the present, as long as it says which is which.

Lose one store at a time

The map answers a hard question quickly: what does each loss cost? Try every store.

Explore

Lose one store: what is gone for good?

Lost for good or broken

0 of 8

Made or filled again

2

Not touched

6

Lost: the vector table and its index. Nothing is lost for good: 2 pieces of data are copies, made or filled again from the originals.

Lost: the vector table and its index

Data	Result	What happens	Its original
A ticket	Not touched	Not in this store.	PostgreSQL tickets, messages
A message	Not touched	Not in this store.	PostgreSQL tickets, messages
A photo a customer attached	Not touched	Not in this store.	object storage (Azurite on your computer), one object per file
A policy document	Not touched	Not in this store.	PostgreSQL documents (each version)
An AI run	Not touched	Not in this store.	PostgreSQL ai_runs
A ticket vector	Made again	Made again from the ticket text: python -m scripts.embed --rebuild	copy, made from the ticket's subject and body, with the same model and version
The similar-ticket index (HNSW)	Made again	Built again from the vectors, after they are made again.	copy, made from the ticket vectors
A cached answer	Not touched	Not in this store.	copy, made from the database's answer

What happens to each piece of help-desk data when one store is lost, by the rules of the project's docs/storage-map.md. 'Lost for good' means that only a backup brings it back.

Text description of the explorer

The explorer has one control: which store is lost. The choices are the vector table and its index, the object storage, the cache, and the whole PostgreSQL database. A table shows each of the eight pieces of data, its result, what happens, and where its original is. Above the table, three numbers count the pieces that are lost for good or broken, made or filled again, and not touched.

The vector table and its index: nothing is lost for good. The ticket vectors are made again from the ticket text, and the HNSW index is built again from the vectors. The other six are not touched.

The object storage: the photos are lost for good, and the attachments rows point to files that are missing. Only a backup of the object storage brings them back.

The cache: nothing is lost. The next request fills it again.

The whole PostgreSQL database: five are lost for good or broken. Tickets, messages, policy documents and AI runs are gone. The photos survive in object storage, but their details rows are gone, so nobody knows whose files they are: orphan files. The vectors and the index are gone too, because they live in PostgreSQL. They can be made again only after a backup brings the tickets back.

Losing the vectors loses nothing for good. Losing PostgreSQL loses almost everything, and it even breaks the files in another store. The originals deserve the backups, and Module 5 tests them.

Make the vectors again (optional)

This part needs an embedding model, which is large. It needs about 1 GB of packages (sentence-transformers 6.1.0 and torch 2.14.1) and a 470 MB model, downloaded once. The model runs on your computer’s CPU. No account or key is needed. If you skip it, read the outputs, and then put the vectors back from the data files with python -m scripts.load --reset.

1. Install the optional packages into the project’s virtual environment. On Linux, install the CPU version of torch first, with the command in requirements-embed.txt.

   python -m pip install -r requirements-embed.txt

2. Your vectors are still deleted. Make sure the original is complete first: python -m scripts.load --reset loads the small data again, with the vectors that came with it. Then compare those stored vectors with new ones, made now from the ticket text:

   python -m scripts.embed --check

   Embedded 200 tickets with intfloat/multilingual-e5-small in 5.3 s.
   Largest difference from the stored vectors: 4.99e-08 (the same, apart from rounding).

   The first run also downloads the model, so it takes longer. A warning about HF_TOKEN and “unauthenticated requests” is harmless: the model is public, and no token is needed.

3. Read the result. For each of the 200 tickets, the script made the 384 numbers again and compared them with the stored ones. The largest difference of all 76,800 numbers is 0.0000000499. The stored vectors were rounded to 7 significant digits when they were saved, so that is rounding, not a change. The copy matches its original.

4. Now make the loss and the rebuild. In psql, run TRUNCATE ticket_embeddings; again, and quit. Then rebuild from the original:

   python -m scripts.embed --rebuild --all

   Embedded 200 tickets with intfloat/multilingual-e5-small in 5.1 s.
   Stored 200 vectors of version e5-small-v1.

5. Check the rebuilt copy, and run the similar-ticket query of the setup lesson in psql again:

   python -m scripts.embed --check

   Embedded 200 tickets with intfloat/multilingual-e5-small in 4.9 s.
   Largest difference from the stored vectors: 4.99e-08 (the same, apart from rounding).

   The query returns T-30002, T-30050, T-30168 and T-30064 again: the same four “Charged twice” tickets as before the loss.

Your times will differ. On the course’s computer, --check took 5.3 s for 200 tickets. The large data’s 10,000 vectors took 24.8 s. In Module 5, the HNSW index over them takes 0.36 s to build. A rebuild costs seconds or minutes, never data.

Without the model, python -m scripts.load --reset also brings the vectors back. That is a different thing: it restores a saved copy of the vectors, which came with the data. A rebuild makes them from the original. Both work here. Only the rebuild still works after the model, the text or the version changes.

Common mistake

The rebuild stores 0 vectors

Symptom: after a loss of all vectors, python -m scripts.embed --rebuild prints this, and the similar-ticket query still returns unrelated tickets:

Embedded 0 tickets with intfloat/multilingual-e5-small in 4.3 s.
Stored 0 vectors of version e5-small-v1.

Cause: --rebuild makes again the vectors that exist: it embeds the tickets that already have a vector. That rule keeps the large data at its newest 10,000 tickets. After a total loss, no ticket has a vector, so the list is empty. The copy was the only record of which tickets it covered.

Diagnose: SELECT count(*) FROM ticket_embeddings; gives 0 before and after the rebuild.

Fix: run python -m scripts.embed --rebuild --all. It embeds every ticket in the original. In the small data, every ticket has a vector, so --all gives back exactly the 200. In the large data, --all would make 300,000 vectors, not the newest 10,000. The rule “which tickets get a vector” belongs in the original’s terms, not only in the copy.

When to keep a copy, and what it costs

Keep a derived copy when it answers a question that the original cannot answer fast or at all. The vectors answer “similar tickets”. The HNSW index makes that fast. A cache saves a repeated, slow answer.

Each copy has a price:

• A recipe to keep. The model name, its revision, the prefix and the version must be recorded with the copy. Module 5 adds an embedding version and a checksum of the text to every vector.
• Time to rebuild. Seconds for 200 tickets, minutes for many thousands. Plan where the search answers from while a copy is rebuilt.
• Copies can go stale. When a ticket’s text changes, its vector is out of date until someone makes it again. Module 5 finds such “stale vectors”.
• Deletion must reach every copy. Grace asks: “If a customer asks us to delete their data, can we?” Yes, but only if you delete the original and every copy. The map lists them all. Module 5 does it.

Module practice: a document assistant

Larkfield’s next project is a document assistant. Staff upload policy PDFs, and the assistant answers questions from them with citations, as in RAG: Building AI That Uses Your Data. For each piece of data, choose a store, and say whether it is an original or a copy. For a copy, say what it is made from.

1. The uploaded PDF files.
2. Each document’s details: title, owner, version, who may read it.
3. The text extracted from each page.
4. The chunks: pieces of the text, about 300 words each.
5. A vector for each chunk.
6. The vector index over the chunk vectors.
7. The conversations: each question, the answer and the cited chunks.
8. Each model call: model, tokens, cost, time.
9. Cached answers to the ten most common questions.

Show an example answer

Data	Store	Original or copy
PDF files	object storage	original
Document details and permissions	relational database	original
Page text	relational database	copy, made from the PDF by a text extractor (keep its version)
Chunks	relational database	copy, made from the page text by a chunking rule (size and overlap)
Chunk vectors	vector index (for example pgvector)	copy, made from the chunks by an embedding model and revision
Vector index	vector index	copy of a copy, built from the chunk vectors
Conversations	relational database	original: an answer that was given cannot be made again
Model calls	relational database	original: tokens and cost happened once
Cached answers	cache	disposable copy; it must expire when a document changes

Three things to check in your own answer. First, the chain of copies starts at an original that has a backup: the PDFs and their details. Second, the conversations are originals, although a model wrote the answers: the same question asked again can get a different answer. Third, a new document version must reach every copy below it: page text, chunks, vectors, the index and the cache.

Module check

You have finished Module 1 when you can show these on your own project:

• python -m scripts.load --reset prints the small counts, and python -m pytest prints 34 passed.
• docs/storage-map.md is in your project, and you can name the original of every row.
• Completion check: you can explain how a derived index is rebuilt from authoritative data. Use the ticket vectors: the original, the recipe, the command, and how you check the result. With the optional model, show it: python -m scripts.embed --check prints a largest difference of about 4.99e-08.

Reset. If your project is broken, rename it and copy the snapshot end-of-m01. Keep your .env. In ~/projects:

mv ticket-api ticket-api-old
cp -R nextia-learning-labs/C20/snapshots/end-of-m01 ticket-api
cp ticket-api-old/.env ticket-api/

On Windows, in PowerShell, in ~/projects: Rename-Item ticket-api ticket-api-old, then Copy-Item -Recurse nextia-learning-labs\C20\snapshots\end-of-m01 ticket-api, then Copy-Item ticket-api-old\.env ticket-api\. Make the virtual environment again, as in the setup lesson, and run python -m scripts.load --reset.

Knowledge check

Check your understanding

In your own words: what is a source of truth, and what makes a copy safe to lose? Use the ticket vectors as the example.

Show the answer

The source of truth is the one store that keeps the original of a fact; if stores disagree, it wins. For a ticket, that is the tickets table in PostgreSQL. A copy is safe to lose only while its original and its recipe still exist. The ticket vectors are made from the ticket text by intfloat/multilingual-e5-small at a fixed revision, with the prefix "query: ". Lose them, and python -m scripts.embed --rebuild --all makes them again from the same text with the same recipe.

Check your understanding

Completion check: explain how the similar-ticket index (HNSW, from Module 5) is rebuilt from authoritative data after the whole vector table is lost. Name each step and how you check it.

Show the answer

The authoritative data is the ticket text in tickets. The vectors and the index are both derived. First make the vectors again from the text, with the recorded model, revision and prefix: python -m scripts.embed --rebuild --all. Use --all, because after a total loss no ticket has a vector to rebuild. Then build the index again from the vectors: drop and create it, or REINDEX. Check with python -m scripts.embed --check: the largest difference is about 4.99e-08. Then check a known search: T-30002’s nearest tickets are T-30050, T-30168 and T-30064.

Check your understanding

Predict: Tomás stores each ticket's text only as metadata in a vector database, and the team changes the embedding model. What must happen to the vectors, and where does the text come from?

Show the answer

Every vector must be made again. Vectors of two models do not mix: Module 5 measured only 0.4% of the right neighbours when they did. To embed again, you need the text. In Tomás’s design, the text exists only next to the old vectors. So the vector database is now the original, in a store that is regularly deleted and rebuilt. One wrong rebuild loses the tickets for good. Keep the text in PostgreSQL, and treat the vectors as a copy.

Recap

• The source of truth keeps the original of a fact (authoritative data). Derived data is a copy made by a known recipe; a disposable copy, such as a cache entry, is not even rebuilt.
• Three rules. Every fact has one original, with a backup. Every copy records what it was made from. A copy is never the only place of a fact.
• A constructed loss of all vectors broke the similar-ticket search, with no error, and lost no data: 200 tickets remained.
• python -m scripts.embed --check proved that the stored vectors match their original: a largest difference of 4.99e-08. --rebuild --all made them again from the ticket text.
• docs/storage-map.md, scripts/embed.py and requirements-embed.txt are in the project, which now matches end-of-m01. python -m pytest prints 34 passed.

You have finished Module 1. In Module 2, the originals get rules: keys, constraints and transactions that keep the data in PostgreSQL correct.

Next: Entities and relationships