Recovery objectives

What you will be able to do

Decide how long the assistant may be down after the database is lost, and how much recent data it may lose. Then choose a backup plan that meets both targets, with numbers: the measured speed of a restore, and the people’s time as an assumption. You finish with a plan for a restore test, because only a test turns those assumptions into facts.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The failure F6 and the single point of failure, from Failure scenarios: The single point of failure.
• The quality attributes Q5 (auditability) and Q7 (recovery), from Quality attributes.
• What a backup and a restore drill are, from Recovery and retention in Databases and Storage for AI Applications. It helps, but it is not required.

What you need on your computer

• Your folder design-pack from the last lesson, with python -m pytest -q printing 42 passed and the file decisions/failure-modes.md.

You do not restore a real database in this lesson. Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A laptop dies at eleven at night

A student writes a long essay on a laptop. At eleven at night, the laptop stops and does not start again. Two questions matter now.

How much work is lost? It depends on the last copy. If the student copies the essay to a USB stick every night at two in the morning, 21 hours of work are gone. If the word processor saves to the cloud every 5 minutes, only 5 minutes are gone.

How long until the student can write again? That depends on other things: finding a second computer, signing in, downloading the file, checking that it opens. Maybe an hour. The download itself takes a few seconds. Most of the hour is people and machines getting ready.

The essay is due tomorrow at nine. So the student has two limits: “I can lose at most an hour of work” and “I must write again within two hours”. The nightly USB copy fails the first limit, however fast it is.

The problem: the nightly copy

Mei’s first plan is the one from the databases course: a nightly dump of the database to file storage, at 02:00. It is simple, and it works.

Omar asks a question. “Suppose the database is lost at 14:00 on a busy day. What do we lose?”

Everything since 02:00. That means every question, answer and upload of the morning, with their audit records. In the base case, the shops ask 1,465 questions a day, so a bad day costs up to a full day of answers. Q5 says that every answer is kept for a year, with its question, citations and document versions. A nightly dump can break Q5 with a single failure, even if the restore is fast.

The plan has a second gap. Nobody knows how long a restore takes, because nobody has tried one on this design. “It is a managed service” is not a time.

The idea: two targets, RTO and RPO

Each gap has a name and a target.

• RPO, the recovery point objective, is how much recent data you may lose, written as time. The “recovery point” is the moment that you go back to: the last copy. An RPO of 60 minutes means “the last copy is never more than 60 minutes old”.
• RTO, the recovery time objective, is how long the service may be down. It runs from the failure until the service works again and someone has checked it.

Q7 in the pack’s requirements sets both, as Omar and Mei agreed. The RTO is 240 minutes: back within 4 hours. The RPO is 60 minutes: lose at most 1 hour of new data.

RPO looks back from the failure to the last copy; RTO looks forward to the moment the service works again. A backup plan sets the first; people and machines set most of the second.

Text description of the diagram

A time line runs from left to right. On the left is the last copy that you can restore. Between it and the failure at 14:00 is the RPO: the data written in that time is lost. After the failure comes the RTO: the assistant is down until it is back online and checked. Two notes hang on the last copy. A nightly dump at 02:00 loses 12 hours on this day, and up to 24 hours. A point-in-time restore loses about 5 minutes, a value from the provider’s documentation that the team must test. A note on the right gives the targets of Q7: RPO 60 minutes and RTO 240 minutes.

A managed PostgreSQL service offers a second kind of backup: point-in-time restore. The service keeps copies and the database’s change log. So you can choose any moment in a retention period, such as “14:00 minus 5 minutes”. The restore makes a new server with the data of that moment. The provider’s documentation describes an RPO of minutes. The pack writes 5 minutes as an assumption with a range of 1 to 15, and the reason says “test it before you promise it”.

A tiny example: the recovery time, step by step

The recovery time is a sum of steps. Here is the base case, with every value visible. The step times are the pack’s assumptions; the copy speed is measured.

1. Detect: the availability alert fires: 15 minutes (assumed).
2. Decide: Mei confirms the loss and decides to restore: 15 minutes (assumed).
3. New server: the service creates the restored server: 20 minutes (assumed; the documentation gives no fixed time).
4. Copy the data: after 12 months, the database holds about 339 MB (the pack’s base case, with the four shops of the growth assumptions). The databases course restored 316 MB in 7.9 s on a laptop, about 40 MB per second (measured). So 339 ÷ 40 = 8.5 seconds, or 0.14 minutes.
5. Check: row counts per tenant and one test question per tenant: 20 minutes (assumed).

Total: 15 + 15 + 20 + 0.14 + 20 = 70 minutes. The copy of the data is 0.2% of it.

The parts of the recovery time in the base case: about 70 minutes. Copying the data (measured speed) takes 0.14 minutes; the other parts are assumptions about people and the provider.

Text description of the chart

A column chart with five columns. Detect: 15 minutes. Decide: 15 minutes. Server, a new server: 20 minutes. Copy, the copy of the data: 0.14 minutes, too small to see. Check: 20 minutes. The total is about 70 minutes. Only the copy is measured; the other four columns are assumptions.

This is the main lesson of the chart. A faster backup format or a bigger disk would change the smallest column. To shorten a recovery, shorten the people’s part: an alert that fires sooner, a written checklist, and a person who has done it before.

Larkfield: two backup plans in the pack

The design pack compares the two plans against Q7. The recovery time is the same for both, because both restore to a new server. The data loss is not.

Guided practice: recovery in the pack

1. In your folder design-pack, with the environment active, ask for the new command:

   python -m costmodel recovery

   This command arrives in Module 5. Download the end-of-module-5 snapshot or keep going.

2. Copy the recovery code and the module’s design file from the snapshot end-of-m05.

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m05
   cp "$S/costmodel/recovery.py" costmodel/
   cp "$S/design.toml" .

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m05"
   Copy-Item "$S\costmodel\recovery.py" costmodel\
   Copy-Item "$S\design.toml" .

   The new design.toml is the old one plus two sections at the end: [recovery] for this lesson and [growth] for the next one.

3. Open design.toml and find [recovery]. The two targets are plain values from Q7. The five step times are ranges (low, base, high), and [recovery.evidence] gives each one a reason that starts with assumed:.

   [recovery]               # Module 5, lesson 2
   rto_target_minutes = 240     # from requirements.md: back within 4 hours
   rpo_target_minutes = 60      # from requirements.md: lose at most 1 hour of new data
   detect_minutes = [5, 15, 60]
   decide_minutes = [5, 15, 30]
   new_server_minutes = [10, 20, 60]
   check_minutes = [10, 20, 40]
   point_in_time_rpo_minutes = [1, 5, 15]

4. Compare the two plans in the base case:

   python -m costmodel recovery

   Recovery (base): targets RTO 240 min, RPO 60 min; database 339 MB restores in about 8.5 s (measured rate, a laptop)
     plan                              RPO min  RTO min  meets both?
     nightly dump to file storage        1,440       70  no (RPO)
     managed point-in-time restore           5       70  yes
   Most of the recovery time is people and a new server, not copying data. Test the restore to know.

   The first line is the tiny example: 339 MB in 8.5 s. The nightly dump’s RPO is 24 hours (1,440 minutes), the worst case.

5. Look at the high case. It takes the high value of every assumption at once: a bigger database, a slow alert at night, a slow server.

   python -m costmodel recovery --scenario high

   Recovery (high): targets RTO 240 min, RPO 60 min; database 2,038 MB restores in about 50.9 s (measured rate, a laptop)
     plan                              RPO min  RTO min  meets both?
     nightly dump to file storage        1,440      191  no (RPO)
     managed point-in-time restore          15      191  yes
   Most of the recovery time is people and a new server, not copying data. Test the restore to know.

   Six times more data adds less than a minute. The people and the server add two hours.

6. Ask “what if”. Suppose that the first restore test shows a new server in 3 hours, not 20 minutes. In design.toml, change one line:

   new_server_minutes = [10, 180, 240]

   Run the base case and the high case again:

   python -m costmodel recovery

   Recovery (base): targets RTO 240 min, RPO 60 min; database 339 MB restores in about 8.5 s (measured rate, a laptop)
     plan                              RPO min  RTO min  meets both?
     nightly dump to file storage        1,440      230  no (RPO)
     managed point-in-time restore           5      230  yes
   Most of the recovery time is people and a new server, not copying data. Test the restore to know.

   python -m costmodel recovery --scenario high

   Recovery (high): targets RTO 240 min, RPO 60 min; database 2,038 MB restores in about 50.9 s (measured rate, a laptop)
     plan                              RPO min  RTO min  meets both?
     nightly dump to file storage        1,440      371  no (RPO, RTO)
     managed point-in-time restore          15      371  no (RTO)
   Most of the recovery time is people and a new server, not copying data. Test the restore to know.

   The base case still passes, with 10 minutes to spare. The high case fails the RTO. One unknown step decides the result, so the plan must measure it. Change the line back to [10, 20, 60].

7. Run the tests:

   python -m pytest -q

   ..........................................                               [100%]
   42 passed in 0.07s

   The module’s new tests arrive with the growth code in the next lesson, because one test file checks both.

Read the result

The nightly dump fails Q7, and it fails on the RPO, not on time. Its recovery takes the same 70 minutes as the other plan. But it can lose 1,440 minutes of answers, 24 times the target.

Point-in-time restore meets Q7 on paper. Its RPO of 5 minutes is far below 60, and its RTO of 70 minutes is far below 240. Even the high case, 191 minutes, passes. But two of its numbers are not measured: the 5-minute RPO comes from the documentation, and the 20-minute server is a guess. The output’s last line says what to do: “Test the restore to know.”

The cost. Point-in-time restore is part of the managed server’s price. The price list also bills backup storage above 100% of the provisioned storage. The design uses about 1% of its 32 GB, so the pack’s backup line is US$0.00. The decision is a point-in-time restore, tested every quarter. Kwame records it in Architecture decisions.

What the restore does not bring back. The database restore returns rows, not the files in file storage. The files need their own protection: soft delete and versions. That was the seventh failure of the last lesson.

Plan a restore test

A backup that nobody has restored is a hope, not a plan. The databases course made this point with a drill. A restore test restores the real backup to a new server, checks it, and records the time of every step. The design states the test as a plan with an owner and a date in each quarter.

A restore test turns the pack's recovery assumptions into measurements. The live server is never touched.

Text description of the diagram

Step 1: pick a moment to restore to and write it down, for example “restore to 10:15”. Step 2: restore the backup to a new server, while the live server keeps working. Step 3: check the restored data: the row counts per tenant, the time of the newest answer, and one test question for each tenant. Step 4: record the real minutes of each step, and how much data was lost: the gap between 10:15 and the newest answer found. Then decide: are the times within the RTO of 240 minutes and the RPO of 60 minutes? If yes, write the real times into design.toml and delete the test server. If no, change the plan and test again.

Three details make the test useful:

• The newest answer measures the RPO. If you restore to 10:15 and the newest answer in the copy is from 10:11, the real data loss was 4 minutes.
• A stopwatch measures the RTO. Write the time when each step starts. The pack’s assumptions then become measured values, with the test’s date as their source.
• The person changes. Mei runs the first test, and Kwame, her backup person, runs the second. A plan that only one person can follow is another single point of failure.

What the test costs. The test server runs for a few hours. A B1ms costs US$0.0185 an hour, so 3 hours cost about US$0.06 at list price. The people’s time is already in the pack. In Build versus buy, the managed database’s labour assumption counts 1 hour a month for reading alerts and testing a restore. That is more than one test a quarter needs.

When to set RTO and RPO, and what they cost

Set them for every store that holds data you cannot rebuild: here, the database with its answers and audit records. Ask the business, not only the engineers. Omar knows what an hour of lost answers costs the shops; Mei knows what a restore takes.

Do not set them for data that you can rebuild. The chunks and vectors come from the files: the worker can make them again. The answer cache can be empty after a restore without any loss. Only its hits are lost for a while.

What tighter targets cost. Each step down costs more. An RPO of minutes comes with the managed service. An RPO near zero, or an RTO of seconds, needs high availability: about US$288.50 a month of compute, as the last lesson showed. Write the target that the business needs, then find the cheapest plan that meets it.

Common mistakes

Common mistake

The code without its design file

Symptom: you copied recovery.py but not design.toml, and python -m costmodel recovery prints:

Input problem: design.toml has no [recovery] section (Module 5, lesson 2)

Cause: the calculator reads the targets and step times from design.toml. Your file is still the one from Module 4.

Fix: copy design.toml from the snapshot end-of-m05, as step 2 shows.

Common mistake

An RPO copied from the documentation into a contract

Symptom: Omar promises Bramble Books “at most 5 minutes of lost data”. In the first real restore, the gap is 12 minutes.

Cause: the 5 minutes was an assumption from the provider’s documentation, never tested on this design.

Fix: promise the target of Q7 (60 minutes), which the plan meets with a margin. Measure the real RPO in the restore test before you promise anything tighter.

Reset. If your pack breaks, copy the snapshot end-of-m04 as the snapshots README shows, then repeat the steps of this module.

Your turn: a tighter promise

Ines asks for a stronger promise: “Bramble Books may lose at most 1 minute of data.” Change rpo_target_minutes in design.toml to 1, run python -m costmodel recovery, and write two sentences for Ines: what the plan can promise, and what it would cost to promise 1 minute. Change the target back to 60 when you finish.

Show a worked solution

Recovery (base): targets RTO 240 min, RPO 1 min; database 339 MB restores in about 8.5 s (measured rate, a laptop)
  plan                              RPO min  RTO min  meets both?
  nightly dump to file storage        1,440       70  no (RPO)
  managed point-in-time restore           5       70  no (RPO)
Most of the recovery time is people and a new server, not copying data. Test the restore to know.

Both plans now fail on the RPO. To Ines: “Our plan loses at most about 5 minutes of data, a value from the provider’s documentation that we test every quarter. We promise 60 minutes, which we meet with a margin. A promise of 1 minute needs a standby server, about US$288.50 a month of compute against US$13.50 today. That is more than the whole budget of US$150 for all shops.”

See it end to endDesign review of an open document-question serviceCase study

The design review compares the two backup plans against a free service’s recovery targets. Its documents are public, so they can be loaded again, and only the question log can be lost.

Knowledge check

Check your understanding

Explain RTO and RPO in one sentence each, and say which one a nightly dump fails for the pack's targets.

Show the answer

RTO, the recovery time objective, is how long the service may be down after a failure, until it works again and someone has checked it. RPO, the recovery point objective, is how much recent data you may lose, written as time back to the last copy. A nightly dump fails the RPO: it can lose 1,440 minutes of data against a target of 60. Its recovery time, about 70 minutes, meets the RTO of 240.

Check your understanding

Predict: the database grows ten times bigger than the base case after 12 months. About how much longer does the recovery take? Use the measured restore speed.

Show the answer

About one minute longer. Ten times 339 MB is about 3,390 MB. At 40 MB a second, the copy takes about 85 seconds instead of 8.5: 76 seconds more. The recovery time goes from about 70 to about 71 minutes. The people and the new server, about 70 minutes of assumptions, do not change with the size.

Check your understanding

Mei says: 'We have point-in-time restore, so we are done.' What is missing, and how does the team get it?

Show the answer

Measurements. The 5-minute RPO comes from the provider’s documentation, and the new server’s 20 minutes and the people’s times are assumptions. Nobody has restored this design yet. The team needs a restore test every quarter. It restores to a chosen moment on a new server and checks the data. It records the real time of every step and the real data lost. Then the measured values go into design.toml. Also, the database restore does not bring back the files, so they need their own protection.

Recap

• RTO is how long the service may be down; RPO is how much recent data you may lose, as time. Q7 sets RTO 240 minutes and RPO 60 minutes.
• The recovery time is a sum: detect, decide, a new server, copy the data, check. Base case: 15 + 15 + 20 + 0.14 + 20 = about 70 minutes. Only the copy is measured (339 MB at about 40 MB a second: 8.5 s).
• A nightly dump fails Q7 on the RPO (1,440 minutes). A point-in-time restore meets it on paper: RPO 5 minutes, an assumption from the documentation.
• One unknown step can decide the result: a 3-hour server still passes in the base case, but fails the RTO in the high case.
• A restore test every quarter turns the assumptions into measurements, with a second person, for about US$0.06 of server time.
• python -m pytest -q prints 42 passed.

Next: Growth plan