Post-incident review of a public outage

What you will be able to do

Read a real outage report the way an operations team reads its own incidents. You will rebuild the timeline from two published reports, check it with a small script, and write the review in this course’s format. Then you will ask what it means for Larkfield: which alerts would fire, which dependency we share, and the one change we would make.

Case studyPortfolio piece

Dataset

Published reports: Cloudflare and Google Cloud (12 June 2025), OpenAI (11 December 2024)

Licence

Our timelines: CC0 1.0. The script: MIT. The reports: © their publishers (read them online; short quotes only)

Time

About 2.5 hours

Runs in

A browser, VS Code and Python 3.10+

Skills used

• Timeline from reports
• Time zones and approximate times
• Dependency chain
• Fail open or fail closed
• Post-incident review
• Runbook review
• Alert mapping
• Options under one measure
• Runbook change

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

This case study is written work, not a notebook. You type the reports’ facts into a CSV, one sourced row per event, and a small script checks and draws them. The rest is reading and writing, in Larkfield’s own formats. The finished project is one complete answer.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• SLOs, the error budget and the severity levels, from Service objectives.
• Owners and who is called first, from Ownership and support.
• The runbook format (detection, triage, mitigation, escalation, recovery checks), from Runbooks.
• The breaker, the queue and the status message in a provider outage, from Provider and dependency outages. The breaker itself comes from the scaling course’s Failure containment.
• The review format and PIR-001, from Post-incident learning.

The page teaches one new idea: fail open or fail closed, what a service does when a check cannot get its data.

What you need on your computer

• A browser, to read the reports.
• Python 3.10 or newer. The script uses only the standard library, so you need no virtual environment and no install.
• For one optional check in “Map it onto Larkfield”: your course project at the end of Module 6, with the saved telemetry in data/telemetry/.

Tested on macOS with Python 3.12.13, from fresh copies of the two zips. The Windows commands are written, not tested.

The situation

A large outage is in the news. Many websites and apps failed for about two and a half hours, and AI products failed with them. At Larkfield, nothing broke: the ticket service uses neither Cloudflare nor Google Cloud.

Sam asks: “It was not our outage. Why spend a morning on it?” Grace answers with the course’s question in a new form: “Could this happen to us, and would we know?”

Kwame owns the provider contracts, so he owns this review. Amira owns the service and its runbooks, and Grace’s agents get the tickets that need a person. The decision belongs to Amira and Grace: should Larkfield change its provider-outage runbook, and how?

What you will build

You will build the folder public-outage-pir/:

File	What it holds
data/cloudflare-2025-06-12.csv	the timeline of both companies, one row per event, each with its source
review/PIR-EXT-001-cloudflare-2025-06-12.md	the review, in the format of PIR-001
review/larkfield-mapping.md	our alerts and runbooks, the shared dependency, three options under one measure, the one change
review/provider-outage-runbook-change.md	Larkfield’s provider-outage runbook with that change
data/openai-2024-12-11.csv, review/PIR-EXT-002-openai-2024-12-11.md	the same method on a new report

PIR-EXT means a review of an external incident. Larkfield numbers its own reviews PIR-001, PIR-002 and so on, so the two series never mix.

Set up the project

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder public-outage-pir.

2. In VS Code, choose File > Open Folder and open public-outage-pir. Then open a terminal: Terminal > New Terminal.

3. Check your Python version:

   macOS / Linux

   python3 --version

   Windows (PowerShell)

   python --version

   You need 3.10 or newer. The page shows python3; on Windows, type python instead.

4. Open the three reports in your browser. Keep them open in tabs: you will read them several times.

The data

The evidence is three reports. No company published data files about these outages, so the reports are the data.

• Cloudflare, Cloudflare service outage June 12, 2025: impact by product, a timeline in UTC, the cause and the actions.
• Google Cloud, incident ow5i3PPK96RduMcb1SsW: status updates, a “mini incident report” and a full report.
• OpenAI, API, ChatGPT & Sora Facing Issues: write-up: the new input at the end.

Source and licence

The three reports are copyright of Cloudflare, Inc., Google and OpenAI. None has an open licence. So this page links them and quotes only short passages, with credit, for review and discussion. The starter does not contain them. The CSVs and the review documents are ours: facts (times and numbers) in our own words, under CC0 1.0. The script is under the MIT licence.

Kwame compared three reports. The Cloudflare outage of 12 June 2025 crossed two companies, as Larkfield depends on a model provider that depends on others. OpenAI’s outage is a model provider’s own, so it is the new input. Cloudflare’s outage of 18 November 2025 stayed inside one company: a “go further” option.

The data dictionary

Each row of the CSV is one event of the incident:

Column	What it means	Example
time_utc	when it happened, converted by you to UTC	2025-06-12T17:49Z
stated	the time as the report wrote it	10:49 PDT, within 10 minutes
approx	yes if the report says “~” or “within”; day if only the day is known; else no	no
org	the company: one lane of the chart	Google Cloud
kind	change, trigger, impact-start, detection, diagnosis, escalation, mitigation, communication, impact, recovery or impact-end	impact-start
label	a short label for the chart, 24 characters or fewer	Impact starts
event	the event in your own words, not the report’s	API requests fail with 503 errors
source	the report’s URL and the section that says it	https://status.cloud.google.com/… (Mini Incident Report)

The kind values follow the runbook’s sections, so the timeline already answers the runbook’s questions.

First checks

The starter’s CSV has four rows: the two impact starts, Cloudflare’s first detection and Cloudflare’s impact end. Run the script on it:

python3 draw_timeline.py data/cloudflare-2025-06-12.csv

The output:

Checks of data/cloudflare-2025-06-12.csv
  4 events, 2 organizations: Google Cloud, Cloudflare
  stated times converted and checked: 4 of 4 (the others are relative, like 'within 10 minutes', or a day)
  approximate times: 0
  no problem found

Timeline (UTC; minutes from the organization's impact start; ~ = approximate):
  06-12 17:49      +0  Google Cloud  impact-start  Impact starts
  06-12 17:52      +0  Cloudflare    impact-start  Impact starts
  06-12 17:52      +0  Cloudflare    detection     WARP team declares
  06-12 20:28    +156  Cloudflare    impact-end    Impact ends

Phases (minutes from the impact start to the first event of each kind):
  Google Cloud: impact starts 17:49 UTC
  Cloudflare: impact starts 17:52 UTC; detection +0, impact-end +156
  Cloudflare: impact lasted 2 h 36 min (156 min) by this timeline

Wrote out/cloudflare-2025-06-12.svg and out/cloudflare-2025-06-12-narrow.svg

What this tells us. The script read the stated column, converted it and compared it with time_utc. Google writes US Pacific time; in June that is UTC−7, so 10:49 PDT is 17:49 UTC. Now make the most common mistake on purpose: type Google’s time as if it were UTC (2025-06-12T10:49Z). The script stops:

  PROBLEM line 2: stated '10:49 PDT' is 17:49 UTC, but time_utc says 10:49
...
Fix the problems first; no chart was written.

That one mistake would put Google’s outage seven hours before Cloudflare’s, and the link between them would disappear.

Kwame’s first reading finds three more things that change the review:

• Cloudflare does not name the provider. The post says only “a third-party cloud provider”. The link to Google Cloud is our reading, from the times (Google’s impact starts 3 minutes earlier) and the press. Neither report says it, so the review labels it each time.
• The stated duration does not match the table. Cloudflare writes that the outage “lasted 2 hours and 28 minutes”. Its own timeline runs from 17:52 to 20:28: 2 h 36 min. The report does not say which two events its number counts between.
• Google gives three durations. The status page says 10:51 to 18:18 PDT. The mini report says 10:49 to 13:49 PDT, “3 hours”. The full report says “up to ~2h 40 mins” for its largest region. Each one counts something different.

So the review fixes one rule: a duration always runs from impact start to impact end, as the CSV defines them.

Frame the problem

The unit is one event, with its time and its source. The questions are the review’s sections: the timeline, the dependency chain, what users saw, the causes and actions, and what it means for Larkfield.

The rules of evidence. Every fact about the outage comes from the reports, with its source. Our own reasoning is labelled “our reading”. Larkfield’s numbers come from the course’s saved runs and fixed facts, and every assumption is named.

Explore what the review needs

The timeline

Fill in the rest of the CSV. Say each event in your own words. Google gives some times only as “within 2 minutes” or “~25 minutes”: count them from 17:49 UTC, the mini report’s start, and mark them approx. The finished CSV has 28 rows. Run the script again:

python3 draw_timeline.py data/cloudflare-2025-06-12.csv

The end of the output:

Phases (minutes from the impact start to the first event of each kind):
  Google Cloud: impact starts 17:49 UTC; detection ~+2, diagnosis ~+10, mitigation ~+25, recovery +119, impact-end +180
  Google Cloud: impact lasted 3 h 00 min (180 min) by this timeline
  Google Cloud: the last recovery is 7 h 29 min after the impact start
  Cloudflare: impact starts 17:52 UTC; detection +0, escalation +14, mitigation +51, recovery +151, impact-end +156
  Cloudflare: impact lasted 2 h 36 min (156 min) by this timeline

Open out/cloudflare-2025-06-12.svg in your browser. It is the chart below.

The two companies' timelines, drawn by draw_timeline.py from the finished CSV. Minutes count from each company's own impact start; ≈ marks a time that the report gives as approximate.

Text description of the diagram

Google’s impact runs from 17:49 to 20:49. Cloudflare’s runs from 17:52 to 20:28, inside it. Google’s triage, root cause and fix all come within about 40 minutes. Cloudflare’s events spread across its band. It detects at once and escalates within 29 minutes. Four mitigations follow, and recovery comes only with the storage.

What this tells us. Read the lanes against each other:

• Detection was fast in both. Google was triaging within about 2 minutes. At Cloudflare, a team saw its own product fail at +0; the first alert came at +13.
• Escalation followed the shared cause. Cloudflare joined the team incidents into one at +14. It raised the priority to P0 at +29, “as severity of impact becomes clear”.
• A fix is not a recovery. Google’s fix was out at about +40, but its AI serving product recovered last, at +449. Restarting tasks overloaded their own database, a “herd effect”. The cure, waits with jitter, is in the scaling course’s Delivery and retries.
• Cloudflare could not fix the cause. It reduced the harm product by product, and recovered when the provider did.

The dependency chain

The timeline says when. The chain says how the failure travelled.

Think of a power cut in one street. The bakery’s oven stops, and so do the bank’s cash machine and the traffic light. Different businesses share one thing that none of them controls.

At Cloudflare, that thing is Workers KV, a key-value store that many products use for configuration, identity and files. Its central store depended on the outside cloud provider.

How the failure travelled. Solid arrows are what the reports say; the dotted arrow is our reading. Orange boxes are AI products.

Text description of the diagram

A bad quota policy crashed Google’s Service Control, and its API requests failed. Linking that to Cloudflare’s third-party provider is our reading. Workers KV then failed every read that missed its cache. The products that read KV failed next: Access, and through it WARP, Gateway and the Dashboard; Workers AI and AutoRAG; AI Gateway. Products that share only part of the storage failed less.

The chain explains the numbers. The report gives one number for each product, each in its own measure:

• Reads its configuration or data from Workers KV
• Shares part of the storage, or is served mostly from a cache (our grouping)

The impact on nine Cloudflare products, as the report states it. The measures differ (a share of logins, a share of requests, a peak or an average), so read the order, not small differences.

Text description of the chart

Access: 100% (identity-based logins that failed). Workers AI: 100% (inference requests that failed). Pages: 100% (peak error rate (about 100%)). AI Gateway: 97% (peak error rate). Workers KV: 90.22% (requests that failed in the window). D1: 22% (peak average error rate). Durable Objects: 22% (peak average error rate (SQLite-backed)). Workers for Platforms: 10% (peak error rate (about 10%)). Workers: 2% (peak error rate (about 2%)). The first five read their configuration or data from Workers KV on each request and failed almost completely. The last four share only part of the storage, or serve most requests from a cache, and failed far less.

What this tells us. How much a product failed depended on how it used the store. Products that read KV on each request failed almost completely. KV itself failed 90.22%, not 100%, because some reads came from its cache. Two facts matter most for an AI service:

• Workers AI failed every inference request, though its models were fine. It “depends on Workers KV for distributing configuration and routing information” (Cloudflare).
• AI Gateway failed up to 97% of requests. Teams use a gateway to switch between model providers. It failed through its own configuration store. A fallback that shares a dependency with the thing that it protects fails with it.

Fail open or fail closed

The report says that Access “is designed to fail closed”. This is the one new idea of the page.

Think of two doors in a power cut. A fire door opens, so that people can leave: it fails open. A bank vault stays locked: it fails closed. Each protects what matters more for that door.

The problem. Every check needs data: a login needs an identity, a triage step needs the model’s answer. When the data cannot be read, the check must do something. If nobody chose in advance, the code chooses by accident.

The idea. Decide, for each check, what happens when its data is missing:

• Fail closed: refuse. Access refused every identity login; rules stayed safe, and users waited.
• Fail open: let it through. Cloudflare’s “kill switches” let visitors pass its challenge pages. The cost, in the report: a valid token could be used more than once.
• Fail to a person: hand the work to someone who can decide without the data, as the course’s TRIAGE_FALLBACK=person does.

Three answers to one question: what does a check do when it cannot get its data?

Text description of the diagram

Failing closed refuses, as Access did. Failing open lets everything through, as Turnstile’s kill switch did, at a cost. Failing to a person hands the work to someone who can decide without the data: Larkfield’s person mode, later on this page.

At Larkfield, the API accepts every ticket in a provider outage: it fails open for intake. The triage step waits for the model: it fails closed for decisions. When to choose which: fail closed where a wrong pass does harm, and fail open where blocking does more harm. Fail to a person where a person can decide. Each choice has a cost; name it.

The review, in the course’s format

Now write review/PIR-EXT-001-cloudflare-2025-06-12.md from templates/review-template.md, which has the sections of PIR-001. The review is blameless: it names systems and conditions, never a person.

1. The header: the reports, their severity (P1, then P0) and ours: SEV-1, because many customers could not be served.
2. The summary, in four or five sentences, with one “our reading” sentence about the unnamed provider.
3. What the reports show: each number with what it counts. “90.22% of requests” and “peaked at 97%” are different kinds of number.
4. The timeline from your CSV, with the durations and the events they count between.
5. The response, section by section of a runbook.
6. Causes and contributing conditions, the cause first; then actions, each with its owner, and how we know it is fixed.

Here is the finished review’s response section, the heart of the comparison with Larkfield:

Runbook section	What the reports show
Detection	Cloudflare: a team saw its own product fail at +0 and declared an incident; the first alert came at +13. Google: triage within 2 minutes.
Triage	Cloudflare joined the team incidents into one at +14, once it saw the shared cause. Google found the root cause within about 10 minutes.
Mitigation	Google: a prepared “red button” turned the failing code path off. Cloudflare could not fix the store. It reduced the harm in each product: kill switches, degraded rules, less load on KV.
Escalation	Cloudflare: P1 at +14, P0 at +29. Google’s first public status post came at +57, late because the status page ran on the failing systems.
Recovery checks	Cloudflare: SLOs back at their level before the incident. Google: region by region, slowed by its own restarts.

The causes, in short. At Google: a quota check without error handling or a feature flag, data copied to every region within seconds, and restarts without randomized backoff. At Cloudflare: one store for many products, on one provider during a migration.

What this tells us. Neither report shows evidence after the recovery, or dates for its actions. We cannot check from outside that the actions are done. We can only change how much our own service depends on them.

Map it onto Larkfield

Now write review/larkfield-mapping.md. The dependency we share in kind is the hosted model provider. Every ticket needs one triage call to get a team, a needs-a-person decision and a draft. We cannot see the provider’s own dependencies, as Cloudflare’s customers could not see Workers KV’s provider. So we plan for a long provider outage, whatever its cause.

The same shape at Larkfield: one hosted model provider, with dependencies that we cannot see. The alerts fire within minutes; the new arrow is the change of this review.

Text description of the diagram

The ticket is accepted at once and waits in the queue, and the status page says that replies take longer. The breaker stops the calls to the failing provider, whose own dependencies Larkfield cannot see. Two alerts fire within minutes. The new arrow, person mode, sends tickets to the agents after 5 minutes of an open breaker.

Which alerts would fire. The course recorded a practice outage of the provider: 5 minutes, caused on purpose, saved in outage.jsonl.gz. Its alerts are the evidence:

Alert	Fires?	When, after the outage began (Grafana)	Runbook
provider-circuit-open (SEV-2, Kwame)	yes	0.85 minutes	provider-outage
drafts-late (SEV-2, Amira)	yes	3.28 minutes	provider-outage
draft-fast-burn	no	it counts finished tickets, and none finish	bad-release
accept-fast-burn	no	the API accepted every ticket	tickets-refused

If you have your course project, you can see both alerts and the breaker in the practice outage’s timeline:

python -m ops incident data/telemetry/outage.jsonl.gz

It lists the outage at +5.1 minutes, provider-circuit-open at +5.7 and drafts-late at +8.2, by the project’s own rules. Grafana’s rules fired at +5.9 and +8.4.

What this tells us. Detection would take under a minute, where Cloudflare’s first alert took 13. We have one dependency, and a breaker that watches it. The gap is in the objectives, not in the alerts. The draft SLO wants a valid draft within 10 minutes. In a long outage it breaks for every waiting ticket, and no alert counts them.

A baseline: the runbook as written

The provider-outage runbook was tested on the practice outage. The queue peaked at 315 tickets, the oldest ticket waited 354 seconds, and the draft SLO held. Its mitigation says “do nothing to the workers”, because the breaker keeps the tickets safe in the queue. Agents read the queue by hand for words such as fire, smoke, burn and child.

What does it do in an outage as long as Cloudflare’s? Larkfield gets 822 tickets a day, one of the course’s fixed facts. We use it as an average; busy hours would give more.

Run Python in your browser

A Cloudflare-length outage at Larkfield's rate

The box computes the tickets that arrive during a 156-minute outage, and those that wait more than 10 minutes. Then it prints how many of them need a person, and how much of the week’s draft budget one outage uses. Then try 5, 148 and 262 minutes.

Python code (you can change it)# Larkfield's average: 822 tickets a day (one week = 5,754 tickets). per_minute = 822 / (24 * 60) needs_person = 0.2513 # sent to a person in the normal week: 1,446 of 5,754 draft_budget = 0.01 * 5754 # the draft SLO (99%) allows 57.5 bad tickets a week outage = 156 # minutes: Cloudflare's impact, 17:52 to 20:28 UTC target = 10 # minutes: a valid draft within 10 minutes tickets = per_minute * outage late = per_minute * max(outage - target, 0) # arrived more than 10 minutes before the end print(f"tickets during the outage: {tickets:.0f}") print(f"wait more than {target} minutes: {late:.0f}") print(f"of those, need a person: {late * needs_person:.0f}") print(f"draft budget used: {late / draft_budget:.0%} of the week's {draft_budget:.1f}") # Try: outage = 5 (the course's practice outage), 148 (the stated # 2 hours 28 minutes) and 262 (the OpenAI outage at the end of this page).

The output:

tickets during the outage: 89
wait more than 10 minutes: 83
of those, need a person: 21
draft budget used: 145% of the week's 57.5

What this tells us. With outage = 5, no ticket waits more than 10 minutes, as in the practice run. At 156 minutes, about 83 tickets miss the draft SLO: one and a half weeks of budget in one afternoon. Worse, about 21 of them need a person, and only the hand scan can find them. One needs-a-person ticket that reaches nobody is SEV-1. (The estimate assumes that the queue drains at once; in the practice outage it took about 2 minutes.)

Alternatives, one measure

A fair comparison needs one measure for every option. The safety journey has no error budget, so Kwame picks a safety measure. It is needs-a-person tickets that wait more than 10 minutes before they reach the agents’ queue, in an outage of 156 minutes. He also shows the drafts and the cost, so that no option hides its price.

Option	Needs-a-person tickets that wait more than 10 minutes	Tickets without a draft	What it costs
A. The runbook as written (the baseline)	about 21, unless the hand scan finds them	about 83	nothing new
B. A second model provider, switched to automatically	0 if it is truly independent; about 21 if it shares a dependency	0, or about 83	a second contract; its model must pass our release gate first; we cannot see its dependencies
C. Person mode: after 5 minutes of an open breaker, every waiting and new ticket goes to a person, until the breaker closes	0 (each waits at most about 5 minutes for the switch)	about 89: no drafts at all	agents sort about 89 tickets by hand, about 67 more than usual; every ticket gets priority 1

Option B sounds strongest, but the report warns against trusting it blindly: AI Gateway, built for this kind of switch, failed up to 97%. A second model at the same provider shares every dependency of the first. And a new model is a new release, so it must pass the gate of Evaluation gates first.

The recommendation

Kwame recommends option C, person mode after 5 minutes of an open breaker, and Amira and Grace accept it. It needs nothing outside our service, it reuses the existing fallback, and it protects the journey with no error budget: safety. It does not save the draft SLO; nothing that we control can, while the provider is down.

The change goes into review/provider-outage-runbook-change.md, with each new line marked (new). The mitigation section gains:

- (new) Person mode. When the breaker has been open for 5 minutes, the worker sends every
  waiting and new ticket to a person (needs_human true, priority 1, outcome sent_to_person),
  with no team and no draft. It stops when the breaker closes.
- (new) In person mode every ticket has priority 1, so the hand scan for safety words is how
  safety tickets come first.
- (new) Do not switch to another model or provider during the outage. A model that has not
  passed the release gate is an unreviewed release.

The actions: Amira builds person mode, done when a 20-minute practice outage shows no needs-a-person ticket waiting over 10 minutes. Grace plans her team for about 90 extra tickets in a long outage. Kwame keeps option B as a later experiment. One question stays open: should a long outage become SEV-1 after a fixed time, as Cloudflare’s became P0 at +29?

End to end on new input

A method is useful only if it works on the next report. Take the OpenAI write-up: a hosted model provider down, Larkfield’s exact dependency. Type its timeline into data/openai-2024-12-11.csv (PST is UTC−8) and run the same script:

python3 draw_timeline.py data/openai-2024-12-11.csv

The end of the output:

Phases (minutes from the impact start to the first event of each kind):
  OpenAI: impact starts 23:16 UTC; detection -3, diagnosis +0, mitigation +11, recovery +80, impact-end +262
  OpenAI: impact lasted 4 h 22 min (262 min) by this timeline

The new input, drawn by the same script: OpenAI's outage of 11 December 2024, from its own write-up.

Text description of the diagram

Alerts fired 3 minutes before customers were affected, and the cause was known at once. The first cluster recovered only at plus 80, and the impact ended at plus 262.

What this tells us. Detection was early, and the cause was known at once. But the fix needed the control plane that had failed: a “locked out effect”, in the report’s words. For Larkfield, the same two alerts would fire. Set outage = 262 in the box above. About 144 tickets wait more than 10 minutes, 36 of them for a person, and the draft SLO uses 250% of a week’s budget. With person mode, none of the 36 waits more than 10 minutes, and agents sort about 150 tickets by hand. The recommendation holds, at a larger cost to Grace’s team. PIR-EXT-002 is the short review.

Insights and limits

• Detection was not the problem in any of the three outages. The time went into mitigation and recovery. Larkfield’s gap, too, was what to do in the second hour.
• A fix and a recovery are two events. Record both, and prove recovery with evidence.
• The numbers are the providers’ own, each counted in its own way, by the company that failed. Larkfield’s numbers are estimates from an average rate.
• Two outages are not a reliability record. Do not rank providers from them.

Responsible use

• Quote, do not copy. The reports are not ours. Link them, credit them, and quote only short passages, as this page does.
• No blame. The reports name their authors. A review names systems and conditions, never the people who wrote the report or made the change.
• Label every inference. “Our reading” is not a fact. The link between Google and Cloudflare is very likely, and still stated by neither.
• Do not test someone else’s service to check a report. Read; do not probe.
• Keep times honest. Convert time zones, mark approximate times, and say which events a duration counts between.

Make it a portfolio piece

This project shows an employer that you can turn a long public report into a checked timeline, a blameless review and one costed runbook change.

What to put in the repository:

public-outage-pir/
  README.md          what it is, how to run the script, the three reports with links
  dataset.md         the sources, the licences, the changes you made, the limits
  LICENSE            MIT for the script; CC0 for your CSVs and documents
  draw_timeline.py   checks and draws a timeline CSV
  data/              one CSV per incident, each row with its source
  review/            the reviews, the mapping and the runbook change
  out/               the charts

Push it to GitHub with the timeline chart and the options table at the top of the README, and say which statements are “our reading”.

In an interview, say it in two sentences. “I rebuilt the timeline of the Cloudflare outage of June 2025 from two companies’ reports, and a script checked every time zone. Then I mapped it onto an AI ticket service and recommended one runbook change, with its cost.”

Go further:

• Review Cloudflare’s outage of 18 November 2025, where a machine-learning feature file crashed the proxy.
• Build person mode in your own course project, and prove it with a 20-minute practice outage.

Knowledge check

Check your understanding

Cloudflare's report says that the outage lasted 2 hours and 28 minutes. Your timeline says 2 h 36 min. Which one goes into the review, and what do you write?

Show the answer

Write the one that you can check: 2 h 36 min. Say what it counts: from the impact start (17:52 UTC) to the impact end (20:28 UTC) in the report’s own table. Add that the report states 2 hours and 28 minutes. It does not say which two events that number counts between. Do not guess a reason. The rule of the review: a duration always runs from impact start to impact end.

Check your understanding

Tomás proposes: 'Add a second model at our provider as a fallback, and switch to it automatically when the breaker opens.' What do you answer, with evidence from the report?

Show the answer

A second model at the same provider shares every dependency of the first, so it would fail in the same outage. The report shows the pattern: AI Gateway, built to route between providers, failed up to 97% because of its own configuration store. A truly separate provider might help. But its model must pass the release gate first, and we cannot see its dependencies. So it is a later experiment, not the change for now.

Check your understanding

Practical check: run the script on your Cloudflare CSV. Which output lines prove that your timeline is consistent, and which numbers does the review quote for Cloudflare?

Show the answer

The checks must end with no problem found. Every row with a clock time must be converted and checked: 23 of 28 in the finished CSV. The phase line for Cloudflare gives detection +0, escalation +14 and impact-end +156. The review quotes the P1 at +14, the P0 at +29 from the timeline, and the duration of 156 minutes. Google’s line shows its approximate times with ~, which the review marks with ≈.

Recap

• You rebuilt one timeline of two companies from their reports, with one sourced row per event, and a script checked every time-zone conversion.
• Cloudflare detected at once and escalated at +14 and +29. It could not fix the cause. It reduced the harm product by product until the provider recovered, after 2 h 36 min.
• The chain explains the numbers: products that read Workers KV on each request failed almost completely, Workers AI and AI Gateway among them.
• A check must fail open, fail closed or fail to a person, and each choice has a cost.
• Larkfield’s alerts would fire within minutes. Its runbook would leave about 21 needs-a-person tickets waiting in an outage of the same length.
• Person mode after 5 minutes of an open breaker brings that to 0. It costs the drafts and about 67 extra tickets for the agents. It held on new input: OpenAI’s outage of 4 h 22 min.

Next, the final assignment: an operations pack for a ticket classifier and document assistant.

Next: Final assignment