LLM/RAG regressions

What you will be able to do

Catch the failures that only a language-model service has. Answers quote an old document or have no text, the provider fails, or it changes the model without telling you. Each one leaves every classic SLI green.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Proxy signals and delayed labels, from Delayed labels.
• Answers to questions with the shop’s documents and the shared cache, from Keys and invalidation. Retrieval quality, from Retrieval metrics.
• The model version in the release manifest, from LLM-specific configuration. Alert rules in Grafana, from Quality and costs.

What you need on your computer

• Your project after the last lesson: python -m pytest prints 207 passed.
• Docker running PostgreSQL and Valkey, and the simulator, the API and the worker in three terminals.

The brochure rack

A tourist office answers visitors’ questions from a rack of leaflets, and it always names the source: “It’s in the bus leaflet.”

One Monday, the bus company changes its timetable, and nobody replaces the leaflets. All week, the staff give clear, confident answers with a source. The answers are wrong, and nothing looks wrong. The same week, the bus company gives the route to a new driver and tells nobody. The buses still run, on another road.

A language-model service has a rack of leaflets too, and a provider that can change the driver.

The problem: right-looking answers, wrong facts

Larkfield’s website answers customers’ questions with the shop’s documents. In this lesson you read the stale-index scenario, a practice run that we made on purpose. During 15 practice minutes, customers asked about 1 question a second, and the service triaged 450 tickets. The answers to tickets are real recordings; the changes are constructed:

• At +4.0 minutes, Grace’s team changed the delivery-policy document: free delivery from 75 dollars instead of 100. Nobody rebuilt the copy of the documents that the answers search.
• At +7.3 minutes, the simulated provider started to answer with another model version, gpt-6-luna-2026-11-04, behind the same deployment name. It went back at +9.8 minutes.

For 1.3 minutes, 22 of 443 answers cited the old version of the delivery policy. For 2.4 minutes, 72 of 450 tickets were triaged by a model version that no evaluation had tested.

The answer-time, accept and draft SLIs stayed at 100%. There was no error, no timeout and no empty answer.

The idea: five regressions that need their own signals

A regression is a change that makes the service worse than it was. RAG, retrieval-augmented generation, means that the service first searches documents (retrieval), then a model writes the answer from them (generation). Five regressions are special to this kind of service:

1. Stale sources. The index that the answers search is older than the documents. The answer is complete and cites a source, but the source is out of date. This is the leaflet in the rack.
2. Citation failures. The answer cites a version that is not current, or it cites no source at all.
3. Answers without text. The model used all its tokens and wrote nothing (finish_reason: length). You met it in “last week”.
4. Tool and provider errors. Calls fail, time out or are given back by the breaker. The work waits, or it is lost.
5. Model-version changes. The provider runs another model behind the same name. Nothing fails, but the answers may change. This is the new driver.

The path of an answer, and where each regression enters. The index is a copy, so it can be older than the documents; the provider controls the model.

Text description of the diagram

Grace’s team edits the documents. The answer index is a copy, made by python -m scripts.reindex; until someone runs it, the copy keeps version 4. A question searches the index for the 2 best documents, and the provider’s model writes the answer from them, with their versions. Three signals watch the path: freshness at the index, provider and model signals at the model, and citation failures at the answer.

A tiny example: one stale source

Three documents, with their current versions, and the index, which still has version 4 of the delivery policy. One answer uses two documents. Count its stale sources:

Run Python in your browser

Count the stale sources of one answer

The box prints that the index is behind on delivery-policy, the answer’s sources delivery-policy@4 and large-item-delivery@2, and stale_sources: 1. Rebuild the index, or change the documents that the answer uses, and run it again.

Python code (you can change it)# The current version of each document, and the copy that the answers search. documents = {"delivery-policy": 5, "large-item-delivery": 2, "returns-policy": 4} index = {"delivery-policy": 4, "large-item-delivery": 2, "returns-policy": 4} behind = [doc for doc in documents if index[doc] != documents[doc]] print("index behind the documents on:", behind) # One answer used the two best documents of the index. used = ["delivery-policy", "large-item-delivery"] source_versions = [f"{doc}@{index[doc]}" for doc in used] stale_sources = sum(index[doc] != documents[doc] for doc in used) print("source_versions:", source_versions) print("stale_sources:", stale_sources) # Try: rebuild the index (index = dict(documents)) and run again, # or let the answer use ["returns-policy", "large-item-delivery"].

index behind the documents on: ['delivery-policy']
source_versions: ['delivery-policy@4', 'large-item-delivery@2']
stale_sources: 1

Two different signals came out, and they answer two questions:

• Source freshness asks: is the index behind the documents? Here yes, on one document. Every answer from this index is at risk.
• A citation failure asks: did this answer use an old version? Here yes, one source. An answer that uses only returns-policy and large-item-delivery would be fine, although the index is behind.

The retrieval index in the project

Before this module, answers searched the documents table directly. Migration 019 adds answer_index: a copy of the current documents, with a search column and an index version. The index version is a fingerprint of all the document versions at the time of the build. It is the same idea as the prompt fingerprint of Module 2.

Now every answer does three things:

• It searches answer_index, not documents.
• It returns source_versions, for example delivery-policy@4, as the index has them.
• It compares each source with the current version in documents, and returns stale_sources: how many are old. It does this every time, also for an answer that comes from the cache.

scripts/update_document.py publishes a new version of a document, as Grace’s team does. It does not rebuild the index. scripts/reindex.py rebuilds it, and --check only compares the two versions. The cache key now contains the index version, so a rebuilt index also gives new cache keys.

Make the index stale, and see it

Start the simulator, the API and the worker again, one terminal for each, as in the last lesson. On Windows, type curl.exe instead of curl, and put each command on one line without the \.

1. Check that the index is up to date. The command exits with 0 when it is.

   python -m scripts.reindex --check

   Index version 75cf862216c3, documents version 75cf862216c3: the index is up to date.

2. Ask a question as customer C-0022:

   curl -X POST http://127.0.0.1:8000/v1/answers -H "Content-Type: application/json" \
     -H "X-Customer-ID: C-0022" -d '{"question": "How much is delivery and when is it free?"}'

   {"answer":"The document \"Delivery policy\" has the details. Your latest ticket is T-30171 (resolved): Question about the garden hose (25 m).","scope":"customer:C-0022","sources":["delivery-policy","large-item-delivery"],"source_versions":["delivery-policy@4","large-item-delivery@2"],"stale_sources":0,"cached":false}

   The answer’s text is the simulator’s template, not a real model answer. The sources are real: version 4 of the delivery policy, and no stale source.

3. Publish a new version of the delivery policy. This is a practice change: free delivery from 75 dollars.

   python -m scripts.update_document delivery-policy --replace "free at 100" "free at 75"

   delivery-policy: version 5 is current (was 4). The answers' index still has the old text: python -m scripts.reindex --check

4. Check the index again, and print its exit code. On Windows, type $LASTEXITCODE instead of $?.

   python -m scripts.reindex --check
   echo $?

   Index version 75cf862216c3, documents version cfc422a3b860: the index is OLDER than the documents.
   1

   The exit code 1 lets a script or a CI job stop on it.

5. Ask the same question again:

   curl -X POST http://127.0.0.1:8000/v1/answers -H "Content-Type: application/json" \
     -H "X-Customer-ID: C-0022" -d '{"question": "How much is delivery and when is it free?"}'

   {"answer":"The document \"Delivery policy\" has the details. Your latest ticket is T-30171 (resolved): Question about the garden hose (25 m).","scope":"customer:C-0022","sources":["delivery-policy","large-item-delivery"],"source_versions":["delivery-policy@4","large-item-delivery@2"],"stale_sources":1,"cached":true}

   The cache gave the same answer ("cached":true), and the check still found the old source: stale_sources is 1. To a customer, this answer looks exactly like the first one.

6. Rebuild the index:

   python -m scripts.reindex

   Rebuilt the index from 34 current documents: index version cfc422a3b860 (was 75cf862216c3).

7. Ask once more:

   curl -X POST http://127.0.0.1:8000/v1/answers -H "Content-Type: application/json" \
     -H "X-Customer-ID: C-0022" -d '{"question": "How much is delivery and when is it free?"}'

   {"answer":"The document \"Delivery policy\" has the details. Your latest ticket is T-30171 (resolved): Question about the garden hose (25 m).","scope":"customer:C-0022","sources":["delivery-policy","large-item-delivery"],"source_versions":["delivery-policy@5","large-item-delivery@2"],"stale_sources":0,"cached":false}

   The new index version made a new cache key ("cached":false), and the answer now cites delivery-policy@5.

Read the stale-index run

ops rag reads the same signals from saved telemetry. Each row is 1.5 practice minutes:

python -m ops rag data/telemetry/stale-index.jsonl.gz --windows 1.5

443 answers to questions, 450 triaged tickets, 0 jobs to retry, from +0.0 min to +15.0 min
minutes  answers  old sources  no source  index behind  tickets  other model  provider errors
-------  -------  -----------  ---------  ------------  -------  -----------  ---------------
0-1.5         42  0 (0%)               0             0       44            0                0
1.5-3         45  0 (0%)               0             0       45            0                0
3-4.5         44  8 (18%)              0            14       45            0                0
4.5-6         44  14 (32%)             0            26       45            0                0
6-7.5         41  0 (0%)               0             0       45            3                0
7.5-9         45  0 (0%)               0             0       45           45                0
9-10.5        45  0 (0%)               0             0       45           24                0
10.5-12       45  0 (0%)               0             0       45            0                0
12-13.5       46  0 (0%)               0             0       45            0                0
13.5-15       46  0 (0%)               0             0       45            0                0
15-16.5        0  -                    0             0        1            0                0
Model versions in the answers: gpt-6-luna-2026-09-22 378, gpt-6-luna-2026-11-04 72 (the release expects gpt-6-luna-2026-09-22)

Read the columns as the tiny example:

• index behind: 14 + 26 = 40 answers came from an index that was older than the documents. That is source freshness.
• old sources: 8 + 14 = 22 of them actually cited the old delivery policy. That is the citation failure. The other 18 used documents that had not changed.
• other model: 3 + 45 + 24 = 72 tickets were triaged by gpt-6-luna-2026-11-04.
• no source and provider errors stayed at 0 in this run.

• Answers that cite an old version
• Tickets answered by another model version

The stale-index run per half minute: answers with old sources from the document change (+4.0) to the rebuild (+5.3), then tickets from another model version from +7.3 to +9.8. A constructed practice run.

Text description of the chart

The x axis shows the 15 practice minutes. The first line counts answers that cite an old version. It is 0 until minute 4, then 8, 8 and 6 per half minute, and 0 after the rebuild. The second line counts tickets from another model version. It is 0 until minute 7, then 3, then 15 per half minute until minute 9.5, then 9, and 0 from minute 10.5. About 15 answers and 15 tickets arrived in each half minute.

Grafana’s alert rules saw both changes. The old-sources alert fired 0.26 minutes after the document change. The model-version-changed alert fired 0.44 minutes after the provider’s change. In the scenario, a scripted on-call step read the first alert and rebuilt the index at +5.3 minutes. The course’s 10 alert rules, and what each one did in this run (from the run’s record of Grafana’s alerts):

Rule	What it watches	In the stale-index run
old-sources	Answers quote old document versions	Fired at +4.27 min, 0.26 min after the document change
model-version-changed	The provider answered with another model version	Fired at +7.78 min, 0.44 min after the provider’s change
no-answer	Answers without text	Quiet
cost-over-budget	AI cost per 1,000 tickets over budget	Quiet
triage-slow	Triage calls slow (p95 over 8 s)	Quiet
reviews-disagree	Agents disagree with the assistant (quality SLO)	Quiet (no reviews in this run)
accept-fast-burn	Tickets refused (accept SLO burns fast)	Quiet
draft-fast-burn	Tickets without a valid draft (draft SLO burns fast)	Quiet
drafts-late	Drafts are late (oldest ticket waits over 2 minutes)	Quiet
provider-circuit-open	AI provider failing (circuit breaker open)	Quiet

Only old-sources and model-version-changed fired. The other eight rules stayed quiet: no error, no slow call, no empty answer.

Write the alert for the signal you mean

The first version of model-version-changed counted every chat call. But the answers to questions report the simulator’s chat-small (simulated) as their model, so the alert fired at +0.4 minutes, before any change. The rule now counts triage calls only (app_prompt_version!=""). An alert that fires on something you did not mean teaches people to ignore it. Test a rule on a period where you know that nothing happened.

A model version that changes under you

A provider can update the model behind a deployment name. In LLM-specific configuration, the manifest got expected_version for this. Each answer reports the model that answered, and the telemetry keeps it as gen_ai.response.model.

The simulator can play that change. Try it on your own service:

1. Tell the simulator to answer with another model version. This is a practice change, made on purpose.

   curl -X POST http://127.0.0.1:8300/admin/model-version -H "Content-Type: application/json" \
     -d '{"model_version": "gpt-6-luna-2026-11-04"}'

   {"model_version":"gpt-6-luna-2026-11-04"}

2. Send 6 practice tickets, and wait until the worker has done them:

   python -m scripts.practice --tickets 6 --rate 2

3. Set the version back:

   curl -X POST http://127.0.0.1:8300/admin/model-version -H "Content-Type: application/json" \
     -d '{"model_version": "gpt-6-luna-2026-09-22"}'

4. Stop the API and the worker with Ctrl+C. Then read your own telemetry:

   python -m ops rag telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl --windows 1

   3 answers to questions, 26 triaged tickets, 0 jobs to retry, from +0.0 min to +2.4 min
   minutes  answers  old sources  no source  index behind  tickets  other model  provider errors
   -------  -------  -----------  ---------  ------------  -------  -----------  ---------------
   0-1            0  -                    0             0       20            0                0
   2-3            3  1 (33%)              0             1        6            6                0
   Model versions in the answers: gpt-6-luna-2026-09-22 20, gpt-6-luna-2026-11-04 6 (the release expects gpt-6-luna-2026-09-22)

   Your telemetry files hold this lesson and the last one: the 20 tickets of Delayed labels, your 3 questions and the 6 new tickets. Your minutes and your number of questions can differ.

The 6 tickets triaged by gpt-6-luna-2026-11-04 are in the “other model” column, and the last line counts the versions. The one old source is your second question in Make the index stale, and see it, before the rebuild. Nothing failed: every ticket got a valid answer, which is why only this signal can show the change.

The fix depends on the cause. If the provider moved the deployment on purpose, the new version needs its own evaluation, and a new release manifest that expects it. If not, ask the provider to pin the version. Until then, the alert is the only sign.

Answers without text and provider errors

The two other regressions are in earlier saved runs:

• Answers without text. python -m ops summary data/telemetry/release-full.jsonl.gz prints Tickets triaged: 2,400 (ok: 2,203, no_answer: 187, dead_letter: 10). 186 came from prompt v3, and one from v2’s single cut-off recording (T-65450). That one also makes the no-answer alert fire on a healthy release: noise, which Module 6 deals with.
• Provider errors. python -m ops rag data/telemetry/outage.jsonl.gz --windows 2 shows 3,138 jobs to retry while the simulated provider was down: 3,102 given back by the open breaker and 36 retries. Module 6 turns this into a runbook.

When to watch these signals, and what they cost

Sam asks, “Why not rebuild the index every night, and stop worrying?”

Tomás answers: “A nightly rebuild leaves up to a day of old answers after a change, and nobody hears if it fails. Rebuild after each change, and keep the check.”

Watch freshness and citations whenever answers come from documents that change, and make reindex --check part of each document change. Watch the model version on every call to a hosted model. Do not alert on them where nothing changes: a fixed, versioned document set needs only the check at release time.

What they cost: one database lookup per answer, one attribute per call, and a few alert rules. The real cost is the follow-up: each stale-source alert needs a rebuild, and each model change needs an evaluation.

Common mistakes

Common mistake

update_document says the text is not there

Symptom: python -m scripts.update_document delivery-policy --replace "free at 100" "free at 75" exits with code 1:

Version 5 of delivery-policy has no text 'free at 100'.

Cause: you ran it twice. Version 5 already says “free at 75”.

Fix: nothing is broken. To start again from version 4, run python -m scripts.load --reset. It loads the documents and rebuilds the index.

Common mistake

The answer is right, but stale_sources is 1

Symptom: after a document change, an answer has "cached":true and "stale_sources":1, although you think the cache is fine.

Cause: the cache key contains the index version, not the documents version. Until the rebuild, the cache gives the old answer, and the service correctly reports its old source.

Fix: rebuild the index with python -m scripts.reindex. Do not empty the cache instead: the index would still be old, and the next answer would cite version 4 again.

Module practice: design two alerts

This module’s practice task: design alerts for a stale index and for a classifier whose decisions get worse. First, a worked example with every step visible. Then you design the second alert yourself.

Worked example: the stale-index alert

An alert has seven parts. Here they are for old sources:

Part	Value	Why
Signal	answers whose sources are not current (stale_sources > 0)	the citation failure that customers get
Where	the counter app_answers_served_total{stale="true"} in Grafana; ops rag on saved telemetry	both read the same events
Rule	any such answer in the last 10 minutes, no wait (for: 0s)	one old source is already wrong; the answer-sources SLO allows only 1%
Severity	SEV-3	wrong but not dangerous; the answer-sources SLO burned at 4.97 in the run
Owner	Tomás	he owns the retrieval index
Action	the runbook stale-index.md: run python -m scripts.reindex, then --check	the fix is one command; a runbook is the written steps for one kind of failure, which Runbooks in Module 6 writes
Test	on the saved run: fired 0.26 minutes after the change; 0 alerts before it	a rule that is never tested is a guess

Add one prevention: after every document change, run python -m scripts.reindex --check, and treat exit code 1 as a failed step. The alert catches what the prevention misses.

Your turn: an alert for worse decisions

Design the alert that Priya needs for performance decay. It must fire in the policy-silent block of the drift run. It must not fire in the normal week, or in the canary run of Module 5, where the reviews look normal. You know from Delayed labels why the 30-minute rule stayed quiet.

1. Write the seven parts, as in the table above.
2. Choose the window as a number of reviews, and say why that number.
3. Test it with python -m ops labels on drift.jsonl.gz, normal-week.jsonl.gz and canary.jsonl.gz, with windows of 250 or 500 tickets (about 120 to 250 reviews in the drift run).
4. Say how late it fires after the policy-silent block starts, and what it cannot see.

Worked solution

Part	Value	Why
Signal	the share of reviews that disagree with the assistant’s team or needs-a-person decision	the quality SLO’s own measure
Where	ticket.reviewed events (app_tickets_reviewed_total); ops labels on saved telemetry	the review and the decision are saved together
Rule	more than 10% disagree in the last 139 reviews	139 gives a margin of about 5 points near 90%; old reviews leave the window as new ones arrive
Severity	SEV-3; SEV-2 if it stays over 10% for a day of tickets	the SLO burns, but customers get valid answers
Owner	Priya	she owns model quality
Action	runbook quality-drop.md: split the bad reviews by slice and release; review that slice by hand; open an experiment record; no automatic change	the cause decides the fix
Test	see below

Test with the windows that ops labels gives:

python -m ops labels data/telemetry/drift.jsonl.gz --windows 250

3,000 tickets, 1,445 reviewed (48%); review delay median 120 s, p95 121 s
window       tickets  reviewed  agrees (95% interval)  no answer  to a person  French
-----------  -------  --------  ---------------------  ---------  -----------  ------
     1-250       250       122  95.1% [89.7%, 97.7%]        0.0%        23.6%    3.6%
   251-500       250       125  93.6% [87.9%, 96.7%]        0.4%        29.7%    6.0%
   501-750       250       112  93.8% [87.7%, 96.9%]        0.0%        24.4%    5.2%
   751-1000      250       115  98.3% [93.9%, 99.5%]        0.0%        22.8%    6.8%
  1001-1250      250       114  97.4% [92.5%, 99.1%]        0.0%        25.2%   23.2%
  1251-1500      250       124  95.2% [89.8%, 97.8%]        0.0%        24.8%   25.6%
  1501-1750      250       121  95.9% [90.7%, 98.2%]        0.0%        18.8%   23.2%
  1751-2000      250       136  94.9% [89.8%, 97.5%]        0.4%        27.3%   27.6%
  2001-2250      250       118  85.6% [78.1%, 90.8%]        0.4%        26.1%    2.8%
  2251-2500      250       122  86.9% [79.8%, 91.8%]        0.0%        25.6%    2.8%
  2501-2750      250       113  81.4% [73.3%, 87.5%]        0.8%        23.0%    3.2%
  2751-3000      250       123  87.0% [79.9%, 91.8%]        0.0%        24.4%    4.8%
Reviews arrive later: the last windows have fewer. For a margin of about 5 points on an agreement near 90%, a window needs 139 reviews.

The first 8 windows agree 93.6% to 98.3%, so no alert. The 4 windows of the policy-silent block agree 81.4% to 87.0%, so the rule fires on the first of them. With --windows 500, every window of normal-week.jsonl.gz that has reviews agrees 92.5% to 96.4%, and every window of canary.jsonl.gz 91.8% to 96.7%. The rule stays quiet in both.

The course’s authors checked the exact rule, review by review, on the saved runs:

Run	Last 139 reviews over 10%?	Last 100 reviews over 10%?
Drift	first at review 992, practice minute 19.1 (14 of 139 disagree)	first at review 988, minute 19.0
Normal week	never	never
Canary	never	once, at minute 18.9: a false alarm
“Last week”	first at minute 7.6, 2.6 minutes after prompt v3	first at minute 7.7

So 139 reviews fire 2.4 practice minutes after the policy-silent block starts. The first reviews take 2 minutes to arrive, and then about 20 reviews are enough. With 100 reviews, the window is noisier and gives a false alarm in the canary run. At Larkfield’s speed, multiply by about 210: a few hours, if reviews arrive at the practice share (an estimate).

What it cannot see: anything before the reviews arrive, a drop too small for 139 reviews (90% → 88%), and why. The next step is to read the disagreeing tickets.

Completion check: drift alone does not prove a mistake

Omar posts in the team’s chat: “Priya’s drift check says LARGE change on French tickets. The assistant is broken for French customers. Roll back to the old release today!”

Write Grace’s reply in five sentences or fewer. Use the saved drift run as evidence. Say what the drift check shows, what it does not show, what the reviews show, and what the team does next.

Worked solution

“The drift check is right that the tickets changed: French went from about 5% to 25%, a PSI of 0.343 on the language guess. Drift says that the inputs changed, not that the decisions are wrong. The agents’ reviews of those same tickets agree with the assistant on 95.8% [93.6%, 97.2%] of 495 reviews, against 95.1% before. So there is nothing to roll back, and a rollback would change a working release. We will review more French tickets on purpose for a week. Separately, the policy-silent tickets are at 85.3% with no language drift, and Priya opens an investigation for those.”

A good answer keeps the two claims apart: the inputs changed (PSI) and the decisions are worse (agreement with an interval). It quotes both, and it does not use drift as proof of a mistake.

Knowledge check

Check your understanding

In the stale-index run, 40 answers came from an index that was behind, and 22 cited an old version. Explain the difference, and say which one customers felt.

Show the answer

“Index behind” is source freshness: the index was older than the documents, so every answer during those minutes was at risk. “Old sources” is the citation failure: the 22 answers that actually used the changed delivery policy. The other 18 used documents that had not changed, so they were right. Customers felt the 22. The freshness signal is still useful, because it fires before a customer asks the question that hits the old document.

Check your understanding

Predict: the provider changes the model version, and the new version is as good as the old one. Should the model-version alert fire? What should the team do?

Show the answer

Yes, it should fire. The alert says “something that no evaluation has tested is answering customers”, not “the answers are worse”. The team runs the release’s evaluation on the new version, through the gate of Module 5. If it passes, they make a new release manifest that expects the new version, and the alert goes quiet. If they never check, the next change may not be as good.

Check your understanding

Practical check: show that your service reports an old source, and that the rebuild clears it. Which outputs prove it?

Show the answer

After python -m scripts.update_document delivery-policy --replace "free at 100" "free at 75", python -m scripts.reindex --check prints the index is OLDER than the documents and exits with 1. The same question then returns "source_versions":["delivery-policy@4", …] and "stale_sources":1. After python -m scripts.reindex, --check says up to date, and the answer cites delivery-policy@5 with "stale_sources":0.

Recap

This module watched behaviour that changes after a release. Now you can:

• Tell data drift, prediction drift, concept drift and performance decay apart. Test input drift with PSI (how much) and a chi-square test (how sure), with a written threshold.
• Show that drift alone does not prove a mistake. In the drift run, French tickets drifted with no loss, and policy-silent tickets lost quality with no drift.
• Plan delayed labels: reviews saved next to the decision, a feedback timeline and proxy signals for the gap. Use 139 reviews for a margin of about 5 points.
• Watch the regressions of a RAG service, each with its own signal: stale sources, citation failures, answers without text, provider errors and model-version changes.
• Design an alert in seven parts, and test it on saved runs before you trust it.

Next, Module 5 uses this evidence before a change reaches customers: an evaluation gate, then a canary that catches prompt v3 on a few tickets.

Next: Evaluation gates