Final assignment
- After the case studies
- about 3 hr
Where we are
In Module 8, we sent the same traces to a second tool and chose views for a team. Each module added one piece and tried it alone: traces, failure patterns, cost, the audit trail and alerts. In this assignment, you use them all at once. You operate a new release candidate for one replayed week, find what it broke from its telemetry alone, and hand in the evidence. We start like an operator on a new shift: with a fresh copy and Grace's question.
What you will be able to do
Operate release candidate 2.2 of Larkfield’s workflow for a replayed week, and find what it broke from its telemetry alone. Then deliver the observability pack: instrumentation with tests, a Collector configuration, a dashboard and alerts, one diagnosed incident with a cost report, and one audit answer. Every claim points to a command and its output.
Before you startWhat to know and to have ready
What you need to know
- Modules 1 to 8. Each part names the lessons that it uses. The certificate also asks for one case study.
What you need on your computer
- The course’s setup: Python 3.12, VS Code, three terminals, and the labs repository in
~/projects/nextia-learning-labs, as in Meet the live workflow. - The snapshot
end-of-m08, copied into a new folder. Your course project stays as it is. - The Collector binary
otelcol-contrib0.162.0 in~/projects/otelcol, from The Collector. Docker and Grafana are optional. - Git, so that
git diffshows every change. No account, key or money.
Tested on macOS (Apple silicon) with Python 3.12.13 and otelcol-contrib 0.162.0. The Windows and Linux commands are written, not run.
The situation: a release candidate for one week
Section titled “The situation: a release candidate for one week”Release 2.2 of the workflow runs as a candidate for one week of Larkfield’s tickets, and the team ships three more candidates during the week. Omar’s switch to chat-strong for hard tickets stays on, and Tomás tries the local model gemma3:4b again. Nobody tells you what each candidate changed.
You are the operator. At the end of the week, Grace asks one question: “Was any customer refunded twice, and who approved it?” Kwame asks a second one: “What does a resolved ticket cost now, and on which model?”
The candidate week is a practice week. The four candidates are constructed: we planted their problems. Every model decision is replayed from the agents course. The slow and broken services are simulated: practice incidents we caused on purpose. The week is seeded: your run IDs, counts and audit heads are ours. Omar’s budget of US$0.002 per resolved ticket is an assumption.
Text description of the diagram
The parts are not in the order of the items. You diagnose in Part 3 before you change the instrumentation in Part 4, so that you keep the “before” evidence.
Label every number in your pack: replayed, simulated, constructed, measured (your own run), price or assumption.
Set up a fresh copy
Section titled “Set up a fresh copy”-
Copy the snapshot into a new folder, and save its first state in Git:
Terminal window cd ~/projectscp -R nextia-learning-labs/C31/snapshots/end-of-m08 candidate-weekcd candidate-weekgit init -q && git add -A && git commit -qm "end-of-m08"Terminal window cd ~\projectsCopy-Item -Recurse nextia-learning-labs\C31\snapshots\end-of-m08 candidate-weekcd candidate-weekgit init -q; git add -A; git commit -qm "end-of-m08"Terminal window cd ~/projectscp -R nextia-learning-labs/C31/snapshots/end-of-m08 candidate-weekcd candidate-weekgit init -q && git add -A && git commit -qm "end-of-m08" -
Make a virtual environment, install
requirements-dev.txtand run the tests, as in Meet the live workflow. The last line says114 passed, 1 skipped, 1 deselected. -
Make an empty
REPORT.mdwith the sections of What to hand in. After each part, paste your commands, outputs and decisions into it.
Part 1: operate the candidate for a week
Section titled “Part 1: operate the candidate for a week”Uses Meet the live workflow, Service levels for agents and Agent failure patterns.
-
Run the candidate week. It replays 316 tickets in about 2 seconds and writes its telemetry, its audit trail and Amira’s log into
work/candidate:Terminal window python -m scenarios week --candidate --out work/candidatecandidate-week: 316 runs in 2 s -> work/candidateresolved 186, right by the course's labels 279Keep
resolved 186: the driver’s count of what happened. The telemetry must agree with it. -
Read the week one view at a time, each with
--data work/candidate:obs runs,obs slo,obs alerts,obs find all,obs cost --by releaseandobs audit verify. Thenpython -m obs check-redaction work/candidate. -
Predict, then check. How many runs does
obscount as resolved? Is that 186? -
Write a table in your report with one row per finding: what you see, from when, which release, the evidence, and who acts.
Do not open
candidate_week()inscenarios/week.pyuntil your table is written. It lists the planted problems.
What the course team saw
release runs resolved cost US$ per run per resolved success2.2.0-rc1 74 52 0.2396 0.00324 0.00461 70.3%2.2.0-rc2 62 16 0.2318 0.00374 0.01449 25.8%2.2.1-rc1 79 19 0.2316 0.00293 0.01219 24.1%2.2.2-rc1 101 61 0.2713 0.00269 0.00445 60.4%total 316 148 0.9743 0.00308 0.00658 46.8%run ticket started outcome model tools appr cost US$ secs model flagsR-1015-042 T-90207 Thu 17:09 waiting_approval 3 3 0 0.00084 4.2 chat-small -R-1015-043 T-90408 Thu 17:11 waiting_approval 2 1 0 0.01178 6.6 chat-strong -R-1015-048 T-90201 Thu 18:48 waiting_approval 3 2 0 0.01844 9.1 chat-strong -45 run(s), 0 flagged (last 3 shown)Checked 1999 span(s), 497 event(s), 178 metric line(s) in work/candidate.218 problem(s): text that must not leave the application: span chat chat-small: attribute gen_ai.input.messages (text) …obs counts 148 resolved runs, not 186, and 45 runs still wait for approval. Their decisions ran under 2.2.0-rc2 or 2.2.1-rc1, which seem to resolve a quarter of their tickets. Release 2.2.2-rc1 puts prompt text on its model spans. Alerts: loops and handoffs from Thursday, cost from Monday. obs find all: five gemma3:4b loops, no duplicate write. The audit trail verifies: 1,342 records, head 6ab932ef5352491b.
Part 2: the week through the Collector
Section titled “Part 2: the week through the Collector”Uses The Collector, Make it tamper-evident and controlled and Send the same traces to Langfuse.
Show that your Collector keeps the prompts of 2.2.2-rc1 out, samples traces but never audit events, and sends to two destinations. Without Docker, the standalone configuration writes files in place of Tempo, Prometheus and Loki; the audit store is the second destination.
-
In terminal 1, start the audit store’s door with its own file:
python -m auditstore serve --store work/collector-audit/audit.jsonl. -
In terminal 2, start the Collector, as in The Collector:
Terminal window mkdir -p work/collector-outOUT_DIR=work/collector-out AUDIT_STORE_URL=http://127.0.0.1:8313 ../otelcol/otelcol-contrib --config collector/standalone.yamlTerminal window New-Item -ItemType Directory -Force work\collector-out | Out-Null$env:OUT_DIR = "work/collector-out"; $env:AUDIT_STORE_URL = "http://127.0.0.1:8313"..\otelcol\otelcol-contrib.exe --config collector\standalone.yamlTerminal window mkdir -p work/collector-outOUT_DIR=work/collector-out AUDIT_STORE_URL=http://127.0.0.1:8313 ../otelcol/otelcol-contrib --config collector/standalone.yaml -
In a third terminal, send the week to the Collector as fast as it can go. It also writes the application’s own copy into
work/candidate-otlp:Terminal window python -m scenarios week --candidate --otlp --speed 1000000 --out work/candidate-otlp -
Wait 15 seconds for the sampling decisions. Stop the Collector and the door with Ctrl+C. Then compare what left the application with what left the Collector:
Terminal window python -m obs check-redaction work/candidate-otlppython -m obs check-redaction work/collector-outpython -m obs runs --data work/collector-outpython -m obs audit verify --audit work/collector-audit/audit.jsonl -
Count the audit records on both sides, and the audit events in the Loki file (
work/collector-out/logs.jsonl). On Windows, use(Get-Content FILE).CountandSelect-String. -
In your report, give a reason for each processor and route: redaction, tail sampling, the audit route and each exporter. Say which traces the sampling must always keep.
What the course team’s Collector let out
Checked 1999 span(s), 497 event(s), 178 metric line(s) in work/candidate-otlp.218 problem(s): text that must not leave the application:Checked 1173 span(s), 497 event(s), 1 metric line(s) in work/collector-out.No text attribute, no e-mail address and no card number found.208 run(s), 5 flaggedThe audit trail is intact: 1342 records, the chain and the anchor match (head 32f85d7348e75efe).The application sent 1,342 audit records, the store holds 1,342, and the Loki file holds none. Tail sampling kept 1,173 of 1,999 spans, from 208 runs. The 10% sample varies, so your counts and the store’s head differ. With Docker, grafana/compose.yaml mounts the same configuration.
Part 3: the refund that obs cannot see
Section titled “Part 3: the refund that obs cannot see”Uses Read a run, Search runs with TraceQL, Tool calls, errors and retries and From a trace to metrics and logs.
obs find duplicate_write finds nothing, and unreconciled-writes is quiet. Do not stop there. Diagnose one incident with trace evidence: we use the refund timeouts; the 45 waiting runs are a second choice.
-
Read the write counter, not the runs. Which refunds timed out on Wednesday, and did a reconcile follow?
Terminal window python -m obs metrics larkfield.writes --data work/candidate --where larkfield.write.outcome=timeout --day wedpython -m obs metrics larkfield.reconciles --data work/candidate --day wedlarkfield.writes: Write calls by outcome, increase per exported interval, larkfield.write.outcome=timeout, wedWed 16:00-17:00 +2 tool.name=request_refund example: ? (trace 5f4710a6)Wed 17:00-18:00 +1 tool.name=request_refund example: ? (trace 3457d66b)2 interval(s) with an increase, 2 with an example tracelarkfield.reconciles: Checks of an uncertain write (did it happen?), increase per exported interval, wed0 interval(s) with an increase, 0 with an example trace?: the example trace has no run ID. -
Save this short script as
follow.py. It starts from a run ID or the start of a trace ID. Then it follows a proposal to the approval that names it, and the approval to its writes, by event ID:"""Follow a run's authority in the audit trail by event IDs: its proposal, the approval that names the proposal,the writes that name the approval, and the close of the same trace.python follow.py AUDIT_FILE RUN_ID_OR_TRACE_ID"""import jsonimport syspath, target = sys.argv[1], sys.argv[2]records = [json.loads(line)["event"] | {"seq": json.loads(line)["seq"]} for line in open(path, encoding="utf-8")]ids = {e["event_id"] for e in records if e["run_id"] == target and e["type"] == "proposal"}ids |= {e["action"]["proposal_event_id"] for e in records if e["trace_id"].startswith(target) and e["type"] == "approval"}traces = set()for e in records:if e["action"].get("proposal_event_id") in ids or e["authority"].get("approval_event_id") in ids:ids.add(e["event_id"])traces.add(e["trace_id"])for e in records:if e["event_id"] in ids or (e["type"] == "run_closed" and e["trace_id"] in traces):a = e["action"]what = {"proposal": lambda: f"{a['outcome']} (rule {a['rule']}): " + ", ".join(f"{x['tool']} {x.get('amount', '')} on {x['order_id']}" for x in a["actions"]),"approval": lambda: f"{a['decision']} by {e['actor']['id']} after {a['wait_s']} s","write": lambda: f"{a['effect']['tool']} {a['effect']['amount']} on {a['effect']['order_id']}: "f"{a['outcome']} (key {a['idempotency_key']})","run_closed": lambda: f"{a['outcome']}, writes done {a['writes_done']}"}.get(e["type"], lambda: a)()print(f"{e['seq']:>5} {e['type']:<10} run {e['run_id'] or '(none)':<10} trace {e['trace_id'][:8]} {what}") -
Start from the metric’s trace, then read the run that it names in two more views:
Terminal window python follow.py work/candidate/audit.jsonl 5f4710a6python -m obs trace R-1014-042 --data work/candidateThen find the run’s line in Amira’s log:
grep R-1014-042 work/candidate/workflow.logon macOS and Linux,Select-String R-1014-042 work\candidate\workflow.logon Windows. -
With Docker (optional): send the week to Grafana as in Search runs with TraceQL, with
--candidate, and run the sheet’sduplicate writequery. It reads the trace, not the run. In the course team’s Grafana run of the candidate week, it returned 1 trace, and{ span.gen_ai.operation.name = "approve" && span.larkfield.run.id = nil }returned the 45 decisions that lost their run. -
Write the incident: what happened, when, on which release, the trace evidence, and what the trace cannot tell you.
The three views of R-1014-042
619 proposal run R-1014-042 trace e0705657 resolve (rule W2, W1): request_refund 9.95 on LK-640763 628 approval run (none) trace 5f4710a6 approved by usr-sam after 1135.0 s 629 write run (none) trace 5f4710a6 request_refund 9.95 on LK-640763: timeout (key cc62a160432a19db) 630 write run (none) trace 5f4710a6 request_refund 9.95 on LK-640763: done (key cc62a160432a19db-retry1) 631 run_closed run (none) trace 5f4710a6 resolved, writes done 1Run R-1014-042 | ticket T-90201 | chat-strong (agent_structured) | release 2.2.1-rc1 | language en3 model call(s), 2 read(s), 0 write call(s), 7313 tokens in, 381 out, US$0.01844… 9.1 s | =| 0.00 s approval.request needs agent
Outcome: waiting_approval2026-10-14T16:48:37-04:00 INFO ticket=T-90201 run=R-1014-042 model=chat-strong status=resolved writes=1 reply=sentThe run’s trace stops at approval.request; the log says resolved with one write. The audit trail says that Sam approved after 19 minutes, the first refund timed out, and a retry with a new key went through. Since 2.2.0-rc2, the decision runs without the run’s context, so its trace and records have no run ID. Since 2.2.1-rc1, the naive retry is back. Together they hide a double refund from every view that reads runs.
Trace 5f4710a6 holds both refund calls. Tempo can find it; obs cannot, because it has no run. No trace shows that the first call was paid: the payment service is the system of record. Of six refund timeouts with no reconcile, the payment service refused five retries as invalid. One customer was refunded twice.
Part 4: tests first, then the instrumentation change
Section titled “Part 4: tests first, then the instrumentation change”Uses Model calls and tokens, Tool calls, errors and retries, Approvals, people and waiting time, Many agents and handoffs and What to record, and what not to.
The release’s settings are switches in resolver/live.py: RESOLVER_APPROVAL_CONTEXT=off stands for the decision job that does not know the run, and OBS_CAPTURE_CONTENT=on for capture. Write the tests first, in tests/test_release_gate.py. One test per area, each with a candidate’s setting:
| Area | The test shows | Setting |
|---|---|---|
| Approvals | The decision trace links to approval.request and carries the run ID, as does every audit event of the run. |
RESOLVER_APPROVAL_CONTEXT=off |
| Tools | A retried refund after a timeout: both calls with their keys in one run, the flag duplicate_write, one unreconciled write. |
off + RESOLVER_RETRY=naive |
| Handoffs | A write that fails after the approval hands the ticket to a person; the handoff audit event keeps the run ID. |
off + --fault request_refund:unavailable:3 |
| Model calls | Outside a developer’s environment, capture puts no text on a span; the span says capture was asked for, and keeps its tokens. | OBS_CAPTURE_CONTENT=on, OBS_ENVIRONMENT=practice |
Use the fixtures of tests/conftest.py and the shape of tests/test_approval_traces.py. Set the variables with monkeypatch.setenv. For capture, set telemetry.CONTENT with monkeypatch.setattr, because the module reads the variable once.
-
Run your tests on the snapshot as it is. Every test that needs a change must fail:
FFFF. [100%]…4 failed, 1 passed in 0.40sOur fifth test passes before and after: with
OBS_ENVIRONMENT=dev, capture still works, so that the Collector’s redaction can be tested. -
Change the instrumentation until they pass. Two changes are enough. The decision takes the run’s ID and the request’s link from the saved run, as the handoff message carries
traceparent. Capture works only whenOBS_ENVIRONMENTisdev; elsewhere the span getslarkfield.capture.refused, not the text. -
Run every test:
119 passed, 1 skipped, 1 deselectedwith our five. Replay the week with your change:python -m scenarios week --candidate --out work/candidate-2. The last line must still sayresolved 186, right by the course's labels 279: your change touches the telemetry, not the decisions. -
Read the new week. Run
obs runs --outcome waiting_approval,obs find allandobs trace R-1014-042, each with--data work/candidate-2, andobs check-redaction work/candidate-2. -
Make a healthy week with your code, for Part 5:
python -m scenarios week --healthy --out work/healthy.
The same run, with the run’s context kept
0 run(s), 0 flaggedR-1014-042 T-90201 Wed 16:29 chat-strong duplicate_write: request_refund LK-640763 9.95 was written twice: key cc62a160432a19db at Wed 16:48:27 and key cc62a160432a19db-retry1 at Wed 16:48:37…6 match(es) in 316 run(s)... waited 18 min 54 s for a person (this trace has a link to 79b9a09ca0dd2c81 in trace e0705657...)Trace 2 5f4710a633154c98b02f7979659e8782 Wed 14 Oct 16:48:27 10.0 s 0.0 s |========================| 10.00 s approve support-resolution approved by usr-sam (agent, limit 100) 0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-640763 9.95 RFD-LATE key cc62a160432a19db 10.0 s | =| 0.00 s execute_tool request_refund done LK-640763 9.95 RFD-LATE key cc62a160432a19db-retry1 10.0 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolvedFlag duplicate_write: request_refund LK-640763 9.95 was written twice: key cc62a160432a19db at Wed 16:48:27 and key cc62a160432a19db-retry1 at Wed 16:48:37No run waits, the flag finds R-1014-042, and check-redaction finds nothing. Trace 2 is the metric’s example of Part 3, 5f4710a6: the same seeded week, now joined. This fixes the telemetry, not the release: the naive retry still paid twice, and goes back to Amira’s team with your tests.
Part 5: a dashboard and three alerts
Section titled “Part 5: a dashboard and three alerts”Uses Service levels for agents, Alerts that lead to action and Budgets, alerts and the expensive run.
-
Run the rules on your new week and on your healthy week:
Terminal window python -m obs alerts --data work/candidate-2python -m obs alerts --data work/healthy -
Choose at least three alerts that fire, and check that each is
quieton the healthy week. For each, write the problem it caught, when, its example run, and what its first step found. A rule of your own for--rules, as in Alerts that lead to action, counts too. -
Add at least one panel to
grafana/dashboards/agent-runs.jsonfor a problem that the course’s panels do not show. Our panel counts runs that closed without a release: a decision that lost its run. Its query issum(increase(larkfield_runs_total{larkfield_release=""}[90s])). Check the file withpython -m json.tool grafana/dashboards/agent-runs.json. -
Show the panel’s data. With Docker, send the week to Grafana and take a screenshot. Without Docker, read the same counter from the files of Part 1:
Terminal window python -m obs metrics larkfield.runs --data work/candidate --where larkfield.release= --day tuelarkfield.runs: Finished runs by outcome, increase per exported interval, larkfield.release=, tueTue 14:00-15:00 +1 request.model=chat-strong design=agent_structured outcome=resolved ticket.language= example: ? (trace bb4c87f7)Tue 15:00-16:00 +1 request.model=chat-small design=agent outcome=resolved ticket.language= example: ? (trace f00476fa)Tue 16:00-17:00 +1 request.model=chat-small design=agent outcome=resolved ticket.language= example: ? (trace 07a0b00b)3 interval(s) with an increase, 3 with an example trace
The alerts on the course team’s run
unreconciled-writes (unreconciled_writes > 0 over the last 45 runs): FIRED 1 time(s) from Wed 14 Oct 16:47 (run 146, R-1014-043) until Fri 16 Oct 10:39 (run 218); peak 6 example runs: R-1014-043 (python -m obs trace RUN_ID)…loop-stops (loop_stops >= 3 over the last 45 runs): FIRED 1 time(s) from Thu 15 Oct 11:24 (run 171, R-1015-016) until Sat 17 Oct 15:32 (run 280); peak 16…cost-per-resolved (cost_per_resolved > 0.002 over the last 45 runs): FIRED 1 time(s) from Mon 12 Oct 12:34 (run 20, R-1012-020) still firing at the end; peak 0.0103 the window held only 20 run(s) when it fired…handoff-rate (handoff_rate > 0.45 over the last 45 runs): FIRED 7 time(s) from Thu 15 Oct 08:40 (run 160, R-1015-003) until Thu 15 Oct 08:49 (run 161); peak 0.4667…On the healthy week, all five are quiet. unreconciled-writes was quiet in Part 1: it reads runs, and the writes had no run. Its example, R-1014-043, is not the double refund: the payment service refused its retry. Cost fires from Monday because Omar’s switch is on all week. Handoffs and loops follow the local model from Thursday, and the payment outage on Saturday.
In Grafana, the course team ran the candidate week as shipped and the healthy week, each at 1,000 times speed, with the model answer’s panel and alert rule. On the candidate week, the model answer’s rule “Runs closed without their release” fired on Tuesday at 19:14, practice time, about 7 hours after rc2 started; the others fired as obs alerts says, a few practice hours later. On the healthy week, the model answer’s rule stayed quiet, and only the handoff rule fired once, on Saturday at 22:11: a time window holds few runs on a weekend evening, as Alerts that lead to action shows. Your times can differ by a few practice hours. The prompt text of release 2.2.2-rc1 was not in Tempo: the Collector’s redaction removed it. The files that the workflow writes, and that check-redaction reads, still have it.
Part 6: the cost report
Section titled “Part 6: the cost report”Uses Cost per run and per step and Cost per customer, agent, tenant and model.
- Split the week’s cost by model and by language:
python -m obs cost --by model --data work/candidate-2, then--by language. - Run the same two commands on
work/candidate, the week as shipped. Predict first: the money is the same. Is the cost per resolved ticket? - Write the report for Kwame: the cost per resolved ticket by model and by slice, what changed it, and one decision with its trade-off.
The cost of the week, with and without the run’s context
model runs resolved cost US$ per run per resolved successchat-small 220 130 0.1250 0.00057 0.00096 59.1%chat-strong 69 55 0.8492 0.01231 0.01544 79.7%gemma3:4b 27 1 0.0000 0.00000 0.00000 3.7%total 316 186 0.9743 0.00308 0.00524 58.9%
language runs resolved cost US$ per run per resolved successde 4 4 0.0023 0.00057 0.00057 100.0%en 293 164 0.8936 0.00305 0.00545 56.0%fr 19 18 0.0784 0.00413 0.00436 94.7%total 316 186 0.9743 0.00308 0.00524 58.9%As shipped, the same US$0.9743 is divided by 148 resolved tickets, not 186: US$0.00658 instead of US$0.00524, and German looks like 1 of 4 resolved instead of 4 of 4. A broken join makes the cost per resolved ticket wrong. And resolved is not right: all four German runs are T-90305, which must go to a person, as in Quality from live runs. chat-strong costs 16 times more per resolved ticket than chat-small, and resolves more. gemma3:4b costs nothing in tokens and resolved 1 of 27: the people who took the other 26 are its real cost. Both numbers are over Omar’s budget, an assumption.
Part 7: Grace’s question
Section titled “Part 7: Grace’s question”Uses Record decisions, approvals and changes, Make it tamper-evident and controlled and Answer an audit question.
Answer Grace from the trail of the week as shipped, work/candidate/audit.jsonl: the trail that Larkfield would really have.
-
Verify it:
python -m obs audit verify --data work/candidate. The answer:The audit trail is intact: 1342 records, the chain and the anchor match (head 6ab932ef5352491b). -
Show that a change is found, as in Make it tamper-evident and controlled. In a copy, change 9.95 into 0.95 in record 630, and verify the copy:
Terminal window cp work/candidate/audit.jsonl work/t.jsonlcp work/candidate/audit.anchor.json work/t.anchor.jsonsed -i.bak '630s/9.95/0.95/' work/t.jsonlpython -m obs audit verify --audit work/t.jsonlTerminal window Copy-Item work\candidate\audit.jsonl work\t.jsonlCopy-Item work\candidate\audit.anchor.json work\t.anchor.json$l = Get-Content work\t.jsonl; $l[629] = $l[629].Replace('9.95', '0.95'); Set-Content work\t.jsonl $l -Encoding utf8NoBOMpython -m obs audit verify --audit work\t.jsonlTerminal window cp work/candidate/audit.jsonl work/t.jsonlcp work/candidate/audit.anchor.json work/t.anchor.jsonsed -i '630s/9.95/0.95/' work/t.jsonlpython -m obs audit verify --audit work/t.jsonlThe audit trail does NOT verify (630 records read):record 630: its content does not match its hash: the record was changed -
Write the answer with the five steps of Answer an audit question: the records, the run, the story, a source for each part, and two answers, one for Grace and one for the customer.
-
Name the gap. Since 2.2.0-rc2, the decision’s records have no run ID and no ticket ID.
obs audit story R-1014-042stops at the proposal. Say how you joined them, and compare with the story onwork/candidate-2.
Fresh-environment run
Section titled “Fresh-environment run”Copy the project without work/ and .venv, make a new virtual environment and install requirements-dev.txt. Run the tests, the candidate week, obs audit verify and obs check-redaction, and paste the last lines. With your change, the head is dc988509b2b0d0d4 and no text is found.
What to hand in
Section titled “What to hand in”Your project folder without .venv and work/, with your changes in Git, and REPORT.md:
# Release candidate 2.2: the observability pack## 1. What the week showed (Part 1: the findings table, each row with its release and evidence)## 2. Instrumentation changes (Part 4: the four tests, failing before and passing after; the change; the week again: no waiting runs, the flag, no prompt text)## 3. Collector configuration (Part 2: a reason per processor and route; text before and after; audit records sent, stored and verified; none in the Loki file)## 4. Dashboard and alerts (Part 5: the panel and its data; three alerts, each with its problem, time, example run and first step; the healthy week quiet)## 5. The incident (Part 3: the timeline, the trace evidence, the audit records, what the trace cannot tell, who acts)## 6. Cost report (Part 6: per resolved ticket by model and by language, as shipped and after the change; one decision)## 7. The audit answer (Part 7: verify, the changed copy, the answer, the gap)## 8. Fresh-environment run## 9. Labels and limits (which numbers are replayed, simulated, constructed, measured, assumed; what this practice week cannot say about production)Include git diff --stat against your first commit. A reviewer reads the diff of the instrumentation, the tests, the dashboard and any rule file.
Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all five must be demonstrated; you can improve an item and send it again. Without a reviewer, use the table on your own pack.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. Instrumentation changes for model calls, tools, approvals and handoffs, with tests (Module 3) | Four tests, one per area, fail on the snapshot and pass after the change; all tests pass. The week again: no waiting run, the flag on R-1014-042, no text. | Nobody saw the tests fail. Or one area has no test. | The candidate’s settings were changed, in scenarios/week.py or the environment, instead of the instrumentation. Or no test. |
| 2. A Collector configuration with redaction, sampling for traces only and two destinations (Module 2, Module 6) | Text before, none after. As many audit records stored as sent, verified, none in the Loki file. A reason for each part. | No reason for the sampling, or no audit counts. | The output has text. Or audit events are sampled. |
| 3. A dashboard and at least three alerts, each fired on its scenario and quiet on a healthy week (Module 7) | Three alerts, each with its problem, time, example run and first step, all quiet on a healthy week made with your code. A new panel with its query and data. | An alert with no first step. Or the saved healthy week, not yours. | An alert that never fired. Or no healthy week. |
| 4. One incident with trace evidence, and the cost report (Module 4, Module 5) | A timeline, the release, the trace evidence and what the trace cannot tell. Cost per resolved ticket by model and language, with a decision. | The cost uses the week as shipped and does not say why it is wrong. | “No double refund” from obs find alone. Or cost per call only. |
| 5. An audit trail that verifies, and one audit answer (Module 6) | The trail verifies and the changed copy does not. The answer names the run, Sam, both keys and the gap. | The answer uses only the trail after your change. | The trail does not verify. Or no sources. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- Telemetry exports prompt text, personal data or secrets. No text in your export or the Collector’s output; no key or real customer data in your pack. (Items 1 and 2.)
- The audit trail can be changed without detection. A changed copy must fail
obs audit verify. (Item 5.) - Audit events are sampled. Every record sent is in the store. (Item 2.)
Common mistake
The tests pass before any change
Symptom: your approvals test passes on the snapshot.
Cause: the test does not use the candidate’s setting. Without RESOLVER_APPROVAL_CONTEXT=off, the decision keeps its context.
Fix: set the variable in the test with monkeypatch.setenv, and check that the test fails first. A test that never failed proves nothing.
Common mistake
The Collector's output has no traces
Symptom: obs runs --data work/collector-out says 0 run(s).
Cause: you stopped the Collector too soon: tail sampling waits 10 seconds for each trace.
Fix: delete work/collector-out and work/collector-audit, or the counts double. Send the week again, and wait 15 seconds before you stop the Collector.
Recap
In this course, you made an agent’s work visible and accountable:
- Record once with OpenTelemetry, with pinned names and no text by default.
- Keep the run’s context across the approval pause, the handoff and the tool call.
- Let the Collector decide where data goes: redact, sample traces only, route audit events to an append-only store.
- Alert on what a person must act on, with a first step, quiet on a healthy week.
- Count cost per resolved ticket, by model and slice, and check the join under it.
- Answer from an audit trail that verifies, and say what it cannot show.
A green dashboard does not mean the agent is working. Evidence that agrees across traces, metrics, logs and the audit trail does.