Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

In Module 8, we sent the same traces to a second tool and chose views for a team. Each module added one piece and tried it alone: traces, failure patterns, cost, the audit trail and alerts. In this assignment, you use them all at once. You operate a new release candidate for one replayed week, find what it broke from its telemetry alone, and hand in the evidence. We start like an operator on a new shift: with a fresh copy and Grace's question.

What you will be able to do

Operate release candidate 2.2 of Larkfield’s workflow for a replayed week, and find what it broke from its telemetry alone. Then deliver the observability pack: instrumentation with tests, a Collector configuration, a dashboard and alerts, one diagnosed incident with a cost report, and one audit answer. Every claim points to a command and its output.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

  • Modules 1 to 8. Each part names the lessons that it uses. The certificate also asks for one case study.

What you need on your computer

  • The course’s setup: Python 3.12, VS Code, three terminals, and the labs repository in ~/projects/nextia-learning-labs, as in Meet the live workflow.
  • The snapshot end-of-m08, copied into a new folder. Your course project stays as it is.
  • The Collector binary otelcol-contrib 0.162.0 in ~/projects/otelcol, from The Collector. Docker and Grafana are optional.
  • Git, so that git diff shows every change. No account, key or money.

Tested on macOS (Apple silicon) with Python 3.12.13 and otelcol-contrib 0.162.0. The Windows and Linux commands are written, not run.

The situation: a release candidate for one week

Section titled “The situation: a release candidate for one week”

Release 2.2 of the workflow runs as a candidate for one week of Larkfield’s tickets, and the team ships three more candidates during the week. Omar’s switch to chat-strong for hard tickets stays on, and Tomás tries the local model gemma3:4b again. Nobody tells you what each candidate changed.

You are the operator. At the end of the week, Grace asks one question: “Was any customer refunded twice, and who approved it?” Kwame asks a second one: “What does a resolved ticket cost now, and on which model?”

The candidate week is a practice week. The four candidates are constructed: we planted their problems. Every model decision is replayed from the agents course. The slow and broken services are simulated: practice incidents we caused on purpose. The week is seeded: your run IDs, counts and audit heads are ours. Omar’s budget of US$0.002 per resolved ticket is an assumption.

At the top, Part 1: operate the candidate for a replayed week and list what the telemetry shows. Below it, five rows. Each row reads left to right: the part of the assignment, the item of the pack that it proves (green border), and the lessons that teach it. Part 2, the week through the Collector, proves item 2, a Collector configuration with redaction, sampling for traces only and two destinations: The Collector, Make it tamper-evident and controlled, Send the same traces to Langfuse. Parts 3 and 6, the refund that obs cannot see and the cost, prove item 4, one incident with trace evidence and the cost per resolved ticket by model and slice: Read a run, Search runs with TraceQL, Agent failure patterns, Cost per customer, agent, tenant and model. Part 4, tests first and then the instrumentation change, proves item 1, instrumentation changes for model calls, tools, approvals and handoffs, with tests: Model calls and tokens, Tool calls, errors and retries, Approvals, people and waiting time, Many agents and handoffs. Part 5, the week again and the healthy week, proves item 3, a dashboard and three alerts that each fire on their scenario and stay quiet when healthy: Service levels for agents, Alerts that lead to action, From a trace to metrics and logs. Part 7, Grace's question, proves item 5, an audit trail that verifies and the answer to one audit question: Record decisions, approvals and changes, Make it tamper-evident and controlled, Answer an audit question.
The pack's five items (green), the part of the week that proves each one, and the lessons that teach it.
Text description of the diagram

The parts are not in the order of the items. You diagnose in Part 3 before you change the instrumentation in Part 4, so that you keep the “before” evidence.

Label every number in your pack: replayed, simulated, constructed, measured (your own run), price or assumption.

  1. Copy the snapshot into a new folder, and save its first state in Git:

    Terminal window
    cd ~/projects
    cp -R nextia-learning-labs/C31/snapshots/end-of-m08 candidate-week
    cd candidate-week
    git init -q && git add -A && git commit -qm "end-of-m08"
  2. Make a virtual environment, install requirements-dev.txt and run the tests, as in Meet the live workflow. The last line says 114 passed, 1 skipped, 1 deselected.

  3. Make an empty REPORT.md with the sections of What to hand in. After each part, paste your commands, outputs and decisions into it.

Uses Meet the live workflow, Service levels for agents and Agent failure patterns.

  1. Run the candidate week. It replays 316 tickets in about 2 seconds and writes its telemetry, its audit trail and Amira’s log into work/candidate:

    Terminal window
    python -m scenarios week --candidate --out work/candidate
    candidate-week: 316 runs in 2 s -> work/candidate
    resolved 186, right by the course's labels 279

    Keep resolved 186: the driver’s count of what happened. The telemetry must agree with it.

  2. Read the week one view at a time, each with --data work/candidate: obs runs, obs slo, obs alerts, obs find all, obs cost --by release and obs audit verify. Then python -m obs check-redaction work/candidate.

  3. Predict, then check. How many runs does obs count as resolved? Is that 186?

  4. Write a table in your report with one row per finding: what you see, from when, which release, the evidence, and who acts.

    Do not open candidate_week() in scenarios/week.py until your table is written. It lists the planted problems.

What the course team saw
release runs resolved cost US$ per run per resolved success
2.2.0-rc1 74 52 0.2396 0.00324 0.00461 70.3%
2.2.0-rc2 62 16 0.2318 0.00374 0.01449 25.8%
2.2.1-rc1 79 19 0.2316 0.00293 0.01219 24.1%
2.2.2-rc1 101 61 0.2713 0.00269 0.00445 60.4%
total 316 148 0.9743 0.00308 0.00658 46.8%
run ticket started outcome model tools appr cost US$ secs model flags
R-1015-042 T-90207 Thu 17:09 waiting_approval 3 3 0 0.00084 4.2 chat-small -
R-1015-043 T-90408 Thu 17:11 waiting_approval 2 1 0 0.01178 6.6 chat-strong -
R-1015-048 T-90201 Thu 18:48 waiting_approval 3 2 0 0.01844 9.1 chat-strong -
45 run(s), 0 flagged (last 3 shown)
Checked 1999 span(s), 497 event(s), 178 metric line(s) in work/candidate.
218 problem(s): text that must not leave the application:
span chat chat-small: attribute gen_ai.input.messages (text)
…

obs counts 148 resolved runs, not 186, and 45 runs still wait for approval. Their decisions ran under 2.2.0-rc2 or 2.2.1-rc1, which seem to resolve a quarter of their tickets. Release 2.2.2-rc1 puts prompt text on its model spans. Alerts: loops and handoffs from Thursday, cost from Monday. obs find all: five gemma3:4b loops, no duplicate write. The audit trail verifies: 1,342 records, head 6ab932ef5352491b.

Uses The Collector, Make it tamper-evident and controlled and Send the same traces to Langfuse.

Show that your Collector keeps the prompts of 2.2.2-rc1 out, samples traces but never audit events, and sends to two destinations. Without Docker, the standalone configuration writes files in place of Tempo, Prometheus and Loki; the audit store is the second destination.

  1. In terminal 1, start the audit store’s door with its own file: python -m auditstore serve --store work/collector-audit/audit.jsonl.

  2. In terminal 2, start the Collector, as in The Collector:

    Terminal window
    mkdir -p work/collector-out
    OUT_DIR=work/collector-out AUDIT_STORE_URL=http://127.0.0.1:8313 ../otelcol/otelcol-contrib --config collector/standalone.yaml
  3. In a third terminal, send the week to the Collector as fast as it can go. It also writes the application’s own copy into work/candidate-otlp:

    Terminal window
    python -m scenarios week --candidate --otlp --speed 1000000 --out work/candidate-otlp
  4. Wait 15 seconds for the sampling decisions. Stop the Collector and the door with Ctrl+C. Then compare what left the application with what left the Collector:

    Terminal window
    python -m obs check-redaction work/candidate-otlp
    python -m obs check-redaction work/collector-out
    python -m obs runs --data work/collector-out
    python -m obs audit verify --audit work/collector-audit/audit.jsonl
  5. Count the audit records on both sides, and the audit events in the Loki file (work/collector-out/logs.jsonl). On Windows, use (Get-Content FILE).Count and Select-String.

  6. In your report, give a reason for each processor and route: redaction, tail sampling, the audit route and each exporter. Say which traces the sampling must always keep.

What the course team’s Collector let out
Checked 1999 span(s), 497 event(s), 178 metric line(s) in work/candidate-otlp.
218 problem(s): text that must not leave the application:
Checked 1173 span(s), 497 event(s), 1 metric line(s) in work/collector-out.
No text attribute, no e-mail address and no card number found.
208 run(s), 5 flagged
The audit trail is intact: 1342 records, the chain and the anchor match (head 32f85d7348e75efe).

The application sent 1,342 audit records, the store holds 1,342, and the Loki file holds none. Tail sampling kept 1,173 of 1,999 spans, from 208 runs. The 10% sample varies, so your counts and the store’s head differ. With Docker, grafana/compose.yaml mounts the same configuration.

Uses Read a run, Search runs with TraceQL, Tool calls, errors and retries and From a trace to metrics and logs.

obs find duplicate_write finds nothing, and unreconciled-writes is quiet. Do not stop there. Diagnose one incident with trace evidence: we use the refund timeouts; the 45 waiting runs are a second choice.

  1. Read the write counter, not the runs. Which refunds timed out on Wednesday, and did a reconcile follow?

    Terminal window
    python -m obs metrics larkfield.writes --data work/candidate --where larkfield.write.outcome=timeout --day wed
    python -m obs metrics larkfield.reconciles --data work/candidate --day wed
    larkfield.writes: Write calls by outcome, increase per exported interval, larkfield.write.outcome=timeout, wed
    Wed 16:00-17:00 +2 tool.name=request_refund example: ? (trace 5f4710a6)
    Wed 17:00-18:00 +1 tool.name=request_refund example: ? (trace 3457d66b)
    2 interval(s) with an increase, 2 with an example trace
    larkfield.reconciles: Checks of an uncertain write (did it happen?), increase per exported interval, wed
    0 interval(s) with an increase, 0 with an example trace

    ?: the example trace has no run ID.

  2. Save this short script as follow.py. It starts from a run ID or the start of a trace ID. Then it follows a proposal to the approval that names it, and the approval to its writes, by event ID:

    """Follow a run's authority in the audit trail by event IDs: its proposal, the approval that names the proposal,
    the writes that name the approval, and the close of the same trace.
    python follow.py AUDIT_FILE RUN_ID_OR_TRACE_ID"""
    import json
    import sys
    path, target = sys.argv[1], sys.argv[2]
    records = [json.loads(line)["event"] | {"seq": json.loads(line)["seq"]} for line in open(path, encoding="utf-8")]
    ids = {e["event_id"] for e in records if e["run_id"] == target and e["type"] == "proposal"}
    ids |= {e["action"]["proposal_event_id"] for e in records if e["trace_id"].startswith(target) and e["type"] == "approval"}
    traces = set()
    for e in records:
    if e["action"].get("proposal_event_id") in ids or e["authority"].get("approval_event_id") in ids:
    ids.add(e["event_id"])
    traces.add(e["trace_id"])
    for e in records:
    if e["event_id"] in ids or (e["type"] == "run_closed" and e["trace_id"] in traces):
    a = e["action"]
    what = {"proposal": lambda: f"{a['outcome']} (rule {a['rule']}): " + ", ".join(
    f"{x['tool']} {x.get('amount', '')} on {x['order_id']}" for x in a["actions"]),
    "approval": lambda: f"{a['decision']} by {e['actor']['id']} after {a['wait_s']} s",
    "write": lambda: f"{a['effect']['tool']} {a['effect']['amount']} on {a['effect']['order_id']}: "
    f"{a['outcome']} (key {a['idempotency_key']})",
    "run_closed": lambda: f"{a['outcome']}, writes done {a['writes_done']}"}.get(e["type"], lambda: a)()
    print(f"{e['seq']:>5} {e['type']:<10} run {e['run_id'] or '(none)':<10} trace {e['trace_id'][:8]} {what}")
  3. Start from the metric’s trace, then read the run that it names in two more views:

    Terminal window
    python follow.py work/candidate/audit.jsonl 5f4710a6
    python -m obs trace R-1014-042 --data work/candidate

    Then find the run’s line in Amira’s log: grep R-1014-042 work/candidate/workflow.log on macOS and Linux, Select-String R-1014-042 work\candidate\workflow.log on Windows.

  4. With Docker (optional): send the week to Grafana as in Search runs with TraceQL, with --candidate, and run the sheet’s duplicate write query. It reads the trace, not the run. In the course team’s Grafana run of the candidate week, it returned 1 trace, and { span.gen_ai.operation.name = "approve" && span.larkfield.run.id = nil } returned the 45 decisions that lost their run.

  5. Write the incident: what happened, when, on which release, the trace evidence, and what the trace cannot tell you.

The three views of R-1014-042
619 proposal run R-1014-042 trace e0705657 resolve (rule W2, W1): request_refund 9.95 on LK-640763
628 approval run (none) trace 5f4710a6 approved by usr-sam after 1135.0 s
629 write run (none) trace 5f4710a6 request_refund 9.95 on LK-640763: timeout (key cc62a160432a19db)
630 write run (none) trace 5f4710a6 request_refund 9.95 on LK-640763: done (key cc62a160432a19db-retry1)
631 run_closed run (none) trace 5f4710a6 resolved, writes done 1
Run R-1014-042 | ticket T-90201 | chat-strong (agent_structured) | release 2.2.1-rc1 | language en
3 model call(s), 2 read(s), 0 write call(s), 7313 tokens in, 381 out, US$0.01844
…
9.1 s | =| 0.00 s approval.request needs agent
Outcome: waiting_approval
2026-10-14T16:48:37-04:00 INFO ticket=T-90201 run=R-1014-042 model=chat-strong status=resolved writes=1 reply=sent

The run’s trace stops at approval.request; the log says resolved with one write. The audit trail says that Sam approved after 19 minutes, the first refund timed out, and a retry with a new key went through. Since 2.2.0-rc2, the decision runs without the run’s context, so its trace and records have no run ID. Since 2.2.1-rc1, the naive retry is back. Together they hide a double refund from every view that reads runs.

Trace 5f4710a6 holds both refund calls. Tempo can find it; obs cannot, because it has no run. No trace shows that the first call was paid: the payment service is the system of record. Of six refund timeouts with no reconcile, the payment service refused five retries as invalid. One customer was refunded twice.

Part 4: tests first, then the instrumentation change

Section titled “Part 4: tests first, then the instrumentation change”

Uses Model calls and tokens, Tool calls, errors and retries, Approvals, people and waiting time, Many agents and handoffs and What to record, and what not to.

The release’s settings are switches in resolver/live.py: RESOLVER_APPROVAL_CONTEXT=off stands for the decision job that does not know the run, and OBS_CAPTURE_CONTENT=on for capture. Write the tests first, in tests/test_release_gate.py. One test per area, each with a candidate’s setting:

Area The test shows Setting
Approvals The decision trace links to approval.request and carries the run ID, as does every audit event of the run. RESOLVER_APPROVAL_CONTEXT=off
Tools A retried refund after a timeout: both calls with their keys in one run, the flag duplicate_write, one unreconciled write. off + RESOLVER_RETRY=naive
Handoffs A write that fails after the approval hands the ticket to a person; the handoff audit event keeps the run ID. off + --fault request_refund:unavailable:3
Model calls Outside a developer’s environment, capture puts no text on a span; the span says capture was asked for, and keeps its tokens. OBS_CAPTURE_CONTENT=on, OBS_ENVIRONMENT=practice

Use the fixtures of tests/conftest.py and the shape of tests/test_approval_traces.py. Set the variables with monkeypatch.setenv. For capture, set telemetry.CONTENT with monkeypatch.setattr, because the module reads the variable once.

  1. Run your tests on the snapshot as it is. Every test that needs a change must fail:

    FFFF. [100%]
    …
    4 failed, 1 passed in 0.40s

    Our fifth test passes before and after: with OBS_ENVIRONMENT=dev, capture still works, so that the Collector’s redaction can be tested.

  2. Change the instrumentation until they pass. Two changes are enough. The decision takes the run’s ID and the request’s link from the saved run, as the handoff message carries traceparent. Capture works only when OBS_ENVIRONMENT is dev; elsewhere the span gets larkfield.capture.refused, not the text.

  3. Run every test: 119 passed, 1 skipped, 1 deselected with our five. Replay the week with your change: python -m scenarios week --candidate --out work/candidate-2. The last line must still say resolved 186, right by the course's labels 279: your change touches the telemetry, not the decisions.

  4. Read the new week. Run obs runs --outcome waiting_approval, obs find all and obs trace R-1014-042, each with --data work/candidate-2, and obs check-redaction work/candidate-2.

  5. Make a healthy week with your code, for Part 5: python -m scenarios week --healthy --out work/healthy.

The same run, with the run’s context kept
0 run(s), 0 flagged
R-1014-042 T-90201 Wed 16:29 chat-strong duplicate_write: request_refund LK-640763 9.95 was written twice: key cc62a160432a19db at Wed 16:48:27 and key cc62a160432a19db-retry1 at Wed 16:48:37
…
6 match(es) in 316 run(s)
... waited 18 min 54 s for a person (this trace has a link to 79b9a09ca0dd2c81 in trace e0705657...)
Trace 2 5f4710a633154c98b02f7979659e8782 Wed 14 Oct 16:48:27 10.0 s
0.0 s |========================| 10.00 s approve support-resolution approved by usr-sam (agent, limit 100)
0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-640763 9.95 RFD-LATE key cc62a160432a19db
10.0 s | =| 0.00 s execute_tool request_refund done LK-640763 9.95 RFD-LATE key cc62a160432a19db-retry1
10.0 s | =| 0.00 s execute_tool send_reply sent
Outcome: resolved
Flag duplicate_write: request_refund LK-640763 9.95 was written twice: key cc62a160432a19db at Wed 16:48:27 and key cc62a160432a19db-retry1 at Wed 16:48:37

No run waits, the flag finds R-1014-042, and check-redaction finds nothing. Trace 2 is the metric’s example of Part 3, 5f4710a6: the same seeded week, now joined. This fixes the telemetry, not the release: the naive retry still paid twice, and goes back to Amira’s team with your tests.

Uses Service levels for agents, Alerts that lead to action and Budgets, alerts and the expensive run.

  1. Run the rules on your new week and on your healthy week:

    Terminal window
    python -m obs alerts --data work/candidate-2
    python -m obs alerts --data work/healthy
  2. Choose at least three alerts that fire, and check that each is quiet on the healthy week. For each, write the problem it caught, when, its example run, and what its first step found. A rule of your own for --rules, as in Alerts that lead to action, counts too.

  3. Add at least one panel to grafana/dashboards/agent-runs.json for a problem that the course’s panels do not show. Our panel counts runs that closed without a release: a decision that lost its run. Its query is sum(increase(larkfield_runs_total{larkfield_release=""}[90s])). Check the file with python -m json.tool grafana/dashboards/agent-runs.json.

  4. Show the panel’s data. With Docker, send the week to Grafana and take a screenshot. Without Docker, read the same counter from the files of Part 1:

    Terminal window
    python -m obs metrics larkfield.runs --data work/candidate --where larkfield.release= --day tue
    larkfield.runs: Finished runs by outcome, increase per exported interval, larkfield.release=, tue
    Tue 14:00-15:00 +1 request.model=chat-strong design=agent_structured outcome=resolved ticket.language= example: ? (trace bb4c87f7)
    Tue 15:00-16:00 +1 request.model=chat-small design=agent outcome=resolved ticket.language= example: ? (trace f00476fa)
    Tue 16:00-17:00 +1 request.model=chat-small design=agent outcome=resolved ticket.language= example: ? (trace 07a0b00b)
    3 interval(s) with an increase, 3 with an example trace
The alerts on the course team’s run
unreconciled-writes (unreconciled_writes > 0 over the last 45 runs): FIRED 1 time(s)
from Wed 14 Oct 16:47 (run 146, R-1014-043) until Fri 16 Oct 10:39 (run 218); peak 6
example runs: R-1014-043 (python -m obs trace RUN_ID)
…
loop-stops (loop_stops >= 3 over the last 45 runs): FIRED 1 time(s)
from Thu 15 Oct 11:24 (run 171, R-1015-016) until Sat 17 Oct 15:32 (run 280); peak 16
…
cost-per-resolved (cost_per_resolved > 0.002 over the last 45 runs): FIRED 1 time(s)
from Mon 12 Oct 12:34 (run 20, R-1012-020) still firing at the end; peak 0.0103
the window held only 20 run(s) when it fired
…
handoff-rate (handoff_rate > 0.45 over the last 45 runs): FIRED 7 time(s)
from Thu 15 Oct 08:40 (run 160, R-1015-003) until Thu 15 Oct 08:49 (run 161); peak 0.4667
…

On the healthy week, all five are quiet. unreconciled-writes was quiet in Part 1: it reads runs, and the writes had no run. Its example, R-1014-043, is not the double refund: the payment service refused its retry. Cost fires from Monday because Omar’s switch is on all week. Handoffs and loops follow the local model from Thursday, and the payment outage on Saturday.

In Grafana, the course team ran the candidate week as shipped and the healthy week, each at 1,000 times speed, with the model answer’s panel and alert rule. On the candidate week, the model answer’s rule “Runs closed without their release” fired on Tuesday at 19:14, practice time, about 7 hours after rc2 started; the others fired as obs alerts says, a few practice hours later. On the healthy week, the model answer’s rule stayed quiet, and only the handoff rule fired once, on Saturday at 22:11: a time window holds few runs on a weekend evening, as Alerts that lead to action shows. Your times can differ by a few practice hours. The prompt text of release 2.2.2-rc1 was not in Tempo: the Collector’s redaction removed it. The files that the workflow writes, and that check-redaction reads, still have it.

Uses Cost per run and per step and Cost per customer, agent, tenant and model.

  1. Split the week’s cost by model and by language: python -m obs cost --by model --data work/candidate-2, then --by language.
  2. Run the same two commands on work/candidate, the week as shipped. Predict first: the money is the same. Is the cost per resolved ticket?
  3. Write the report for Kwame: the cost per resolved ticket by model and by slice, what changed it, and one decision with its trade-off.
The cost of the week, with and without the run’s context
model runs resolved cost US$ per run per resolved success
chat-small 220 130 0.1250 0.00057 0.00096 59.1%
chat-strong 69 55 0.8492 0.01231 0.01544 79.7%
gemma3:4b 27 1 0.0000 0.00000 0.00000 3.7%
total 316 186 0.9743 0.00308 0.00524 58.9%
language runs resolved cost US$ per run per resolved success
de 4 4 0.0023 0.00057 0.00057 100.0%
en 293 164 0.8936 0.00305 0.00545 56.0%
fr 19 18 0.0784 0.00413 0.00436 94.7%
total 316 186 0.9743 0.00308 0.00524 58.9%

As shipped, the same US$0.9743 is divided by 148 resolved tickets, not 186: US$0.00658 instead of US$0.00524, and German looks like 1 of 4 resolved instead of 4 of 4. A broken join makes the cost per resolved ticket wrong. And resolved is not right: all four German runs are T-90305, which must go to a person, as in Quality from live runs. chat-strong costs 16 times more per resolved ticket than chat-small, and resolves more. gemma3:4b costs nothing in tokens and resolved 1 of 27: the people who took the other 26 are its real cost. Both numbers are over Omar’s budget, an assumption.

Uses Record decisions, approvals and changes, Make it tamper-evident and controlled and Answer an audit question.

Answer Grace from the trail of the week as shipped, work/candidate/audit.jsonl: the trail that Larkfield would really have.

  1. Verify it: python -m obs audit verify --data work/candidate. The answer: The audit trail is intact: 1342 records, the chain and the anchor match (head 6ab932ef5352491b).

  2. Show that a change is found, as in Make it tamper-evident and controlled. In a copy, change 9.95 into 0.95 in record 630, and verify the copy:

    Terminal window
    cp work/candidate/audit.jsonl work/t.jsonl
    cp work/candidate/audit.anchor.json work/t.anchor.json
    sed -i.bak '630s/9.95/0.95/' work/t.jsonl
    python -m obs audit verify --audit work/t.jsonl
    The audit trail does NOT verify (630 records read):
    record 630: its content does not match its hash: the record was changed
  3. Write the answer with the five steps of Answer an audit question: the records, the run, the story, a source for each part, and two answers, one for Grace and one for the customer.

  4. Name the gap. Since 2.2.0-rc2, the decision’s records have no run ID and no ticket ID. obs audit story R-1014-042 stops at the proposal. Say how you joined them, and compare with the story on work/candidate-2.

Copy the project without work/ and .venv, make a new virtual environment and install requirements-dev.txt. Run the tests, the candidate week, obs audit verify and obs check-redaction, and paste the last lines. With your change, the head is dc988509b2b0d0d4 and no text is found.

Your project folder without .venv and work/, with your changes in Git, and REPORT.md:

# Release candidate 2.2: the observability pack
## 1. What the week showed (Part 1: the findings table, each row with its release and evidence)
## 2. Instrumentation changes (Part 4: the four tests, failing before and passing after; the change;
the week again: no waiting runs, the flag, no prompt text)
## 3. Collector configuration (Part 2: a reason per processor and route; text before and after;
audit records sent, stored and verified; none in the Loki file)
## 4. Dashboard and alerts (Part 5: the panel and its data; three alerts, each with its problem,
time, example run and first step; the healthy week quiet)
## 5. The incident (Part 3: the timeline, the trace evidence, the audit records, what the
trace cannot tell, who acts)
## 6. Cost report (Part 6: per resolved ticket by model and by language, as shipped and
after the change; one decision)
## 7. The audit answer (Part 7: verify, the changed copy, the answer, the gap)
## 8. Fresh-environment run
## 9. Labels and limits (which numbers are replayed, simulated, constructed, measured, assumed;
what this practice week cannot say about production)

Include git diff --stat against your first commit. A reviewer reads the diff of the instrumentation, the tests, the dashboard and any rule file.

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all five must be demonstrated; you can improve an item and send it again. Without a reviewer, use the table on your own pack.

Required item Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Instrumentation changes for model calls, tools, approvals and handoffs, with tests (Module 3) Four tests, one per area, fail on the snapshot and pass after the change; all tests pass. The week again: no waiting run, the flag on R-1014-042, no text. Nobody saw the tests fail. Or one area has no test. The candidate’s settings were changed, in scenarios/week.py or the environment, instead of the instrumentation. Or no test.
2. A Collector configuration with redaction, sampling for traces only and two destinations (Module 2, Module 6) Text before, none after. As many audit records stored as sent, verified, none in the Loki file. A reason for each part. No reason for the sampling, or no audit counts. The output has text. Or audit events are sampled.
3. A dashboard and at least three alerts, each fired on its scenario and quiet on a healthy week (Module 7) Three alerts, each with its problem, time, example run and first step, all quiet on a healthy week made with your code. A new panel with its query and data. An alert with no first step. Or the saved healthy week, not yours. An alert that never fired. Or no healthy week.
4. One incident with trace evidence, and the cost report (Module 4, Module 5) A timeline, the release, the trace evidence and what the trace cannot tell. Cost per resolved ticket by model and language, with a decision. The cost uses the week as shipped and does not say why it is wrong. “No double refund” from obs find alone. Or cost per call only.
5. An audit trail that verifies, and one audit answer (Module 6) The trail verifies and the changed copy does not. The answer names the run, Sam, both keys and the gap. The answer uses only the trail after your change. The trail does not verify. Or no sources.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • Telemetry exports prompt text, personal data or secrets. No text in your export or the Collector’s output; no key or real customer data in your pack. (Items 1 and 2.)
  • The audit trail can be changed without detection. A changed copy must fail obs audit verify. (Item 5.)
  • Audit events are sampled. Every record sent is in the store. (Item 2.)

Common mistake

The tests pass before any change

Symptom: your approvals test passes on the snapshot.

Cause: the test does not use the candidate’s setting. Without RESOLVER_APPROVAL_CONTEXT=off, the decision keeps its context.

Fix: set the variable in the test with monkeypatch.setenv, and check that the test fails first. A test that never failed proves nothing.

Common mistake

The Collector's output has no traces

Symptom: obs runs --data work/collector-out says 0 run(s).

Cause: you stopped the Collector too soon: tail sampling waits 10 seconds for each trace.

Fix: delete work/collector-out and work/collector-audit, or the counts double. Send the week again, and wait 15 seconds before you stop the Collector.

Recap

In this course, you made an agent’s work visible and accountable:

  • Record once with OpenTelemetry, with pinned names and no text by default.
  • Keep the run’s context across the approval pause, the handoff and the tool call.
  • Let the Collector decide where data goes: redact, sample traces only, route audit events to an append-only store.
  • Alert on what a person must act on, with a first step, quiet on a healthy week.
  • Count cost per resolved ticket, by model and slice, and check the join under it.
  • Answer from an audit trail that verifies, and say what it cannot show.

A green dashboard does not mean the agent is working. Evidence that agrees across traces, metrics, logs and the audit trail does.