Could the records have answered?
- Case study 3
- 180 min
- Optional · 1 case study needed for the certificate
Where we are
In the audit module, we answered Grace's question about the double refund from our own trail. Real organisations have had to defend what their AI assistants said, in public. In this case study, we take one documented case, a city's business chatbot, list the questions it raised, and test four designs of records against them. At the end, a new complaint arrives, and we answer it from the records alone.
Case study 3 of 3 · Observability for AI Agents: Traces, Costs and Audit TrailsCould the records have answered?
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Take a public case in which a city had to defend its AI chatbot’s answers. Turn the questions that it raised into questions for a system’s records. Then build four designs of records for a practice copy of the chatbot, and ask every question of every design. Recommend one, with its costs and its limits.
The starter is a small Python project: a practice assistant, a pilot of 190 practice days, three designs of records, and the case’s questions as code. You build the fourth design. The finished project is the result, for when you are stuck. Nothing needs an account, a key or a model call: the model’s answers are recordings, replayed.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- What each signal is for, and who reads it, from What to record and what not to.
- Sampling, from The Collector.
- The audit trail’s four rules, from An audit trail is not a log.
- The schema and what stays out, from Record decisions, approvals and changes.
- The chain, the anchor and pseudonyms, from Make it tamper-evident and controlled.
- The five-step method, from Answer an audit question.
What you need on your computer
- Python 3.12 or newer, VS Code, and about 60 MB of disk space for the virtual environment and the records.
- No Larkfield project: this case study has its own folder,
could-the-records-answer. To start again, delete itswork/folder.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
The situation
Section titled “The situation”In October 2023, New York City opened a pilot AI chatbot on its MyCity Business site. It was to answer business owners’ questions from “more than 2,000 NYC Business web pages and articles” (the city’s announcement). Five months later, reporters tested it. Here is the case in our own words, with its sources:
| When | What the sources report |
|---|---|
| 29 March 2024 | The Markup, with THE CITY and Documented, reports answers that the reporters say contradict local rules: on tips, on taking cash, on housing vouchers and more. One reporter got one answer to the voucher question; ten other staff members got the opposite one. The bot’s page says that the city reviews questions and otherwise deletes data within 30 days. (The Markup) |
| 2 April 2024 | Asked whether the chatbot should come down, the mayor says “We’re going to fix them”. Officials call it “a pilot”. (the city’s transcript) |
| 2 April 2024 | The chatbot stays online with a new warning: “Do not use its responses as legal or professional advice”. A reader’s screenshot shows a flat “no” to a tenant’s question. (The Markup) |
| January–February 2026 | The next mayor calls the bot “functionally unusable”, at a cost of “around half a million dollars”, and the city takes it down. (The Markup and THE CITY) |
The AI Incident Database lists the case as incident 714. We draw no legal conclusion from it. We do not know what the city recorded, and nothing on this page says what it recorded.
What we can do is ask a narrower question. Suppose you ran a chatbot like this one. Which records would let you answer, on the day the report comes out, what the reporters, the officials and the critics asked?
Our practice city is Harbourton. Nadia (she) leads its digital services, and the reporters will call her. Felix (he) built the city’s Business Helper and runs it. The people and the city are made up.
The objective. Recommend a design of records for the Business Helper. It must answer the case’s ten questions on the day after the report, from the records alone. It must keep as little of people’s own words as it can. And it must still answer a new complaint months later.
What you will build
Section titled “What you will build”Four designs record the same conversations. The case study compares them under one measure: how many of the ten questions each design answers in full, from its records alone, on a given day.
Text description of the diagram
The top box is the conversation: a question about cash, the two pages that the search found, and the helper’s answer. Below it, four boxes. A keeps a log line, and the words for 30 days only. B keeps the question and the answer, with the session, for ever. C keeps a trace only for one conversation in ten, for 14 days, without text. D keeps three audit records with fixed fields, and the answer’s text once, by its fingerprint.
| Design | What it keeps |
|---|---|
| A. The app log (the baseline) | one log line per conversation (90 days); the chat (30 days); a feedback form with no link to a chat |
| B. Every question and answer | the question, the answer and the person’s account or session, for ever |
| C. Traces only | OpenTelemetry traces with the course’s names: 10% of conversations, 14 days, no text |
| D. Audit trail, answers and metrics | audit events with the course’s schema, never sampled, 7 years; each distinct answer once, 2 years; daily counters, 13 months; plus C’s traces |
The 30 days of A come from the one public statement about the real bot’s data. Every other period is an assumption, written in the code so that you can change it.
The data
Section titled “The data”No records of the real chatbot are public. So the data is a practice copy that we made, and the dataset card says what is real and what is constructed:
- Harbourton’s rule pages (
data/pages/): eleven short pages that we wrote, one file per version. They are not any real city’s rules and not legal advice. - The Business Helper (
helper/): a keyword search finds the two pages that share most words with the question. The model answers in JSON: a short answer (yes,no,it_dependsorrefer), at most 60 words, and the pages it used. - The model’s answers are real recordings. We sent each of the 17 distinct requests ten times to the course’s
chat-smalldeployment. The 170 answers are replayed byte for byte, as in the rest of the course. The recording cost US$0.0105 at the course’s price table. - The traffic is constructed (
pilot.py, seeded): 190 practice days from Monday 5 January, 18 conversations a weekday and 6 a weekend day. A quarter of the people sign in. - The labels (
data/labels.json): a person read each distinct answer once against its page.
The release log (data/releases.json) holds the pilot’s story. We built two mistakes into it, on purpose:
| Day | What happens |
|---|---|
| 1 | Release 1.0: prompt p1, index i1, warning d1. The index holds an old copy of the cash page, version 1, which says that a shop may stop taking cash. |
| 148 | Eleven reporters ask the housing-voucher question. |
| 150 | The report is published. |
| 152 | Release 1.1 for 20% of visitors: prompt p2, index i2 (the current cash page), warning d2 (to accept), and a guardrail that sends housing questions to the housing office. |
| 155 | Release 1.1 for everybody. |
| 171 | The minimum-wage page changes on the website: 18.10 from 1 July. |
| 178 | 1 July. The index still holds the old page, version 3: 17.40. |
| 185 | Release 1.2: index i3 with the new page. |
| 186 | A business owner complains. |
The questions
Section titled “The questions”Each question comes from what someone asked in public. The project writes them in casefile/case.py:
| The question to the records | Raised by | |
|---|---|---|
| Q1 | A reader sends a screenshot. Did the helper really give this answer, and when? | the reader’s screenshot |
| Q2 | How many answers were wrong, on which topic, from which day to which day? | the reporters’ tests |
| Q3 | Which pages, in which version, was each wrong answer built from: a wrong page, or a wrong model? | “2,000 web pages” |
| Q4 | Which release gave the wrong answers, and did the fix change them? | “We’re going to fix them” |
| Q5 | Which warning did each person see, and did they accept it? | the new warning |
| Q6 | Why did the same question get different answers? | one reporter against ten |
| Q7 | Did a person check any answer before it went out? | the critics |
| Q8 | Can the city reach the people who got a wrong answer? | a wrong answer may have been followed |
| Q9 | Did people report the wrong answers, and when was the first report? | the feedback pop-up |
| Q10 | What did one answer cost, and one right answer? | “around half a million dollars” |
We ask on day 156, Tuesday 9 June: six days after the report, one day after the fix reached everybody.
Set up the project
Section titled “Set up the project”-
Download the starter from the box at the top of this page, and unzip it. Open the folder
could-the-records-answerin VS Code, then open a terminal: Terminal > New Terminal. -
Make a virtual environment, and install the packages:
Terminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements-dev.txtTerminal window python -m venv .venv.venv\Scripts\Activate.ps1python -m pip install -r requirements-dev.txtTerminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements-dev.txtThe packages are the course project’s: OpenTelemetry 1.45.1, pytest and ruff.
-
Run the pilot. It takes about 5 seconds and writes
work/pilot/:Terminal window python pilot.py2783 conversations, 3359 answers over 190 days; 263 wrong by the practice pages. Records in work/pilot/ -
Run the tests:
python -m pytest -q. The last line says11 passed, 9 deselected. The nine left out check design D, which you build in Part 3.
What this tells us. The pilot is seeded, so your records are the same as ours, byte for byte. Of 3,359 answers, 263 were wrong by the practice pages. That count comes from work/pilot/truth.json, which no design may read. It is the answer key: we use it only to check the designs.
Part 1: the baseline
Section titled “Part 1: the baseline”Design A is what a quick pilot keeps. Ask it the screenshot question:
python -m casefile ask Q1 --design applogA. The app log: not answered The chats of day 41 were deleted after 30 days (the oldest kept: day 127). app.log shows 6 conversations that day, with no text.The screenshot is from day 41. On day 156, the chats from before day 127 are gone. The log knows that 6 conversations happened that day, and nothing about what was said. Now count the wrong answers, and the reports:
A. The app log: partly 35 wrong answers of 549 kept; days 127–154 77 distinct answer texts to label. Only the last 30 days are kept (from day 127): earlier wrong answers are gone.A. The app log: partly The form has 64 problem reports (first on day 3), but no link to a conversation or an answer.What this tells us. Design A sees 35 wrong answers; the pilot gave 250 by day 156. The feedback form shows problem reports from day 3, but nobody can say which answer they were about. Run python -m casefile matrix: design A answers 0 of the 10 questions in full. This is the baseline.
Part 2: two alternatives
Section titled “Part 2: two alternatives”Design B is Tomás’s plan from the course: “we log every prompt and answer, so we can explain anything”. Ask it the same two questions (ask Q1 --design transcripts, then Q8):
B. Every question and answer: answered 1 answer(s) with exactly this text on day 41: C-041-04 at 13:52 to S-d26125cf The file also holds the person's own question, in their words.B. Every question and answer: answered 247 people; 51 with an account ID, in clear text, next to their own question.B answers the screenshot, the count and who to call. But it cannot say which pages the helper read, which release answered, or why two people got different answers. And it keeps 2,778 questions in people’s own words, for ever.
Design C is the engineer’s first idea: traces with the GenAI names of the conventions lesson, no text, head sampling at 10%, and the trace store’s 14 days. Ask it about the pages:
C. Traces only: partly In the sampled spans of the last 14 days: 1 cash 'yes' answer(s) from BP-102@1, BP-106@1That one span has the right clue, the old cash page BP-102@1. But it is one of the 30 conversations that the sample kept from the last 14 days. Traces are right for “how did this run go?” and wrong for “how many, since when?”. C answers 0 questions in full.
Part 3: design D
Section titled “Part 3: design D”Start from the questions
Section titled “Start from the questions”The schema of Record decisions, approvals and changes gives the shape: who, what, when, on whose authority, with what evidence. The case’s questions decide what goes into each field. Here are the records of the screenshot’s conversation in the finished design, field by field:
Text description of the diagram
Ten rows. Each row is one field of a record and the question that needs it. The proposal holds the pages with their versions, the short answer and topic, the release with its prompt and index, and the review rule. The write holds the answer’s fingerprint and the warning. Every record holds the person’s pseudonym and the trace and span IDs. The close holds the feedback. The cost comes from daily metrics, not from the trail.
Three choices need a word:
- The answer is a
write. In Larkfield, a write changed an order. Here, the change in the world is what a person reads and may act on. So the reply that the person sees is the write, and the model’s draft is the proposal. - The question stays out. It is the person’s own words. The trail keeps a topic code from a fixed list, made from the question’s words by
helper/pages.py, never from the pages that the search found. - The review rule is a field. “Nobody checked” must be a fact in the record,
"review": "none", not a missing record. A missing approval can mean “not recorded”.
A new idea: the answer store
Section titled “A new idea: the answer store”Your bank sends the same few letters to thousands of people. It does not keep thousands of copies. It keeps each letter once, with a reference number, and its records say “customer 18, letter R-7, 3 March”. To prove what someone received, it shows letter R-7.
The trail has the same problem. Q1 and Q2 need the words of the answer. But the schema refuses text, and for good reasons: text is long, it may hold personal data, and nobody can count it.
The idea: keep each distinct answer once, in an answer store, under its digest: the first 16 hex digits of the SHA-256 of its text. The write record holds only the digest, a short ID that the schema accepts. The same text always has the same digest, so a screenshot finds its records. The answers are the helper’s own words, so the store holds nothing that a person typed.
Text description of the diagram
Three write records point to two texts. The answer store holds each text once, with the day it was first given. A person labels each text once, here “wrong”, and the trail counts how often each one was given: 23 and 24 times by day 156.
On day 156, the trail holds 2,778 answers, but there are only 77 distinct texts. So a person reads 77 answers once, not 2,778 conversations. The store costs little: 101 lines for the whole pilot. It has a limit: a model that writes a new text every time makes the store as long as the trail. Then keep the short answer and the pages, which the proposal holds anyway.
Write the events
Section titled “Write the events”Open records/audit.py. The starter writes one event: run_closed, at the end of each conversation. The refusal and the handoff of the housing guardrail are there too, as an example. Fill in the three TODOs. Our version of the two events:
else: m["input_tokens"] += a.input_tokens m["output_tokens"] += a.output_tokens first = self.event("proposal", at, conv, helper, ids["answer"], {"topic": a.topic, "short_answer": a.short_answer, "answer_digest": d, "cites": a.cites}, {"release": rel["id"], "prompt_version": rel["prompt"], "index": rel["index"], "guardrail": rel["guardrail"], "review": rel["review"]}, [f"page:{p}" for p in a.pages]) self.event("write", at, conv, web, ids["reply"], {"answer_digest": d, "topic": a.topic, "disclaimer": a.disclaimer, "disclaimer_accepted": accept, "channel": "web", "language": "en", "proposal_event_id": first}, {"release": rel["id"], "review": rel["review"]})And the answer store, in keep_answer():
d = digest(text) if d not in self.seen: if PERSONAL.search(text): raise ValueError(f"answer {d} looks like it holds personal data: not stored") with self.answers_path.open("a", encoding="utf-8") as f: f.write(json.dumps({"digest": d, "first_day": day, "text": text}, ensure_ascii=False) + "\n") self.seen.add(d) return dEach event is written in its span, ids["answer"] or ids["reply"], so its trace_id and span_id lead to the details. That works even when the sampler drops the trace: the IDs exist for every conversation.
Common mistake
The schema refuses a version code
Symptom: your first proposal names the prompt’s version "prompt": rel["prompt"]. python pilot.py stops at the first answer:
auditstore.schema.AuditEventError: event.authority.prompt: text fields are not allowed in an audit eventCause: the course’s schema refuses any field named prompt, whatever its value. The value here is p1, a code, but the schema checks names, not meaning. We made this mistake ourselves when we built the case study.
Diagnose: the last line names the field: event.authority.prompt.
Recover: rename the field prompt_version. Do not remove prompt from the schema’s list: the next field called prompt may really hold a prompt. A refused event stops the pilot before any answer goes out unrecorded, which is the safe way to fail.
Run the pilot again, then the tests of design D:
python pilot.pypython -m pytest -q -m design_dpython -m casefile verify9 passed, 11 deselected in …The audit trail is intact: 9550 records, the chain and the anchor match (head a452596501fd0848).The tests check that every answer has a proposal or a refusal and a write, that no question text is in design D, and that the store refuses an e-mail address. They also check that a changed record breaks the chain. The pilot uses a fixed seed, so your head must be the same as ours. If it differs, compare your fields with the code above.
Part 4: the fair comparison
Section titled “Part 4: the fair comparison”Ask every question of every design: python -m casefile matrix.
The case's questions on day 156: what each design's records can answer
A. B. C. D.Q1 no yes no yesQ2 partly yes partly yesQ3 no no partly yesQ4 partly partly partly yesQ5 partly partly partly yesQ6 no no partly yesQ7 no no no yesQ8 no yes no yesQ9 partly no no yesQ10 no no partly yesyes 0 3 0 10
A. The app log people's own words kept: 549 questions (30 days)B. Every question and answer people's own words kept: 2778 questions (all)C. Traces only people's own words kept: noneD. Audit trail, answers and metrics people's own words kept: noneThen ask again a year later: python -m casefile --day 400 matrix.
- Day 156 (after the report)
- Day 400 (a year later)
Text description of the chart
On day 156, design A answers 0 questions in full, B 3, C 0 and D 10. On day 400, A answers 0, B 3, C 0 and D 9.
Read what design D says, question by question (python -m casefile ask Q3 --design audit, and so on). Four answers matter most:
Q3 cash: 250 answer(s) from BP-102@1, BP-106@1; a replaced page: BP-102@1Q4 release 1.0 (from day 1): 250 wrong of 2729 release 1.1 (from day 152 (part of the visitors first)): 0 wrong of 49Q6 food_permit, release 1.0, the same pages (BP-107@1, BP-105@1): 8 distinct answers, short answers {'it_depends': 4, 'yes': 4}: the model's own variationQ9 cash: 38 reports of a wrong answer; the first on day 3 (Wed 7 Jan)What this tells us.
- Every wrong answer came from a page, not from the model. All 250 wrong cash answers used the old copy of the cash page. The model repeated its page faithfully. Without the page versions in the record, a team would blame the model and change the prompt. That fix would change nothing.
- The same question got different answers from the same pages. Ten recorded samples of the coffee-permit request gave “yes” and “it depends” in turn. The record proves that the pages and the release were the same, so the cause is the model’s own variation. That is what Q6 asks about the one reporter and the ten.
- The warning was the first report. People reported a wrong cash answer on day 3, 147 days before the press did. Design A had the reports too, but with no link to an answer, so nobody could act on them.
- D’s own retention shows too. On day 400, Q10 is only “partly”: the metrics are kept 13 months.
Is the comparison fair? All four designs saw the same 2,783 conversations, through the same code, and were asked the same ten questions on the same day. The measure has two weak points. We wrote the questions and the designs, so read the matrix as a check of a design against stated questions, not as a score. And “answered” means “the records give the whole answer”. It does not mean that a person would agree with every label.
D also costs the most bytes: 7.2 MB, against 1.1 MB for B and 0.4 MB for C. Most of it is the trail’s fixed fields, hashes and the pseudonym keys of 2,780 people (sessions and accounts). None of it is a person’s words.
The recommendation
Section titled “The recommendation”For Nadia, in her words: keep design D, and fix the index process.
- Record each answer as a write, with its pages and their versions, the release, the warning and the review rule. Keep each distinct answer once. Never keep the question’s words.
- Link every feedback report to the conversation and its answer. The reports came from day 3. A weekly count of reports by answer would have found the old cash page in the first week.
- Rebuild the index when a page changes. Both failures came from old pages, not from the model.
- Decide whether a person checks answers about rules and money before they go out, and record the decision. The records can show it; they cannot make it.
Do not use design D to say that an answer was legally right or wrong. The labels compare answers with practice pages only.
Part 5: new data, end to end
Section titled “Part 5: new data, end to end”On day 186, a business owner writes. Read the complaint first:
cat work/pilot/complaint.jsonGet-Content work\pilot\complaint.jsoncat work/pilot/complaint.json "says": "On Wednesday 1 July your business helper told me that I could pay a new cashier 17.40 an hour. I did. Now I owe back pay. Why did it say that?", "screenshot_text": "Yes. Harbourton’s minimum wage is $17.40 an hour for every worker aged 16 or over, including new workers."This is a new failure, after the fix: the minimum-wage page. Answer it with the method of Answer an audit question, from design D’s records alone:
python -m casefile investigate1. The record: the audit trail is intact, 9550 records, head a452596501fd0848.2. The conversation: C-178-11 on day 178, person cust:81900cf8d97b43dd; the screenshot's digest 6d0d25df2ed577da matches write record 8949.3. The story: record 8948 2026-07-01 16:16 UTC proposal the helper proposed the answer 6d0d25df2ed577da (minimum_wage, short answer yes) from BP-104@3, BP-101@2; release 1.1, prompt p2, index i2, review none record 8949 2026-07-01 16:16 UTC write the reply 6d0d25df2ed577da was shown with warning d2 (accepted: yes) record 8950 2026-07-01 16:16 UTC run_closed the conversation closed: 1 answer(s), feedback reported_problem4. Sources: the answer store: the text of 6d0d25df2ed577da ("Yes. Harbourton’s minimum wage is $17.40 an hour for every w…") the pages: BP-104@3 says "The minimum wage in Harbourton is 17.40 an hour for every …"; BP-104@4 (on the website from day 171) replaced it from day 178: "From 1 July, the minimum wage in Harbourton is 18.10 an ho…" the release log: release 1.2 (index i3, without BP-104@3) from day 185 the trail: 13 wrong minimum_wage answers, days 178–184, to 13 people; 3 signed in the trail: 2 of those conversations reported a problem; the first on day 178 gaps: the trail keeps no question text (only the topic code), and nothing about what the person did next.The command then writes the two answers: a short letter for the owner, and a table of records for an auditor. The story in plain words: the new page was on the website from day 171, but the helper’s index kept the old one until day 185. Between 1 and 7 July, 13 people got the old wage. Three of them signed in, so the city can write to them. The owner reported the problem on the same day, and the index was rebuilt a week later.
The records answered a complaint that nobody had planned for. That is the real test of a design.
- Wrong answers
- Reports about a wrong answer
Text description of the chart
A line chart over the 28 weeks of the pilot. Wrong answers run between 6 and 24 a week from week 1 to week 22, all about cash. They fall to 0 in weeks 23 to 25, after the fix. In weeks 26 and 27 they come back: 8 and 5 wrong minimum-wage answers. After the new index, week 28 has 0. Reports about a wrong answer come in every few days, between 0 and 3 a week, from week 1.
Insights and limits
Section titled “Insights and limits”- Ask the records before you build them. Each field came from a question. “Log everything” answered three of ten and kept every person’s words.
- Absence is not evidence. “Did a person check?” has an answer only when the record says
"review": "none". - Retention is a decision about future questions. Thirty days of chats could not answer a report that came after five months.
- Limits. Harbourton is not New York, and nothing here says what the real bot did or kept. The traffic is constructed: eight fixed questions, one language, made-up rates. In this setup the model followed its pages. A real assistant with 2,000 pages and free questions also makes model errors. Its topic codes make mistakes too. We made the labels. No record shows what a person did after reading an answer.
Responsible use
Section titled “Responsible use”- Keep people’s words out by design, not by promise, and test it, as
test_no_question_text_in_design_ddoes. - A pseudonym is still personal data while its key exists. Protect the keys file as in Make it tamper-evident and controlled, and ask your legal team what a deletion request means for it.
- A record is not a verdict. Whether an answer broke a rule is for people who know the rules.
- Talk about real cases fairly. Link to the sources, use your own words, quote only short phrases, and say what you do not know. Never present a practice copy as the real system.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This project shows what a team needs from an engineer who runs AI assistants: records designed from the questions that will be asked, tested, and measured against the alternatives.
What to put in the repository:
could-the-records-answer/ README.md the case, the four designs, how to run them, your matrix for day 156 and day 400 CASE.md the public case in your own words, with links (no copied articles) DATASET.md what is real, what is constructed, the limits LICENSE MIT records/audit.py your design D, with the reason for each field casefile/ the questions as code: each one a query over the records tests/ 20 tests, including "no question text in the trail"- Make a new Git repository from your finished folder, and push it to GitHub.
.gitignorekeepswork/out: anyone can make the records again withpython pilot.py. - In the README, add your matrix and three sentences: what the case asked, which design you chose, and what it cannot answer.
- Add one question of your own to
casefile/case.py, for example “which answers did people rate one star?”, and the field that answers it.
In an interview, in two sentences: “I turned the questions of a public chatbot case into ten queries over an assistant’s records. My audit trail answered all ten without keeping anybody’s words, and showed that every wrong answer came from an out-of-date page.”
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says that design B, every question and answer for ever, is the safest: it answers the screenshot and the count. Give two questions that B cannot answer, and one cost that D does not have.
Show the answer
B cannot say which pages and versions an answer was built from (Q3), so it cannot tell a wrong page from a wrong model. It cannot say which release answered, which warning was shown, or why the same question got different answers (Q4 to Q6). And it cannot link a feedback report to an answer. Its cost: it keeps 2,778 questions in people’s own words for ever, which may hold personal data. D keeps none, and stores each of the helper’s answers once, by its digest.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
All 250 wrong cash answers came from release 1.0. Felix proposes to fix the prompt. What do the records say, and what is the right fix?
Show the answer
Every wrong cash answer’s proposal names page BP-102@1 as evidence, and the page history says that version 2 replaced it before the pilot. So the model only repeated its page, and a new prompt would not help. Fix the index instead: release 1.1 removed the old copy and gave 0 wrong cash answers. Then rebuild the index each time a page changes.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Independent task: ask Q9 of design D on day 186. Which topics have reports, and when did the first minimum-wage report come? What would a weekly count of reports by answer digest have done?
Show the answer
Run python -m casefile --day 186 ask Q9 --design audit. Cash has 38 reports of a wrong answer, from day 3 on. The minimum wage has two, from day 178 on. Counted by digest each week, the reports find both early: the cash answers in week 1, the wage answer within days. The team could have found the old pages before any reporter or complaint.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The public case gave ten questions, and each field of the records came from one of them.
- On day 156, the app log answered 0 questions in full, every question and answer 3, traces only 0, and the audit design 10. A year later, the audit design answered 9.
- The answer is the helper’s write. The trail keeps its digest, and the answer store keeps each text once, so no person’s words are kept.
- Every wrong answer came from an out-of-date page. Only the page versions in the records showed it, and the first report came on day 3.
- A new complaint, months later, was answered from the records alone: 13 people, 3 reachable, the cause and the fix.
Next, the final assignment: operate a new release with planted problems, and deliver an observability pack. Nobody tells you what the problems are: your traces, your cost panel and your audit trail must find them.
Next: Final assignment